Information processing device and method, program, and storage medium

Abstract

The present disclosure provides an information processing device capable of generating appropriate haptic information corresponding to an image capture state or object. The information processing device comprises: a detecting unit that detects an object from an image; a determining unit that determines the image capture state of the object; an acquiring unit that acquires first haptic information stored in association with the type of the object; a generating unit that generates second haptic information on the basis of the image; and a selecting unit that selects either the first haptic information or the second haptic information to be used as haptic information of the object on the basis of the determination result obtained for the object by the determining unit.

Description

Information Processing Apparatus and Method, Program, Storage Medium

[0001] The present disclosure relates to a technique for presenting tactile information for an image.

[0002] Conventionally, a haptics technique is known that reproduces the tactile sensation of an object by giving skin sensation feedback such as vibration when a user touches a touch panel or the like.

[0003] For example, Patent Document 1 discloses a method for generating tactile information of an object based on visual characteristics in a displayed image.

[0004] Also, Patent Document 2 discloses a technique for presenting tactile information corresponding to enlargement and reduction by correcting tactile information prepared in advance according to enlargement and reduction of an image.

[0005] Japanese Patent Application Laid-Open No. 2020-201926, Japanese Patent Application Laid-Open No. 2018-129102

[0006] However, in the technique disclosed in Patent Document 1, since the tactile sensation is determined based on visual characteristics, there is a problem that correct tactile information cannot be obtained when, for example, a photographed image or the like is input and there is blurring or shaking.

[0007] In the technique disclosed in Patent Document 2, since the tactile information is corrected according to enlargement and reduction, there is a possibility that preferable tactile information for each object cannot be presented.

[0008] The present disclosure has been made in view of the above problems, and provides an information processing apparatus capable of generating appropriate tactile information according to the photographing state of an image and an object.

[0009] The information processing device relating to this disclosure is characterized by comprising: detection means for detecting an object from an image; determination means for determining the shooting state of the object; acquisition means for acquiring first tactile information stored in association with the type of object; generation means for generating second tactile information based on the image; and selection means for selecting whether to use the first tactile information or the second tactile information as the tactile information of the object based on the determination result of the determination means for the object.

[0010] This disclosure makes it possible to provide an information processing device that can generate appropriate tactile information according to the image capture conditions and objects.

[0011] Other features and advantages of this disclosure will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral.

[0012] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments of the present disclosure and are used to explain the principles of the present disclosure together with their descriptions. Block diagram showing the configuration of an information processing device relating to the first embodiment of the present disclosure. Plan view showing the configuration of an image sensor. Plan view showing the configuration of a display unit and a vibration unit. Flowchart showing the operation of tactile information generation. Schematic diagram explaining object detection by the image analysis unit. Schematic diagram explaining object detection by the image analysis unit. Diagram explaining the contents of the tactile database. Flowchart explaining the analysis operation of the shooting state by the image analysis unit. Schematic diagram explaining the reduction in contrast due to blur. Schematic diagram explaining the reduction in contrast due to blur. Flowchart showing the operation of tactile information generation by the tactile information generation unit. Diagram explaining the selection of a tactile information generation method based on image analysis results. Flowchart explaining the operation during image playback. Flowchart showing the operation of tactile information generation in the second embodiment. Flowchart showing the operation of tactile information generation in the second embodiment. Diagram explaining the selection of a tactile information generation method based on image analysis results in the second embodiment.

[0013] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claims. While the embodiments describe multiple features, not all of these features are necessary, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0014] (First Embodiment) Figure 1 is a block diagram showing the configuration of an information processing device 100 that generates tactile information according to the first embodiment of the present disclosure.

[0015] In Figure 1, the light beam passing through the lens 101 is received by the image sensor 102 to generate an image signal. The generated image signal undergoes various image processing, such as white balance correction, by the image processing unit 103, and is then stored as image data in the storage unit 104 via the CPU 110.

[0016] The memory unit 104 is a storage device for holding various types of data, consisting of RAM and registers. The memory unit 104 stores image data, tactile information associated with the image data, a tactile information database (tactile DB) described later, and the operating program for the CPU 110.

[0017] The image analysis unit 105 is a group of circuits that detects objects contained in the input image and also analyzes the image capture conditions.

[0018] The tactile information generation unit 106 is a group of circuits that generate tactile information for a target image based on the image analysis results from the image analysis unit 105.

[0019] The communication unit 107 is a communication unit that communicates with an external network via wireless communication such as Wi-Fi and transmits and receives image data and other data.

[0020] The display unit 108 is a display device composed of an organic EL, LCD, or the like. The display unit 108 also includes a touch panel, which is configured to display images and detect user touches.

[0021] The vibration unit 109 is an actuator that is stacked on the display unit 108 and generates vibrations according to the user's contact state with the display unit 108.

[0022] The CPU 110 is a control unit that controls the entire information processing device 100 by executing programs stored in the memory unit 104. Hereafter, the transfer of various data and processing operations in the information processing device 100 will be described as being realized by control by the CPU 110.

[0023] Figure 2 shows the configuration of the image sensor 102.

[0024] The image sensor 102 has unit pixels 200 arranged in a two-dimensional grid. Each unit pixel 200 is composed of a pair of photoelectric conversion units 202 and 203 positioned below a microlens 201. The signal group acquired by the pair of photoelectric conversion units 202 and 203 enables focus detection calculation using the image plane phase-difference method. This focus detection calculation using the image plane phase-difference method enables focusing during shooting and acquisition of a depth information map representing the distance (distance information) of the subject. Furthermore, an image signal can be obtained by adding the signals from the pair of photoelectric conversion units.

[0025] Figure 3 is a view of the display unit 108 and the vibration unit 109 from above.

[0026] A vibration unit 109 is stacked on the lower part of the display unit 108. Vibration elements 300 are arranged in a two-dimensional grid pattern in the vibration unit 109.

[0027] Next, we will explain the generation of tactile information by the information processing device 100. Figure 4 is a flowchart showing the processing procedure for generating tactile information by the information processing device 100.

[0028] First, in step S400, the user takes a photograph, and image data is generated.

[0029] In step S401, the CPU 110 analyzes the image data generated in step S400 using the image analysis unit 105 to detect objects contained in the image data and determine the shooting state.

[0030] Object detection by the image analysis unit 105 can be achieved using known methods such as R-CNN (Region-based CNN), YOLO (You Only Look Once), and SSD (Single Shot Detector), which utilize neural network models.

[0031] Figures 5A and 5B are schematic diagrams illustrating object detection.

[0032] Figure 5A shows an example of an input image, and Figure 5B shows the category information (object type information) and object regions extracted by performing object detection processing on the input image shown in Figure 5A. The information for classifying into categories is registered in the tactile database pre-stored in the storage unit 104, and each object region is assigned to a category according to its characteristics.

[0033] Figure 6 shows an example of a tactile database. The tactile database registers category names, their characteristics, and tactile information corresponding to the category names. The tactile information is waveform data for vibrating the vibrating unit 109. The characteristics registered include the features of each object, which are used for comparison with the object-specific characteristics detected by image analysis. The tactile database allows referencing corresponding tactile information using the category name as a key.

[0034] Next, we will explain the analysis process of the shooting conditions by the image analysis unit 105.

[0035] Figure 7 is a flowchart showing the procedure for analyzing the shooting conditions by the image analysis unit 105.

[0036] In step S700, the CPU 110 inputs the image data and object region information to be analyzed to the image analysis unit 105.

[0037] In step S701, the image analysis unit 105 calculates the contrast evaluation value of the object region in the image. The contrast evaluation value can be calculated from the difference between the maximum pixel value and the minimum pixel value within the reference region.

[0038] In step S702, the image analysis unit 105 determines whether or not the contrast evaluation value of the object area has reached a predetermined threshold value (predetermined threshold). If the contrast evaluation value is greater than the threshold value, the CPU 110 proceeds to step S703 and determines that the shooting state of the corresponding object area is "good". If the contrast evaluation value is less than or equal to the threshold value, the CPU 110 proceeds to step S704 and determines that the shooting state of the corresponding object area is "bad".

[0039] In step S705, the CPU 110 stores the shooting state determined as described above in association with the corresponding object area.

[0040] In step S706, it is determined whether or not the analysis of the shooting state for all object areas has been completed. If the analysis of all object areas has not been completed, the CPU 110 returns the process to step S701 and analyzes the shooting state of the next object area. If the analysis of the shooting state for all object areas has been completed, the analysis process of the shooting state is terminated.

[0041] Here, referring to FIGS. 8A and 8B, the case where the contrast is calculated to be low will be described.

[0042] FIG. 8A shows an example of normal image data. FIG. 8B shows an example of an image in the same composition as FIG. 8A when defocusing has occurred (the focusing state is low).

[0043] In the image shown in FIG. 8B, due to the influence of defocusing, the edges of the subject are lost, so the contrast within the area becomes low. Such loss of edge information also occurs due to subject blur or camera shake. Therefore, in the present embodiment, the shooting state is determined using contrast (contrast information) as an index, and for object areas that have not reached a certain contrast, the shooting state is determined to be "bad".

[0044] Returning to the description of FIG. 4, in step S402, the CPU 110 generates tactile information using the tactile information generation unit 106.

[0045] Here, referring to FIG. 9, the haptic information generation process by the haptic information generation unit 106 will be described. FIG. 9 is a flowchart showing the procedure of the haptic information generation process by the haptic information generation unit 106.

[0046] In step S900, the CPU 110 inputs image data to the haptic information generation unit 103.

[0047] In step S901, the CPU 110 inputs the detection result of the object for each region in the image and the determination result of the shooting state for each region by the image analysis unit 105 to the haptic information generation unit 103.

[0048] In step S902, the haptic information generation unit 103 selects a method for generating haptic information according to the table shown in FIG. 10.

[0049] FIG. 10 is a correspondence table between the method for generating haptic information by the image analysis unit 105 and the image analysis result by the image analysis unit 105. As shown in FIG. 10, when the shooting state of the object is "good", haptic information is generated from the image. When the shooting state of the object is "bad", haptic information is generated from the haptic information database (haptic DB) without generating haptic information from the image. Also, when it is not an object region, haptic information is not generated.

[0050] In step S903, the haptic information generation unit 103 switches the method for generating haptic information according to the selection in step S902. When the method for generating haptic information from the image is selected, that is, when it is determined that it is an object region and the shooting conditions are good, the process proceeds to step S904. Then, based on the image data of the corresponding object region, the shape and texture of the object are estimated to generate haptic information.

[0051] As a method for generating haptic information, it can be realized by aligning the position of the depth information map that can be obtained during shooting for the object region and generating haptic information simulating concavo-convex information according to the depth information. Also, the present disclosure is not limited to this, and the estimation of the object shape and texture based on the image data may be realized using known techniques.

[0052] If it is determined in step S903 that tactile information should not be generated from the image, the tactile information generation unit 103 proceeds to step S905. It then refers to the tactile information associated with the category name to which the corresponding object belongs in the tactile database and uses it as tactile data within the object area.

[0053] Returning to Figure 4, in step S403, the CPU 110 stores the tactile information of each generated object in the storage unit 104, associating it with each region of the image data.

[0054] The above describes the series of processes from image capture by the information processing device 100 to the generation of tactile information.

[0055] Next, we will explain the operation during image playback with reference to Figure 11. Figure 11 is a flowchart showing the playback operation of captured image data.

[0056] When the system switches to the playback mode of an image specified by the user, in step S1100, the CPU 110 reads the specified image data and the haptic information associated with the image.

[0057] In step S1101, the CPU 110 displays the image data on the display unit 108.

[0058] In step S1102, the CPU 110 determines whether the user is touching the display unit 108. If the user is touching the display unit 108, the CPU 110 proceeds to step S1103; otherwise, it proceeds to step S1104.

[0059] In step S1103, the CPU 110 drives the vibration unit 109 based on tactile information associated with the region corresponding to the user's touch detection coordinates in the displayed image, and provides tactile stimulation as feedback to the user.

[0060] In step S1104, the CPU 110 determines whether or not an instruction to end image display has been given. If no instruction to end image display has been given, steps S1101 to S1103 are repeated. If an instruction to end image display has been given, the image playback process is terminated.

[0061] Furthermore, if zooming in or out is performed while the image is displayed, the processes described in steps S401 to S403 in Figure 4 are executed based on the image signal after zooming in or out, thereby generating tactile information that corresponds to the changes in image features that occur in response to zooming in or out.

[0062] As described above, according to this embodiment, when the object is captured in good condition, it is possible to present the detailed shape and texture of the object by generating tactile information based on the image signal. On the other hand, when the capture condition is poor, it is thought that it may not be possible to correctly generate tactile information based on the image. In such cases, it is possible to present tactile information without compromising quality by referring to tactile information in a pre-associated database.

[0063] Furthermore, if the image is modified in response to zooming operations while it is displayed, it becomes possible to provide appropriate haptic feedback that corresponds to the changes in the image by performing image analysis and generating haptic information again on the modified image.

[0064] (Second Embodiment) In the second embodiment, a process for correcting tactile information for image data received from an external source in the information processing device will be described. The configuration of the information processing device in this embodiment is the same as that of the information processing device 100 described in the first embodiment, so the description will be omitted.

[0065] Figure 12 is a flowchart illustrating the tactile information correction process in the second embodiment.

[0066] In step S1200, the CPU 110 receives image data and tactile information associated with the image data from an external source via the communication unit 107 and stores them in the storage unit 104.

[0067] In step S1201, the CPU 110 inputs the received image data to the image analysis unit 105 and performs object detection and shooting condition analysis within the image data.

[0068] In step S1202, the CPU 110 generates tactile information using the tactile information generation unit 106.

[0069] Figure 13 is a flowchart showing the procedure for generating tactile information in the second embodiment. Parts identical to those described in Figure 9 of the first embodiment are given the same step numbers and their explanations are omitted. In step S1302, a method for generating tactile information is selected based on Figure 14. Figure 14 is a correspondence table between the tactile information generation method by the image analysis unit 105 and the image analysis results by the image analysis unit 105.

[0070] In this embodiment, as shown in Figure 14, if the object's image quality is "good," tactile information is generated from the image. If the object's image quality is "poor," tactile information is not generated from the image, and the tactile data stored in association with the received data is used as is.

[0071] In this way, for each object in the received data whose tactile sensation can be estimated based on the image data, the received tactile information can be replaced with tactile information generated based on the image.

[0072] Returning to Figure 12, in step S1203, the CPU 110 associates the tactile information and image data of each generated object and stores them in the storage unit 104.

[0073] As explained above, by performing haptic information generation processing on image data and haptic information transmitted from an external source, according to the object and shooting conditions, it becomes possible to provide more appropriate haptic information to the user.

[0074] (Third Embodiment) The third embodiment describes a case in which the generation of tactile sensations from an image is limited to a specific object.

[0075] In the first and second embodiments, if the image quality of the object obtained from the image analysis was good, a process was performed to generate tactile information based on the image.

[0076] However, in the case of certain subjects, such as human skin, it may be undesirable to present detailed tactile information.

[0077] In such cases, a flag may be added to the haptic database to allow the generation of haptic information based on images, enabling more detailed configuration of the haptic information generation method for each object.

[0078] Furthermore, by associating each object with its own unique evaluation metrics and thresholds and maintaining them in the tactile database, the selection criteria for generating tactile information can be changed for each object.

[0079] (Other Embodiments) The Disclosure can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0080] This disclosure is not limited to the embodiments described above, and various modifications and alterations are possible without departing from the spirit and scope of the disclosure. Accordingly, the claims are attached to make the scope of the disclosure public.

[0081] This application claims priority based on Japanese Patent Application No. 2024-165648, filed on September 24, 2024, and all of its contents are incorporated herein by reference.

Claims

1. An information processing device comprising: detection means for detecting an object from an image; determination means for determining the shooting state of the object; acquisition means for acquiring first tactile information stored in association with the type of object; generation means for generating second tactile information based on the image; and selection means for selecting whether to use the first tactile information or the second tactile information as the tactile information of the object based on the determination result of the determination means for the object.

2. The information processing apparatus according to claim 1, characterized in that the tactile information is information relating to the unevenness of the object.

3. The information processing apparatus according to claim 2, characterized in that the generation means calculates distance information of the object and obtains information regarding the unevenness of the object based on focus detection information at the time the image was captured.

4. The information processing apparatus according to claim 1, characterized in that the determination means determines the shooting state based on the focus state or blur state of the object in the image.

5. The information processing apparatus according to claim 1, characterized in that the determination means determines the shooting state based on the contrast information of the object in the image.

6. The information processing apparatus according to claim 5, characterized in that the determination means determines the shooting state to be good when the contrast evaluation value of the object in the image is greater than a predetermined threshold, and determines the shooting state to be poor when the contrast evaluation value is less than or equal to a predetermined threshold.

7. The information processing apparatus according to claim 6, characterized in that the selection means selects the second tactile information as the tactile information of the object when the shooting condition is determined to be good, and selects the first tactile information as the tactile information of the object when the shooting condition is determined to be poor.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the selection means selects whether to use the first tactile information or the second tactile information based on an evaluation index unique to each object.

9. The information processing apparatus according to claim 1, characterized in that when the image is enlarged or reduced, the detection means detects an object, the determination means makes a determination, and the generation means generates the second tactile information for the enlarged or reduced image.

10. The information processing apparatus according to claim 1, further comprising a presentation means for presenting tactile information selected by the selection means.

11. The information processing apparatus according to claim 10, characterized in that the presentation means comprises a display means for displaying the image and a vibrating means stacked on the display means.

12. The information processing apparatus according to claim 11, characterized in that the vibration means is configured such that vibration elements are arranged in a two-dimensional grid on the display means.

13. The information processing apparatus according to claim 12, characterized in that the vibrating element exhibits vibrations of a predetermined waveform.

14. The information processing apparatus according to claim 1, further comprising an imaging means for capturing an image.

15. An information processing method characterized by comprising: a detection step of detecting an object from an image; a determination step of determining the shooting state of the object; an acquisition step of acquiring first tactile information stored in association with the type of object; a generation step of generating second tactile information based on the image; and a selection step of selecting whether to use the first tactile information or the second tactile information as the tactile information of the object based on the determination result of the determination step for the object.

16. A program for causing a computer to execute each step of the information processing method described in claim 15.

17. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method described in claim 15.

Images