Information processing device and method, program, storage medium

Abstract

The present invention provides an information processing device that can generate appropriate tactile information according to the image capture conditions and objects. [Solution] The system comprises a detection unit that detects objects from an image, a determination unit that determines the shooting state of the object, an acquisition unit that acquires first tactile information stored in association with the type of object, a generation unit that generates second tactile information based on the image, and a selection unit that selects whether to use the first tactile information or the second tactile information as the tactile information of the object based on the determination result of the determination unit for the object.

Description

Technical Field

[0001] The present invention relates to a technique for presenting tactile information for an image.

Background Art

[0002] Conventionally, there is known a haptics technique that reproduces the tactile sensation of an object by giving skin sensation feedback such as vibration when a user touches a touch panel or the like.

[0003] [[ID=1,5]] For example, Patent Document 1 discloses a method for generating tactile information of an object based on the visual characteristics in the displayed image.

[0004] Further, Patent Document 2 discloses a technique for presenting tactile information corresponding to enlargement and reduction by correcting the tactile information prepared in advance according to the enlargement and reduction of an image. [[ID=2,1]]

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the technique disclosed in Patent Document 1, since the tactile sensation is determined based on visual characteristics, for example, when an input is a photographed image or the like, if there is blurring or shaking, there is a problem that correct tactile information cannot be obtained.

[0007] In the technique disclosed in Patent Document 2, since the tactile information is corrected according to enlargement and reduction, there is a possibility that preferable tactile information for each object cannot be presented. ]

[0008] The present invention has been made in view of the above-mentioned problems, and its objective is to provide an information processing device that can generate appropriate tactile information according to the image capture conditions and objects. [Means for solving the problem]

[0009] The information processing device according to the present invention is characterized by comprising: detection means for detecting an object from an image; determination means for determining the shooting state of the object; acquisition means for acquiring first tactile information stored in association with the type of object; generation means for generating second tactile information based on the image; and selection means for selecting whether to use the first tactile information or the second tactile information as the tactile information of the object based on the determination result of the determination means for the object. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an information processing device that can generate appropriate tactile information according to the image capture conditions and objects. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing the configuration of an information processing device according to a first embodiment of the present invention. [Figure 2] A plan view showing the configuration of the image sensor. [Figure 3] A plan view showing the configuration of the display unit and the vibration unit. [Figure 4] A flowchart illustrating the process of generating tactile information. [Figure 5] A schematic diagram illustrating object detection by the image analysis unit. [Figure 6] A diagram illustrating the contents of the tactile database. [Figure 7] A flowchart illustrating the image analysis process performed by the image analysis unit. [Figure 8] A schematic diagram illustrating the decrease in contrast due to blurring. [Figure 9] Flowchart showing the operation of generating tactile information by the tactile information generation unit. [Figure 10] Diagram explaining the selection of the tactile information generation method based on the image analysis result. [Figure 11] Flowchart explaining the operation during image playback. [Figure 12] Flowchart showing the operation of tactile information generation in the second embodiment. [Figure 13] Flowchart showing the operation of tactile information generation in the second embodiment. [Figure 14] Diagram explaining the selection of the tactile information generation method based on the image analysis result in the second embodiment.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant explanations are omitted.

[0013] (First Embodiment) FIG. 1 is a block diagram showing the configuration of an information processing apparatus 100 that generates tactile information according to the first embodiment of the present invention.

[0014] In FIG. 1, the light beam that has passed through the lens 101 is received by the image sensor 102 to generate an image signal. The generated image signal is subjected to various image processes such as white balance correction by the image processing unit 103, and then stored in the storage unit 104 as image data via the CPU 110.

[0015] The memory unit 104 is a storage device for holding various types of data, consisting of RAM, registers, and the like. The memory unit 104 stores image data, tactile information associated with the image data, a tactile information database (tactile DB) described later, and the operating program for the CPU 110, etc.

[0016] The image analysis unit 105 is a group of circuits that detects objects contained in the input image and also analyzes the image capture conditions.

[0017] The tactile information generation unit 106 is a group of circuits that generate tactile information for a target image based on the image analysis results from the image analysis unit 105.

[0018] The communication unit 107 is a communication unit that communicates with an external network via wireless communication such as Wi-Fi to send and receive image data and other data.

[0019] The display unit 108 is a display device composed of an organic EL, LCD, or the like. Furthermore, the display unit 108 is equipped with a touch panel, capable of displaying images and detecting user touches.

[0020] The vibration unit 109 is an actuator that is stacked on the display unit 108 and generates vibrations according to the user's contact state with the display unit 108.

[0021] The CPU 110 is a control unit that controls the entire information processing device 100 by executing programs stored in the memory unit 104. Hereafter, the transfer of various data and processing operations in the information processing device 100 will be described assuming that they are implemented by the control of the CPU 110.

[0022] Figure 2 shows the configuration of the image sensor 102.

[0023] The image sensor 102 has unit pixels 200 arranged in a two-dimensional grid. Each unit pixel 200 is composed of a pair of photoelectric conversion units 202 and 203 positioned below a microlens 201. The signal group acquired by the pair of photoelectric conversion units 202 and 203 enables focus detection calculation using the image plane phase-difference method. This focus detection calculation using the image plane phase-difference method enables focusing during shooting and acquisition of a depth information map representing the distance (distance information) of the subject. Furthermore, an image signal can be obtained by adding the signals from the pair of photoelectric conversion units.

[0024] Figure 3 is a view of the display unit 108 and the vibration unit 109 from above.

[0025] A vibration unit 109 is stacked on the lower part of the display unit 108. Vibration elements 300 are arranged in a two-dimensional grid pattern in the vibration unit 109.

[0026] Next, we will explain the generation of tactile information by the information processing device 100. Figure 4 is a flowchart showing the processing procedure for generating tactile information by the information processing device 100.

[0027] In the first step, S400, the user takes a picture, and image data is generated.

[0028] In step S401, the CPU 110 analyzes the image data generated in step S400 using the image analysis unit 105 to detect objects contained in the image data and determine the shooting state.

[0029] Object detection by the image analysis unit 105 can be achieved using known methods such as R-CNN (Region-based CNN), YOLO (You Only Look Once), and SSD (Single Shot Detector), which utilize neural network models.

[0030] Figure 5 is a schematic diagram illustrating object detection.

[0031] Figure 5(a) shows an example of an input image, and Figure 5(b) shows the category information (object type information) and object regions extracted by performing object detection processing on the input image shown in Figure 5(a). The information for classifying into categories is registered in the tactile DB pre-stored in the storage unit 104, and each object region is assigned to a category according to its characteristics.

[0032] Figure 6 shows an example of a tactile database. The tactile database registers category names, their characteristics, and tactile information corresponding to the category names. The tactile information is waveform data for vibrating the vibrating unit 109. The characteristics registered include the features of each object, which are used for comparison with the object-specific characteristics detected by image analysis. The tactile database allows referencing corresponding tactile information using the category name as the key.

[0033] Next, we will explain the image analysis process performed by the image analysis unit 105.

[0034] Figure 7 is a flowchart showing the procedure for analyzing the shooting conditions by the image analysis unit 105.

[0035] In step S700, the CPU 110 inputs the image data and object region information to be analyzed to the image analysis unit 105.

[0036] In step S701, the image analysis unit 105 calculates the contrast evaluation value of the object region in the image. The contrast evaluation value can be calculated from the difference between the maximum pixel value and the minimum pixel value within the reference region.

[0037] In step S702, the image analysis unit 105 determines whether the contrast evaluation value of the object region has reached a predetermined threshold (a specified threshold). If the contrast evaluation value is greater than the threshold, the CPU 110 proceeds to step S703 and determines that the imaging state of the object region is "good". If the contrast evaluation value is less than or equal to the threshold, the CPU 110 proceeds to step S704 and determines that the imaging state of the object region is "poor".

[0038] In step S705, the CPU 110 stores the shooting state determined as described above, associating it with the corresponding object area.

[0039] In step S706, it is determined whether the analysis of the shooting state for all object regions has been completed. If the analysis of all object regions has not been completed, the CPU 110 returns to step S701 and analyzes the shooting state of the next object region. If the analysis of the shooting state for all object regions has been completed, the shooting state analysis process is terminated.

[0040] Now, referring to Figure 8, we will explain the cases where the contrast is calculated to be low.

[0041] Figure 8(a) shows an example of normal image data. Figure 8(b) shows an example of an image with the same composition as Figure 8(a) but with blurring (low focus).

[0042] In the image shown in Figure 8(b), the edges of the subject are lost due to the effect of blurring, resulting in low contrast within the area. Such loss of edge information can also occur due to subject blur or camera shake. Therefore, in this embodiment, the shooting state is determined using contrast (contrast information) as an indicator, and object areas that do not reach a certain level of contrast are determined to be in a "poor" shooting state.

[0043] Returning to the explanation of Figure 4, in step S402, the CPU 110 generates tactile information using the tactile information generation unit 106.

[0044] Now, with reference to Figure 9, the tactile information generation process by the tactile information generation unit 106 will be explained. Figure 9 is a flowchart showing the procedure for the tactile information generation process by the tactile information generation unit 106.

[0045] In step S900, the CPU 110 inputs image data to the haptic information generation unit 103.

[0046] In step S901, the CPU 110 inputs the object detection results for each region in the image and the shooting state determination results for each region from the image analysis unit 105 to the tactile information generation unit 103.

[0047] In step S902, the tactile information generation unit 103 selects a method for generating tactile information according to the table shown in Figure 10.

[0048] Figure 10 is a correspondence table between the method of generating tactile information by the image analysis unit 105 and the image analysis results by the image analysis unit 105. As shown in Figure 10, if the object's shooting condition is "good," tactile information is generated from the image. If the object's shooting condition is "poor," tactile information is not generated from the image, but rather from the tactile information database (tactile DB). Also, if it is not an object area, no tactile information is generated.

[0049] In step S903, the tactile information generation unit 103 switches the method of generating tactile information according to the selection made in step S902. If the method of generating tactile information from an image is selected, that is, if it is an object area and the shooting conditions are determined to be good, the process proceeds to step S904. Then, the shape and texture of the object are estimated based on the image data of the object area and tactile information is generated.

[0050] One method for generating tactile information is to align a depth information map, which can be acquired during shooting, with the object region, and then generate tactile information that mimics unevenness according to the depth information. However, the present invention is not limited to this, and the estimation of object shape and texture based on image data may also be achieved using known techniques.

[0051] If it is determined in step S903 that tactile information should not be generated from the image, the tactile information generation unit 103 proceeds to step S905. It then refers to the tactile information associated with the category name to which the relevant object belongs in the tactile database and uses it as tactile data within the object area.

[0052] Returning to Figure 4, in step S403, the CPU 110 stores the tactile information of each generated object in the storage unit 104, associating it with each region of the image data.

[0053] The above describes the series of processes from image capture by the information processing device 100 to the generation of tactile information.

[0054] Next, we will explain the operation during image playback, referring to Figure 11. Figure 11 is a flowchart showing the playback operation of captured image data.

[0055] When the system switches to the playback mode of an image specified by the user, in step S1100, the CPU 110 reads the specified image data and the haptic information associated with the image.

[0056] In step S1101, the CPU 110 displays the image data on the display unit 108.

[0057] In step S1102, the CPU 110 determines whether the user is touching the display unit 108. If the user is touching the display unit 108, the CPU 110 proceeds to step S1103; otherwise, it proceeds to step S1104.

[0058] In step S1103, the CPU 110 drives the vibration unit 109 based on tactile information associated with the region corresponding to the user's touch detection coordinates in the displayed image, and provides tactile stimulation feedback to the user.

[0059] In step S1104, the CPU 110 determines whether or not an instruction to end image display has been given. If no instruction to end image display has been given, steps S1101 to S1103 are repeated. If an instruction to end image display has been given, the image playback process is terminated.

[0060] Furthermore, if zooming in or out is performed while the image is displayed, the process described in steps S401 to S403 in Figure 4 is executed based on the image signal after zooming in or out, thereby generating tactile information that corresponds to the changes in image features that occur in response to zooming in or out.

[0061] As described above, according to this embodiment, when the object is captured in good condition, it is possible to present the detailed shape and texture of the object by generating tactile information based on the image signal. On the other hand, when the capture condition is poor, it is thought that it may not be possible to correctly generate tactile information based on the image. In such cases, it is possible to present tactile information without compromising quality by referring to tactile information in a pre-associated database.

[0062] Furthermore, if the image is modified in response to zooming operations while it is displayed, it becomes possible to provide appropriate haptic feedback that corresponds to the changes in the image by performing image analysis and generating haptic information again on the modified image.

[0063] (Second embodiment) In the second embodiment, a process for correcting tactile information in image data received from an external source will be described in the information processing device. The configuration of the information processing device in this embodiment is the same as that of the information processing device 100 described in the first embodiment, so the description will be omitted.

[0064] Figure 12 is a flowchart illustrating the tactile information correction process in the second embodiment.

[0065] In step S1200, the CPU 110 receives image data and tactile information associated with the image data from an external source via the communication unit 107 and stores them in the storage unit 104.

[0066] In step S1201, the CPU 110 inputs the received image data to the image analysis unit 105 and performs object detection and analysis of the shooting conditions contained in the image data.

[0067] In step S1202, the CPU 110 generates tactile information using the tactile information generation unit 106.

[0068] Figure 13 is a flowchart showing the procedure for generating tactile information in the second embodiment. Parts identical to those described in Figure 9 of the first embodiment are given the same step numbers and their explanations are omitted. In step S1302, a method for generating tactile information is selected based on Figure 14. Figure 14 is a correspondence table between the method for generating tactile information by the image analysis unit 105 and the image analysis results by the image analysis unit 105.

[0069] In this embodiment, as shown in Figure 14, if the object's image quality is "good," tactile information is generated from the image. If the object's image quality is "poor," tactile information is not generated from the image, and the tactile data stored in association with the received data is used as is.

[0070] In this way, for each object in the received data whose tactile sensation can be estimated based on the image data, the received tactile information can be replaced with tactile information generated based on the image.

[0071] Returning to Figure 12, in step S1203, the CPU 110 associates the tactile information and image data of each generated object and stores them in the storage unit 104.

[0072] As explained above, by performing haptic information generation processing on image data and haptic information transmitted from an external source, according to the object and shooting conditions, it becomes possible to provide more appropriate haptic information to the user.

[0073] (Third embodiment) A third embodiment describes a case where haptic generation from an image is limited to a specific object.

[0074] In the first and second embodiments, if the image quality of the object obtained from the image analysis was good, a process was performed to generate tactile information based on the image.

[0075] However, in the case of certain subjects, such as human skin, it may be undesirable to present detailed tactile information.

[0076] In such cases, a flag may be added to the haptic database to allow the generation of haptic information based on images, enabling more detailed configuration of how haptic information is generated for each object.

[0077] Furthermore, by associating each object with its own unique evaluation metrics and thresholds in the haptic database, the selection criteria for generating haptic information can be changed for each object.

[0078] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0079] The disclosures herein include the following information processing devices and methods, programs, and storage media.

[0080] (Item 1) A detection means for detecting objects from an image, A determination means for determining the shooting state of the object, An acquisition means for acquiring first tactile information stored in association with the type of object, A generation means for generating second tactile information based on the aforementioned image, A selection means that, based on the determination result of the determination means for the object, selects whether to use the first tactile information or the second tactile information as the tactile information of the object, An information processing device characterized by comprising:

[0081] (Item 2) The information processing device according to item 1, characterized in that the tactile information is information relating to the unevenness of the object.

[0082] (Item 3) The information processing device according to item 2, characterized in that the generation means calculates distance information of the object and obtains information regarding the unevenness of the object based on focus detection information at the time the image was captured.

[0083] (Item 4) The information processing device according to any one of items 1 to 3, characterized in that the determination means determines the shooting state based on the focus state or blur state of the object in the image.

[0084] (Item 5) The information processing device according to any one of items 1 to 4, characterized in that the determination means determines the shooting state based on the contrast information of the object in the image.

[0085] (Item 6) The information processing apparatus according to item 5, characterized in that the determination means determines the shooting state to be good when the contrast evaluation value of the object in the image is greater than a predetermined threshold, and determines the shooting state to be poor when the contrast evaluation value is less than or equal to a predetermined threshold.

[0086] (Item 7) The information processing device according to item 6, characterized in that the selection means selects the second tactile information as the tactile information of the object when the shooting condition is determined to be good, and selects the first tactile information as the tactile information of the object when the shooting condition is determined to be poor.

[0087] (Item 8) The information processing device according to any one of items 1 to 7, characterized in that the selection means selects whether to use the first tactile information or the second tactile information based on an evaluation index unique to each object.

[0088] (Item 9) The information processing device according to any one of items 1 to 8, characterized in that when the image is enlarged or reduced, the detection means detects an object, the determination means makes a determination, and the generation means generates the second tactile information for the enlarged or reduced image.

[0089] (Item 10) The information processing device according to any one of items 1 to 9, further comprising a presentation means for presenting tactile information selected by the selection means.

[0090] (Item 11) The information processing apparatus according to item 10, characterized in that the presentation means comprises a display means for displaying the image and a vibration means stacked on the display means.

[0091] (Item 12) The information processing apparatus according to item 11, characterized in that the vibration means is configured such that vibration elements are arranged in a two-dimensional grid on the display means.

[0092] (Item 13) The information processing device according to item 12, characterized in that the vibration element exhibits vibrations of a predetermined waveform.

[0093] (Item 14) An information processing device according to any one of items 1 to 13, further comprising an imaging means for capturing an image.

[0094] (Item 15) A detection process for detecting objects from an image, A determination step for determining the shooting state of the object, An acquisition step of acquiring first tactile information stored in association with the type of the object, A generation process for generating second tactile information based on the aforementioned image, A selection step to select which of the first tactile information and the second tactile information to use as the tactile information of the object, based on the determination result of the determination step for the object; An information processing method characterized by having the following features.

[0095] (Item 16) A program that causes a computer to execute each step of the information processing method described in item 15.

[0096] (Item 17) A computer-readable storage medium containing a program that causes a computer to execute each step of the information processing method described in item 15. [Explanation of Symbols]

[0097] 100: Information processing unit, 101: Lens, 102: Image sensor, 103: Image processing unit, 104: Memory unit, 105: Image analysis unit, 106: Tactile information generation unit, 107: Communication unit, 108: Display unit, 109: Vibration unit, 110: CPU

Claims

1. A detection means for detecting objects from an image, A determination means for determining the shooting state of the object, An acquisition means for acquiring first tactile information stored in association with the type of object, A generation means for generating second tactile information based on the aforementioned image, A selection means that, based on the determination result of the determination means for the object, selects whether to use the first tactile information or the second tactile information as the tactile information of the object, An information processing device characterized by comprising:

2. The information processing apparatus according to claim 1, characterized in that the tactile information is information relating to the unevenness of the object.

3. The information processing apparatus according to claim 2, characterized in that the generation means calculates distance information of the object and obtains information regarding the unevenness of the object based on focus detection information at the time the image was captured.

4. The information processing apparatus according to claim 1, characterized in that the determination means determines the shooting state based on the focus state or blur state of the object in the image.

5. The information processing apparatus according to claim 1, characterized in that the determination means determines the shooting state based on the contrast information of the object in the image.

6. The information processing apparatus according to claim 5, characterized in that the determination means determines the shooting state to be good when the contrast evaluation value of the object in the image is greater than a predetermined threshold, and determines the shooting state to be poor when the contrast evaluation value is less than or equal to a predetermined threshold.

7. The information processing apparatus according to claim 6, characterized in that the selection means selects the second tactile information as the tactile information of the object when the shooting condition is determined to be good, and selects the first tactile information as the tactile information of the object when the shooting condition is determined to be poor.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the selection means selects whether to use the first tactile information or the second tactile information based on an evaluation index unique to each object.

9. The information processing apparatus according to claim 1, characterized in that when the image is enlarged or reduced, the detection means detects an object, the determination means makes a determination, and the generation means generates the second tactile information for the enlarged or reduced image.

10. The information processing apparatus according to claim 1, further comprising a presentation means for presenting tactile information selected by the selection means.

11. The information processing apparatus according to claim 10, characterized in that the presentation means comprises a display means for displaying the image and a vibrating means stacked on the display means.

12. The information processing apparatus according to claim 11, characterized in that the vibration means is configured such that vibration elements are arranged in a two-dimensional grid on the display means.

13. The information processing apparatus according to claim 12, characterized in that the vibration element exhibits vibrations of a predetermined waveform.

14. The information processing apparatus according to claim 1, further comprising an imaging means for capturing an image.

15. A detection process for detecting objects from an image, A determination step for determining the shooting state of the object, An acquisition step of acquiring first tactile information stored in association with the type of the object, A generation process for generating second tactile information based on the aforementioned image, A selection step to select which of the first tactile information and the second tactile information to use as the tactile information of the object, based on the determination result of the determination step for the object; An information processing method characterized by having the following features.

16. A program for causing a computer to execute each step of the information processing method described in claim 15.

17. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method described in claim 15.

Images