Control apparatus and control method thereof

Abstract

To present tactile information corresponding to a scene.SOLUTION: Tactile information corresponding to an object whose image data is detected by an object detection part 113 is determined by a tactile information determination part 114. The tactile information determined by the tactile information determination unit 114 is presented to the user by the presentation unit 120.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Concerning the presentation of tactile information. [Background technology]

[0002] Patent Document 1 proposes a method for generating vibration information from an acoustic signal of an object according to a scene of video content. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-116109 Summary of the Invention [Problem to be solved by the invention]

[0004] However, Patent Document 1 does not take into consideration the presentation of tactile information according to the scene.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a control device and a control method that can present tactile information according to the scene. [Means for solving the problem]

[0006] The present invention is characterized by being configured to have an object detection unit that detects an object from image data, a determination unit that determines tactile information corresponding to the object detected by the object detection unit, and a presentation unit that presents the tactile information determined by the determination unit to a user. [Effects of the Invention]

[0007] According to the present invention, it is possible to present tactile information according to the scene. [Brief explanation of the drawings]

[0008] [Figure 1]1 is a block diagram showing the configuration of a tactile information presentation device according to a first embodiment. [Figure 2] 4 is a flowchart illustrating processing of the tactile information presentation device according to the first embodiment. [Figure 3] 10 is a flowchart illustrating processing of a tactile information presentation device according to a second embodiment. [Figure 4] 10 is a flowchart illustrating processing of a tactile information presentation device according to a third embodiment. [Figure 5] 10 is a flowchart illustrating the processing of a tactile information presentation device according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment to which the present invention is applied will be described with reference to the accompanying drawings.

[0010] First Embodiment In an embodiment of the present invention, the tactile information presentation device 100 (control device) detects an object from input image data, identifies the type of object, and references a tactile information table stored in advance in memory based on the type of object. This determines tactile information corresponding to the relevant object region in the tactile information presentation device 100, and controls the tactile information corresponding to the relevant object region to present tactile information to the user of the tactile information presentation device 100. In this embodiment, the case where the tactile information is vibration information will be mainly described, but this is not limited to this. The tactile information may also be unevenness information, temperature / cold information, sound information, etc.

[0011] 1 is a block diagram showing a tactile information presentation device 100 according to an embodiment of the present invention. The tactile information presentation device 100 includes an image processing unit 110, a presentation unit 120, a system control unit 130, an operation unit 131, a nonvolatile memory 132, and a system memory 133.

[0012] The image processing unit 110 includes an image data input unit 111 , a memory 112 , an object detection unit 113 , an object tactile information determination unit 114 , and a presentation control unit 115 .

[0013] The memory 112 includes an image memory 112_A, an object information memory 112_B, a tactile information table memory 112_C, and a tactile information presentation memory 112_D.

[0014] The image data input unit 111 has an I / F for inputting digital image data signals, and is equipped with, for example, an HDMI terminal, receives a video signal conforming to the HDMI standard, extracts image data from the received video signal, and writes the extracted image data to image memory 112_A. The image data input unit 111 has a physical I / F into which a memory card capable of reading and writing high-resolution image data at high speed, such as an SD card or a CFexpress card, can be inserted and removed. The data input unit 111 may be configured to read image data recorded on an inserted memory card and write the read image data to image memory 112_A.

[0015] The image memory 112_A is, for example, a DRAM or an SRAM, and records image data.

[0016] The object detection unit 113 reads image data recorded in the image memory 112_A and detects one or more object regions from the read image data. For each detected region, the object information associated with the position and size in the image data, the object type (car, train, airplane, bird, insect, human body, head, eye, cat, dog, ball, etc.), and the reliability of the type is recorded in the object information memory 112_B. The object detection unit 113 can detect candidate regions using known techniques for detecting feature regions such as the facial regions of people or animals. For example, the object detection unit 113 may be configured as a class classifier trained using machine learning training data. There are no particular limitations on the classification algorithm. The object detection unit 113 can be realized by training a classifier that implements multi-class logistic regression, support vector machine, random forest, neural network, etc.

[0017] The object information memory 112_B is, for example, a DRAM or an SRAM, and records the object information obtained by the object detection unit 113.

[0018] The tactile information table memory 112_C is, for example, a DRAM or SRAM, and tactile information prepared in advance for each type of object is written to the tactile information table memory 112_C as a table list. For example, in the case of vibration, the amplitude and period of vibration are defined and recorded in advance for each type of object. Also, if the type of object is, for example, "dog," vibration information on the amplitude and period corresponding to a running dog, amplitude and period corresponding to a heartbeat, amplitude and period corresponding to a bark, etc. may be recorded. In this way, multiple pieces of tactile information for the same type of object may be recorded in the tactile information table memory 112_C in advance.

[0019] Based on the type of object read from object information memory 112_B, tactile information determination unit 114 (determination unit) reads tactile information prepared for each object type from tactile information table memory 112_C and outputs it to tactile information presentation memory 112_D. In addition, the intensity of tactile information may be adjusted according to the size of the detected object read from object information memory 112_B and output to tactile information presentation memory 112_D. For example, in the case of vibration, if the tactile sensation is expressed by the period and amplitude, the amplitude is adjusted based on the size of the detected object and output to tactile information presentation memory 112_D. This allows for the presentation of tactile information with a high degree of freedom according to the detected size, even for objects of the same type.

[0020] The haptic information determination unit 114 may further include a haptic information control determination unit that determines whether to control the haptic information. It may also determine whether to control the presentation of haptic information for an object area detected by the object detection unit 113 based on the type of object read from the object information memory 112_B. For example, in a haptic information presentation device for vehicles, if the type of object detected and identified by the object detection unit 113 is a "car" or a "train," the haptic information control determination unit will determine to control the haptic information. If the type of object detected and identified by the object detection unit 113 is an animal such as a "dog" or a "bird," the haptic information control determination unit will determine not to control the haptic information.

[0021] Furthermore, the tactile information determination unit 114 is configured to store not only the object type read from the object information memory 112_B, but also multiple tactile information presentation modes for the same object type as described above. Different patterns of tactile information for the same object type may be read out by a user's prior setting. For example, if the tactile information is vibration and the tactile information presentation mode is the "movement mode," vibration information with amplitude and period corresponding to the detected object's movement is read out. If the tactile information is vibration and the tactile information presentation mode is the "picture book mode," vibration information with amplitude and period corresponding to the detected object's heartbeat is read out. If the tactile information is vibration and the tactile information presentation mode is the "resonance mode," vibration information with amplitude and period corresponding to the detected object's cry is read out. Based on the object type and the set tactile information presentation mode, the corresponding tactile information is read out from the tactile information table memory 112_C and output to the tactile information presentation memory 112_D.

[0022] In addition to reading tactile information from the tactile information table memory 112_C, the tactile information determination unit 114 may further include a tactile information input unit that inputs tactile information corresponding to image data from, for example, an external device outside the tactile information presentation device 100. The tactile information for the corresponding object region may be extracted from the input tactile information and output to the tactile information presentation memory 112_D. Furthermore, if the corresponding object region is composed of multiple pieces of tactile information of the same type, the information may be smoothed and output to the tactile information presentation memory 112_D, or one piece of tactile information may be selected from the multiple pieces and output to the tactile information presentation memory 112_D. Alternatively, the image data may be input to a tactile information inferring device that infers tactile information according to a learned type using learning data, and the resulting inference tactile information may be output to the tactile information presentation memory 112_D as tactile information for the target object. Furthermore, if there are multiple means for acquiring tactile information, such as reading from tactile information table memory 112_C, tactile information input from a tactile information input unit, or inference by a tactile information inference device, the acquisition means may be switched depending on the type of object.

[0023] The tactile information presentation device 100 is configured such that the image data input unit 101 further includes an object information input unit that inputs from the outside an object region in image data, an object type associated with the object region, and tactile information of the object region. When the object type in the input object information matches a type arbitrarily determined in advance by the user of the tactile information presentation device 100, the presentation control unit 115 may adopt the input object region, object type, and tactile information as information about the corresponding object.

[0024] The presentation control unit 115 reads the tactile information determined by the tactile information determination unit 114 from the tactile information presentation memory 112_D and controls the tactile sensations presented by the presentation unit 120. Furthermore, the presentation control unit 115 superimposes object information read from the object information memory 112_B, such as the object area, object type, and number of objects, on the image data read from the image memory 112_A. The superimposition of object information may be, for example, a frame display of the detected object area or an icon representing the object identification result. Furthermore, symbols, characters, icons, or marks that notify the user of the presentation target of the tactile information determined by the tactile information determination unit 114 may be superimposed.

[0025] The presentation unit 120 is a liquid crystal display, a touch display, a tactile device, etc. The presentation unit 120 may be configured as a device in which a display and a tactile device are integrated, such as a smartphone, or as a device in which a display that presents images and object information and a tactile device that presents tactile information to the user are separate.

[0026] Haptic devices include, for example, vibration devices made up of two-dimensionally arranged eccentric motors and linear resonant actuators, and uneven devices in which multiple two-dimensionally arranged pins each move up and down independently. There are also thermal sensation devices made up of two-dimensionally arranged semiconductor thermoelectric elements called Peltier elements. Furthermore, haptic devices can also be ring-shaped or stick-shaped devices that can be worn on the hand.

[0027] The presentation unit 120 is a device that can present tactile information, image data, and object information, and any presentation device may be used as long as it can achieve the object of the present invention.

[0028] The nonvolatile memory 132 is an electrically erasable and recordable memory, and may be, for example, an EEPROM, etc. Constants, programs, etc. for the operation of the system control unit 130 are stored in the nonvolatile memory 132.

[0029] The system control unit 130 controls the entire tactile information presentation device 100. Each process is realized by executing the program recorded in the nonvolatile memory 132 described above.

[0030] The system memory 133 uses RAM, and develops constants and variables for the operation of the system control unit 130, programs read from the nonvolatile memory 132, and the like.

[0031] The operation unit 131 is an operation means for inputting various operation instructions to the system control unit 130. The operation members of the operation unit 131 can be used to intuitively perform various settings by selecting and operating various function icons displayed on the presentation unit 120, or by operating various function buttons and four-way keys (up, down, left, and right). The operation unit 131 may also be configured with the operation members on, for example, a ring-shaped tactile device, and various settings may be performed using gesture operations such as tapping, swiping, and pinching with the hand wearing the ring, or button operations.

[0032] FIG. 2 shows a flowchart of the tactile information presentation control of the tactile information presentation device 100 in this embodiment.

[0033] In S200, the image data input unit 111 records the input image data in the image memory 112_A.

[0034] In S201, the object detection unit 113 reads image data from the image memory 112_A and performs object detection.

[0035] In S202, if the object detection unit 113 has not detected any objects, the process ends. If an object has been detected, the object detection unit 113 records the object information (position, size, and number of detected objects) in the object information memory 112_B, and proceeds to S203.

[0036] In S203, the object detection unit 113 identifies the type of the detected object and records the identification result in the object information memory 112_B.

[0037] In S204, the tactile information determination unit 114 reads the identification result from the object information memory 112_B and determines whether the type of the detected object matches the type of object that is the target of tactile information presentation control. If they match, the process proceeds to S205. If they do not match, the process ends.

[0038] In S205, the tactile information determination unit 114 reads tactile information corresponding to the object type from the tactile information table list in the tactile information table memory 112_C, links the object information with the tactile information, and records the linked information in the tactile information presentation memory 112_D.

[0039] In S206, the presentation control unit 115 reads out tactile information from the tactile information presentation memory 112_D and controls the tactile sensation presented by the presentation unit 120. The user of the tactile information presentation device 100 receives the presentation of tactile information by touching the presentation unit 120 with a finger, a palm, or the like.

[0040] As described above, according to this embodiment, an object is detected from image data input to the tactile information presentation device 100, the type of the detected object is identified, and tactile information corresponding to the identified object region is determined based on the object identification results. By controlling the tactile information corresponding to the object region, it is possible to present tactile information with a high degree of freedom, limited to the target intended by the user of the tactile information presentation device 100.

[0041] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist thereof. For example, the flowchart of this embodiment describes a method for reading tactile information from a tactile information table list stored in tactile information table memory 112_C based on the identification result. Here, a tactile information inferring device that infers tactile information according to a learned type using learning data may be used, and the tactile information obtained by the inference by the tactile information inferring device may be presented to the user regardless of the object detected.

[0042] <Second embodiment> Next, a second embodiment will be described. Here, the differences from the first embodiment will be mainly described. In the second embodiment, vibration is presented as an example of tactile information, and vibration adjustment control is performed to adjust the amplitude of the vibration presented based on the detection result and identification result of the object detection unit 113, based on the user's operation on the operation unit 131.

[0043] The flow from image data input to object detection, object identification, and tactile information control is the same as in the first embodiment, so a description thereof will be omitted.

[0044] FIG. 3 shows a flowchart of vibration adjustment control in this embodiment.

[0045] In S300, the presentation unit 120 presents vibration of the object that is the vibration presentation target.

[0046] In S301, the user of the tactile information presentation device 100 performs a tap operation using the operation unit 131.

[0047] In S302, if the tap operation in S301 was a single tap, it is recognized as a control request to amplify the amplitude, and the process proceeds to S303. If the tap operation in S301 was a double tap, it is recognized as a control request to decrease the amplitude, and the process proceeds to S304.

[0048] In S303, the presentation control unit 115 amplifies the amplitude of the vibration presented by the presentation unit 120 by one step.

[0049] In S304, the presentation control unit 115 decreases the amplitude of the vibration presented by the presentation unit 120 by one step.

[0050] As described above, according to this embodiment, an object in image data input to the tactile information presentation device 100 is detected, the type of the detected object is identified, and tactile information corresponding to the area of ​​the identified object is determined based on the identified object type. Tactile control of the tactile information presentation area is then performed, an object in image data input to the tactile information presentation device 100 is detected, and the type of the detected object is identified. Tactile information corresponding to the identified object area is determined based on the object identification result, the tactile information corresponding to the object area is controlled, and vibration adjustment control is performed through operation of the tactile information presentation device 100 by the user. This allows for the presentation of tactile information with a high degree of freedom.

[0051] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications and alterations are possible within the scope of the gist thereof. In the flowchart of the vibration adjustment control of this embodiment, a configuration in which a tap operation is performed as a user operation has been described. However, the user may also request vibration adjustment control by sliding left and right, up and down on the display, or by pinching in and out. Furthermore, if the operation unit 131 is in a form that is worn on the hand, such as a ring-shaped device, a request for vibration adjustment control may be made by a hand gesture operation.

[0052] <Third embodiment> Next, a third embodiment will be described. In the third embodiment, the tactile information to be presented is determined and controlled based on the detection results and identification results of the object detection unit 113, as well as the mode setting determined by the user of the tactile information presentation device 100. Figure 4 shows a flowchart of the tactile information presentation control of the tactile information presentation device 100 in this embodiment.

[0053] In S400, the image data input unit 111 records the input image data in the image memory 112_A.

[0054] In S401, the object detection unit 113 reads image data from the image memory 112_A and performs object detection.

[0055] In S402, if the object detection unit 113 has not detected any objects, the process ends. If an object has been detected, the object detection unit 113 records the object information (position, size, and number of detected objects) in the object information memory 112_B, and proceeds to S403.

[0056] In S403, the object detection unit 113 identifies the type of the detected object and records the identification result in the object information memory 112_B.

[0057] In S404, the tactile information determination unit 114 reads the identification result from the object information memory 112_B and determines whether the type of the detected object matches the type of object that is the target of tactile information presentation control. If they match, the process proceeds to S405. If they do not match, the process ends.

[0058] In S405, if the tactile information presentation mode has been set in advance by the user, the process proceeds to S406. If the tactile information presentation mode has not been set by the user, the process proceeds to S408.

[0059] In S406, if tactile information corresponding to the set tactile information presentation mode is present in tactile information table memory 112_C, the process proceeds to S407. If tactile information corresponding to the set tactile information presentation mode is not present in tactile information table memory 112_C, the process proceeds to S408.

[0060] In S407, the tactile information determination unit 114 reads tactile information corresponding to the type of object and tactile information presentation mode from the tactile information table list in the tactile information table memory 112_C, links the object information with the tactile information, and outputs it to the tactile information presentation memory 112_D.

[0061] In S408, the tactile information determination unit 114 reads tactile information corresponding to the object type from the tactile information table list in the tactile information table memory 112_C, links the object information with the tactile information, and outputs the linked information to the tactile information presentation memory 112_D.

[0062] In S409, the presentation control unit 115 reads out tactile information from the tactile information presentation memory 112_D and controls the tactile sensation presented on the presentation unit 120. The user of the tactile information presentation device 100 receives the presentation of tactile information by touching the presentation unit 120 with a finger, a palm, or the like.

[0063] The "movement mode" and "picture book mode" which are examples of the tactile presentation mode of this embodiment will be described below.

[0064] In the "motion mode," for example, if the target object is a horse, the presentation unit 120 presents vibrations with amplitudes and periods corresponding to the speed and intensity of the horse's movement. For example, even if the target is the same horse, by presenting a larger amplitude to a running horse compared to a walking horse, the user can more easily experience the magnitude of its movement. Also, even if the target is the same horse, by presenting a longer periodic vibration to a horse running from a distance compared to a horse running closer, the user can more easily experience the sense of distance of the horse's movement. Furthermore, the amplitude and period presented by the presentation unit 120 may be configured to change depending on the size of the object. For example, if the target is a horse, by making the amplitude larger compared to when the target is a rabbit, the user can more easily experience the dynamism of the movement due to the larger size of the target.

[0065] Furthermore, in the "picture book mode," for example, if the target object is a horse, the presentation unit 120 presents vibrations with amplitude and period corresponding to the horse's heartbeat. At this time, the amplitude and period presented by the presentation unit 120 may be configured to change depending on the size of the object. For example, if the target is a horse, the presentation unit 120 presents vibrations with a larger amplitude than if the target is a rabbit, making it easier to experience the difference in size of the object.

[0066] As described above, according to this embodiment, an object in image data input to the tactile information presentation device 100 is detected and the type of the detected object is identified. Then, based on the object identification result and the tactile information presentation mode preset by the user, tactile information corresponding to the identified object region is determined and the tactile information corresponding to the object region is controlled. This makes it possible to present tactile information that is limited to the target intended by the user of the tactile information presentation device 100 and with a high degree of freedom.

[0067] <Fourth embodiment> Next, a fourth embodiment will be described. In the fourth embodiment, the detection results and identification results of the object detection unit 113 are presented to the presentation unit 120, and the tactile information presentation target to be presented is switched when the user selects a tactile information presentation target. Figure 5 shows a flowchart of tactile information presentation control of the tactile information presentation device 100 in this embodiment.

[0068] In S500, the image data input unit 111 records the input image data in the image memory 112_A.

[0069] In S501, the object detection unit 113 reads image data from the image memory 112_A and performs object detection.

[0070] In S502, if the object detection unit 113 has not detected any objects, the process ends. If an object has been detected, the object detection unit 113 records the object information (position, size, and number of detected objects) in the object information memory 112_B, and proceeds to S403.

[0071] In S503, the object detection unit 113 identifies the type of the detected object and records the identification result in the object information memory 112_B.

[0072] In S504, the tactile information determination unit 114 reads the identification result from the object information memory 112_B and determines whether the type of the detected object matches the type of object that is the target of tactile information presentation control. If they match, S505 is executed. If they do not match, the process ends.

[0073] In S505, the presentation control unit 115 reads out the object detection result from the object information memory 112_B, and presents the result on the presentation unit 120 by superimposing a detection frame based on the object position on the image data read out from the image memory 112_A.

[0074] In S506, the user selects an object for which tactile information is to be presented from the object detection results presented on the presentation unit 120.

[0075] In S507, the tactile information determination unit 114 reads out tactile information corresponding to the type of object to which tactile information is to be presented, selected from the tactile information table list in the tactile information table memory 112_C, links the object information to the tactile information, and outputs the information to the tactile information presentation memory 112_D.

[0076] In S508, the presentation control unit 115 reads out tactile information from the tactile information presentation memory 112_D and controls the tactile sensation presented by the presentation unit 120. The user of the tactile information presentation device 100 receives the presentation of tactile information by touching the presentation unit 120 with a finger, a palm, or the like.

[0077] In S509, if the target for tactile information presentation is to be reselected, the process proceeds to S505; if not, the process ends.

[0078] As described above, according to this embodiment, the detection results and identification results are presented to the presentation unit 120, and the user selects a target for tactile information presentation and switches the target for tactile information presentation to be presented. This allows tactile information presentation to be limited to the target intended by the user of the tactile information presentation device 100, while still allowing for a high degree of freedom in the presentation of tactile information.

[0079] <Other embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.Furthermore, the present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0080] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0081] 100 Tactile information presentation device 110 Image processing section 111 Image data input unit 112 memory 113 Object detection unit 114 Tactile Information Determination Unit 115 Presentation control unit 120 Presentation section 130 System control unit 131 Operation section 132 Non-volatile memory 133 system memory

Claims

1. an object detection unit that detects an object from image data; a determination unit that determines tactile information according to the object detected by the object detection unit; a presentation unit that presents the tactile information determined by the determination unit to a user.

2. The control device according to claim 1 , wherein the determination unit determines the tactile information of the object region according to the type of object detected by the object detection unit and a preset tactile information presentation mode.

3. The control device according to claim 2 , wherein the tactile presentation mode includes a first mode in which the presentation unit presents tactile information corresponding to the movement of the object detected by the object detection unit.

4. The control device according to claim 3 , wherein the tactile presentation mode includes a second mode in which the presentation unit presents tactile information that is not based on the movement of the object detected by the object detection unit.

5. The control device according to claim 1 , wherein the determination unit determines tactile information according to the size of the object detected by the object detection unit.

6. The control device according to claim 1 , wherein the determination unit determines the tactile information by a tactile information determination means that uses machine learning.

7. The control device according to claim 1 , wherein the presentation unit presents tactile information corresponding to the object detected by the object detection unit determined by the determination unit in association with an area of ​​the image data corresponding to the object.

8. 2. The control device according to claim 1, wherein an object capable of presenting tactile information is notified by displaying on the presentation unit.

9. The control device according to claim 1 , further comprising an operation unit that accepts an operation to switch the tactile information presented by the presentation unit.

10. The control device according to claim 9 , wherein the operation unit switches the tactile information to be presented based on an operation by the user.

11. 10. The control device according to claim 9, wherein the operation of the operation unit to switch the tactile information is an operation to switch the object to which the tactile information is presented by the presentation unit.

12. The control device according to claim 9 , wherein the operation of the operation unit to switch the tactile information is an operation to switch the intensity of the tactile information presented by the presentation unit.

13. The control device according to claim 1, wherein the tactile information is at least one of vibration information, unevenness information, sound information, and temperature information.

14. A control method for a control device having a presentation unit, an object detection step of detecting an object from the image data; a tactile information determination step of determining tactile information corresponding to the object detected in the object detection step; a presentation step of presenting the tactile information determined in the tactile information determination step to a user via a presentation unit.

Images