Haptic feedback apparatus, control method, storage medium, and computer program product

Abstract

The invention discloses a haptic feedback apparatus, a control method, a storage medium, and a computer program product. A haptic feedback device that generates a tactile sensation on an operation unit capable of operating an operated unit obtains haptic information indicating the tactile sensation when the operated unit contacts an object, detects an object of interest to a user who operates the operation unit from among objects within a predetermined range including the object that the operated unit is contacting, and calculates a feedback amount for generating a tactile sensation in the operation unit on the basis of the information of the object concerned by the user and the tactile information.

Description

TECHNICAL FIELD

[0001] The present application relates to a haptic feedback device, a control method, a storage medium, and a computer program product, and particularly relates to the technical field of a technique for haptic feedback based on a user's degree of attention. BACKGROUND

[0002] Haptic feedback is a technique for transmitting the results and states of operations by applying physical stimuli such as vibrations and / or pressures to the hands or fingers of a user from an operation target object. Haptic feedback is used to confirm that operations have been properly performed by the user feeling vibrations and / or movements when the user operates an operation target object, or to enhance the usability and accessibility of a user interface by providing feedback as an important information source to a user with visual or auditory impairments via haptics. Users also use haptic feedback to properly perform operations of a robot arm or the like based on haptic information, or to improve the sense of immersion in a virtual space.

[0003] Japanese Patent Application Publication No. 2015-215894 describes a technique for obtaining haptic information by a tactile sensor when a user focuses on an object to which the tactile sensor is attached and performing haptic feedback.

[0004] In Japanese Patent Application Publication No. 2015-215894, haptic feedback can be performed only when the user focuses on an object to which the tactile sensor is attached, and haptic feedback cannot be performed when the user wants to obtain haptic information from an object to which the tactile sensor is not attached. Therefore, a technique capable of obtaining haptic information associated with an object on which a user focuses is desired. SUMMARY

[0005] The present application addresses the above-described problems and achieves a technique capable of performing haptic feedback according to a user's degree of attention.

[0006] To address the problem described above, the present application provides a haptic feedback device that generates a tactile sensation on an operation unit that can operate an operated unit, the haptic feedback device including: an obtaining unit that obtains haptic information representing a tactile sensation when the operated unit contacts an object; a detecting unit that detects an object on which a user operating the operation unit is focusing from among objects within a predetermined range including the object that the operated unit is contacting; and a calculating unit that calculates a feedback amount for generating a tactile sensation in the operation unit based on information of the object on which the user is focusing and the haptic information.

[0007] In order to solve the above-mentioned problems, the present invention provides a control method for a tactile feedback device, wherein the tactile feedback device generates a tactile sensation in an operating unit capable of operating the operated unit, the method comprising: a step of obtaining tactile information representing the tactile sensation when the operated unit contacts an object; a step of detecting an object focused on by a user operating the operating unit from objects within a predetermined range including the object being contacted by the operated unit; and a step of calculating a feedback amount for generating the tactile sensation in the operating unit based on information about the object focused on by the user and the tactile information.

[0008] The present invention provides a computer-readable storage medium storing a program for causing a computer to function as the above-mentioned tactile feedback device.

[0009] The present invention provides a computer program product, which includes a program for causing a computer to function as the above-mentioned tactile feedback device.

[0010] According to the present invention, tactile feedback can be performed based on the user's attention level.

[0011] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram schematically showing the appearance of a tactile feedback device according to the first embodiment.

[0013] Figure 2 is a block diagram showing the configuration of a tactile feedback device according to the first embodiment.

[0014] Figure 3 is a flowchart illustrating a haptic feedback process according to the first embodiment.

[0015] Figure 4 is a flowchart illustrating attention object detection processing according to the first embodiment.

[0016] Figure 5 is a block diagram showing a configuration of a tactile feedback apparatus including a gaze detection device according to the first embodiment.

[0017] Figure 6 is a flowchart illustrating gaze detection processing according to the first embodiment.

[0018] Figure 7 A and B are diagrams illustrating the gaze detection method according to the first embodiment.

[0019] Figure 8 is a diagram showing an eyeball image according to the first embodiment.

[0020] Figure 9 : is a flowchart illustrating attention object detection processing based on gaze information according to the first embodiment.

[0021] Figures 10A-10C is a diagram illustrating attention degree calculation processing based on gaze information according to the first embodiment.

[0022] Figure 11 is a flowchart illustrating tactile feedback amount calculation processing according to the first embodiment.

[0023] Figure 12A and Figure 12B is a diagram schematically showing the appearance of a tactile feedback device according to a second embodiment.

[0024] Figure 13 is a block diagram showing the configuration of a tactile feedback device according to a second embodiment.

[0025] Figure 14 is a flowchart illustrating a haptic feedback process according to the second embodiment.

[0026] Figure 15 is a flowchart illustrating haptic feedback amount calculation processing according to the second embodiment. DETAILED DESCRIPTION

[0027] The following embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but this does not limit the invention to one requiring all such features, and a plurality of such features may be appropriately combined. Furthermore, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0028] [First embodiment]

[0029] In the first embodiment, a tactile feedback device that generates a tactile sensation in an operating unit capable of operating a robot arm as an operated unit will be described.

[0030] <Device Configuration>

[0031] First, refer to Figure 1 and Figure 2 The configuration and functions of the tactile feedback device according to the first embodiment are described.

[0032] Figure 1 is a diagram schematically showing the appearance of a tactile feedback device according to the first embodiment.

[0033] The tactile feedback device 1 includes a controller 100 , a robot arm 101 and an operating unit 102 .

[0034] The operation unit 102 is wearable on the hand of the user. When the user moves his / her hand or finger while wearing the operation unit 102, the arm portion 101a and the hand portion 101b of the robot arm 101 can move in accordance with the movement. When the user moves his / her hand or finger while wearing the operation unit 102, the controller 100 receives operation information corresponding to the user's operation from the operation unit 102, and outputs drive information corresponding to the operation information to the robot arm 101. The robot arm 101 drives the arm portion 101a and the hand portion 101b of the robot arm 101 on the basis of the drive information received from the controller 100.

[0035] The controller 100 is provided with a display unit 104 that displays a GUI (graphical user interface of the controller 100). In addition, the display unit 104 displays an image of the hand portion 101b of the robot arm 101 and the periphery of the hand portion 101b, which is captured by the camera unit 108.

[0036] A tactile information detection unit 106 is provided on the hand portion 101b of the robot arm 101. The tactile information detection unit 106 is capable of detecting tactile information applied to the hand portion 101b when the robot arm 101 holds an object or the like. The tactile information detection unit 106 includes a force sensor that is capable of detecting force information as tactile information, and outputs the force information detected by the force sensor as tactile information from the robot arm 101 to the controller 100.

[0037] The controller 100 calculates a tactile feedback amount on the basis of tactile information obtained from the tactile information detection unit 106 and information about an object on which the user focuses (focus object information), and outputs the tactile feedback amount to a tactile generation unit 110.

[0038] The tactile generation unit 110 is formed of a piezoelectric element or the like that applies physical stimulation such as vibration and / or pressure to the hand or finger of the user who wears the operation unit 102, and is provided on the hand or finger of the operation unit 102. The tactile generation unit 110 generates a tactile sensation such as vibration and / or pressure corresponding to the tactile feedback amount received from the controller 100, and transmits the tactile sensation to the user via the operation unit 102.

[0039] Figure 2 is a block diagram showing a hardware configuration of a tactile feedback device according to a first embodiment.

[0040] The controller 100 includes a control unit 103 and a display unit 104. The control unit 103 includes a processor (CPU) that performs arithmetic processing and control processing of the haptic feedback device 1, a volatile memory (ROM) that stores a program executed by the processor, a program read from a nonvolatile memory, and a work memory (RAM) that loads constants and variables for executing the program. The control unit 103 controls the respective components of the haptic feedback device 1 by loading the program stored in the ROM into the RAM and executing the program.

[0041] Instead of the control unit 103 controlling the entire device, a plurality of hardware can share processing to control the entire device.

[0042] The display unit 104 includes a liquid crystal panel, an organic EL panel, or the like, and displays images and various information.

[0043] The robot arm 101 includes a force sensor 105, a haptic information detection unit 106, a driving unit 107, and a camera unit 108.

[0044] The force sensor 105 detects force information applied to the hand 101b when the robot arm 101 holds an object or the like.

[0045] The haptic information detection unit 106 outputs haptic information detected by the force sensor 105 to the control unit 103.

[0046] The driving unit 107 drives the arm 101a and the hand 101b of the robot arm 101 based on driving information received from the control unit 103.

[0047] The camera unit 108 includes an image sensor and an A / D converter, converts an object image into an electric signal, and outputs image data including a digital signal to the control unit 103. The control unit 103 controls the display unit 104 to display an image of the hand 101b and a periphery of the hand 101b of the robot arm 101 photographed by the camera unit 108, and an image within a predetermined range including a portion contacted by the hand 101b when the hand 101b holds an object.

[0048] The operation unit 102 includes an operation information transmission unit 109 and a haptic generation unit 110.

[0049] The operation information transmission unit 109 outputs operation information of a user wearing the operation unit 102 to the control unit 103. The control unit 103 outputs driving information corresponding to the operation information received from the operation information transmission unit 109 to the driving unit 107. The driving unit 107 drives the arm 101a and the hand 101b of the robot arm 101 based on the driving information received from the control unit 103.

[0050] The tactile sensation generating unit 110 applies a tactile sensation such as vibration and / or pressure to the operating unit 102 according to the amount of tactile feedback received from the control unit 103 , and feeds back the tactile sensation to the user.

[0051] exist Figure 1 and Figure 2 In the embodiment, the operation unit 102 and the control unit 103 are provided separately, but they may be integrated into the same device.

[0052] <Control Processing>

[0053] Next, we will refer to Figure 3 A tactile feedback process based on the user's attention level according to the first embodiment is described.

[0054] Figure 3 The processing is realized by the control unit 103 of the controller 100 executing the program stored in the ROM and controlling the display unit 104, the robot arm 101 and the operation unit 102. The same applies to the later described Figure 4 、 Figure 6 、 Figure 9 and Figure 11 .

[0055] In step S301, the control unit 103 detects information about an object of interest. Figure 4 The description focuses on the details of the object detection process.

[0056] In step S302 , the control unit 103 obtains tactile information from the tactile information detection unit 106 .

[0057] In step S303 , the control unit 103 calculates a tactile feedback amount for causing the tactile sense generation unit 110 to generate a tactile sensation based on the attention object information obtained in step S301 and the tactile sense information obtained in step S302 .

[0058] In step S304 , the control unit 103 transmits the tactile feedback amount calculated in step S303 to the tactile sense generating unit 110 .

[0059] In step S305 , the control unit 103 causes the tactile sense generation unit 110 to generate a tactile sensation corresponding to the tactile feedback amount.

[0060] <Focus on Object Detection Processing>

[0061] Next, about Figure 3 Regarding the attention object detection processing in step S301, examples of processing using image information and processing using gaze information will be described.

[0062] <Attention Object Detection Processing Using Image Information>

[0063] Next, a focus object detection process using image information will be described with reference to Figure 4

[0064] In step S401, the control unit 103 detects an object in an image captured by the camera unit 108 and displayed on the display unit 104, and identifies the region of the object.

[0065] In step S402, the control unit 103 initializes the focus degree to 0. The value of the focus degree is an index indicating the degree of focus of the user.

[0066] In step S403, the control unit 103 determines whether there is an object that overlaps the hand 101b (tactile information detection unit 106) for a predetermined period of time or more among the objects detected in step S401. When it is determined that there is an object that overlaps the hand 101b for a predetermined period of time or more, the process proceeds to step S404, otherwise, the process skips the process of step S404 and proceeds to step S405.

[0067] In step S404, the control unit 103 adds 1 to the focus degree of the object that overlaps the hand 101b for a predetermined period of time or more.

[0068] In step S405, the control unit 103 determines whether there is an object that is located within a predetermined region from the center of the display unit 104 for a predetermined period of time or more among the objects detected in step S401. When it is determined that there is an object that is located within a predetermined range from the center of the image for a predetermined period of time or more, the process proceeds to step S406, otherwise, the process skips the process of step S406 and proceeds to step S407.

[0069] In step S406, the control unit 103 adds 1 to the value of the focus degree of the object that is located within a predetermined region from the center of the image for a predetermined period of time or more.

[0070] In step S407, the control unit 103 determines whether there is an object whose region size is equal to or greater than a predetermined size among the objects detected in step S401. When it is determined that there is an object whose region size is equal to or greater than a predetermined size, the process proceeds to step S408, otherwise, the process skips the process of step S408 and proceeds to step S409.

[0071] In step S408, the control unit 103 adds 1 to the value of the focus degree of the object whose region size is equal to or greater than a predetermined size.

[0072] ​In step S409, the control unit 103 determines whether there is an object whose value of the attention degree is equal to or greater than 1. When it is determined that there is an object whose value of the attention degree is equal to or greater than 1, the process proceeds to step S410, otherwise, the process proceeds to step S411.

[0073] In step S410, the control unit 103 determines that the object having the largest area size among the objects having the highest attention degree is the attention object.

[0074] In step S411, the control unit 103 determines that there is no attention object.

[0075] <Attention object detection processing using gaze information>

[0076] Next, the attention object detection processing using gaze information will be described with reference to Figures 5 to 8

[0077] Figure 5 is a block diagram showing a hardware configuration of a haptic feedback device including a gaze detection device.

[0078] The controller 100 is provided with an eyeball imaging unit 201 and an illumination light source 202 as a device for detecting a user's gaze.

[0079] The eyeball imaging unit 201 captures an image of the user's eyeball and outputs the captured image to the control unit 103. The control unit 103 performs gaze detection of the user based on the eyeball image.

[0080] The illumination light source 202 emits infrared light to the observer's eyeball so that the eyeball imaging unit 201 can capture an image of appropriate brightness for gaze detection.

[0081] Next, the gaze detection processing will be described with reference to Figure 6 and Figure 7

[0082] Figure 6 is a flowchart showing the gaze detection processing. Figure 7 A and B of FIG. 7 are diagrams showing a gaze detection method.

[0083] In step S601, the control unit 103 emits infrared light to the observer's eyeball 71 through the illumination light source 202. An image of the observer's eyeball illuminated with the infrared light is formed on the eyeball imaging unit 201. The eyeball imaging unit 201 generates an eyeball image signal obtained by converting the eyeball image formed on the eyeball imaging unit 201 into an electric signal.

[0084] In step S602, the control unit 103 obtains the eyeball image signal from the eyeball imaging unit 201.

[0085] ​​In step S603, the control unit 103 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe and the pupil center c of the illumination light source 202 shown in A of FIG. 6 from the information of the eye image signal obtained in step S602. Figure 7 The coordinates of the points corresponding to the corneal reflection images Pd and Pe and the pupil center c of the illumination light source 202 shown in A of FIG. 6 are obtained from the information of the eye image signal.

[0086] Figure 7 A of FIG. 6 shows the corneal reflection images obtained from the eye image. Figure 7 B of FIG. 6 shows the corneal reflection images obtained from Figure 7 The luminance information obtained from the region a of the eye image of A of FIG. 6.

[0087] The infrared light emitted from the illumination light source 202 illuminates the cornea of the observer's eye 71, and the corneal reflection images Pd and Pe formed by the portion of the infrared light reflected on the corneal surface form images on the eye imaging unit 201 (Pd' and Pe' in A of FIG. 6). Similarly, the light beams from the ends a and b of the pupil 72 form images on the eye imaging unit 201. Figure 7 The coordinates of the points corresponding to the corneal reflection images Pd and Pe and the pupil center c of the illumination light source 202 shown in A of FIG. 6 are obtained from the information of the eye image signal.

[0088] In A and B of FIG. 6, the horizontal direction is the X-axis, the vertical direction is the Y-axis, and the coordinates of the images Pd' and Pe' formed by the corneal reflection images of the illumination light source 202 in the X-axis direction (horizontal direction) are Xd and Xe. In addition, in A and B of FIG. 6, the coordinates of the images a', b' formed by the light beams from the ends of the pupil 72 in the X-axis direction are taken as Xa and Xb. Figure 7 Figure 7 In A and B of FIG. 6, the horizontal direction is the X-axis, the vertical direction is the Y-axis, and the coordinates of the images Pd' and Pe' formed by the corneal reflection images of the illumination light source 202 in the X-axis direction (horizontal direction) are Xd and Xe. In addition, in A and B of FIG. 6, the coordinates of the images a', b' formed by the light beams from the ends of the pupil 72 in the X-axis direction are taken as Xa and Xb.

[0089] In A and B of FIG. 6, the horizontal direction is the X-axis, the vertical direction is the Y-axis, and the coordinates of the images Pd' and Pe' formed by the corneal reflection images of the illumination light source 202 in the X-axis direction (horizontal direction) are Xd and Xe. In addition, in A and B of FIG. 6, the coordinates of the images a', b' formed by the light beams from the ends of the pupil 72 in the X-axis direction are taken as Xa and Xb. Figure 8 ​In the luminance information of B, extremely high luminance is obtained at positions Xd and Xe corresponding to the images Pd' and Pe', at which the corneal reflection images of the illumination light source 202 are formed. The region between the coordinates Xa and Xb corresponding to the region of the pupil 72 results in extremely low levels of luminance other than the positions Xd and Xe. On the other hand, in the regions having X coordinate values lower than Xa and the regions having X coordinate values higher than Xb corresponding to the regions of the iris 73 outside the pupil 72, intermediate values of the two levels of luminance are obtained. From the change information of the luminance levels with respect to the X coordinate positions, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of the illumination light source 202, and the X coordinates Xa and Xb of the images a' and b' of the pupil end can be obtained. In addition, in the case where θx of the optical axis of the eyeball 14 with respect to the optical axis of the eyeball imaging unit 201 is small, the coordinate Xc of the portion corresponding to the pupil center c formed as an image on the eyeball imaging unit 201, which is referred to as c', can be expressed as Xc ≒ (Xa + Xb) / 2. In this way, the X coordinate of c' corresponding to the pupil center formed as an image on the eyeball imaging unit 201 and the coordinates of the corneal reflection images Pd' and Pe' of the eyeball imaging unit 201 can be estimated.

[0090] In step S604, the control unit 103 calculates the image formation magnification β of the eyeball image, which is determined by the position of the eyeball 71 with respect to the optical system of the eyeball imaging unit 201, and can be obtained substantially from the distance (Xd - Xe) of the corneal reflection images Pd' and Pe'.

[0091] In step S605, since the X coordinate of the midpoint of the corneal reflection images Pd' and Pe' substantially coincides with the X coordinate of the center of curvature O of the cornea 74, the control unit 103 can obtain θx of the optical axis of the eyeball 71 in the Z-X plane from Equation 1 assuming that the normal distance between the center of curvature O of the cornea 74 and the center c of the pupil 72 is Oc.

[0092] (Equation 1)

[0093] B * Oc * SIN θx ≒ {(Xd + Xe) / 2} - Xc

[0094] Figure 9 An example of calculating the rotation angle θx when the eyeball of the observer rotates in a plane perpendicular to the Y axis is shown, which is equally applicable to the calculation method of the rotation angle θy when the eyeball of the observer rotates in a plane perpendicular to the X axis.

[0095] In step S606, the control unit 103 reads the gaze correction coefficients Ax, Bx, Ay, and By from the ROM to correct for individual differences in the user's gaze. The gaze correction coefficients Ax, Bx, Ay, and By represent the sensitivity coefficients for the offset amount and rotation angle in the x-direction and the offset amount and rotation angle in the y-direction. These values ​​are obtained by performing a calibration operation and are stored in the ROM.

[0096] In step S607, the control unit 103 uses θx and θy to obtain the gaze position (line of sight position) of the user on the display unit 104. Assuming that the coordinates corresponding to the center c of the pupil 72 on the display unit 104 are the left eye (Hx_L, Hy_L) and the right eye (Hx_R, Hy_R), respectively, the gaze position can be calculated according to Equation 2.

[0097] (Equation 2)

[0098] Hx_L=m*(Ax*θx+Bx)*nLx

[0099] Hx_R=m*(Ax*θx+Bx)*nRx

[0100] Hy_L=m*(Ay*θy+By)*nLy

[0101] Hy_R=m*(Ay*θy+By)*nRy

[0102] The coefficient m is a constant determined by the configuration of the optical system of the gaze detection device and is a conversion coefficient for converting the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 72 in the display unit 104. The correction coefficients (nLx, nLy, nRx, nRy) and the coefficient m for correcting the reliability of the left eye and the right eye when determining the gaze position are determined in advance.

[0103] Then, the control unit 103 calculates the total gaze position (Hx, Hy) based on Hx_L, Hx_R, Hy_L, and Hy_R.

[0104] Next, we will refer to Figures 6 to 8 Describe the use as reference Figure 9 The detected gaze information is used for object of interest detection processing.

[0105] Figures 10A-10C : is a flowchart showing the attention object detection process using gaze information.

[0106] In step S901 , the control unit 103 detects an object(s) from an image displayed on the display unit 104 and determines a region of each object.

[0107] In step S902, the control unit 103 detects the gaze direction of the user and determines the position at which the user is gazing at the display unit 104.

[0108] In step S903, the control unit 103 initializes the value of the attention degree to 0.

[0109] In step S904, the control unit 103 determines whether there is an object (gazed object) at which the user is continuously gazing for a predetermined period of time or more from the current time among the objects detected in step S901. When it is determined that there is a gazed object, the process proceeds to step S905, otherwise, the process skips the process of step S905 and proceeds to step S906.

[0110] In step S905, the control unit 103 adds 1 to the value of the attention degree of the gazed object.

[0111] In step S906, the control unit 103 compares the speed and / or the direction of travel of the user's gaze direction with the speed and / or the direction of travel of the gazed object, respectively, and determines whether the difference between the user's gaze direction and the speed and / or the direction of travel of the gazed object falls within a predetermined difference. When it is determined that the difference between the user's gaze direction and the speed and / or the direction of travel of the gazed object falls within the predetermined difference, the process proceeds to step S907, otherwise, the process skips the process of step S907 and proceeds to step S908.

[0112] In step S907, the control unit 103 adds 1 to the value of the attention degree of the gazed object.

[0113] In step S908, the control unit 103 determines whether there is a gazed object in the gaze direction for a total of a predetermined period of time or more while the gazed object is being detected, and determines whether there is an object at which the user is gazing for a long time. When it is determined that there is an object at which the user is gazing for a long time, the process proceeds to step S909, otherwise, the process skips the process of step S909 and proceeds to step S910.

[0114] In step S909, the control unit 103 adds 1 to the value of the attention degree of the object at which the user is gazing for a long time.

[0115] Here, the determination processes of steps S904 and S908 will be described with reference to Figures 10A-10C Figs. 10A to 10C.

[0116] Figures 10A-10C The time transition of the object in the gaze direction at which the user is gazing is shown. In Figure 10A the detection period of the gazed object is 10 seconds from the current time, but the present application is not limited to this, and a different period can be set.

[0117] Hereinafter, the judgment threshold of step S904 will be described as 4 seconds, and the judgment threshold of step S908 will be described as 6 seconds.

[0118] In Figure 10B , since the user is gazing at object 1 for 4 seconds or more from the current time, and the judgment condition of step S904 is satisfied, the value of the attention degree of object 1 is incremented by 1. Further, since no object is gazed at for more than 6 seconds, there is no object whose attention degree is incremented in step S908.

[0119] In Figure 10C , since there is no object that the user is gazing at for 4 seconds or more from the current time, there is no object that satisfies the judgment condition of step S904 and whose attention degree is incremented. Further, since the user is gazing at object 1 for a total of 6 seconds during the judgment period, in step S908, the value of the attention degree of object 1 is incremented by 1.

[0120] In Figure 11 , since the user is gazing at object 1 for 5 seconds or more from the current time, and the judgment condition of step S904 is satisfied, the value of the attention degree of object 1 is incremented by 1. Further, since the user is gazing at object 1 for a total of 9 seconds during the judgment period, in step S908, the value of the attention degree of object 1 is incremented by 1.

[0121] In step S910, the control unit 103 judges whether there is an object whose attention degree is 1 or more. When it is judged that there is an object whose attention degree is 1 or more, the process proceeds to step S911, otherwise, the process proceeds to step S912.

[0122] In step S911, the control unit 103 judges that the object having the largest region size among the objects having the highest attention degree is the attention object.

[0123] In step S912, the control unit 103 judges that there is no attention object.

[0124] <Calculation of the amount of tactile feedback>

[0125] Next, with reference to Figure 3 , the process of calculating the amount of tactile feedback in step S303 will be described. Figure 3

[0126] When tactile information is detected in step S302 of Figure 11 , the process of Figure 4 begins, and the attention object judgment process is performed in Figure 9 or Figure 12A .

[0127] ​In step S1101, the control unit 103 determines whether or not there is an object of interest. When it is determined that there is an object of interest, the process proceeds to step S1102, otherwise, the process proceeds to step S1105.

[0128] In step S1102, the control unit 103 determines whether or not the object of interest and the hand 101b are in contact with each other based on the image displayed on the display unit 104. When it is determined that the object of interest and the hand 101b are in contact with each other, the process proceeds to step S1103, otherwise, the process proceeds to step S1105.

[0129] In step S1105, the control unit 103 sets the amount of haptic feedback to 0, and does not perform haptic feedback.

[0130] In step S1103, the control unit 103 obtains haptic information from the haptic information detection unit 106.

[0131] In step S1104, the control unit 103 calculates the amount of haptic feedback based on the interest degree of the object of interest. Assuming that the amount of haptic feedback is H_fb, the haptic information is H_in, and the interest degree is Int, the amount of haptic feedback is calculated as in Equation 3.

[0132] (Equation 3)

[0133] H_fb = H_in (0.6 + 0.2 * Int)

[0134] As described above, according to the first embodiment, appropriate haptic feedback can be realized by calculating the amount of haptic feedback based on the interest degree of the user with respect to the object in the real space.

[0135] [Second Embodiment]

[0136] In the first embodiment, an example of haptic feedback based on the interest degree of the user with respect to the object in the real space has been described. In the second embodiment, an example of haptic feedback based on the interest degree of the user with respect to the object present in the virtual space will be described.

[0137] In the second embodiment, a haptic feedback device that generates a tactile sensation in an operation unit that is capable of operating an avatar in a virtual space as a manipulated unit will be described.

[0138] Figure 12B and Figure 12A is a front perspective view schematically showing the appearance of a haptic feedback device according to the second embodiment. Figure 12B is a front perspective view schematically showing the appearance of a haptic feedback device according to the second embodiment. Figure 13 is a front perspective view schematically showing the appearance of a haptic feedback device according to the second embodiment.

[0139] The haptic feedback device 2 of the second embodiment includes a controller 300, a goggle-type device 301 that is wearable on the head of a user, and an operation unit 302. The goggle-type device 301 includes a gaze detection device that detects a gaze direction of each of the left eyeball and the right eyeball. The left eye and the right eye are also referred to as the binoculars.

[0140] The goggle-type device 301 generates and visually displays an object such as an avatar in a virtual space.

[0141] The operation unit 302 has a left operation unit 302a and a right operation unit 302b that are wearable on the left and right hands of the user, and is used to operate the avatar in the virtual space viewed through the goggle-type device 301.

[0142] The goggle-type device 301 is provided with an imaging unit 305 and a light measurement unit 307.

[0143] The goggle-type device 301 is provided with an illumination light source 313a and 313b for each of the left eyeball and the right eyeball. The illumination light sources 313a and 313b are light sources such as light emitting diodes that emit infrared light to which the user is not sensitive, and each of the light sources illuminates the left eyeball and the right eyeball of the user. For each of the left eyeball and the right eyeball, a portion of the illumination light reflected by the eyeball converges on an eyeball imaging unit 315.

[0144] Further, the goggle-type device 301 is provided with a display unit 308a and 308b for each of the left eyeball and the right eyeball.

[0145] Figure 6 is a block diagram showing a hardware configuration of the haptic feedback device according to the second embodiment.

[0146] The controller 300 includes a control unit 103, a memory unit 304, a communication unit 306, a display control unit 311, a gaze detection unit 312, a display unit 341, and operation members 342 and 343.

[0147] The control unit 303 includes a processor (CPU) that performs arithmetic processing and control processing of the haptic feedback device 2, a volatile memory (ROM) that stores a program executed by the processor, a program read from a non-volatile memory, and a work memory (RAM) that loads constants and variables used to execute the program. The control unit 303 controls the respective components of the haptic feedback device 2 by loading the program stored in the ROM into the RAM and executing the program.

[0148] Instead of the control unit 303 controlling the entire device, a plurality of hardware can share the processing to control the entire device.

[0149] The memory unit 304 stores an image signal from the eyeball imaging unit 315, gaze correction data for correcting individual differences in gaze, and attention degree calculation information set for each object.

[0150] The communication unit 306 includes an interface that realizes communication with the Internet or an external device. The communication method can be wired or wireless.

[0151] The gaze detection unit 312 obtains an eyeball image from the eyeball imaging unit 315 and outputs the result of gaze detection to the control unit 303. The control unit 303 detects the gaze direction of the user in accordance with the algorithm described above. Figure 12A The gaze direction of the user is detected.

[0152] The display control unit 311 controls the display unit 341 on the basis of a display control signal from the control unit 303. The display unit 341 includes a liquid crystal panel, an organic EL panel, or the like, and displays an image and various information.

[0153] The operation members 342 and 343 accept user operations and transmit operation information corresponding to the user operations to the controller 300.

[0154] The eyewear-type device 301 includes display units 308a and 308b, an eyeball imaging unit 315, illumination light sources 313a and 313b, an imaging unit 305, and a light meter unit 307.

[0155] The display units 308a and 308b display an avatar in a virtual space for each of the left and right eyeballs.

[0156] The eyeball imaging unit 315 photographs an eyeball image of the user and outputs the photographed image to the control unit 303.

[0157] The illumination light sources 313a and 313b emit infrared light to the eyeballs of the observer. Therefore, the eyeball imaging unit 315 can photograph an image of appropriate brightness for gaze detection.

[0158] The imaging unit 305 photographs an image of a subject site viewable by the user and outputs an imaging signal generated thereby to the control unit 303. The light meter unit 307 as a light meter amplifies a luminance signal output corresponding to the luminance of the subject site on the basis of the imaging signal generated by the imaging unit 305, performs logarithmic compression and A / D conversion, and outputs the result as subject site luminance information to the control unit 303.

[0159] The eyewear-type device 301 can be a non-transmission type such as VR (Virtual Reality) or a transmission type such as AR (Augmented Reality).

[0160] In Figure 12B and Figure 13 andFigure 14 In the embodiment, the goggle-like device 301 and the control unit 303 are provided separately, but they may be integrated into the same device.

[0161] The operation units 302 a and 302 b include an operation information transmission unit 309 and a tactile sense generation unit 310 .

[0162] The operation information transmission unit 309 outputs the operation information of the user wearing the operation units 302a and 302b to the control unit 303. The control unit 303 reproduces the movement of the avatar in the virtual space according to the operation information received from the operation information transmission unit 309.

[0163] The tactile sensation generating unit 310 applies a tactile sensation such as vibration or pressure to the operation units 302 a and 302 b according to the amount of tactile feedback received from the control unit 303 , and feeds back the tactile sensation to the user.

[0164] <Haptic Feedback Processing>

[0165] Next, we will refer to Figure 14 A haptic feedback process according to the second embodiment is described.

[0166] Figure 15 The processing is realized by the control unit 303 of the controller 300 executing the program stored in the ROM and controlling the goggle-type device 301 and the operation units 302a and 302b. The same applies to the later described Figures 4 to 10A .

[0167] In step S1401, the control unit 303 detects an object of interest using the image displayed on the display unit 308. Figure 10C - Figure 15 The attention object detection process according to the first embodiment has been described.

[0168] In step 1402 , the control unit 303 calculates tactile information based on the object of interest detected in step 1401 and the movement of the avatar operated by the operation units 302 a and 302 b .

[0169] In step 1403 , the control unit 303 calculates a tactile feedback amount based on the attention object information and the tactile information obtained in step 1402 .

[0170] In step S1404 , the control unit 303 transmits the tactile feedback amount calculated in step S1403 to the tactile sense generating unit 310 .

[0171] In step S1405 , the control unit 303 causes the tactile sense generation unit 310 to generate a tactile sensation corresponding to the tactile feedback amount.

[0172] Figure 14 is a flowchart showing Figure 14 the tactile feedback calculation processing of steps S1402 and S1403.

[0173] In step S1501, the control unit 303 determines whether there is an object of interest, and when it is determined that there is an object of interest, the processing proceeds to step S1502, otherwise, the processing proceeds to step S1501.

[0174] In step S1502, the control unit 303 determines whether the object of interest and the avatar operated by the operation units 302a and 302b overlap each other. The control unit 303 uses the position information and shape information of the object of interest to be used when generating an image in a virtual space and the position information and shape information of the avatar being operated by the operation units 302a and 302b to determine whether the object of interest and the avatar operated by the operation units 302a and 302b overlap each other. When it is determined that the object of interest and the avatar overlap each other, the processing proceeds to step S1503, otherwise, the processing proceeds to step S1505.

[0175] In step S1505, the control unit 103 sets the amount of tactile feedback to 0, and does not perform tactile feedback.

[0176] In step S1503, the control unit 303 calculates tactile information based on the physical calculation parameters of the object of interest and the avatar. The control unit 303 calculates the force applied to each portion of the avatar by using the speed, stiffness parameter, and the like of the object of interest and the avatar.

[0177] In step S1504, the control unit 303 calculates the amount of tactile feedback based on the degree of interest of the object of interest. In ​ step S1404, the amount of tactile feedback generated as described above is transmitted to the tactile generation unit 310, and in step S1405, tactile feedback is performed on the user by the tactile generation unit 310 that generates a tactile sensation.

[0178] As described above, according to the second embodiment, appropriate tactile feedback can be realized by calculating the amount of tactile feedback based on the degree of interest of the user with respect to the object present in the virtual space.

[0179] Other Embodiments

[0180] Embodiments of the present application can also be realized by a method in which a computer (Central Processing Unit (CPU), Micro Processing Unit (MPU)) of a system or an apparatus reads out and executes a computer program prepared for the above-mentioned embodiments from a storage medium or a variety of storage media to function in accordance with the computer program.

[0181] While the application has been described with reference to the example embodiments thereof, it is to be understood that the application is not limited to the example embodiments disclosed. The scope of the appended claims covers all such modifications and equivalents as falling within the spirit and scope of the application.

Claims

1. A tactile feedback device that generates a tactile sensation on an operating unit capable of operating an operated unit, the tactile feedback device comprising: an obtaining unit for obtaining tactile information representing a tactile sensation when the operated unit contacts an object; a detection unit configured to detect an object of interest to a user operating the operating unit from objects within a predetermined range including the object being contacted by the operating unit; as well as A calculation unit is configured to calculate a feedback amount for generating the tactile sensation in the operation unit based on information about the object focused on by the user and the tactile information. 2 . The tactile feedback device according to claim 1 , further comprising a setting unit for setting a degree of attention of an object within the predetermined range.

3. The tactile feedback device according to claim 2, further comprising a display unit for displaying an image within the predetermined range. in, The setting unit sets the degree of attention using the image displayed on the display unit.

4. The tactile feedback device according to claim 3, wherein: The setting unit sets the attention degree of an object overlapping the operated unit for a predetermined period of time or longer, the attention degree of an object within the predetermined range, and the attention degree of an object having a predetermined size or larger among objects to a value higher than the attention degree of other objects.

5. The tactile feedback device according to claim 4, wherein The calculation unit calculates the feedback amount using object information of an object having a largest size among the objects having the highest degree of attention.

6. The tactile feedback device according to any one of claims 2 to 4, wherein: When there is no object for which the attention degree is set, the calculation unit does not calculate the feedback amount.

7. The tactile feedback device according to claim 2, further comprising a display unit for displaying an image within the predetermined range and a gaze detection unit for detecting a gaze direction of the user. in, The setting unit sets the degree of attention based on the image displayed on the display unit and the user's gaze.

8. The tactile feedback device according to claim 7, wherein: The setting unit sets the attention level of the object in the gaze direction to a value higher than the attention levels of other objects.

9. The tactile feedback device according to claim 8, wherein: The setting unit sets the attention level of an object in the gaze direction for a predetermined time period or longer, the attention level of the object whose difference between the speed and / or travel direction of the gaze direction and the speed and / or travel direction of the object falls within a predetermined difference, and the total attention level of objects in the gaze direction for a predetermined time period or longer to a value higher than the attention level of other objects.

10. The tactile feedback device according to claim 9, wherein The calculation unit calculates the feedback amount using object information of an object having a largest size among the objects having the highest degree of attention.

11. The tactile feedback device according to any one of claims 7 to 9, wherein: When there is no object for which the attention degree is set, the calculation unit does not calculate the feedback amount.

12. The tactile feedback device according to any one of claims 1 to 5 and 7 to 10, wherein: The operation unit includes a tactile sensation generating unit configured to apply the tactile sensation based on the feedback amount.

13. The tactile feedback device according to any one of claims 1 to 5 and 7 to 10, wherein: The operated unit is a robot arm, and the tactile information is generated based on information detected by a sensor provided in the robot arm.

14. The tactile feedback device according to any one of claims 1 to 5 and 7 to 10, wherein: The operated unit is an avatar in a virtual space, and the tactile information is generated based on position information of an object in the virtual space and the avatar operated by the operating unit.

15. A method for controlling a tactile feedback device, the tactile feedback device generating a tactile sensation in an operating unit capable of operating an operated unit, the control method comprising: a step of obtaining tactile information representing the tactile sensation when the operated unit contacts an object; a step for detecting an object focused on by a user operating the operating unit from objects within a predetermined range including the object being contacted by the operating unit; as well as a step of calculating a feedback amount for generating the tactile sensation in the operation unit based on information of the object to which the user is paying attention and the tactile information. 16 . A computer-readable storage medium storing a program for causing a computer to function as the tactile feedback device according to claim 1 .

17. A computer program product comprising a program for causing a computer to function as a tactile feedback device according to any one of claims 1 to 14.

Images