Haptic feedback device, control method, and program

Abstract

To perform haptic feedback on the basis of a user's degree of interest.SOLUTION: A haptic feedback device generates a tactile sense in an operating unit that can operate a unit to be operated, and has: acquisition means that acquires tactile information representing a tactile sense when the unit to be operated touches an object; detection means that detects, from objects located within a predetermined range including the object touched by the unit to be operated, an object in which a user who operates the operating unit has an interest; and calculation means that calculates the amount of feedback for generating the tactile sense in the operating unit on the basis of the tactile information and object information on the object in which the user has an interest.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for providing haptic feedback based on a user's interest level. [Background technology]

[0002] Haptic feedback is a technology that conveys the results and status of an operation by providing physical stimuli such as vibrations and pressure to the user's hands or fingers from an object being operated. Haptic feedback allows the user to feel vibrations or movement when operating an object, allowing them to confirm that the operation was performed correctly. For users with visual or hearing impairments, tactile feedback is an important source of information, improving the usability and accessibility of user interfaces. Haptic feedback is also used to enable users to properly operate robotic arms and other devices based on tactile information, and to improve the sense of immersion in virtual spaces.

[0003] Patent Document 1 describes a technology in which, when a user shows interest in an object to which a tactile sensor is attached, the tactile sensor acquires tactile information and provides tactile feedback. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-215894 Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, tactile feedback is only possible when the user focuses on an object equipped with a tactile sensor, and it is not possible to obtain tactile information from an object that does not have a tactile sensor attached. Therefore, a technology that allows the user to obtain tactile information associated with the object the user is focusing on is desired.

[0006] The present invention has been made in view of the above-mentioned problems, and its object is to realize a technology that can provide haptic feedback based on the user's interest level. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the present invention provides a tactile feedback device that generates a tactile sensation in an operating unit that can operate an operated unit, and includes an acquisition means that acquires tactile information that represents the tactile sensation when the operated unit touches an object, a detection means that detects an object that is of interest to a user operating the operating unit from objects within a predetermined range that includes the object that the operated unit is touching, and a calculation means that calculates the amount of feedback for generating a tactile sensation in the operating unit based on the object information of the object that the user is interested in and the tactile information. [Effects of the Invention]

[0008] According to the present invention, haptic feedback can be provided based on the user's interest level. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an external view schematically illustrating a haptic feedback device according to a first embodiment. [Figure 2] FIG. 1 is a block diagram illustrating the configuration of a haptic feedback device according to a first embodiment. [Figure 3] 4 is a flowchart illustrating a haptic feedback process according to the first embodiment. [Figure 4] 1 is a flowchart illustrating an example of an object of interest detection process according to the first embodiment; [Figure 5] 1 is a block diagram illustrating the configuration of a haptic feedback device including a gaze detection device according to a first embodiment. [Figure 6] 4 is a flowchart illustrating an example of gaze detection processing according to the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of the gaze detection method according to the first embodiment. [Figure 8]FIG. 2 is an explanatory diagram of an eyeball image according to the first embodiment. [Figure 9] 10 is a flowchart illustrating an example of an object-of-interest detection process based on line-of-sight information according to the first embodiment. [Figure 10] FIG. 10 is an explanatory diagram of interest level calculation processing based on gaze information according to the first embodiment; [Figure 11] 6 is a flowchart illustrating a haptic feedback amount calculation process according to the first embodiment. [Figure 12] FIG. 10 is an external view schematically showing a haptic feedback device according to a second embodiment. [Figure 13] FIG. 10 is a block diagram illustrating the configuration of a haptic feedback device according to a second embodiment. [Figure 14] 10 is a flowchart illustrating a haptic feedback process according to the second embodiment. [Figure 15] 10 is a flowchart illustrating a haptic feedback amount calculation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] In the first embodiment, a haptic feedback device that generates a tactile sensation on an operating unit that can operate a robot arm as an operated unit will be described.

[0012] <Device configuration> First, the configuration and functions of the haptic feedback device of the first embodiment will be described with reference to FIGS.

[0013] FIG. 1 is a schematic external view of the haptic feedback device of the first embodiment.

[0014] The haptic feedback device 1 includes a controller 100, a robot arm 101, and an operation unit .

[0015] The operation unit 102 can be worn by a user on their hand. When the user moves their hand or fingers while wearing the operation unit 102, the operation unit 102 can move the arm unit 101a and hand unit 101b of the robot arm 101 in accordance with the movement. When the user moves their hand or fingers while wearing the operation unit 102, the controller 100 receives operation information corresponding to the user operation from the operation unit 102 and outputs drive information corresponding to the operation information to the robot arm 101. The robot arm 101 drives the arm unit 101a and hand unit 101b of the robot arm 101 based on the drive information received from the controller 100.

[0016] The display unit 104 is provided in the controller 100 and displays a GUI (Graphical User Interface) of the controller 100. The display unit 104 also displays an image of the hand unit 101b of the robot arm 101 and the surroundings of the hand unit 101b captured by the camera unit 108.

[0017] A tactile information detection unit 106 is provided on the hand unit 101b of the robot arm 101. The tactile information detection unit 106 is capable of detecting tactile information applied to the hand unit 101b when the robot arm 101 grasps an object or the like. The tactile information detection unit 106 has a force sensor that can detect force information as tactile information, and outputs the force information detected by the force sensor from the robot arm 101 to the controller 100 as tactile information.

[0018] The controller 100 calculates the amount of haptic feedback from the haptic information acquired from the haptic information detection unit 106 and information about the object in which the user is interested (interest object information), and outputs the amount to the haptic generation unit 110.

[0019] The tactile sense generating unit 110 is composed of a piezoelectric element or the like that applies physical stimuli such as vibration and pressure to the hand or fingers of the user wearing the operation unit 102, and is provided on the hand or fingers of the operation unit 102. The tactile sense generating unit 110 generates a tactile sensation such as vibration or pressure according to the amount of tactile feedback received from the controller 100, and conveys the tactile sensation to the user via the operation unit 102.

[0020] FIG. 2 is a block diagram illustrating the hardware configuration of the haptic feedback device of the first embodiment.

[0021] The controller 100 includes a control unit 103 and a display unit 104. The control unit 103 includes a processor (CPU) that performs arithmetic processing and control processing of the haptic feedback device 1, a volatile memory (ROM) that stores programs executed by the processor, and a work memory (RAM) into which programs read from the non-volatile memory and constants and variables for executing the programs are loaded. The control unit 103 controls each component of the haptic feedback device 1 by loading the programs stored in the ROM into the RAM and executing them. Note that instead of the control unit 103 controlling the entire device, the entire device may be controlled by multiple hardware components sharing the processing load.

[0022] The display unit 104 includes a liquid crystal panel or an organic EL panel, and displays images and various information.

[0023] The robot arm 101 includes a force sensor 105, a tactile information detection unit 106, a driving unit 107, and a camera unit 108.

[0024] The force sensor 105 detects force information applied to the hand unit 101b when the robot arm 101 grasps an object or the like.

[0025] The tactile information detection unit 106 outputs the force information detected by the force sensor 105 to the control unit 103 .

[0026] The driving unit 107 drives the arm unit 101 a and the hand unit 101 b of the robot arm 101 based on driving information received from the control unit 103 .

[0027] The camera unit 108 has an image sensor and an A / D converter, converts the subject image into an electrical signal, and outputs image data consisting of a digital signal to the control unit 103. The control unit 103 displays on the display unit 104 the image of the hand unit 101b of the robot arm 101 and the area around the hand unit 101b captured by the camera unit 108, and, if the hand unit 101b is gripping an object, an image within a predetermined range including the area where the hand unit 101b is touching.

[0028] The operation unit 102 includes an operation information transmission unit 109 and a tactile sensation generation unit 110 .

[0029] The operation information transmitting unit 109 outputs operation information of a user wearing the operation unit 102 to the control unit 103. The control unit 103 outputs drive information corresponding to the operation information received from the operation information transmitting unit 109 to the drive unit 107. The drive unit 107 drives the arm unit 101a and the hand unit 101b of the robot arm 101 based on the drive information received from the control unit 103.

[0030] The tactile sensation generating unit 110 applies a tactile sensation such as vibration or pressure to the operation unit 102 in accordance with the amount of tactile feedback received from the control unit 103, and provides the feedback to the user.

[0031] 1 and 2, the operation unit 102 and the control unit 103 are configured as separate units, but they may be configured as being incorporated into the same device.

[0032] <Control processing> Next, the haptic feedback processing based on the user's interest level according to the first embodiment will be described with reference to FIG.

[0033] The processing in Figure 3 is realized by the control unit 103 of the controller 100 executing a program stored in ROM and controlling the display unit 104, robot arm 101, and operation unit 102. The same applies to Figures 4, 6, 9, and 11, which will be described later.

[0034] In step S301, the control unit 103 detects information about an object that the user is interested in (interest object information). Details of the interest object detection process will be described later with reference to FIG.

[0035] In step S302, the control unit 103 acquires tactile information from the tactile information detection unit .

[0036] In step S303, the control unit 103 calculates the amount of haptic feedback for generating a tactile sensation in the haptic sensation generating unit 110 based on the object of interest information acquired in step S301 and the haptic sensation information acquired in step S302.

[0037] In step S304, the control unit 103 transmits the amount of haptic feedback calculated in step S003 to the haptic generation unit 110.

[0038] In step S305, the control unit 103 causes the haptic sensation generating unit 110 to generate a haptic sensation according to the amount of haptic feedback.

[0039] <Interest object detection processing> Next, examples of the object of interest detection process in step S301 in FIG. 3 will be described, including a process using image information and a process using line-of-sight information.

[0040] <<Object of interest detection processing using image information>> FIG. 4 explains the processing for detecting an object of interest using image information.

[0041] In step S401, the control unit 103 detects an object in the image captured by the camera unit 108 and displayed on the display unit 104, and determines the area of ​​the object.

[0042] In step S402, the control unit 103 initializes the value of the interest level to 0. The value of the interest level is an index that indicates the degree of interest of the user.

[0043] In step S403, the control unit 103 determines whether or not any of the objects detected in step S401 has been overlapping the hand unit 101b (tactile information detection unit 106) for a predetermined period of time or longer. If the control unit 103 determines that any object has been overlapping the hand unit 101b for a predetermined period of time or longer, the process proceeds to step S404; otherwise, the process skips step S404 and proceeds to step S405.

[0044] In step S404, the control unit 103 adds 1 to the value of the interest level of the object that has been overlapping with the hand unit 101b for a predetermined period of time or more.

[0045] In step S405, the control unit 103 determines whether or not any of the objects detected in step S401 has been positioned within a predetermined range from the center of the screen of the display unit 104 for a predetermined time or longer. If the control unit 103 determines that any object has been positioned within a predetermined range from the center of the screen for a predetermined time or longer, the process proceeds to step S406; otherwise, the process skips step S406 and proceeds to step S407.

[0046] In step S406, the control unit 103 adds 1 to the value of the interest level of an object that has been positioned within a predetermined range from the center of the screen for a predetermined period of time or more.

[0047] In step S407, the control unit 103 determines whether or not the area size of the object detected in step S401 is equal to or larger than a predetermined size. If the control unit 103 determines that an object equal to or larger than the predetermined size is present, the process proceeds to step S408; otherwise, the control unit 103 skips the process of step S408 and proceeds to step S409.

[0048] In step S408, the control unit 103 adds 1 to the value of the interest level of an object whose area size is equal to or larger than a predetermined size.

[0049] In step S409, the control unit 103 determines whether or not there is an object whose level of interest value is equal to or greater than 1. If the control unit 103 determines that there is an object whose level of interest value is equal to or greater than 1, the process proceeds to step S410; otherwise, the process proceeds to step S411.

[0050] In step S410, the control unit 103 determines the object with the largest area size among the objects with the highest interest level as the object of interest.

[0051] In step S411, the control unit 103 determines that there is no object of interest.

[0052] <<Detection of objects of interest using gaze information>> Next, the processing for detecting an object of interest using line-of-sight information will be described with reference to FIGS.

[0053] FIG. 5 is a block diagram illustrating a hardware configuration of a haptic feedback device including a gaze detection device.

[0054] The controller 100 is provided with an eyeball imaging unit 201 and an illumination light source 202 as devices for detecting the user's line of sight.

[0055] The eyeball imaging unit 201 captures an image of the user's eyeballs and outputs the image to the control unit 103. The control unit 103 detects the user's line of sight based on the eyeball image.

[0056] The illumination light source 202 emits infrared light to the observer's eyeball, allowing the eyeball imaging unit 201 to capture an image with appropriate brightness for line of sight detection.

[0057] Next, the gaze detection process will be described with reference to FIGS.

[0058] Fig. 6 is a flowchart illustrating an example of the gaze detection process, and Fig. 7 is an explanatory diagram of the gaze detection method.

[0059] In step S601, the control unit 103 causes the illumination light source 202 to irradiate infrared light toward the observer's eyeball 71. An image of the observer's eyeball illuminated by the infrared light is formed on the eyeball imaging unit 201. The eyeball imaging unit 201 converts the eyeball image formed on the eyeball imaging unit 201 into an electrical signal to generate an eyeball image signal.

[0060] In step S602, the control unit 103 acquires an eyeball image signal from the eyeball imaging unit 201.

[0061] In step S603, the control unit 103 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the illumination light source 202 and the pupil center c shown in FIG. 7 from the information of the eyeball image signal acquired in step S602.

[0062] Fig. 7(a) illustrates a corneal reflection image obtained from an eyeball image, and Fig. 7(b) illustrates brightness information obtained from region α of the eyeball image in Fig. 7(a).

[0063] Infrared light emitted from the illumination light source 202 illuminates the cornea of ​​the observer's eyeball 71, and corneal reflection images Pd and Pe formed by part of the infrared light reflected from the surface of the cornea are focused on the eyeball imaging unit 201 (points Pd' and Pe' in FIG. 7). Similarly, light beams from ends a and b of the pupil 72 are also focused on the eyeball imaging unit 201.

[0064] 7(a), the horizontal direction is the X-axis and the vertical direction is the Y-axis, and the coordinates in the X-axis direction (horizontal direction) of images Pd', Pe' formed by the corneal reflection images of illumination light source 202 are designated as Xd and Xe. Also, in FIG. 7(a), the coordinates in the X-axis direction of images a', b' formed by the light beam from the edge of pupil 72 are designated as Xa and Xb.

[0065] In the luminance information in Figure 7(b), extremely high levels of luminance are obtained at positions Xd and Xe, which correspond to images Pd' and Pe' formed by the corneal reflection of the illumination light source 202. In the region between coordinates Xa and Xb, which corresponds to the area of ​​the pupil 72, extremely low levels of luminance are obtained except for the positions Xd and Xe. In contrast, in the region corresponding to the area of ​​the iris 73 outside the pupil 72, which has an X coordinate value lower than Xa and an X coordinate value higher than Xb, intermediate values ​​between the two luminance levels are obtained. From information on the fluctuation of the luminance level with respect to the X coordinate position, the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection of the illumination light source 202 and the X coordinates Xa and Xb of the images a' and b' at the pupil edge can be obtained. Furthermore, when the rotation angle θx of the optical axis of the eyeball 14 with respect to the optical axis of the eyeball imaging unit 201 is small, the coordinate Xc of a point (assumed to be c') corresponding to the pupil center c imaged on the eyeball imaging unit 201 can be expressed as Xc ≈ (Xa + Xb) / 2. In this manner, the X coordinate of c' corresponding to the pupil center imaged on the eyeball imaging unit 201 and the coordinates of the corneal reflection images Pd' and Pe' of the eyeball imaging unit 201 can be estimated.

[0066] In step S604, the control unit 103 calculates the imaging magnification β of the eyeball image. β is a magnification determined by the position of the eyeball 71 relative to the optical system of the eyeball imaging unit 201, and can be calculated substantially as a function of the distance (Xd-Xe) between the corneal reflection images Pd' and Pe'.

[0067] In step S605, the control unit 103 determines that the X coordinate of the midpoint of the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea are approximately the same, and therefore, if the standard distance from the center of curvature O of the cornea to the center c of the pupil 72 is Oc, the rotation angle θX of the optical axis of the eyeball 71 in the ZX plane can be calculated from Equation 1. (Formula 1) β*Oc*SINθX≒{(Xd+Xe) / 2}-Xc FIG. 8 shows an example of calculating the rotation angle θX when the observer's eyeball rotates in a plane perpendicular to the Y axis, but the method for calculating the rotation angle θy when the observer's eyeball rotates in a plane perpendicular to the X axis is similar.

[0068] In step S606, the control unit 103 reads out from the ROM the gaze correction coefficients Ax, Bx, Ay, and By that correct for individual differences in the user's gaze. The gaze correction coefficients Ax, Bx, Ay, and By represent the offset amount in the x direction, the sensitivity coefficient for the rotation angle, the offset amount in the y direction, and the sensitivity coefficient for the rotation angle. These values ​​are assumed to be acquired by performing a calibration operation and stored in the ROM.

[0069] In step S607, the control unit 103 uses θx and θy to determine the user's line of sight (gaze position) on the display unit 104. The gaze position can be calculated from Equation 2, assuming that the coordinates corresponding to the center c of the pupil 72 on the display unit 104 are (Hx_L, Hy_L) and (Hx_R, Hy_R) for the left eye and right eye, respectively. (Formula 2) Hx_L=m*(Ax*θx+Bx)*nLx Hx_R=m*(Ax*θx+Bx)*nRx Hy_L=m*(Ay*θy+By)*nLy Hy_R=m*(Ay*θy+By)*nRy The coefficient m is a constant determined by the configuration of the optical system of the gaze detection device, and is a conversion coefficient that converts the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 72 on the display unit 104. The coefficient m and the correction coefficients (nLx, nLy, nRx, nRy) that correct the reliability of the left and right eyes when determining the gaze position are determined in advance.

[0070] Then, the control unit 103 calculates the overall gaze position (Hx, Hy) from Hx_L, Hx_R, Hy_L, and Hy_R.

[0071] Next, with reference to FIG. 9, an object of interest detection process using line-of-sight information detected as described with reference to FIGS. 6 to 8 will be described.

[0072] FIG. 9 is a flowchart illustrating an example of an object of interest detection process using line-of-sight information.

[0073] In step S901, the control unit 103 detects objects from the image displayed on the display unit 104 and determines the area of ​​each object.

[0074] In step S902, the control unit 103 detects the direction of the user's line of sight and determines the position on the display unit 104 at which the user is gazing.

[0075] In step S903, the control unit 103 initializes the value of the interest level to zero.

[0076] In step S904, the control unit 103 determines whether or not there is an object (gaze object) that has been gazed at continuously for a predetermined time or more from the current time among the objects detected in step S901. If the control unit 103 determines that there is a gaze object, the process proceeds to step S905; otherwise, the process skips step S905 and proceeds to step S906.

[0077] In step S905, the control unit 103 adds 1 to the value of the interest level of the gazed-at object.

[0078] In step S906, the control unit 103 compares the speed and direction of the user's line of sight with the speed and direction of the gazed object, respectively, and determines whether they are within a predetermined difference. If the control unit 103 determines that they are within the predetermined difference, the process proceeds to step S907; otherwise, the process skips step S907 and proceeds to step S908.

[0079] In step S907, the control unit 103 adds 1 to the value of the interest level of the gazed-at object.

[0080] In step S908, the control unit 103 determines whether or not there is a gazed object in the line of sight for a total of a predetermined time or more while the gazed object is detected, and determines whether or not there is an object that the user has been gazing at for a long time. If the control unit 103 determines that there is an object that the user has been gazing at for a long time, the control unit 103 proceeds to step S909; otherwise, the control unit 103 skips the process of step S909 and proceeds to step S910.

[0081] In step S909, the control unit 103 adds 1 to the value of the interest level of the object that the user has been gazing at for a long time.

[0082] Here, the determination processing in steps S904 and S908 will be described with reference to FIG.

[0083] Fig. 10 shows an example of the time transition of an object in the line of sight of the user. In the example of Fig. 10, the detection period for the gazed object is 10 seconds from the current time, but this is not limited to this and a different period may be set.

[0084] In the following description, the determination threshold in step S904 is set to 4 seconds, and the determination threshold in step S908 is set to 6 seconds.

[0085] In Figure 10(a), object 1 has been gazed at for 4 seconds from the current time, which satisfies the judgment condition in step S904 and causes the interest level of object 1 to be incremented by 1. Also, no object has been gazed at for 6 seconds or more, so no object has its interest level incremented in step S908.

[0086] 10(b), there is no object that has been gazed at for 4 seconds or more since the current time, so there is no object that satisfies the judgment condition in step S904 and has its interest level added. Also, because object 1 has been gazed at for a total of 6 seconds during the judgment period, 1 is added to the value of the interest level for object 1 in step S908.

[0087] In Figure 10(c), object 1 has been gazed at for 5 seconds from the current time, which satisfies the judgment condition in step S904 and the interest level value for object 1 is incremented by 1. Also, object 1 has been gazed at for a total of 9 seconds during the judgment period, so the monitoring level for object 1 is incremented by 1 in step S908.

[0088] In step S910, the control unit 103 determines whether or not there is an object whose interest level is equal to or greater than 1. If the control unit 103 determines that there is an object whose interest level is equal to or greater than 1, the process proceeds to step S911; otherwise, the process proceeds to step S912.

[0089] In step S911, the control unit 103 determines the object with the largest area size among the objects with the highest interest level as the object of interest.

[0090] In step S112, the control unit 103 determines that there is no object of interest.

[0091] <Haptic feedback amount calculation process> Next, the process of calculating the amount of haptic feedback in step S303 of FIG. 3 will be described with reference to FIG.

[0092] The processing in FIG. 11 starts when tactile information is detected in step S302 in FIG. 3 and the object of interest determination processing in FIG. 4 or FIG. 9 is performed.

[0093] In step S1101, the control unit 103 determines whether or not an object of interest is present, and if it is determined that an object of interest is present, the process proceeds to step S1102, and if not, the process proceeds to step S1105.

[0094] In step S1102, the control unit 103 determines whether or not the object of interest and the hand unit 101b are in contact with each other from the image displayed on the display unit 104. If the control unit 103 determines that the object of interest and the hand unit 101b are in contact with each other, the control unit 103 proceeds to step S1103, and if not, the control unit 103 proceeds to step S1105.

[0095] In step S1105, the control unit 103 sets the amount of haptic feedback to 0, and does not execute haptic feedback.

[0096] In step S1103, the control unit 103 acquires tactile information from the tactile information detection unit .

[0097] In step S1104, the control unit 103 calculates the amount of haptic feedback based on the level of interest in the object of interest. When the amount of haptic feedback is H_fb, the haptic information is H_in, and the level of interest is Int, the amount of haptic feedback is calculated as shown in Equation 3. (Formula 3) H_fb=H_in(0.6+0.2*Int) As described above, according to the first embodiment, the amount of haptic feedback is calculated based on the user's level of interest in an object in real space, thereby realizing appropriate haptic feedback.

[0098] [Embodiment 2] In the first embodiment, an example of providing haptic feedback based on the user's level of interest in an object in real space has been described. In the second embodiment, an example of providing haptic feedback based on the user's level of interest in an object in virtual space will be described.

[0099] In the second embodiment, a haptic feedback device that generates a tactile sensation on an operating unit that can operate an avatar in a virtual space as an operated unit will be described.

[0100] 12A and 12B are external views schematically showing a tactile feedback device of embodiment 2. Fig. 12A is a front perspective view of the tactile feedback device of embodiment 2. Fig. 12B is a rear perspective view of the tactile feedback device of embodiment 2.

[0101] The haptic feedback device 2 of the second embodiment includes a controller 300, a goggle-type device 301 that can be worn on the user's head, and an operation unit 302. The goggle-type device 301 is equipped with a gaze detection device that detects the gaze direction for each of the left and right eyes. The left and right eyes are also referred to as both eyes.

[0102] The goggle-type device 301 generates an object such as an avatar in a virtual space and displays it so that it can be seen.

[0103] The operation unit 302 has left and right operation units 302 a and 302 b that can be worn on the left and right hands of the user, and is used to operate an avatar in the virtual space that is viewed through the goggle-type device 301 .

[0104] The goggle-type device 301 is provided with an imaging unit 305 and a photometry unit 307 .

[0105] The goggle-type device 301 is also provided with illumination light sources 313a and 313b for the left and right eyeballs. The illumination light sources 313a and 313b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the user, and each light source illuminates the left and right eyeballs of the user. A portion of the illumination light reflected by the eyeballs is collected onto the eyeball imaging unit 315 for each eyeball.

[0106] The goggle-type device 301 is provided with display units 308a and 308b for the left and right eyes, respectively.

[0107] FIG. 13 is a block diagram illustrating the hardware configuration of the haptic feedback device of the second embodiment.

[0108] The controller 300 includes a control unit 103 , a memory unit 304 , a communication unit 306 , a display control unit 311 , a gaze detection unit 312 , a display unit 341 , and operation members 342 and 343 .

[0109] The control unit 303 includes a processor (CPU) that performs arithmetic processing and control processing of the haptic feedback device 2, a volatile memory (ROM) that stores programs executed by the processor, and a work memory (RAM) into which programs read from the non-volatile memory and constants and variables for executing the programs are loaded. The control unit 303 controls each component of the haptic feedback device 2 by loading the programs stored in the ROM into the RAM and executing them. Note that instead of the control unit 303 controlling the entire device, the entire device may be controlled by multiple hardware components sharing the processing load.

[0110] The memory unit 304 stores image signals from the eyeball imaging unit 315, line-of-sight correction data for correcting individual differences in line of sight, and interest level calculation information set for each subject.

[0111] The communication unit 306 includes an interface that enables communication with the Internet or external devices. The communication method may be wired or wireless.

[0112] The gaze detection unit 312 acquires an eyeball image from the eyeball imaging unit 315 and outputs the gaze detection result to the control unit 303. The control unit 303 detects the gaze direction of the user according to the algorithm explained in FIG.

[0113] The display control unit 311 controls the display unit 341 based on a display control signal from the control unit 303. The display unit 341 includes a liquid crystal panel or an organic EL panel, and displays images and various information.

[0114] The operation members 342 and 343 accept user operations and transmit operation information corresponding to the user operations to the controller 300.

[0115] The goggle-type device 301 includes display units 308 a and 308 b , an eyeball imaging unit 315 , illumination light sources 313 a and 313 b , an imaging unit 305 , and a photometry unit 307 .

[0116] The display units 308a and 308b display avatars in a virtual space for each of the left and right eyeballs.

[0117] The eyeball imaging unit 315 captures an image of the user's eyeball and outputs it to the control unit 303 .

[0118] The illumination light sources 313a and 313b radiate infrared light to the observer's eyeballs, which allows the eyeball imaging unit 315 to capture an image with appropriate brightness for line of sight detection.

[0119] The imaging unit 305 captures an image of the object scene visible to the user and outputs the generated imaging signal to the control unit 303. The photometry unit 307, as a photometric sensor, amplifies a luminance signal output corresponding to the brightness of the object scene based on the imaging signal generated by the imaging unit 305, then performs logarithmic compression and A / D conversion, and outputs the amplified luminance signal to the control unit 303 as object scene luminance information.

[0120] The goggle-type device 301 may be a non-transparent type such as VR (virtual reality) or a transparent type such as AR (augmented reality).

[0121] 12 and 13, the goggle-type device 301 and the control unit 303 are configured as separate units, but they may be configured as being incorporated into the same device.

[0122] The operation units 302 a and 302 b include an operation information transmission unit 309 and a tactile sensation generation unit 310 .

[0123] Operation information transmission unit 309 outputs operation information of the user wearing operation units 302a and 302b to control unit 303. Control unit 303 reproduces the movement of the avatar in the virtual space in accordance with the operation information received from operation information transmission unit 309.

[0124] The tactile sensation generating unit 310 applies a tactile sensation such as vibration or pressure to the operation units 302a and 302b in accordance with the amount of tactile feedback received from the control unit 303, and provides the tactile sensation as feedback to the user.

[0125] <Haptic feedback processing> Next, the haptic feedback processing of the second embodiment will be described with reference to FIG.

[0126] 14 is realized by the control unit 303 of the controller 100 executing a program stored in ROM and controlling the goggle-type device 301 and operation units 302a and 302b. The same applies to FIG. 15, which will be described later.

[0127] In step S1401, the control unit 303 performs object of interest detection using the image displayed on the display unit 308. The object of interest detection process is as described in the first embodiment with reference to FIGS.

[0128] In step S1402, the control unit 303 calculates tactile information based on the object of interest detected in step S1401 and the movement of the avatar operated by the operation units 302a and 302b.

[0129] In step S1403, the control unit 303 calculates the amount of haptic feedback based on the object of interest information and haptic information acquired in step S1402.

[0130] In step S1404, the control unit 303 transmits the amount of haptic feedback calculated in step S1403 to the haptic generation unit 310.

[0131] In step S1405, the control unit 303 causes the haptic sensation generating unit 110 to generate a haptic sensation according to the amount of haptic feedback.

[0132] FIG. 15 is a flowchart illustrating the haptic feedback calculation process in steps S1402 and S1403 of FIG.

[0133] In step S1501, the control unit 303 determines whether or not an object of interest is present, and if it is determined that an object of interest is present, the process proceeds to step S1502, and if not, the process proceeds to step S1501.

[0134] In step S1502, the control unit 303 determines whether the object of interest and the avatar operated by the operation units 302a and 302b overlap. The control unit 303 determines whether the object of interest and the avatar operated by the operation units 302a and 302b overlap using position information and shape information of the object of interest used when generating an image of the virtual space and the avatar operated by the operation units 302a and 302b. If the control unit 303 determines that the object of interest and the avatar overlap, the process proceeds to step S1503; otherwise, the process proceeds to step S1505.

[0135] In step S1505, the control unit 103 sets the amount of haptic feedback to 0, and does not execute haptic feedback.

[0136] In step S1503, the control unit 303 calculates tactile information based on the physical calculation parameters of the object of interest and the avatar. The control unit 303 calculates the force applied to each part of the avatar using the speed and stiffness parameters of the object of interest and the avatar.

[0137] In step S1504, the control unit 303 calculates the amount of haptic feedback based on the interest level of the object of interest. The amount of haptic feedback generated as described above is transmitted to the haptic generation unit 310 in step S1404 of FIG. 14, and in step S1405 the haptic generation unit 310 generates a tactile sensation, thereby providing haptic feedback to the user.

[0138] As described above, according to the second embodiment, the amount of haptic feedback is calculated based on the user's level of interest in an object in the virtual space, thereby realizing appropriate haptic feedback.

[0139] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0140] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0141] The disclosure of this specification includes the following haptic feedback device, control method, and program. [Configuration 1] A tactile feedback device that generates a tactile sensation on an operating unit that can operate an operated unit, an acquisition means for acquiring tactile information representing a tactile sensation when the operated unit touches an object; a detection means for detecting an object that is of interest to a user operating the operation unit from among objects within a predetermined range including the object that the operated unit is touching; a calculation means for calculating a feedback amount for generating a tactile sensation in the operation unit based on information about an object in which the user is interested and the tactile information. [Configuration 2] The tactile feedback device according to configuration 1, further comprising a setting means for setting a degree of interest in an object within the predetermined range. [Configuration 3] a display means for displaying an image within the predetermined range, 3. The haptic feedback device according to configuration 2, wherein the setting means sets the interest level using an image displayed on the display means. [Configuration 4] The tactile feedback device described in configuration 3 is characterized in that the setting means sets the interest level of an object that overlaps the operated part for more than a predetermined time, the interest level of an object that is within the predetermined range, and the interest level of an object that is larger than a predetermined size to a higher value than the interest level of other objects. [Configuration 5] 5. The haptic feedback device according to configuration 4, wherein the calculation means calculates the amount of feedback using object information of the largest object among the objects with the highest interest level. [Configuration 6] 5. The haptic feedback device according to any one of configurations 2 to 4, wherein when there is no object for which the interest level is set, the calculation means does not calculate the feedback amount. [Configuration 7] a display unit that displays an image within the predetermined range, and a gaze detection unit that detects a gaze direction of a user, 3. The haptic feedback device according to configuration 2, wherein the setting means sets the interest level based on the image displayed on the display means and the direction of the user's line of sight. [Configuration 8] 8. The haptic feedback device according to configuration 7, wherein the setting means sets the interest level of an object in the line of sight direction to a value higher than the interest levels of other objects. [Configuration 9] The tactile feedback device of configuration 8, characterized in that the setting means sets the interest level of an object that is in the line of sight direction for more than a predetermined time, the interest level of an object whose difference between the line of sight direction and the speed and direction of travel of the object is within a predetermined difference, and the interest level of an object that is in the line of sight direction for a total of a predetermined time to a value higher than the interest levels of other objects. [Configuration 10] 10. The haptic feedback device according to configuration 9, wherein the calculation means calculates the amount of feedback using object information of the largest object among the objects with the highest interest level. [Configuration 11] 10. The haptic feedback device according to any one of configurations 7 to 9, wherein when there is no object for which the interest level is set, the calculation means does not calculate the feedback amount. [Configuration 12] 12. The tactile feedback device according to any one of configurations 1 to 11, wherein the operation unit includes tactile sensation generating means for applying the tactile sensation based on the amount of feedback. [Configuration 13] the operated unit is a robot arm, 13. The haptic feedback device according to any one of configurations 1 to 12, wherein the haptic information is generated based on information detected by a sensor provided on the robot arm. [Configuration 14] the operated unit is an avatar in a virtual space, 13. The haptic feedback device according to any one of configurations 1 to 12, wherein the haptic information is generated based on position information of an object in the virtual space and an avatar operated by the operation unit. [Configuration 15] A control method for a tactile feedback device that generates a tactile sensation on an operating unit that can operate an operated unit, comprising: acquiring tactile information representing a tactile sensation when the operated unit touches an object; detecting an object that is of interest to a user operating the operating unit from among objects within a predetermined range including the object touched by the operated unit; and calculating a feedback amount for generating a tactile sensation in the operation unit based on object information of the object in which the user is interested and the tactile information. [Configuration 16] A program for causing a computer to function as each means of the haptic feedback device according to any one of configurations 1 to 14. [Explanation of symbols]

[0142] 1, 2... tactile feedback device, 100, 300... controller, 101... robot arm, 301... goggle-type device, 102, 302a, 302b... operation unit, 103, 303... control unit, 106... tactile information detection unit, 110, 310... tactile generation unit

Claims

1. A tactile feedback device that generates a tactile sensation on an operating unit that can operate an operated unit, an acquisition means for acquiring tactile information representing a tactile sensation when the operated unit touches an object; a detection means for detecting an object that is of interest to a user operating the operation unit from among objects within a predetermined range including the object that the operated unit is touching; a calculation means for calculating a feedback amount for generating a tactile sensation in the operation unit based on information about an object in which the user is interested and the tactile information.

2. 2. The tactile feedback device according to claim 1, further comprising a setting means for setting a degree of interest of an object within said predetermined range.

3. a display means for displaying an image within the predetermined range, 3. The haptic feedback device according to claim 2, wherein the setting means sets the interest level using an image displayed on the display means.

4. The tactile feedback device according to claim 3, characterized in that the setting means sets the interest level of an object that overlaps the operated portion for a predetermined period of time or more, the interest level of an object within the predetermined range, and the interest level of an object that is larger than a predetermined size to a higher value than the interest level of other objects.

5. 5. The haptic feedback device according to claim 4, wherein the calculation means calculates the amount of feedback using object information of the largest object among the objects with the highest interest level.

6. 3. The haptic feedback device according to claim 2, wherein when there is no object for which the interest level is set, the calculation means does not calculate the feedback amount.

7. a display unit that displays an image within the predetermined range, and a gaze detection unit that detects a gaze direction of a user, 3. The haptic feedback device according to claim 2, wherein the setting means sets the interest level based on the image displayed on the display means and the user's line of sight.

8. 8. The haptic feedback device according to claim 7, wherein the setting means sets the interest level of an object in the line of sight direction to a value higher than the interest levels of other objects.

9. The tactile feedback device of claim 8, wherein the setting means sets the interest level of an object that is in the line of sight for more than a predetermined time, the interest level of an object whose difference between the line of sight and the object's respective speed and direction of travel is within a predetermined difference, and the interest level of an object that is in the line of sight for a total of a predetermined time to a value higher than the interest levels of other objects.

10. 10. The haptic feedback device according to claim 9, wherein the calculation means calculates the amount of feedback using object information of the largest object among the objects with the highest interest level.

11. 8. The haptic feedback device according to claim 7, wherein when there is no object for which the interest level is set, the calculation means does not calculate the feedback amount.

12. 2. The tactile feedback device according to claim 1, wherein the operation unit includes a tactile sensation generating means for applying the tactile sensation based on the amount of feedback.

13. the operated unit is a robot arm, 2. The haptic feedback device according to claim 1, wherein the haptic information is generated based on information detected by a sensor provided on the robot arm.

14. the operated unit is an avatar in a virtual space, 2. The haptic feedback device according to claim 1, wherein the haptic information is generated based on position information of an object in the virtual space and an avatar operated by the operation unit.

15. A control method for a tactile feedback device that generates a tactile sensation on an operating unit that can operate an operated unit, comprising: acquiring tactile information representing a tactile sensation when the operated unit touches an object; detecting an object that is of interest to a user operating the operating unit from among objects within a predetermined range including the object that the operated unit is touching; and calculating a feedback amount for generating a tactile sensation in the operation unit based on object information of the object in which the user is interested and the tactile information.

16. A program for causing a computer to function as each of the means of the haptic feedback device according to any one of claims 1 to 14.

Images