Head-mounted display device, head-mounted display device control method, and program

Abstract

To provide a head-mounted display device in which a virtual object that a user is trying to hold or operate is selected with high precision and a tactile sense of the selected virtual object is given to the user.SOLUTION: A head-mounted display device includes acquisition means for acquiring information about motion of a user, and control means for performing control to reproduce a tactile sense of a second virtual object which is a virtual operation body to act on a first virtual object designated by the user and is selected on the basis of the information about motion of the user.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a head-mounted display device, a control method for a head-mounted display device, and a program. [Background technology]

[0002] Some head-mounted display devices using technologies such as mixed reality (MR), augmented reality (AR), and virtual reality (VR) have a haptic feedback function. The haptic feedback function is a function that, when a user touches a virtual object arranged in a virtual space, gives the haptic sensation of the virtual object to the hand or finger touching the virtual object. The haptic feedback function allows the user to feel as if he or she is handling a real object when manipulating a virtual object. When there are multiple virtual objects, the haptic sensation of which virtual object is to be given is determined, for example, using the user's line of sight information. Patent Document 1 discloses a technology that provides haptic feedback by giving the haptic effect of an object on which the user is concentrating his or her attention when the user operates a virtual object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-215894 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the virtual object that the user is gazing at is not necessarily the virtual object that the user wants to operate. The user may try to grasp or operate an object other than the object that the user is looking at by feeling around. If the haptic effect of the virtual object that the user is gazing at is prioritized over the virtual object that the user wants to grasp, the user may become confused.

[0005] An object of the present invention is to provide a head-mounted display device that can accurately select a virtual object that a user wishes to grasp or manipulate, and provide the user with a tactile sensation of the selected virtual object. [Means for solving the problem]

[0006] The head-mounted display device of the present invention is characterized by having an acquisition means for acquiring information on a user's movement, and a control means for controlling a virtual operating body that acts on a first virtual object specified by the user, and that reproduces the tactile sensation of a second virtual object selected based on the information on the user's movement. Effect of the Invention

[0007] According to the present invention, it is possible to provide a head-mounted display device that can accurately select a virtual object that a user wishes to grasp or manipulate, and provide the user with a tactile sensation of the selected virtual object. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a head-mounted display device. [Diagram 2] FIG. 1 is a block diagram of a head-mounted display device. [Diagram 3] 1A and 1B are diagrams illustrating the principle of a gaze detection method. [Figure 4] FIG. 2 is a schematic diagram of an eye image projected onto an eye imaging element. [Diagram 5] 11 is a flowchart illustrating an example of a gaze detection process. [Figure 6] 11 is a flowchart illustrating a haptic feedback process. [Figure 7] 11A and 11B are diagrams illustrating a specific example of selecting a virtual object for reproducing a haptic sensation. [Figure 8] 13 is a flowchart illustrating a detection process of a first virtual object according to the second embodiment. [Figure 9] 13 is a flowchart illustrating a tactile reproduction process according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of a head-mounted display device according to the present invention will be described with reference to the drawings.

[0010] [Embodiment 1] The head-mounted display device according to the first embodiment detects a virtual object in which a user is interested, and selects a virtual object that the user intends to operate from among candidates of virtual objects that act on the virtual object, based on information on the user's movement (gesture). The head-mounted display device has a haptic feedback function and reproduces the haptic sensation of the selected virtual object. The head-mounted display device can provide the user with the haptic effect of the virtual object that the user intends to operate.

[0011] <Configuration Description> 1(A) and 1(B) show the appearance of a head-mounted display device 100 according to the first embodiment. The head-mounted display device 100 has a haptic feedback function. The head-mounted display device 100 will be described as an optical see-through head-mounted display (HMD) assuming MR. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view.

[0012] The lens 10 is an optical member facing the user's eye. The user can view the outside world through the lens 10. The display device 11 is a display element that displays virtual objects under control (display control) from the CPU 2 described later. The display device 11 displays virtual images (computer graphics, CG) of virtual objects, which are digital information, superimposed on the field of view of both eyes (both right and left eyes) of the user who views the outside world through the optical system (lens 10). The virtual objects include graphical user interfaces (GUIs) such as buttons and icons. The user can view the displayed virtual objects as if they exist in the outside world. When displaying graphics, the position of the depth direction (direction moving away from the user) of the CG as seen by the user can be adjusted by adjusting the left-right positional relationship (parallax) between the display position for the right eye and the display position for the left eye.

[0013] In the optical see-through method, the user views real space through a lens 10 that serves as a display surface. For example, the lens 10 has a prism or a half mirror, and the display device 11 projects CG onto the lens 10. The display device 11 may project CG onto the user's retina. The lens 10 may be a non-light-shielding transmissive display having a display function (the function of the display device 11). The user views real space through the display, and a virtual object is superimposed on the display (lens 10).

[0014] The head-mounted display device 100 may be a video see-through type head-mounted display device that blocks light from the lens 10 and displays a live view or a recorded video captured by an external world imaging unit 20 (described later) directly or after processing it. The head-mounted display device 100 may also be a head-mounted display device that displays a virtual space such as VR.

[0015] The light source driving circuit 12 drives the light sources 13a and 13b. Each of the light sources 13a and 13b is a light source that illuminates the user's eye, and is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user. A part of the light emitted from the light sources 13a and 13b and reflected by the user's eye is collected by the light receiving lens 16 onto the eye imaging element 17. The eye imaging element 17 is an imaging sensor (imaging element) that images the user's eye. The imaging sensor is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0016] The lens 10, the display device 11, the light source driving circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye imaging element 17 are provided for each of the right and left eyes. The user's line of sight information can be acquired using the light source driving circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye imaging element 17. The line of sight information is information related to the line of sight, and indicates, for example, at least one of the viewpoint, the line of sight direction (direction of the line of sight), and the convergence angle (angle between the line of sight of the right eye and the line of sight of the left eye). The viewpoint can be regarded as the position where the line of sight is fixed, the position where the user is looking, or the line of sight position. A method for acquiring line of sight information will be described later with reference to Figs. 3 to 5.

[0017] The external imaging unit 20 includes an imaging sensor (image capture element) that captures an image of the external world in the direction in which the user is facing. The external imaging unit 20 has various functions for acquiring information about the external world. The imaging sensor is, for example, a CCD sensor or a CMOS sensor.

[0018] For example, the external world imaging unit 20 acquires information on the user's movements (gestures) operating the GUI. The information on the user's movements may include information on the movements of the user's hands, the movements of the user's fingers, and the movements of a wand held by the user. The external world imaging unit 20 has a function for detecting and tracking the user's hands, for example, and detecting a specific movement of the user's hands.

[0019] Specific hand movements of the user can be detected using various known techniques such as hand tracking. For example, by linking a finger tapping movement on a GUI button to the operation of pressing the button, the user can perform a gesture operation on the button.

[0020] The external world imaging unit 20 also has a function of detecting the distance from the head mounted display device 100 to a real object existing in the external world. The external world imaging unit 20 can acquire information (distance information) relating to the distance from the head mounted display device 100 to the real object. The external world imaging unit 20 detects depth information of the user's hand, thereby making it possible to more accurately grasp the relationship between the position of the user's hand and the position of a virtual object arranged in the virtual space.

[0021] The function of detecting distance can be realized by using a known technology. For example, the external imaging unit 20 may use a method of irradiating a light wave such as a laser light or an LED (Light-Emitting Diode) light to an object and measuring the time it takes for the light wave to be reflected by the object and return. Instead of light waves, sound waves or radio waves may be used. The external imaging unit 20 may use a distance measuring sensor different from an imaging sensor. The distance measuring sensor is, for example, a LiDAR (Light Detection And Ranging) sensor using a laser light or a TOF (Time Of Flight) sensor using an LED light. The external imaging unit 20 may use a method of calculating the distance from an image of the external world. For example, the external imaging unit 20 can calculate the distance from an image taken by a stereo camera that measures distance using two cameras. In addition, the external imaging unit 20 may use phase difference AF (AutoFocus), which is a method of measuring distance using different pixels (multiple photoelectric conversion elements) in the imaging plane of an imaging sensor. In the first embodiment, phase difference AF is used.

[0022] The haptic feedback device 28 reproduces the haptic sensation of the virtual object being operated by the user, thereby providing the haptic sensation of the virtual object being operated to a part of the body of the user, such as the hand or finger, performing the operation. By feeding back the haptic sensation of the virtual object being operated, the user can obtain the sensation of handling a real object in the virtual space.

[0023] The haptic feedback device 28 may be a device using known technology. The haptic feedback device 28 includes, for example, an encounter type in which the haptic sense imparting device moves in accordance with the position of the user's hand, a grip type in which the user grasps a variable object having a haptic sense imparting function, or a wearable type in which the user wears a glove having a haptic sense imparting function.

[0024] The haptic feedback method may be a method using known technology. For example, the haptic feedback method may be a method that reproduces the sense of touch by applying vibrations using a motor or an actuator. The haptic feedback method may also be a method that uses air, fluid, or ultrasonic waves. The haptic feedback method may also be a method that uses electrical stimulation of a living body within the range of technologies whose safety has been confirmed, such as electrical muscle stimulation (EMS).

[0025] FIG. 2 is a block diagram showing an electrical configuration of the head-mounted display device 100. In FIG. 2, the same numbers are used for the configurations corresponding to those shown in FIG. 1, and detailed explanations are omitted. The CPU 2 is a central processing unit of a microcomputer built into the head-mounted display device 100, and controls the entire head-mounted display device 100. The head-mounted display device 100 includes a memory unit 3, a display device 11, a light source driving circuit 12, a gaze detection circuit 15, an external world imaging unit 20, a head posture detection circuit 23, and a haptic feedback control circuit 27, and these configurations are connected to the CPU 2. The head-mounted display device 100 also includes light sources 13a and 13b, an eye imaging element 17, a distance detection circuit 21, a gesture detection circuit 22, a first object detection circuit 24, an operating body detection circuit 25, and a second object selection circuit 26. The head-mounted display device 100 is connected to a haptic feedback device 28.

[0026] The memory unit 3 has a function of storing data transmitted from each component, such as a video signal from the eye imaging element 17. The data stored in the memory unit 3 can be transmitted to each component via the CPU 2.

[0027] The gaze detection circuit 15 is a digital serial interface circuit that performs A / D conversion on the output of the eye imaging element 17 (an image of the eye captured by the eye imaging element 17) when an optical image of the eye is formed on the eye imaging element 17, and transmits the result to the CPU 2. The CPU 2 extracts feature points used for gaze detection from the eye image according to a predetermined algorithm, and detects the user's gaze from the positions of the feature points. The CPU 2 can identify (detect) the virtual object the user is looking at from the gaze detection result (gaze information).

[0028] The distance detection circuit 21, for example, A / D converts signals (voltages) from a plurality of pixels (a plurality of photoelectric conversion elements) for phase difference detection included in an imaging sensor in the external world imaging unit 20, and sends the converted signals to the CPU 2. The CPU 2 uses the signals from the plurality of pixels to calculate the distance to a real object (subject) corresponding to each distance detection point.

[0029] The gesture detection circuit 22 detects and tracks the user's movement (gesture) from an image (image of the outside world) obtained by the outside world imaging unit 20, and transmits the obtained information on the user's movement to the CPU 2. The CPU 2 determines whether the user's gesture is a specific movement, and if it is, executes a process associated with the specific movement.

[0030] Head posture detection circuit 23 has, for example, an acceleration sensor, and transmits a detection signal from the acceleration sensor to CPU 2. CPU 2 analyzes the detection signal and detects the posture of the user's head (for example, the amount of tilt) at the time the signal is detected. A change in head posture can be considered as a change in the viewing direction (the direction the user is facing) in the coordinate system of real space (world coordinate system). Based on the detection result of the head posture, CPU 2 adjusts the display of the GUI so as to synchronize with the change in the viewing direction. The display position can be changed.

[0031] The first object detection circuit 24 is a circuit that detects (selects) a virtual object in which the user shows interest. A virtual object in which the user shows interest is a virtual object on which the user intends to exert an action using another virtual object. A virtual object in which the user shows interest is also referred to as a first virtual object hereinafter. In addition, another virtual object that acts on the first virtual object is also referred to as a virtual operating body. The first virtual object is previously associated with at least one or more virtual operating bodies. The user can exert an action on the first virtual object by operating one of the virtual operating bodies.

[0032] The first object detection circuit 24 can detect the first virtual object in which the user shows interest, for example, based on the user's gaze information. The first object detection circuit 24 may detect the first virtual object based on the movement of the user's hand or finger pointing at the virtual object, or the user's operation of specifying the virtual object with a controller for operating the head-mounted display device 100. The user can specify the first virtual object in which the user shows interest by the gaze, the user's hand or finger, or the controller. The following describes the case where the user specifies the first virtual object by the gaze.

[0033] The operating body detection circuit 25 detects another virtual object (virtual operating body) that can act on the first virtual object. The operating body detection circuit 25 acquires candidates for the virtual operating body that is previously associated with the first virtual object.

[0034] As a specific example, if the first virtual object in which the user shows interest is a "screw," the operation object detection circuit 25 acquires virtual objects such as a "screwdriver" and an "electric screwdriver," which are tools for turning the "screw," as candidates for the virtual operation object. If there are multiple "screwdrivers" with different tip shapes and sizes, the operation object detection circuit 25 may be configured to acquire a "screwdriver" that matches the shape and size of the head of the "screw" as a candidate. Whether or not the shape and size match may be determined in advance, or may be determined by analyzing an image.

[0035] The relationship between the first virtual object in which the user is interested and the virtual control body acting on the first virtual object may be predefined or updated based on the usage status. For example, if the virtual control body has not been used to control the first virtual object for a predetermined period of time, the relationship between the virtual control body and the first virtual object may be released.

[0036] The operating body detection circuit 25 may acquire a virtual object located within a predetermined distance from the user as a candidate for a virtual operating body acting on the first virtual object. The operating body detection circuit 25 may acquire candidates for a virtual operating body based on various conditions, such as a distance from the first virtual object, without being limited to a distance from the user.

[0037] The second object selection circuit 26 selects a virtual object (hereinafter also referred to as a second virtual object) that the user intends to grasp or operate based on information on the user's movement. The second object selection circuit 26 first acquires information (gesture information) on the movement of the user intending to grasp or operate a virtual operating body acting on the first virtual object. The gesture information may be acquired by the gesture detection circuit 22. The second object selection circuit 26 searches for candidates of the virtual operating body acquired by the operating body detection circuit 25, and determines whether or not there is a virtual object that matches the gesture information. If there is a virtual object that matches the user's gesture information, the second object selection circuit 26 selects (selects) the virtual object as the second virtual object.

[0038] As a specific example, the first virtual object is a "screw" and the virtual manipulation acting on the first virtual object is If the candidates for the operating object are a "screwdriver" and an "electric screwdriver", the second object selection circuit 26 selects one of the virtual operating objects based on the gesture information. If the diameter of the gripping part of the "electric screwdriver" is larger than the diameter of the gripping part of the "screwdriver", and the user makes a gesture of opening the hand wider than the diameter of the gripping part of the "electric screwdriver" to grip it, the "electric screwdriver" is selected.

[0039] The haptic feedback control circuit 27 controls the haptic feedback device 28 to reproduce the haptic sensation of the second virtual object selected by the second object selection circuit 26, and gives a haptic sensation to the user's hand or finger, etc. When other virtual objects exist around the second virtual object, the haptic feedback control circuit 27 may control to reproduce the haptic sensation of the second virtual object that the user is holding and operating, and not to reproduce the haptic sensation of the other virtual objects around the second virtual object. On the other hand, the haptic feedback control circuit 27 may control to reproduce the haptic sensation of the other virtual objects existing around the second virtual object together with the haptic sensation of the second virtual object. In this case, the haptic feedback control circuit 27 may control to reproduce the haptic sensation of the second virtual object by weighting it so that it is larger than the haptic sensation of the other virtual objects. The haptic feedback device 28 gives the haptic sensation of the second virtual object, etc. to the user according to an instruction from the haptic feedback control circuit 27.

[0040] In the following example, the head-mounted display device 100 detects a first virtual object in which the user is interested by gaze information. The head-mounted display device 100 detects candidates for virtual operating bodies that act on the first virtual object, and selects a second virtual object to be operated from among the candidates for virtual operating bodies based on information on the user's movement.

[0041] <Gaze detection processing> The process of detecting the user's line of sight and acquiring line of sight information will be described with reference to Figures 3 to 5. The line of sight of both the right eye and the left eye can be detected by the following line of sight detection method.

[0042] FIG. 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in FIG. 3, light sources 13a and 13b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the user. Light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 16, and illuminate user's eyeball 140. A part of the light emitted from light sources 13a and 13b and reflected by eyeball 140 is collected by light-receiving lens 16 on eye imaging element 17. Head-mounted display device 100 has left and right eye imaging elements 17 as shown in FIG. 1, and acquires left and right eye images.

[0043] Fig. 4(A) is a schematic diagram of an eye image captured by eye imaging element 17 (optical image projected onto eye imaging element 17). Fig. 4(B) is a diagram showing the output intensity of the CCD in eye imaging element 17.

[0044] 5 is a flow chart illustrating the gaze detection process. When the gaze detection process starts, in step S001, the light sources 13a and 13b irradiate infrared light toward the user's eyeball 140. An optical image of the user's eye illuminated by the infrared light is formed on the eye imaging element 17 through the light receiving lens 16 and photoelectrically converted by the eye imaging element 17. This results in an electrical signal of the eye image that can be processed. In step S002, the CPU 2 acquires the eye image (electrical signal of the eye image; image data of the eye image) from the eye imaging element 17 via the gaze detection circuit 15.

[0045] In step S003, the CPU 2 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c shown in FIG. 3 from the eye image obtained in step S002. The infrared light emitted from light sources 13a and 13b illuminates a cornea 142 of a user's eyeball 140. Corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are collected by light receiving lens 16 and focused on eye imaging element 17 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from ends a and b of pupil 141 are focused on eye imaging element 17 to become pupil end images a' and b' in the eye image.

[0046] Fig. 4(A) shows an example of an eye image of a reflection image obtained from eye imaging element 17. Fig. 4(B) shows luminance information (luminance distribution) of area α in the eye image of Fig. 4(A). In Fig. 4(B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. The coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe. The coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb.

[0047] 4B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from coordinates Xb to Xa, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the eye imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd and Xe.

[0048] In contrast, a luminance intermediate between the above two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, the luminance in the region where the X coordinate is smaller than Xb and the region where the X coordinate is larger than Xa is approximately intermediate between the above two types of luminance.

[0049] From the luminance distribution as shown in Fig. 4(B), the CPU 2 can obtain the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b'. Specifically, the CPU 2 can obtain the coordinates where the luminance is extremely high as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates of the boundary where the luminance is extremely low as the coordinates of the pupil edge images a' and b'.

[0050] When the rotation angle θx of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 16 is equal to or smaller than a predetermined angle, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of a light beam from the pupil center c on the eye imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. The predetermined angle can be determined, for example, so that the corneal reflection images Pd', Pe' are formed on the eye imaging element 17. The X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the CPU 2 can estimate the coordinates of the corneal reflection images Pd', Pe' and the coordinate of the pupil center image c'.

[0051] In step S004, CPU 2 calculates the imaging magnification β of the eye image. The imaging magnification β is determined by the position of eyeball 140 relative to light receiving lens 16, and can be obtained as a function of the distance (Xd-Xe) between corneal reflection images Pd', Pe'.

[0052] In step S005, CPU 2 acquires the rotation angle of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 16. The X coordinate of the midpoint between corneal reflection images Pd' and Pe' approximately coincides with the X coordinate of the center of curvature O of cornea 142. If the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of the optical axis of eyeball 140 in the ZX plane (plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of eyeball 14 in the ZY plane (plane perpendicular to the X axis) can be calculated in the same manner as the rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0053] In step S006, the CPU 2 acquires the user's line of sight (hereinafter also referred to as the viewpoint) on the lens 10 using the rotation angles θx and θy calculated in step S005. If the coordinates (Hx, Hy) of the point correspond to the center c of the pupil 141 on the lens 10, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following Equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0054] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system for performing the gaze detection process, and is a conversion coefficient for converting the rotation angles θx, θy into coordinates corresponding to the center c of the pupil 141 on the lens 10. Parameter m is determined in advance and stored in the memory unit 3. Gaze correction coefficients Ax, Bx, Ay, and By are parameters for correcting individual differences in the user's gaze, and are obtained by performing a calibration operation. Gaze correction coefficients Ax, Bx, Ay, and By are stored in the memory unit 3 before the gaze detection process starts.

[0055] In step S007, the CPU 2 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 3, and ends the gaze detection process. Note that the process in FIG. 5 shows an example in which the corneal reflection images of the light sources 13a and 13b are used to obtain the rotation angle of the eyeball, and the coordinates of the viewpoint on the lens 10 are obtained, but this is not limiting. A method for obtaining the rotation angle of the eyeball from an eye image may be a method for measuring the gaze from the pupil center position. Also, the gaze detection method may be a method that does not use an eye image, such as a method for detecting an electrooculogram and detecting the gaze based on the electrooculogram.

[0056] <Haptic feedback processing> 6(A) to 6(C) are flowcharts illustrating haptic feedback processing. The haptic feedback processing is processing from when a user starts an action of grasping or manipulating a second virtual object to affect a first virtual object to when the user receives haptic feedback from the second virtual object. Even when the user tries to manipulate the second virtual object by hand, the user can receive the haptic sensation of the second virtual object and can cause the manipulation of the second virtual object to affect the first virtual object.

[0057] Fig. 6(A) shows a flowchart illustrating a haptic feedback process. Fig. 6(B) is a flowchart illustrating details of a process for detecting a first virtual object in the haptic feedback process. Fig. 6(C) is a flowchart illustrating details of a process for reproducing the haptic sensation of a second virtual object in the haptic feedback process.

[0058] The processes in Fig. 6(A) to 6(C) are executed when the user wears the head-mounted display device 100 and the head-mounted display device 100 is activated. In addition, initial settings such as calibration for correcting individual differences in the line of sight are performed before the process in Fig. 6(A) is started. The process shown in Fig. 6(A) is executed periodically, for example at a set cycle, while the user is using the head-mounted display device.

[0059] In step S101, the first object detection circuit 24 detects a first virtual object in which the user shows interest. The first virtual object is specified by the user. The user can specify the first virtual object by, for example, his / her line of sight. In this case, the first object detection circuit 24 can detect a virtual object present at the line of sight position of the user detected by the line of sight detection circuit 15 as the first virtual object. Also, the user can specify the first virtual object using a hand or a controller. In this case, the first object detection circuit 24 can detect the virtual object specified by the user as the first virtual object based on the information of the user's movement detected by the gesture detection circuit 22.

[0060] In step S102, the operation body detection circuit 25 judges whether or not there is a candidate for a virtual operation body that can act (operate) on the first virtual object detected in step S101. The virtual operation body is a virtual object that is paired with the first virtual object and is previously selected as a candidate for the virtual operation body. The first virtual object is stored in the memory unit 3 or the like. The first virtual object can be linked to a plurality of virtual operating bodies. The operating body detection circuit 25 searches for virtual objects other than the first virtual object, and determines whether or not there is a candidate for a virtual operating body linked to the first virtual object. If there is a candidate for a virtual operating body, the process proceeds to step S103. If there is no candidate for a virtual operating body, the process returns to step S101.

[0061] In step S103, if there is at least one candidate for the virtual operating object, it is presented as a candidate for the virtual operating object. The presentation here means that the virtual operating object is placed in the virtual space, including the case where it is placed outside the range of the user's field of view.

[0062] In step S104, the gesture detection circuit 22 acquires information on the user's movement (gesture information). The gesture information is information on the user's movement of trying to grasp a virtual operation body acting on the first virtual object while the user is showing an interest in the first virtual object.

[0063] In step S105, the second object selection circuit 26 determines whether or not there is a candidate for a virtual operation body that matches the user's movement detected in step S104. That is, the second object selection circuit 26 determines whether or not there is a candidate for a virtual operation body that can be grasped or operated by the detected user's movement (gesture). For example, by previously storing a virtual operation body and a gesture that can grasp or operate the virtual operation body in association with each other, the second object selection circuit 26 can search for a virtual operation body that matches the user's gesture. If a candidate that matches the user's movement is present, the process proceeds to step S106. If a candidate that matches the user's movement is not present, the process proceeds to step S106.

[0064] In step S106, the second object selection circuit 26 selects a candidate for a virtual operation body that matches the user's movement as a second virtual object to be operated by the user. If there are multiple candidates for a virtual operation body that matches the user's movement, the second object selection circuit 26 may select, for example, a candidate that is closer to the position of the user's hand (the position where the user's movement is detected) as the second virtual object.

[0065] In step S107, the haptic feedback control circuit 27 controls the haptic feedback device 28 to reproduce the haptic sensation of the second virtual object selected in step S106. The haptic feedback control circuit 27 can improve the sense of operation by providing the user with the haptic sensation of the second virtual object.

[0066] FIG. 6(B) is a flowchart illustrating the details of a process for detecting a first virtual object in which a user shows interest. In the example of FIG. 6(B), the first object detection circuit 24 uses the user's gaze information to select the first virtual object. The first object detection circuit 24 can select the first virtual object based on, for example, the time the user's gaze is directed at the virtual object. Specifically, the first object detection circuit 24 can select, as the first virtual object, a virtual object on which the user's gaze is directed for a time longer than a predetermined time.

[0067] In step S201, the first object detection circuit 24 acquires the user's line of sight information detected by the line of sight detection circuit 15. In step S202, the first object detection circuit 24 determines whether or not a virtual object is present at the user's line of sight. If a virtual object is present at the user's line of sight, the process proceeds to step S203. If a virtual object is not present at the user's line of sight, the process returns to step S201.

[0068] In step S203, the first object detection circuit 24 measures the time during which the user is looking at the virtual object, and determines whether the measured time is longer than a predetermined time. If the time that the user is looking at the virtual object is longer than the predetermined time, the process proceeds to step S204. If the time that the user is looking at the virtual object is equal to or shorter than the predetermined time, the process returns to step S201.

[0069] In step S204, the first object detection circuit 24 sets the virtual object that the user has been looking at for a period of time that exceeds a predetermined period of time as the first virtual object in which the user is interested.

[0070] The first object detection circuit 24 may select a virtual object present at the line of sight position as the first virtual object regardless of the time the user is looking at the virtual object. The first object detection circuit 24 may also set a virtual object designated by the user by a method other than the line of sight as the first virtual object. For example, the first object detection circuit 24 may set a virtual object designated by the user's hand or finger, or a virtual object designated by a controller for operating the head-mounted display device 100 as the first virtual object.

[0071] Fig. 6C is a flow chart illustrating the details of the process of reproducing the haptic sensation of the second virtual object. In the example of Fig. 6C, when another virtual object exists around the second virtual object, the haptic feedback control circuit 27 controls the reproduction so that the haptic intensity of the second virtual object is weighted to be stronger than the haptic intensity of the other virtual object.

[0072] In step S301, the haptic feedback control circuit 27 determines whether or not there is another virtual object other than the first virtual object and the second virtual object. The haptic feedback control circuit 27 can determine whether or not there is another virtual object based on, for example, the results of searching for a virtual object by the first object detection circuit 24, the operation body detection circuit 25, and the second object selection circuit 26. If there is another virtual object, the process proceeds to step S302. If there is no other virtual object, the process proceeds to step S303.

[0073] In step S302, the haptic feedback control circuit 27 sets weighting so that the haptic intensity of the second virtual object is output stronger than the haptic intensities of the other virtual objects. In step S303, the haptic feedback control circuit 27 controls so that the haptic intensity of the second virtual object is reproduced based on the weighting set in step S302.

[0074] Even if there are virtual objects other than the second virtual object, the user can reduce the influence of the tactile sensation caused by the other virtual objects and appropriately provide the user with the tactile sensation of the second virtual object. The processes shown in Figures 6(A) to 6(C) are merely examples, and the order of the steps and the processing method may be changed as appropriate within the scope in which similar results can be obtained.

[0075] 7 is a diagram for explaining a specific example of selecting a second virtual object for reproducing a haptic sensation. Example 1 and Example 2 shown in FIG. 7 specifically exemplify a process of selecting a second virtual object from virtual objects present around a user.

[0076] Example 1 in the upper part is a specific example of a scene in which a screw is tightened with a screwdriver. Virtual objects 701 present around the user are a screw, a nut, a screwdriver, an electric screwdriver, a spanner, and a wrench. A first virtual object 702 is a virtual object in which the user shows interest and which is detected by the first object detection circuit 24. The first virtual object 702 is specified by the user. For example, the user can specify the first virtual object 702 by directing his or her gaze toward the screw. In Example 1, the first object detection circuit 24 detects the screw specified by the user as the first virtual object 702.

[0077] The candidate virtual operating object 703 is a candidate for a virtual operating object that can act on the first virtual object 702. The screw, which is the first virtual object 702, is associated with a screwdriver and an electric screwdriver as virtual operating objects that can act on the screw. The operating object detection circuit 25 detects the virtual operating object A screwdriver and an electric screwdriver are presented as candidates 703 for the work object.

[0078] The gesture information 704 is information on the user's movement detected by the gesture detection circuit 22. The gesture information 704 is information on the user's movement to grasp or operate a second virtual object acting on a first virtual object 702 in which the user shows interest. In Example 1, the gesture information 704 is information on the movement of grasping an object having a diameter (thickness) of about the handle of an electric screwdriver. The user only needs to open his / her hand wider than when grasping a screwdriver and make a movement as if grasping a virtual object thicker than a screwdriver.

[0079] The second virtual object 705 is information on a second virtual object detected based on the gesture information 704. In the first example, the second object selection circuit 26 specifies the power screwdriver as the second virtual object.

[0080] Example 2 in the lower part is a specific example of a scene in which a keyboard is operated while looking at a monitor. Virtual objects 701 present around the user are a monitor, a tablet, a keyboard, a mouse, a watch, and a touch pen. In Example 2, the first object detection circuit 24 detects a monitor designated by the user as a first virtual object 702. The monitor, which is the first virtual object 702, is associated with a keyboard and a mouse as virtual operating objects that can act on the monitor. The operating object detection circuit 25 presents the keyboard and the mouse as candidates 703 for the virtual operating object.

[0081] In Example 2, the gesture information 704 is information on a movement of extending both hands with fingers open and palms facing downward. In Example 2, the second object selection circuit 26 specifies a keyboard as the second virtual object from the candidate 703 of a virtual operating body acting on the first virtual object and the gesture information 704.

[0082] The haptic feedback control circuit 27 weights the haptic intensity of the second virtual object specified in Example 1 and Example 2 so that it is stronger than the haptic intensity of the other virtual objects, and provides the user with a haptic sensation. The user can feel a stronger sense of gripping and manipulating the second virtual object.

[0083] According to the above-mentioned embodiment 1, the head-mounted display device 100 can improve the selectability of a virtual object that the user is trying to grasp or operate, even when the user grasps the virtual object by feeling around without directing his / her gaze. Moreover, the head-mounted display device 100 can improve the sense of operation of the virtual object by giving the user a tactile sensation of the virtual object that the user is trying to grasp or operate by feeling around.

[0084] [Embodiment 2] The second embodiment differs from the first embodiment in the method of detecting (selecting) the first virtual object in which the user shows interest. The configuration of the head-mounted display device 100 and the method of gaze detection in the second embodiment are the same as those in the first embodiment.

[0085] Fig. 8 is a flowchart illustrating the details of the process of detecting a first virtual object in which a user is interested. The process illustrated in Fig. 8 is the details of the process of step S101 in Fig. 6(A), and illustrates a specific example different from that in Fig. 6(B).

[0086] The first object detection circuit 24 selects the first virtual object using the gaze information as in the first embodiment. In the second embodiment, the first object detection circuit 24 selects the first virtual object based on the number of times the user's gaze is directed toward the virtual object, in addition to the user's gaze position.

[0087] In step S211, the first object detection circuit 24 acquires the user's line of sight information detected by the line of sight detection circuit 15. In step S212, the first object detection circuit 24 determines whether or not a virtual object is present at the user's line of sight. If a virtual object is present at the user's line of sight, the process proceeds to step S203. If a virtual object is not present at the user's line of sight, the process returns to step S201.

[0088] In step S213, when a virtual object is present at the gaze position, the first object detection circuit 24 counts up the number of times the user's gaze is directed to the virtual object. When multiple virtual objects are present, the first object detection circuit 24 counts the number of times the user's gaze is directed to each virtual object.

[0089] In step S214, the first object detection circuit 24 determines whether or not to select a first virtual object in which the user is interested. The first object detection circuit 24 can determine to select a first virtual object when, for example, a predetermined time has elapsed since the previous selection of the first virtual object.

[0090] Furthermore, the first object detection circuit 24 may determine to select the first virtual object when there is a virtual object to which the user's gaze has been directed a predetermined number of times. In this case, in steps S215 and S216, the first object detection circuit 24 detects the virtual object to which the user's gaze has been directed a predetermined number of times as the virtual object to which the gaze has been directed the most times, and selects it as the first virtual object.

[0091] Furthermore, the first object detection circuit 24 may determine to select the first virtual object when receiving a voice instruction or a gesture instruction from the user. In this case, in steps S215 and S216, the first object detection circuit 24 detects the virtual object to which the user's gaze is most frequently directed when receiving the instruction from the user, and selects it as the first virtual object.

[0092] If the first virtual object is to be selected, the process proceeds to step S215. If the first virtual object is not to be selected, the process returns to step S211. In a situation where the user is looking at multiple virtual objects present in the field of view, the number of times the user's gaze is directed at each virtual object is incremented. Since the processes from step S211 to step S214 are executed periodically, the number of times is incremented for each period for the virtual object that the user continues to look at.

[0093] In step S215, the first object detection circuit 24 searches for a virtual object that the user's gaze is directed to the most times within a predetermined time. It can be determined that the more times the user's gaze is directed to a virtual object, the higher the user's interest is.

[0094] In step S216, the first object detection circuit 24 selects the virtual object to which the user's gaze has been directed the most number of times within a predetermined time period as the first virtual object in which the user is interested. When the first virtual object is detected (selected) in step S216, the number of times the user's gaze has been directed to each virtual object may be initialized.

[0095] In the above-mentioned second embodiment, the first virtual object in which the user shows interest is selected using time-series information such as the number of times the user's gaze is directed. The second embodiment is an embodiment effective when it is difficult for the user to continue looking at the first virtual object. For example, the second embodiment is effective when the user alternates between looking at the first virtual object and a second virtual object acting on the first virtual object, and when the user wants to look at the second virtual object in order to actually grasp the second virtual object. The process shown in FIG. 8 is an example, and the order of each step and the processing method may be changed as appropriate within a range in which similar results can be obtained.

[0096] Note that the first object detection circuit 24 may select the first virtual object using time-series information on a gesture or instruction using a user's hand, a user's finger, or a controller, other than the line of sight.

[0097] According to the second embodiment, the head-mounted display device 100 can appropriately select the first virtual object in which the user is interested by using time-series information, even if the target specified by the user changes. Also, even if the user grasps the virtual object by feel without directing his / her gaze, the selectability of the virtual object that the user intends to grasp or operate can be improved.

[0098] [Embodiment 3] The third embodiment differs from the first embodiment in the method of providing (reproducing) the tactile sensation of a second virtual object acting on a first virtual object in which a user is interested. The configuration of the head-mounted display device 100 and the method of gaze detection in the third embodiment are the same as those in the first embodiment.

[0099] 9(A) and 9(B) are flow charts illustrating the details of the process of reproducing the tactile sensation of the second virtual object. The processes illustrated in FIGS. 9(A) and 9(B) are the details of the process of step S107 in FIG. 6(A), and show specific examples different from FIG. 6(C). FIG. 9(A) is a flow chart illustrating the process of controlling the rearrangement and remote operation of the second virtual object based on the distance between the second virtual object and the hand. FIG. 9(B) is a flow chart illustrating the process of reproducing the tactile sensation of the second virtual object by weighting based on the distance between the second virtual object and the gaze position.

[0100] In step S311 of FIG. 9A, the haptic feedback control circuit 27 determines whether the distance between the user's hand operating the second virtual object and the second virtual object is shorter than a predetermined threshold. The predetermined threshold can be set arbitrarily based on, for example, the types of the first virtual object and the second virtual object, the sizes in the virtual space, and the like. If the distance between the second virtual object and the user's hand is shorter than the predetermined threshold, the process proceeds to step S316. If the distance between the second virtual object and the user's hand is equal to or greater than the predetermined threshold, the process proceeds to step S312. Note that even if the distance between the second virtual object and the user's hand is shorter than the predetermined threshold, if the second virtual object is present on the back of the hand rather than the palm side, the haptic feedback control circuit 27 may proceed to step S312.

[0101] In step S312, the haptic feedback control circuit 27 determines whether to rearrange the second virtual object. The haptic feedback control circuit 27 rearranges the second virtual object, thereby enabling the user to grasp or manipulate the second virtual object with the hand.

[0102] Whether or not to rearrange the second virtual object when the distance between the second virtual object and the user's hand is equal to or greater than a predetermined threshold value can be set in advance. Whether or not to rearrange the second virtual object may be set in advance for each of the first virtual object or the second virtual object. In addition, the haptic feedback control circuit 27 may determine that the second virtual object is to be rearranged when, for example, the second virtual object is within the reach of the user's hand. Furthermore, whether or not to rearrange the second virtual object may be determined based on whether or not the user has given an instruction by voice or an instruction by gesture. If the second virtual object is to be rearranged, the process proceeds to step S313. If the second virtual object is not to be rearranged, the process proceeds to step S314.

[0103] In step S313, in response to an instruction from the haptic feedback control circuit 27, the display device 11 places the second virtual object at the position of the user's hand. The path 27 may control the position of the second virtual object and the position of the user's hand (the position where the user's movement is detected) to have a predetermined positional relationship. The predetermined positional relationship is a positional relationship where the user can grasp or operate the second virtual object. The display device 11 rewrites the position information of the virtual object placed in the real space coordinate system, moves the second virtual object to a position where the user can grasp or operate it, and redraws it. For example, the display device 11 may redraw the second virtual object to a position where the user can grasp it with the palm side. By relocating the second virtual object, the user can smoothly start the operation without searching for the second virtual object even when the user tries to operate the second virtual object at a position away from the second virtual object.

[0104] In step S314, the haptic feedback control circuit 27 determines whether or not to remotely control the second virtual object. When it is determined that the second virtual object is not to be rearranged, whether or not to remotely control the second virtual object can be set in advance. Whether or not to remotely control the second virtual object may be set in advance for each of the first virtual object or the second virtual object. In addition, whether or not to remotely control the second virtual object may be determined based on whether or not a voice instruction or a gesture instruction is given from the user. When the second virtual object is to be remotely controlled, the process proceeds to step S315. When the second virtual object is not to be remotely controlled, the process proceeds to step S316.

[0105] In step S315, the haptic feedback control circuit 27 activates a linking function between the second virtual object and the user's hand. That is, the haptic feedback control circuit 27 controls so that the haptic sensation of the second virtual object is reproduced even if the second virtual object is away from the position of the user's hand (the position where the user's movement is detected). By activating the linking function, the haptic feedback control circuit 27 can make the user's movement (such as the action of holding or manipulating with the hand) act on the second virtual object even if the second virtual object is not disposed at a position where the second virtual object is in contact with the user's hand. The haptic feedback control circuit 27 does not move the second virtual object to the position of the hand, but moves the position of the user's hand (the position where the user's movement is detected) to a position where the second virtual object can be held or manipulated, and reproduces the haptic sensation. In this way, the haptic feedback control circuit 27 can link the second virtual object with the user's movement.

[0106] In step S316, the haptic feedback control circuit 27 reproduces the haptic sensation of the second virtual object, and provides the reproduced haptic sensation to the hand or the like of the user that is holding or manipulating the second virtual object.

[0107] The haptic feedback control circuit 27 moves the position of the second virtual object or the position of the user's movement (hand position) to reproduce the haptic sensation of the second virtual object, thereby realizing a smooth operation even if the position of the second virtual object and the position of the user's movement are separated. The haptic feedback control circuit 27 can appropriately provide the user with the haptic sensation of the second virtual object by rearranging the second virtual object or enabling a linking function based on the distance between the position of the second virtual object and the position of the user's movement.

[0108] 9B is a flowchart illustrating a process for setting the weighting of the haptic sense and the priority (processing performance) of the haptic sense processing based on the distance between the second virtual object and the gaze position. In step S321, the haptic feedback control circuit 27 acquires the distance between the user's gaze position and the second virtual object.

[0109] In step S322, the haptic feedback control circuit 27 determines whether or not to set a haptic weighting factor based on the distance acquired in step S321. Whether or not to set a haptic weighting factor may be set in advance, or may be set or changed by the user. If a haptic weighting factor is to be set, the process proceeds to step S323. If a haptic weighting factor is not to be set, the process proceeds to step S324.

[0110] In step S323, the haptic feedback control circuit 27 sets a haptic weighting coefficient for the second virtual object based on the distance between the user's gaze position and the second virtual object. The haptic feedback control circuit 27 sets a larger haptic weighting coefficient as the distance (the distance between the user's gaze position and the second virtual object) increases within the distance range for which the weighting coefficient is set.

[0111] When detecting an object by touch, if the object is outside the range of the field of view, the user may not notice the touch if the tactile intensity is weak. Therefore, the tactile feedback control circuit 27 sets a weighting coefficient so that the tactile intensity of the second virtual object increases as the distance between the line of sight and the second virtual object increases. This makes it easier for the user to notice the touch of the second virtual object.

[0112] The value of the weighting factor and the distance range for which the weighting factor is set can be set arbitrarily. The value of the weighting factor may be set so that there is a positive correlation between the distance and the magnitude of the weighting factor within the distance range for which the weighting factor is set.

[0113] In step S324, the haptic feedback control circuit 27 determines whether or not to set a priority (processing performance) for the process of reproducing a haptic sensation based on the distance acquired in step S321. Whether or not to set a priority for the process of reproducing a haptic sensation may be set in advance, or may be set or changed by the user. If the priority for the process of reproducing a haptic sensation is set, the process proceeds to step S325. If the priority for the process of reproducing a haptic sensation is not set, the process proceeds to step S326.

[0114] In step S325, the haptic feedback control circuit 27 sets a priority of a process for reproducing a haptic sensation of the second virtual object based on the distance between the user's gaze position and the second virtual object. The haptic feedback control circuit 27 sets a higher priority of a process for reproducing a haptic sensation as the distance (the distance between the user's gaze position and the second virtual object) becomes longer within the distance range for setting the priority of the process.

[0115] When detecting an object by contact, if the object is outside the range of the field of view, the gain reduction and time lag when reproducing the haptic sensation are undesirable. Therefore, the haptic feedback control circuit 27 increases the priority of the process of reproducing the haptic sensation of the second virtual object as the distance between the line of sight and the second virtual object increases. This makes it easier for the user to notice contact with the second virtual object.

[0116] The priority of the processing can be changed, for example, by the occupancy rate of the CPU 2, the priority of the task to be processed, the priority of access to the memory unit 3, etc. The distance range for setting the priority of the processing can be set arbitrarily. The priority of the processing can be set so that there is a positive correlation between the distance and the priority within the distance range for setting the priority of the processing. Furthermore, in order to raise the priority of the processing for reproducing the haptic sensation, the priority of the processing of other parallel modules may be lowered, as long as the impact on the usability of the head-mounted display device 100 is not an issue.

[0117] In step S326, the haptic feedback control circuit 27 reproduces the haptic sensation of the second virtual object, and provides the reproduced haptic sensation to the hand or the like of the user that is holding or manipulating the second virtual object.

[0118] By setting the weighting of the tactile sensation based on the distance between the user's gaze position and the second virtual object, the user can easily notice contact with the second virtual object located away from the gaze position. Also, by setting the priority of the process of reproducing the tactile sensation based on the distance between the user's gaze position and the second virtual object, the user can more accurately detect the tactile sensation of the second virtual object located away from the gaze position. According to the process shown in FIG. 9(B), Thus, the user can smoothly grasp or manipulate virtual objects even if they are outside the field of view.

[0119] The processes shown in FIGS. 9(A) and (B) are merely examples, and the order of the steps and the processing method may be changed as appropriate within the scope in which similar results can be obtained.

[0120] According to the head-mounted display device 100 of the third embodiment, even if the second virtual object is far from the position of the user's hand (the position where the user's movement is detected) or the gaze position, the user can smoothly grasp or operate the intended virtual object. Moreover, the head-mounted display device 100 can improve the selectability of the virtual object that the user is trying to grasp or operate, even when the user grasps the virtual object by feeling around without directing his or her gaze. Furthermore, the head-mounted display device 100 can improve the sense of operation of the virtual object by giving the user a tactile sensation of the virtual object that the user is trying to grasp or operate by feeling around.

[0121] In the third embodiment, the distance between the user's gaze position and the second virtual object is described. However, the weighting and processing priority may be set based on the distance between the first virtual object and the second virtual object, instead of the distance from the gaze position.

[0122] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. In addition, parts of the above-described embodiments may be combined as appropriate.

[0123] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0124] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An acquisition means for acquiring information on a user's movement; a control means for controlling so as to reproduce a haptic sensation of a second virtual object, which is a virtual operating body acting on a first virtual object designated by the user and is selected based on information on the movement of the user; A head-mounted display device comprising: (Configuration 2) The first virtual object is specified by the user's line of sight. 2. A head-mounted display device according to configuration 1. (Configuration 3) A selection means for selecting the first virtual object is further provided, The first virtual object is selected based on a time period during which the user's gaze is directed at the first virtual object. 3. The head-mounted display device according to configuration 2. (Configuration 4) A selection means for selecting the first virtual object is further provided, The first virtual object is selected based on the number of times the user's gaze is directed at the first virtual object. 3. The head-mounted display device according to configuration 2. (Configuration 5) The first virtual object is a virtual object to which the user's gaze is directed most frequently within a predetermined time period. 5. The head-mounted display device according to configuration 4. (Configuration 6) The first virtual object is a virtual object to which the user's gaze has been directed a predetermined number of times. 5. The head-mounted display device according to configuration 4. (Configuration 7) The first virtual object is a virtual object to which the user's gaze is most frequently directed when an instruction is received from the user. 5. The head-mounted display device according to configuration 4. (Configuration 8) The first virtual object is specified by the user's hand, the user's finger, or a controller for operating the head-mounted display device. 2. A head-mounted display device according to configuration 1. (Configuration 9) The control means controls the position of the second virtual object and the position where the user's movement is detected so as to have a predetermined positional relationship. 9. The head-mounted display device according to any one of configurations 1 to 8. (Configuration 10) The control means performs control so as to reproduce the haptic sensation of the second virtual object even if the second virtual object is away from a position where the movement of the user is detected. 9. The head-mounted display device according to any one of configurations 1 to 8. (Configuration 11) The control means performs control so as to weight the haptic intensity of the second virtual object so as to be stronger than the haptic intensities of the other virtual objects, and reproduces the haptic sensation of the second virtual object. 11. The head-mounted display device according to any one of configurations 1 to 10. (Configuration 12) The control means performs control so as to reproduce the haptic sensation of the second virtual object by weighting the haptic sensation of the second virtual object so that the intensity of the haptic sensation of the second virtual object increases as the distance between the user's gaze position and the second virtual object increases. 12. The head-mounted display device according to any one of configurations 1 to 11. (Configuration 13) The control means performs control so as to reproduce the haptic sensation of the second virtual object by weighting the haptic sensation of the second virtual object so that the intensity of the haptic sensation of the second virtual object increases as the distance between the first virtual object and the second virtual object increases. 12. The head-mounted display device according to any one of configurations 1 to 11. (Configuration 14) The control means performs control so that the haptic sensation of the second virtual object is reproduced by giving a higher priority to a process of reproducing the haptic sensation of the second virtual object as the distance between the gaze position of the user and the second virtual object becomes longer. 14. The head-mounted display device according to any one of configurations 1 to 13. (Configuration 15) The control means performs control so that the haptic sensation of the second virtual object is reproduced by giving a higher priority to a process of reproducing the haptic sensation of the second virtual object as the distance between the first virtual object and the second virtual object becomes longer. 14. The head-mounted display device according to any one of configurations 1 to 13. (method) acquiring user movement information; a step of controlling the reproduction of a haptic sensation of a second virtual object, which is a virtual operating body acting on a first virtual object designated by the user and is selected based on information of the user's movement; A method for controlling a head-mounted display device comprising: (program) A program for causing a computer to function as each of the means of the head-mounted display device according to any one of configurations 1 to 15. [Explanation of symbols]

[0125] 100: head mounted display device, 2: CPU, 22: gesture detection circuit, 24: first object detection circuit, 26: second object selection circuit, 27: haptic feedback control circuit

Claims

1. An acquisition means for acquiring information on a user's movement; a control means for controlling so as to reproduce a haptic sensation of a second virtual object, which is a virtual operating body acting on a first virtual object designated by the user and is selected based on information on the movement of the user; A head-mounted display device comprising:

2. The first virtual object is specified by the line of sight of the user.

2. The head-mounted display device according to claim 1 .

3. The device further includes a selection means for selecting the first virtual object, The first virtual object is selected based on a time period during which the user's gaze is directed at the first virtual object.

3. The head-mounted display device according to claim 2.

4. The device further includes a selection means for selecting the first virtual object, The first virtual object is selected based on the number of times the user's gaze is directed at the first virtual object.

3. The head-mounted display device according to claim 2.

5. The first virtual object is a virtual object to which the user's gaze is directed most frequently within a predetermined time period.

5. The head-mounted display device according to claim 4.

6. The first virtual object is a virtual object to which the user's gaze has been directed a predetermined number of times.

5. The head-mounted display device according to claim 4.

7. The first virtual object is a virtual object to which the user's gaze is most frequently directed when an instruction is received from the user.

5. The head-mounted display device according to claim 4.

8. The first virtual object is specified by the user's hand, the user's finger, or a controller for operating the head-mounted display device.

2. The head-mounted display device according to claim 1 .

9. The control means controls the position of the second virtual object and the position where the user's movement is detected so as to have a predetermined positional relationship.

2. The head-mounted display device according to claim 1 .

10. The control means performs control so as to reproduce the haptic sensation of the second virtual object even if the second virtual object is away from a position where the movement of the user is detected.

2. The head-mounted display device according to claim 1 .

11. The control means performs control so as to weight the haptic intensity of the second virtual object so as to be stronger than the haptic intensities of the other virtual objects, and reproduces the haptic sensation of the second virtual object.

2. The head-mounted display device according to claim 1 .

12. The control means performs control so as to reproduce the haptic sensation of the second virtual object by weighting the haptic sensation of the second virtual object so that the intensity of the haptic sensation of the second virtual object increases as the distance between the line of sight of the user and the second virtual object increases.

2. The head-mounted display device according to claim 1 .

13. The control means performs control so as to reproduce the haptic sensation of the second virtual object by weighting the haptic sensation of the second virtual object so that the intensity of the haptic sensation of the second virtual object increases as the distance between the first virtual object and the second virtual object increases.

2. The head-mounted display device according to claim 1 .

14. The control means controls the reproduction of the haptic sensation of the second virtual object by giving a higher priority to the process of reproducing the haptic sensation of the second virtual object as the distance between the line of sight of the user and the second virtual object becomes longer.

2. The head-mounted display device according to claim 1 .

15. The control means controls the reproduction of the haptic sensation of the second virtual object by giving a higher priority to the process of reproducing the haptic sensation of the second virtual object as the distance between the first virtual object and the second virtual object becomes longer.

2. The head-mounted display device according to claim 1 .

16. acquiring user movement information; a step of controlling so as to reproduce a haptic sensation of a second virtual object, which is a virtual operating body acting on a first virtual object designated by the user and is selected based on information of the user's movement; A method for controlling a head-mounted display device comprising:

17. A program for causing a computer to function as each of the means of the head-mounted display device according to any one of claims 1 to 15.

Images