Control device

Abstract

The control apparatus according to the present invention is a control apparatus configured to control a head-mounted display device of an optical see-through type, the control apparatus comprising: an acquisition unit configured to acquire line-of-sight information of a user; a detection unit configured to detect a real object being viewed by the user based on the line-of-sight information; and a control unit configured to control the head-mounted display apparatus in a case where the real object detected by the detection unit is a specific real object, and a processor configured to receive the detected real object from the head-mounted display device to display a virtual object in a direction near a direction from the head-mounted display device toward the detected real object and at an imaging distance corresponding to a distance from the head-mounted display device to the detected real object.

Description

Technical Field

[0001] The present invention relates to a control device for controlling an optical see-through type head-mounted display device. Background Art

[0002] With the spread of head-mounted display devices, technologies for fusing the real world (real space) and the virtual world (virtual space), such as augmented reality (AR) and mixed reality (MR), have advanced.

[0003] Patent Document 1 discloses a head-up display device that displays a virtual image in front of a driver and adjusts the imaging distance from the driver to the virtual image according to the driving state (e.g., speed) of the vehicle. Patent Document 2 discloses a see-through type head-mounted display device that sets the imaging distance of a projected image in the middle of the depth range of a work space.

[0004] [Citation List]

[0005] [Patent Documents]

[0006] [Patent Document 1] JP 2020-37351A

[0007] [Patent Document 2] JP 2010-139575 A Summary of the Invention

[0008] Problems to be Solved by the Invention

[0009] However, in the prior art, depending on the real object (real object) viewed by the user, the virtual object (virtual objects such as the above virtual image and projected image) appears blurred. For example, in the technology disclosed in Patent Document 1, depending on the distance from the driver to the real object the driver is gazing at, the virtual image appears blurred. In the technology disclosed in Patent Document 2, when the observer is looking at the middle of the depth range of the work space, the projected image can be seen with little blur. When the depth range is narrow, the observer can see the projected image with a small amount of blur at any position in the depth range. However, when the depth range is wide (e.g., in the case where the work space is outdoors), the projected image may exhibit a large blur. For example, in the case of viewing the frontmost real object and in the case of viewing the rearmost real object, the projected image appears blurred.

[0010] An object of the present invention is to provide a technique that can show a virtual object to a user with a small amount of blur even when the user is looking at any real object.

[0011] Solutions to the Problems

[0012] In a first aspect of the present invention, there is provided a control device configured to control an optical see-through head-mounted display device. The control device includes: an acquisition unit configured to acquire the gaze information of a user; a detection unit configured to detect a real object being viewed by the user based on the gaze information; and a control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object when the real object detected by the detection unit is a specific real object.

[0013] In a second aspect of the present invention, there is provided a control device configured to control an optical see-through head-mounted display device. The control device includes: an acquisition unit configured to acquire the gaze information of a user; a detection unit configured to detect a real object being viewed by the user based on the gaze information; a determination unit configured to determine the relevance between each virtual object among a plurality of displayable virtual objects and the real object detected by the detection unit; and a control unit configured to control the head-mounted display device to display the virtual object with a relevance higher than a threshold in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

[0014] In a third aspect of the present invention, there is provided a control device configured to control an optical see-through head-mounted display device. The control device includes: a control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction towards the real object being viewed by the user. The head-mounted display device includes an adjustment lens for adjusting the imaging distance of the virtual object. The control unit further controls the head-mounted display device to change the imaging distance by changing the position of the adjustment lens relative to the head-mounted display device, and the position of the adjustment lens relative to the head-mounted display device is different between the case where the user is viewing a first real object and the case where the user is viewing a second real object.

[0015] In a fourth aspect of the present invention, there is provided a control method for controlling an optical see-through head-mounted display device. The control method includes the following steps: obtaining gaze information of a user; detecting a real object being viewed by the user based on the gaze information; and when the real object detected in the detecting step is a specific real object, controlling the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

[0016] In a fifth aspect of the present invention, there is provided a control method for controlling an optical see-through head-mounted display device. The control method includes the following steps: obtaining gaze information of a user; detecting a real object being viewed by the user based on the gaze information; determining the relevance between each virtual object among a plurality of displayable virtual objects and the detected real object; and controlling the head-mounted display device to display the virtual object having a relevance higher than a threshold value in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

[0017] In a sixth aspect of the present invention, there is provided a program that causes a computer to function as each unit of the above control device. In a seventh aspect of the present invention, there is provided a computer-readable medium that stores a program that causes a computer to function as each unit of the above control device.

[0018] Effects of the Invention

[0019] According to the present invention, even when a user is looking at any real object, a virtual object can be shown to the user with a small amount of blur. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A and Figure 1 B of [the figure] are external views of the display device, and Figure 1 C of [the figure] is a schematic diagram showing the optical system of the display device.

[0021] Figure 2 is a block diagram of the display device.

[0022] Figure 3 is a diagram describing the principle of the gaze detection method.

[0023] Figure 4 A of [the figure] is a diagram showing an eye image, and Figure 4 B of [the figure] is a diagram showing the brightness distribution of the eye image.

[0024] Figure 5 It is a flowchart of gaze detection processing.

[0025] Figure 6 It is a flowchart of overall processing according to the first embodiment.

[0026] Figure 7 It is a table showing the correspondence between distance and lens driving amount.

[0027] Figure 8 A of Figure 8 B of

[0028] Figure 9 It is a flowchart of overall processing according to the second embodiment.

[0029] Figure 10 It is a schematic diagram showing a real object and a virtual object.

[0030] Figure 11 It is a modified example of the flowchart of overall processing according to the second embodiment. Detailed implementation mode

[0031] <<First Embodiment>>

[0032] The first embodiment of the present invention is described. In the first embodiment, an example in which the present invention is applied to an optical see-through type head-mounted display device (HMD, head-mounted display) will be described. The optical see-through type head display device is, for example, smart glasses (augmented reality (AR) glasses). The optical see-through type head display device has a lens (optical member) similar to that of ordinary glasses, for example, and projects a graphic (for example, a virtual object) onto the lens. A user wearing the optical see-through type head-mounted display device can see the outside world (real space) or a graphic projected (displayed) by the head-mounted display device through the optical member (lens). In the first embodiment, an example in which the present invention is applied to a head-mounted display device in which a user views a graphic with both eyes will be described, but the present invention is also applicable to a head-mounted display device in which a user views a graphic with one eye.

[0033] The present invention is applicable to a control device for controlling an optical see-through type head-mounted display device, and the control device can be provided in the head-mounted display device or can be provided in an electronic device separated from the head-mounted display device. For example, the present invention is also applicable to a controller or a personal computer (PC) connected to the optical see-through type head-mounted display device.

[0034] <Configuration description>

[0035] Figure 1 A of Figure 1 B of Figure 1A is a front perspective view, and Figure 1 B is a rear perspective view. The display device 100 is an optical see-through type head-mounted display device. The display device 100 can separately detect the line of sight of the user's right eye and the line of sight of the user's left eye when the display device 100 is worn on the head. Hereinafter, the user who wears the display device 100 on the head will be simply referred to as the user.

[0036] The lens unit 10 faces the user's eyes, and the user can visually recognize the outside world through the lens unit 10. The display device 11 displays a graphic (for example, a virtual image of a virtual object) for both eyes (both the right eye and the left eye) of the user under the control (display control) from the CPU 2 which will be described later. For example, the display device 11 projects the graphic onto the lens unit 10 through the optical system described later with reference to Figure 1 C. The display device 11 displays, for example, a graphical user interface (GUI, such as a button or an icon) as a graphic. The user can view the displayed graphic (virtual object) as if the displayed graphic exists in the outside world. When displaying the graphic, the depth position (position in the depth direction (direction away from the user)) of the graphic viewed by the user can be adjusted by adjusting the positional relationship (parallax) in the left-right direction between the display position relative to the right eye and the display position relative to the left eye.

[0037] The light source drive circuit 12 drives the light sources 13a and 13b. The light sources 13a and 13b are each a light source that irradiates the user's eyes, and are, for example, infrared light-emitting diodes that emit infrared light to which the user is insensitive. A part of the light emitted from the light sources 13a and 13b and reflected by the user's eyes is focused on the eye imaging element 17 by the light receiving lens 16. The eye imaging element 17 is an imaging sensor (imaging element) that images the user's eyes. The imaging sensor is, for example, a charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

[0038] The lens unit 10, the display device 11, the light source drive circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye imaging element 17 are provided for each of the right eye and the left eye. The light source drive circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye imaging element 17 can be used to obtain the user's line-of-sight information. The line-of-sight information is information related to the line of sight, and indicates at least one of, for example, the viewpoint, the line-of-sight direction (direction of the line of sight), and the convergence angle (angle formed by the line of sight of the right eye and the line of sight of the left eye). The viewpoint can also be considered as the position where the line of sight is focused, the position where the user is looking, or the line-of-sight position. Details of the method for obtaining the line-of-sight information will be described later.

[0039] External imaging elements 18 are also provided for the right eye and the left eye respectively. The external imaging element 18 is an imaging sensor (imaging element) that images the external scene in the direction the user is facing. The imaging sensor is, for example, a CCD sensor or a CMOS sensor. For example, the external imaging element 18 is used to detect the real object that the user is viewing. Various known techniques can be used to detect the real object. By using two external imaging elements 18 (a stereo camera), the distance from the user (display device 100) to the real object can also be detected (measured). Various known techniques can be used for distance measurement, and sensors different from the two external imaging elements 18 (stereo camera) can be used. For example, a light detection and ranging (LiDAR) sensor using a laser or a time-of-flight (TOF) sensor using light from a light-emitting diode (LED) can be utilized. A single imaging sensor can be used to measure the distance through phase difference AF (autofocus).

[0040] Figure 1 C of FIG. is a schematic diagram showing the optical system of the display device 100. The lens unit 10 includes a beam splitter 21, an eyepiece lens 22, and an eyepiece unit 23. The beam splitter 21 includes a half mirror or a prism, transmits light from the outside world toward the eyepiece lens 22, and reflects light from the display device 11 toward the eyepiece lens 22. The light from the outside world and the light from the display device 11 enter the user's eye (right eye or left eye) through the eyepiece lens 22 and the eyepiece unit 23. This allows the user to view both the outside world and the graphics (virtual objects) displayed by the display device 11.

[0041] An adjustment lens 24 is provided between the display device 11 and the beam splitter 21. By adjusting the position of the adjustment lens 24 between the display device 11 and the beam splitter 21, the imaging distance of the graphics displayed by the display device 11 can be adjusted. This is a process of adjusting the depth position at which the graphics are in focus (the depth position at which the graphics can be viewed with substantially no blur), and is different from the process of adjusting the depth position of the graphics themselves. For example, in the above method for adjusting parallax, the depth position of the graphics themselves is adjusted, and the stereoscopic effect of the graphics is adjusted, but the blur of the graphics is not adjusted. In the method of adjusting the imaging distance, the stereoscopic effect of the graphics is not adjusted, but the blur of the graphics is adjusted.

[0042] Figure 2 FIG. is a block diagram showing the electrical configuration of the display device 100. The CPU 2 is the central processing unit of the microcomputer incorporated in the display device 100, and overall controls the display device 100. The display device 11, the light source drive circuit 12, the digital interface circuit 15, the external imaging element 18, and the memory unit 3 are connected to the CPU 2.

[0043] The memory unit 3 has a storage function for the video signal from the eye imaging element 17 and a storage function for the line-of-sight correction parameters (parameters for correcting individual differences in the line of sight), which will be described later.

[0044] In a state where the optical image of the eye is formed on the eye imaging element 17, the digital interface circuit 15 performs A / D conversion on the output of the eye imaging element 17 (the eye image obtained by imaging the eye) and sends the result to the CPU 2. The CPU 2 extracts the feature points required for line-of-sight detection from the eye image according to a predetermined algorithm, which will be described later, and detects the user's line of sight from the positions of the feature points. The CPU 2 can display information (virtual object) related to the real object viewed by the user on the display device 11 according to the detection result of the GUI fixation.

[0045] <Description of line-of-sight detection processing>

[0046] Will be referred to Figure 3 、 Figure 4 of A, Figure 4 of B, and Figure 5 describe the line-of-sight detection processing (line-of-sight detection method). Both the line of sight of the right eye and the line of sight of the left eye are detected by the following line-of-sight detection method. Figure 3 is a diagram showing the principle of the line-of-sight detection method and is a schematic diagram of the optical system for detecting the line of sight. As Figure 3 shown, the light sources 13a and 13b are arranged substantially symmetrically with respect to the optical axis of the light receiving lens 16 and irradiate the user's eyeball 140. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 140 is focused by the light receiving lens 16 on the eye imaging element 17. Figure 4 of A is a schematic diagram of the eye image (the optical image of the eye projected on the eye imaging element 17) captured by the eye imaging element 17, and Figure 4 of B is a diagram showing the output intensity of the eye imaging element 17. Figure 5 is a flowchart of the line-of-sight detection processing.

[0047] When Figure 5 the line-of-sight detection processing in starts, in step S1, the CPU 2 drives the light sources 13a and 13b using the light source drive circuit 12 to emit infrared light toward the user's eyeball 140. The optical image of the user's eye irradiated by the infrared light is formed on the eye imaging element 17 through the light receiving lens 16, and is photoelectrically converted by the eye imaging element 17. Thus, an electrical signal of the eye image that can be processed is obtained.

[0048] In step S2, the CPU 2 acquires the eye image (image data and image signal) from the eye imaging element 17 via the digital interface circuit 15.

[0049] In step S3, the CPU 2 detects the coordinates of points corresponding to the corneal reflection images Pd and Pe of the light sources 13a and 13b and the pupil center c from the eye image obtained in step S2.

[0050] The infrared light emitted from the light sources 13a and 13b irradiates the cornea 142 of the user's eyeball 140. At this time, the corneal reflection images Pd and Pe formed by a part of the infrared light reflected from the surface of the cornea 142 are focused by the light receiving lens 16 and formed on the eye imaging element 17, thereby forming the corneal reflection images Pd' and Pe' in the eye image. Similarly, the light beams from the ends a and b of the pupil 141 are also imaged on the eye imaging element 17, thereby forming the pupil end images a' and b' in the eye image.

[0051] Figure 4 B of shows Figure 4 the luminance information (luminance distribution) of the region α in the eye image shown in A of. Figure 4 B of shows the luminance distribution in the X-axis direction when the horizontal direction of the eye image is the X-axis direction and the vertical direction is the Y-axis direction. In the first embodiment, the coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil end images a' and b' in the X-axis direction are Xa and Xb. As Figure 4 shown in B of, extremely high levels of luminance are obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from the coordinate Xa to the coordinate Xb corresponding to the region of the pupil 141 (the region of the pupil image obtained by forming the light beam from the pupil 141 on the eye imaging element 17), extremely low levels of luminance are obtained except at the coordinates Xd and Xe. Then, in the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by imaging the light beam from the iris 143), intermediate luminance between the above two luminances is obtained. For example, in the region where the X coordinate (coordinate in the X-axis direction) is greater than the coordinate Xa and the region where the X coordinate is less than the coordinate Xb, intermediate luminance between the two luminances is obtained.

[0052] The X coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X coordinates Xa and Xb of the pupil end images a' and b' can be obtained from Figure 4It is obtained from the luminance distribution shown in B. For example, coordinates with extremely high luminance can be obtained as the coordinates of the corneal reflection images Pd' and Pe', and coordinates with extremely low luminance can be obtained as the coordinates of the pupil edge images a' and b'. When the rotation angle θx of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 16 is small, the coordinates Xc of the pupil center image c' (the center of the pupil image) obtained by imaging the light beam from the pupil center c on the eye imaging element 17 can be expressed as Xc≈(Xa + Xb) / 2. That is, the coordinates Xc of the pupil center image c' can be calculated based on the X coordinates Xa and Xb of the pupil edge images a' and b'. In this way, the coordinates of the corneal reflection images Pd' and Pe' and the coordinates of the pupil center image c' can be estimated.

[0053] In step S4, the CPU 2 calculates the imaging magnification β of the eye image. The imaging magnification β is the magnification determined by the position of the eyeball 140 relative to the light receiving lens 16, and can be calculated using a function of the interval (Xd - Xe) between the corneal reflection images Pd' and Pe'.

[0054] In step S5, the CPU 2 calculates the rotation angle of the optical axis of the eyeball 140 relative to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the corneal reflection image Pd and the corneal reflection image Pe is approximately the same as the X coordinate of the curvature center O of the cornea 142. Therefore, when the standard distance from the curvature center O of the cornea 142 to the center c of the pupil 141 is defined as Oc, the rotation angle θx of the eyeball 140 in the Z - X plane (the plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of the eyeball 140 in the Z - Y plane (the plane perpendicular to the X axis) can also be calculated by a method similar to the method for calculating the rotation angle θx.

[0055] β×Oc×SINθx≈{(Xd + Xe) / 2}-Xc···(Formula 1)

[0056] In step S6, the CPU 2 estimates the user's viewing point on the lens unit 10 using the rotation angles θx and θy calculated in step S5. Assuming that the coordinates (Hx, Hy) of the viewing point are the coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewing point can be calculated by the following formulas 2 and 3.

[0057] Hx = m × (Ax × θx + Bx) ··· (Formula 2)

[0058] Hy = m × (Ay × θy + By) ··· (Formula 3)

[0059] The parameter m in Formulas 2 and 3 is a constant determined by the configuration of the optical system for performing the line-of-sight detection process, and is a conversion coefficient for converting the rotation angles θx and θy into coordinates corresponding to the pupil center c on the lens unit 10. It is assumed that the parameter m is predetermined and stored in the memory unit 3. The parameters Ax, Bx, Ay, and By are line-of-sight correction parameters for correcting individual differences in the line of sight, and are obtained by performing calibration of the line-of-sight detection. It is assumed that before the start of the line-of-sight detection process, the line-of-sight correction parameters Ax, Bx, Ay, and By are stored in the memory unit 3.

[0060] In step S7, the CPU 2 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 3 and ends the line-of-sight detection process.

[0061] Note that the line-of-sight detection method is not limited to the above method. For example, any method can be used as long as it is a method for obtaining line-of-sight information from an eye image. The line of sight can be detected by a method that does not use an eye image, such as a method of detecting the eye potential without using an eye image and detecting the line of sight based on the eye potential. As the final line-of-sight information, information indicating the line-of-sight direction (direction of the line of sight) can be obtained instead of information indicating the viewpoint. For example, the process can be performed until the rotation angle (Ax×θx + Bx or Ay×θy + By) is obtained without obtaining the coordinates (Hx, Hy) of the viewpoint.

[0062] <Description of the overall process>

[0063] Figure 6 is a flowchart of the overall process according to the first embodiment. For example, when the user wears the display device 100 and starts the display device 100, the overall process in Figure 6 starts. The CPU 2 repeats Figure 6 all the processes.

[0064] In step S101, the CPU 2 detects a real object from the image captured by the external imaging element 18 (an image of the outside world corresponding to the user's field of view). For example, the CPU 2 uses a trained model (a training model trained by deep learning) to detect a real object of a predetermined type. Note that, as described above, the method of detecting a real object is not particularly limited, and various known techniques can be used. For example, the CPU 2 can detect a real object by performing template matching.

[0065] In step S102, the CPU 2 detects (measures) the distance from the user (display device 100) to the real object detected in step S101 based on the difference information between the two images captured by the two external imaging elements 18. Note that, as described above, the method for measuring the distance is not particularly limited, and various known techniques can be used.

[0066] In step S103, the CPU 2 obtains the user's gaze information by performing Figure 5 the gaze detection process.

[0067] In step S104, the CPU 2 detects (selects) the real object (gaze object) that the user is viewing from the one or more real objects detected in step S101 based on the gaze information obtained in step S103. Note that in the case where there are multiple real objects in the user's line of sight, the CPU 2 can calculate the priority of each real object among the multiple real objects. For example, as the frequency of the viewpoint entering the area of the real object increases, the CPU 2 can calculate a higher priority, or as the time the viewpoint exists in the area of the real object increases, the CPU 2 can calculate a higher priority. Then, the CPU 2 can detect (select) the real object with the highest priority as the gaze object.

[0068] In step S105, the CPU 2 determines whether the gaze object (the real object selected in step S104) is a specific real object. The specific real object will be described later. In the case where the CPU 2 determines that the gaze object is a specific real object, the process proceeds to step S106, and in the case where the CPU 2 determines that the gaze object is not a specific real object, Figure 6 the overall process ends. The determination method is not particularly limited, and various known techniques can be used. For example, the CPU 2 can determine whether the gaze object is a specific real object by using a trained model (a trained model trained by deep learning). The CPU 2 can detect the real object by performing template matching. When performing the process of step S101, the CPU 2 can obtain information related to the type of the detected real object. Then, the CPU 2 can determine whether the gaze object is a specific real object based on the information obtained in step S101.

[0069] In step S106, the CPU 2 drives the adjustment lens 24 so that the virtual object can be displayed at an imaging distance corresponding to the distance from the user (display device 100) to the gaze object (the distance detected in step S102). For example, Figure 7 the table shown is pre-stored in the memory unit 3. Figure 7shows the correspondence between the distance from the user (display device 100) to the real object and the driving amount (position adjustment amount) of the adjustment lens 24 required to achieve the imaging distance (target imaging distance) corresponding to that distance. The CPU 2 reads from Figure 7 the table of and reads the driving amount corresponding to the distance from the user (display device 100) to the fixation object, and drives the adjustment lens 24 with the read driving amount. Instead of this table, a function indicating the correspondence between the distance to the real object and the driving amount of the adjustment lens 24 can be used. The target imaging distance is, for example, the imaging distance at which the user can see the virtual object with good visibility (with a small amount of blur) while viewing the fixation object. The target imaging distance is preferably substantially the same as the distance from the user (display device 100) to the fixation object.

[0070] In step S107, the CPU 2 controls the display device 11 to display the virtual object in a direction near the direction from the user (display device 100) toward the fixation object. The virtual object is displayed in a direction near the direction from the user (display device 100) toward the fixation object and at the imaging distance adjusted in step S106. The timing of hiding the virtual object is, for example, the timing when the user's line of sight deviates from the fixation object (the real object selected in step S104). In a state where the user cannot see the real object selected in step S104, the CPU 2 can hide the virtual object at the timing when a predetermined time (for example, 3 to 5 seconds) has elapsed.

[0071] Assume that the first real object and the second real object are at different distances from the user (display device 100). According to Figure 6 the overall processing, the position of the adjustment lens 24 relative to the display device 100 is different between the case where the user is viewing the first real object and the case where the user is viewing the second real object.

[0072] <Description of a specific real object>

[0073] The specific real object used for the determination in step S105 is, for example, a real object that the user frequently views or touches.

[0074] The specific real object may include a real object having a display unit (display surface) that displays various images or various types of information. Examples of real objects including a display unit are a personal computer (PC), a television, a smart phone, a tablet terminal, or an operation panel of a machine tool installed in a factory.

[0075] The specific real object may include a real object having an operation unit that the user can operate. For example, real objects including an operation unit are a lighting switch in a room, a keyboard of a PC, or a switch of a machine tool installed in a factory.

[0076] <Description of virtual object>

[0077] The virtual object displayed in step S107 is based on information related to, for example, the real object that the user is viewing. The method of displaying (generating method) the virtual object will be described later. For example, when the user operates the operation panel or switch of a machine tool, a manual of the operation method or a screen indicating the operation state of the machine tool can be displayed as a virtual object. When the user confirms a failure of a machine tool (factory production line), a design drawing (wiring diagram) can be displayed as a virtual object. When the user views video content on a television, PC, smart phone, or tablet terminal, a program guide can be displayed as a virtual object. An introduction screen of recommended video content (e.g., popular video content or video content related to the video content being viewed) can be displayed as a virtual object. When the user operates a PC, smart phone, or tablet terminal (e.g., when working or using a website or application for communication), an email screen can be displayed as a virtual object. An address book or schedule screen can be displayed as a virtual object. In a case where the user needs to confirm multiple documents during work, a part of the multiple documents can be displayed on the display unit, and the remaining documents can be displayed as virtual objects.

[0078] <Description of virtual object display method (generating method)>

[0079] For example, the CPU 2 acquires information related to a specific real object that the user is viewing, and based on the acquired information, displays a virtual object on the display device 11. For multiple real objects, information related to the real objects can be pre-stored in the memory unit 3, and the CPU 2 can acquire information related to a specific real object that the user is viewing from the memory unit 3. The display device 100 can have a communication function (communication interface) for communicating with an external device. When the user is viewing a specific external device with a communication function (e.g., a PC, television, smart phone, tablet terminal, or operation panel), the CPU 2 can acquire information related to the external device by communicating with the external device. This communication can be short-range communication or two-way communication.

[0080] For example, in the case of short-range communication, the external device periodically sends information about itself. The CPU 2 generates a virtual object based on the information received from the specific external device that the user is viewing, and displays the virtual object on the display device 11. When information is received from multiple external devices, the CPU 2 selects information related to the specific external device that the user is viewing from the multiple pieces of information, and generates a virtual object based on the selected information.

[0081] For example, in the case of two-way communication, the CPU 2 requests a specific external device that the user is viewing to send information. The external device that has received the request sends information related to the external device to the display device 100. Then, the CPU 2 generates a virtual object based on the information received from the specific external device that the user is viewing, and displays the virtual object on the display device 11.

[0082] The information sent from the specific external device that the user is viewing may or may not be the information indicated by the displayed virtual object (for example, the image data of the virtual object). For example, the information sent from the specific external device that the user is viewing may be information indicating the display screen of the external device. In this case, the CPU 2 determines the information to be displayed based on the information received from the specific external device that the user is viewing, and generates a virtual object indicating the determined information. When generating a virtual object, the CPU 2 may obtain necessary information (information to be displayed) from a server. As described above, for example, a manual, a screen indicating the operating state of a machine tool, a plan, an email screen, an address book, a schedule screen, or a document is displayed as a virtual object.

[0083] <Description of environment consideration control>

[0084] The display device 100 may have an environment detection function for detecting the environment around the user (display device 100). For example, the CPU 2 may detect the environment (scene) around the user based on the image captured by the external imaging element 18. Then, the CPU 2 may determine (change) a specific real object to be used for the determination in step S105 according to the environment around the user (detected environment). For example, when it is determined that the user is in an office, the CPU 2 determines a real object having a display unit as a specific real object, and when it is determined that the user is in a factory, the CPU 2 determines a real object having an operation unit as a specific real object.

[0085] <Summary>

[0086] As described above, according to the first embodiment, the display direction and imaging distance of the virtual object are adjusted based on the position of the real object viewed by the user. As a result, even when the user looks at any real object, the virtual object can be shown to the user with a small amount of blur. Even when the virtual object is displayed in a direction near the direction where the real object viewed by the user exists, when the imaging distance of the virtual object deviates greatly from the distance to the real object, the virtual object looks very blurry. Therefore, when the imaging distance of the virtual object is constant (fixed) without performing Figure 6 the overall processing, the virtual object may look very blurry (as shown in A of Figure 8 ). When performing Figure 6When performing overall processing, the imaging distance of the virtual object is adjusted to the imaging distance corresponding to the real object being viewed by the user, so that the virtual object can be shown to the user with a small amount of blur (as shown in Figure 8 B of Figure 8 A of Figure 8 and B of

[0087]

[0087]

[0088] <<Second Embodiment>>

[0089] The second embodiment of the present invention is described. Hereinafter, descriptions of the same points as those in the first embodiment (for example, the same configurations and processes as those in the first embodiment) will be omitted, and the points different from those in the first embodiment will be described. In the first embodiment, a virtual object is displayed in response to the user viewing a specific real object, but in the second embodiment, one or more virtual objects are displayed in advance.

[0090] <Description of Overall Processing>

[0091] Figure 9 is a flowchart of the overall processing according to the second embodiment. For example, when the user wears the display device 100 and starts the display device 100, the overall processing of Figure 9 starts. The CPU 2 repeats the overall processing of Figure 9 Before starting the overall processing of Figure 9 the CPU 2 displays a plurality of virtual objects on the display device 11.

[0092] Steps S201 to S204 are the same as steps S101 to S104 of the first embodiment ( Figure 3 ).

[0093] In step S205, the CPU 2 determines the relevance between each of the plurality of displayed virtual objects and the fixation object (the real object selected in step S204 and viewed by the user). A method for determining the relevance will be described later.

[0094] In step S206, the CPU 2 selects, from among the plurality of displayed virtual objects, the virtual objects whose relevance determined in step S205 is higher than the threshold Th1. Then, the CPU 2 drives the adjustment lens 24 to change the imaging distance of the selected virtual object to an imaging distance corresponding to the distance from the user (display device 100) to the fixation object (the distance detected in step S202).

[0095] In step S207, the CPU 2 controls the display device 11 to change the display direction of the virtual object selected in step S206 to a direction near the direction from the user toward the fixation object. The virtual object whose relevance is higher than the threshold Th1 is displayed at the imaging distance adjusted in step S206 in the direction near the direction from the user toward the fixation object.

[0096] <Method for determining relevance>

[0097] Before starting Figure 9 the overall processing, the CPU 2 may display, on the display device 11, virtual objects corresponding to the information displayed on the display unit of a real object (e.g., a PC, a television, a smart phone, a tablet terminal, or an operation panel). For example, when the real object displays a plurality of pieces of information on the display unit, the CPU 2 may display, on the display device 11, a plurality of virtual objects corresponding to the plurality of pieces of information. Then, the CPU 2 may set the relevance higher than the threshold Th1 as the relevance of the virtual object corresponding to the information being viewed by the user (the information displayed on the display unit). The CPU 2 may set the relevance lower than the threshold Th1 as the relevance of the remaining virtual objects.

[0098] Before starting Figure 9 the overall processing, the CPU 2 may display, on the display device 11, a plurality of virtual objects corresponding to a plurality of real objects, respectively. Then, the CPU 2 may set the relevance higher than the threshold Th1 as the relevance of the virtual object corresponding to the real object being viewed by the user. The CPU 2 may set the relevance lower than the threshold Th1 as the relevance of the remaining virtual objects. The real object and the virtual object may not be in one-to-one correspondence. A plurality of virtual objects may be associated with one real object. For example, a plurality of virtual objects corresponding to a plurality of pieces of information displayed on the display unit of a real object may be regarded as virtual objects corresponding to the real object.

[0099] For each of the plurality of virtual objects, a default imaging distance may be determined in advance (before starting Figure 9imaging distance before overall processing). For example, the imaging distance corresponding to the distance from the user (display device 100) to the real object can be predetermined as the default imaging distance of the virtual object corresponding to the real object. Then, as the default imaging distance of the virtual object gets closer to the distance from the user to the real object the user is viewing, the CPU 2 can determine a higher value as the relevance of the virtual object.

[0100] For example, as Figure 10 shown, when the user is viewing department store 41a, a relevance higher than threshold Th1 is set for virtual object 41b corresponding to department store 41a. The content of virtual object 42b corresponding to cinema 42a is not similar to the content of virtual object 41b, but since cinema 42a is close to department store 41a, the possibility that virtual object 42b is useful to the user is high. Therefore, a relevance higher than threshold Th1 is also set for virtual object 42b. Then, the content of virtual object 43b corresponding to restaurant 43a is similar to the content of virtual object 41b, but since restaurant 43a is far from department store 41a, the possibility that virtual object 43b is useful to the user is low. Therefore, a relevance lower than threshold Th1 is also set for virtual object 43b.

[0101] The CPU 2 can detect, based on the line-of-sight information, the virtual object at which the user repeatedly moves the line of sight from the real object, and set a relevance higher than threshold Th1 for the virtual object. The CPU 2 can set the relevance of the remaining virtual objects to be lower than threshold Th1.

[0102] As described above, according to the second embodiment, the display direction and imaging distance of the virtual object whose relevance to the real object the user is viewing is higher than threshold Th1 are adjusted. As a result, the virtual object (information) useful to the user can be shown to the user with a small amount of blur.

[0103] Note that an example has been described in which a plurality of virtual objects are displayed before the start of the overall processing in Figure 9 , but the present invention is not limited thereto. At the start of the overall processing in Figure 9 , it is sufficient that there are a plurality of virtual objects that can be displayed, and a plurality of virtual objects may not be displayed.

[0104] In addition, the CPU 2 can display the virtual object whose relevance is lower than threshold Th2 on the display device 11 with reduced saliency. By doing so, it is possible to prevent the user from being distracted by the virtual object (information) that is useless to the user. Threshold Th2 (a threshold related to whether the relevance is low) can be equal to or less than threshold Th1 (equal to or less than a threshold related to whether the relevance is high), and can be equal to threshold Th1. Saliency can be considered as the degree of prominence or visibility.

[0105] The CPU 2 can perform Figure 11 overall processing. For example, when the user wears the display device 100 and activates the display device 100, Figure 11 overall processing starts. The CPU 2 repeats Figure 11 overall processing. Before starting Figure 11 overall processing, the CPU 2 displays a plurality of virtual objects on the display device 11.

[0106] Steps S301 to S307 are the same as Figure 9 steps S201 to S207 in

[0107] Note that the above embodiments (including modified examples) are merely examples, and configurations obtained by appropriately deforming or changing the configurations of the above embodiments within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the above embodiments are also included in the present invention.

[0108] <<Other Embodiments>>

[0109] The present invention can also be implemented by supplying a program for realizing one or more functions of the above embodiments to a system or device via a network or a storage medium and causing one or more processors in a computer of the system or device to read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for realizing one or more functions.

[0110] The disclosure of the present embodiment includes the following configurations, methods, programs, and media.

[0111] (Configuration 1)

[0112] A control device configured to control an optical see-through head-mounted display device, the control device comprising:

[0113] An acquisition unit configured to acquire gaze information of a user;

[0114] A detection unit configured to detect a real object being viewed by the user based on the gaze information; and

[0115] A control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object when the real object detected by the detection unit is a specific real object.

[0116] (Configuration 2)

[0117] The control device according to Configuration 1, wherein

[0118] The imaging distance is substantially consistent with the distance from the head-mounted display device to the detected real object.

[0119] (Configuration 3)

[0120] The control device according to Configuration 1 or 2, wherein

[0121] The specific real object includes a real object having a display unit.

[0122] (Configuration 4)

[0123] The control device according to any one of Configurations 1 to 3, wherein

[0124] The specific real object includes a real object having an operation unit.

[0125] (Configuration 5)

[0126] The control device according to any one of Configurations 1 to 4, further comprising:

[0127] A second acquisition unit configured to acquire information related to the specific real object,

[0128] wherein the control unit controls the head-mounted display device to display a virtual object based on the information acquired by the second acquisition unit.

[0129] (Configuration 6)

[0130] The control device according to Configuration 5, wherein

[0131] The specific real object includes a specific external device having a communication function, and

[0132] when the detected real object is the specific external device, the second acquisition unit acquires the information by performing two-way communication with the detected real object.

[0133] (Configuration 7)

[0134] The control device according to any one of Configurations 1 to 6 further includes:

[0135] An environment detection unit configured to detect the environment around the user,

[0136] wherein the control unit determines a specific virtual object based on the environment detected by the environment detection unit.

[0137] (Configuration 8)

[0138] A control device configured to control an optical see-through head-mounted display device, the control device including:

[0139] An acquisition unit configured to acquire the user's gaze information;

[0140] A detection unit configured to detect a real object that the user is viewing based on the gaze information;

[0141] A determination unit configured to determine the relevance between each virtual object among a plurality of displayable virtual objects and the real object detected by the detection unit; and

[0142] A control unit configured to control the head-mounted display device to display, in a direction near the direction from the head-mounted display device toward the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object, the virtual object having a relevance higher than a threshold.

[0143] (Configuration 9)

[0144] The control device according to Configuration 8, wherein

[0145] the control unit:

[0146] controls to display the plurality of virtual objects; and

[0147] controls the head-mounted display device to change the display direction and imaging distance of the virtual object having a relevance higher than the threshold to a direction near the direction from the head-mounted display device toward the detected real object and an imaging distance corresponding to the distance from the head-mounted display device to the detected real object, respectively.

[0148] (Configuration 10)

[0149] The control device according to Configuration 8 or 9, wherein

[0150] for each virtual object among the plurality of virtual objects, a default imaging distance is determined in advance, and

[0151] As the default imaging distance of the virtual object gets closer to the distance from the head-mounted display device to the detected real object, the determining unit determines a higher value as the relevance of the virtual object.

[0152] (Configuration 11)

[0153] The control device according to any one of Configurations 8 to 10, wherein,

[0154] The control unit controls the head-mounted display device to display the virtual object with a relevance lower than a second threshold with reduced saliency, wherein the second threshold is equal to or less than the threshold.

[0155] (Configuration 12)

[0156] The control device according to Configuration 11, wherein,

[0157] The control unit controls the head-mounted display device to hide the virtual object with a relevance lower than the second threshold.

[0158] (Configuration 13)

[0159] The control device according to any one of Configurations 8 to 12, wherein,

[0160] The control unit controls to display the plurality of virtual objects,

[0161] The control device further includes a second detection unit configured to detect, based on the line-of-sight information, the virtual object to which the user repeatedly moves the line of sight from the detected real object, and

[0162] The control unit controls the head-mounted display device to change the display direction and imaging distance of the virtual object detected by the second detection unit to a direction near the direction from the head-mounted display device toward the detected real object and an imaging distance corresponding to the distance from the head-mounted display device to the detected real object, respectively.

[0163] (Configuration 14)

[0164] A control device configured to control an optical see-through type head-mounted display device, the control device includes:

[0165] A control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device toward the real object being viewed by the user,

[0166] wherein the head-mounted display device includes an adjustment lens for adjusting the imaging distance of the virtual object,

[0167] The control unit also controls the head-mounted display device to change the imaging distance by changing the position of the adjustment lens relative to the head-mounted display device, and

[0168] the position of the adjustment lens relative to the head-mounted display device is different between the case where the user is viewing a first real object and the case where the user is viewing a second real object.

[0169] (Method 1)

[0170] A control method for controlling an optical see-through type head-mounted display device, the control method comprising the steps of:

[0171] Obtaining the gaze information of the user;

[0172] Detecting a real object being viewed by the user based on the gaze information; and

[0173] When the real object detected in the detecting step is a specific real object, controlling the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

[0174] (Method 2)

[0175] A control method for controlling an optical see-through type head-mounted display device, the control method comprising the steps of:

[0176] Obtaining the gaze information of the user;

[0177] Detecting a real object being viewed by the user based on the gaze information;

[0178] Determining the relevance between each virtual object among a plurality of displayable virtual objects and the detected real object; and

[0179] Controlling the head-mounted display device to display the virtual object having a relevance higher than a threshold in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

[0180] (Program)

[0181] A program for causing a computer to function as each unit of the control device according to any one of Configurations 1 to 14.

[0182] (Medium)

[0183] A computer-readable medium stores a program for causing a computer to function as each unit of the control device according to any one of Configurations 1 to 14.

[0184] The present invention is not limited to the above embodiments, and various modifications and changes can be made without departing from the spirit and scope of the present invention. Therefore, the appended claims are provided to disclose the scope of the present invention.

[0185] This application claims priority based on Japanese Patent Application No. 2022-166091 filed on October 17, 2022, the entire disclosure of which is incorporated herein by reference.

[0186] [List of Reference Numerals]

[0187] 100 Display device

[0188] 2 CPU

Claims

1. A control device configured to control an optical see-through head-mounted display device, the control device comprises: an acquisition unit configured to acquire the gaze information of a user; a detection unit configured to detect a real object that the user is viewing based on the gaze information; and a control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object when the real object detected by the detection unit is a specific real object.

2. The control device according to claim 1, wherein the imaging distance is substantially the same as the distance from the head-mounted display device to the detected real object.

3. The control device according to claim 1, wherein the specific real object includes a real object having a display unit.

4. The control device according to claim 1, wherein the specific real object includes a real object having an operation unit.

5. The control device according to claim 1, further comprises: a second acquisition unit configured to acquire information related to the specific real object, wherein the control unit controls the head-mounted display device to display a virtual object based on the information acquired by the second acquisition unit.

6. The control device according to claim 5, wherein the specific real object includes a specific external device having a communication function, and when the detected real object is the specific external device, the second acquisition unit acquires the information by performing two-way communication with the detected real object.

7. The control device according to claim 1, further comprises: an environment detection unit configured to detect the environment around the user, wherein the control unit determines a specific virtual object according to the environment detected by the environment detection unit.

8. A control device configured to control an optical see-through head-mounted display device, the control device comprises: an acquisition unit configured to acquire the gaze information of a user; a detection unit configured to detect a real object that the user is viewing based on the gaze information; a determination unit configured to determine the relevance between each virtual object among a plurality of displayable virtual objects and the real object detected by the detection unit; and a control unit configured to control the head-mounted display device to display a virtual object with a relevance higher than a threshold value in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

9. The control device according to claim 8, wherein the control unit: controls to display the plurality of virtual objects; and Controlling the head-mounted display device to change the display direction and imaging distance of the virtual object with a relevance higher than the threshold to a direction near the direction from the head-mounted display device towards the detected real object and an imaging distance corresponding to the distance from the head-mounted display device to the detected real object, respectively.

10. The control device according to claim 8, wherein, for each of the plurality of virtual objects, a default imaging distance is determined in advance, and as the default imaging distance of the virtual object gets closer to the distance from the head-mounted display device to the detected real object, the determination unit determines a higher value as the relevance of the virtual object.

11. The control device according to claim 8, wherein, the control unit controls the head-mounted display device to display the virtual object with a relevance lower than a second threshold with reduced saliency, where the second threshold is equal to or less than the threshold.

12. The control device according to claim 11, wherein, the control unit controls the head-mounted display device to hide the virtual object with a relevance lower than the second threshold.

13. The control device according to claim 8, wherein, the control unit controls to display the plurality of virtual objects, the control device further includes a second detection unit configured to detect, based on the line-of-sight information, the virtual object to which the user repeatedly moves the line of sight from the detected real object, and the control unit controls the head-mounted display device to change the display direction and imaging distance of the virtual object detected by the second detection unit to a direction near the direction from the head-mounted display device towards the detected real object and an imaging distance corresponding to the distance from the head-mounted display device to the detected real object, respectively.

14. A control device configured to control an optical see-through type head-mounted display device, the control device comprises: a control unit configured to control the head-mounted display device to display a virtual object in a direction near the direction from the head-mounted display device towards the real object being viewed by the user, wherein the head-mounted display device includes an adjustment lens for adjusting the imaging distance of the virtual object, the control unit further controls the head-mounted display device to change the imaging distance by changing the position of the adjustment lens relative to the head-mounted display device, and the position of the adjustment lens relative to the head-mounted display device is different between the case where the user is viewing a first real object and the case where the user is viewing a second real object.

15. A control method for controlling an optical see-through type head-mounted display device, the control method comprises the following steps: obtaining the line-of-sight information of the user; detecting the real object being viewed by the user based on the line-of-sight information; and When the real object detected in the detection step is a specific real object, the head-mounted display device is controlled to display a virtual object in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

16. A control method for controlling an optical see-through type head-mounted display device, the control method comprises the following steps: Obtain the gaze information of the user; Detect a real object that the user is viewing based on the gaze information; Determine the relevance between each virtual object among a plurality of displayable virtual objects and the detected real object; and Control the head-mounted display device to display a virtual object with a relevance higher than a threshold value in a direction near the direction from the head-mounted display device towards the detected real object and at an imaging distance corresponding to the distance from the head-mounted display device to the detected real object.

17. A program for causing a computer to function as each unit of the control device according to any one of claims 1 to 14.

18. A computer-readable medium storing a program for causing a computer to function as each unit of the control device according to any one of claims 1 to 14.

Images