Glance detector and control method of glance detector
The line-of-sight detection device corrects virtual object positions in head-mounted displays to maintain consistency with external objects, addressing misperceptions caused by eye closure, thereby improving user experience.
Patent Information
- Application Number
- JP2023076267
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing head-mounted display devices fail to account for the change in visual appearance when a user closes one eye, leading to misperception of virtual objects relative to external objects due to positional discrepancies.
A line-of-sight detection device that corrects the display position of virtual objects based on the user's eye state transition from both eyes to one eye, using a line-of-sight detection method and display control means to maintain the positional relationship between virtual and external objects.
Prevents misperceptions by ensuring the relative position of virtual objects remains consistent with external objects when the user transitions from viewing with both eyes to one eye, enhancing user experience and accuracy.
Smart Images

Figure 2024160840000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a gaze detection device and a control method for a gaze detection device. [Background technology]
[0002] In recent years, head-mounted display devices with gaze detection functions, such as glasses-type devices using MR (Mixed Reality) or AR (Augmented Reality), have become more automated and intelligent. One type of head-mounted display device is an optical see-through type that allows a user to see the outside world (real space) through a transparent optical member (lens) and see graphics (e.g., virtual objects) displayed by the head-mounted display device. Patent Document 1 discloses a technology that detects the direction of the user's gaze and controls an external device such as a television according to the option that the gaze is directed to in a video showing the options. Patent Document 2 discloses a technology that improves the accuracy of gaze detection even when only one eye can be detected from a face image. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2003-230539 A [Patent Document 2] Patent Publication No. 2021-77265 Summary of the Invention [Problem to be solved by the invention]
[0004] When a user wearing a head-mounted display device closes one eye while viewing an external object and a virtual object with both eyes through a transparent optical member, the position of the virtual object appears to move relative to the position of the external object in the background. Patent documents 1 and 2 do not take into consideration the change in appearance caused by the user closing one eye, and there is a risk that the user may mistakenly recognize that the virtual object has moved relative to the external object when closing one eye.
[0005] In view of the above, the present invention aims to provide a technology that controls the relationship between the position of an external object that a user sees and the position of a virtual object so that it does not change when the user is looking with both eyes and when the user is looking with one eye. [Means for solving the problem]
[0006] A first aspect of the present invention is a gaze detection device comprising a display control means for controlling the display of a virtual object in relation to an object a user is looking at, and a detection means for detecting the user's gaze position, wherein the display control means corrects the display position of the virtual object when the user transitions from one of a state where the object is viewed with both eyes and a state where the object is viewed with one eye to the other.
[0007] A second aspect of the present invention is a control method for a gaze detection device, comprising a display control step of controlling a virtual object to be displayed in relation to an object that a user is looking at, and a detection step of detecting a gaze position of the user, wherein the display control step corrects the display position of the virtual object when the user transitions from one of a state where the object is viewed with both eyes and a state where the object is viewed with one eye to the other.
[0008] A third aspect of the present invention is a program for causing a computer to function as each of the means of the gaze detection device described above.A fourth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the gaze detection device described above. Effect of the Invention
[0009] According to the present invention, it is possible to provide a technology that controls the relationship between the position of an external object that a user sees and the position of a virtual object so that it does not change depending on whether the user is looking with both eyes or one eye. [Brief description of the drawings]
[0010] [Figure 1] FIG. 2 is an external view of a display device. [Diagram 2] FIG. 2 is a block diagram of a display device. [Diagram 3] FIG. 11 is an external view of another example of the display device. [Figure 4] 4 is a flowchart illustrating a virtual object display process according to the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating gaze detection. [Figure 6] FIG. 2 is a diagram for explaining how a virtual object appears when viewed with both eyes. [Figure 7] FIG. 1 is a diagram illustrating how a virtual object appears when viewed with one eye. [Figure 8] FIG. 1 is a diagram illustrating a block matching method. [Figure 9] 1A and 1B are diagrams illustrating the principle of a gaze detection method. [Figure 10] FIG. 2 is a schematic diagram of an eye image projected onto an eye imaging element. [Figure 11] 11 is a flowchart illustrating an example of a gaze detection process. [Figure 12] FIG. 11 is a diagram illustrating calibration. [Figure 13] FIG. 11 is a diagram illustrating a method for detecting an open eye state. [Figure 14] 10A and 10B are diagrams illustrating a method for determining a gaze position based on the eyeball angles of the left and right eyes. [Figure 15] 13A to 13C are diagrams illustrating correction of the display position of a virtual object in the second embodiment. [Figure 16] 13A to 13C are diagrams illustrating correction of the display position of a virtual object according to the third embodiment. [Figure 17] 11 is a diagram illustrating a restricted area in which the display of a virtual object is restricted. FIG. [Figure 18] 13 is a flowchart illustrating a virtual object display process according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
[0012] <Embodiment 1> A first embodiment of the present invention will be described. The present invention is applicable to an optical see-through head-mounted display device (HMD, head-mounted display). An optical see-through head-mounted display device is, for example, smart glasses (AR (Augmented Reality) glasses). An optical see-through head-mounted display device has, for example, a lens (optical member) similar to the lens of general glasses, and projects a graphic (for example, a virtual object) onto the lens or the user's retina. A user wearing an optical see-through head-mounted display device can see the outside world (real space) through the optical member (lens) and can see a graphic projected (displayed) by the head-mounted display device.
[0013] <Configuration Description> 1(A) and 1(B) show the appearance of a display device 100 as a gaze detection device according to the first embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. The display device 100 is a head-mounted display device of an optical see-through type. The display device 100 can separately detect the gazes of the right eye and the left eye of a user wearing the display device 100 on his / her head. Hereinafter, the user wearing the display device 100 on his / her head will be simply referred to as the user.
[0014] The lens 10 is an optical member facing (directly facing) the user's eyes. The user can view the outside world through the lens 10. The display device 11 displays graphics (e.g., virtual images of virtual objects) to both eyes (both right and left eyes) of the user under control (display control) from a CPU 2 described later. For example, the lens 10 may be a prism or a half mirror. and the display device 11 projects a graphic onto the lens 10. The display device 11 may project the graphic onto the user's retina. The lens 10 may be a transmissive display (display unit) having a display function (function of the display device 11). The display device 11 displays, for example, a GUI (Graphical User Interface, for example, a button or an icon) as the graphic. The user can see the displayed virtual object as if it exists in the outside world. When displaying the graphic, the depth position of the graphic as seen by the user (position in the depth direction (direction moving away from the user)) can be adjusted by adjusting the left-right positional relationship (parallax) between the display position for the right eye and the display position for the left eye.
[0015] The light source driving circuit 12 drives the light sources 13a and 13b. Each of the light sources 13a and 13b is a light source that illuminates the user's eye, and is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user. A part of the light emitted from the light sources 13a and 13b and reflected by the user's eye is collected by the light receiving lens 16 onto the eye imaging element 17. The eye imaging element 17 is an imaging sensor (imaging element) that captures an image of the user's eye. The imaging sensor is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.
[0016] The lens 10, the display device 11, the light source driving circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye image sensor 17 are provided for each of the right and left eyes. The light source driving circuit 12, the light sources 13a and 13b, the light receiving lens 16, and the eye image sensor 17 can be used to acquire the gaze information of the user. The gaze information is information related to the gaze, and indicates, for example, at least one of the viewpoint, the gaze direction (the direction of the gaze), and the convergence angle (the angle between the gaze of the right eye and the gaze of the left eye). The viewpoint can be regarded as the position where the gaze is fixed, the position where the user is looking, or the gaze position. Details of the method of acquiring the gaze information will be described later.
[0017] The external imaging unit 20 includes an imaging sensor (imaging element) that captures an external scene in the direction in which the user is facing. Furthermore, the external imaging unit 20 has various functions for acquiring external information (information about the external world). The imaging sensor is, for example, a CCD sensor or a CMOS sensor.
[0018] For example, the external world imaging unit 20 has a function for detecting and tracking an operating object (for example, a user's hand, a user's finger, or a wand held by a user) that operates the GUI, and detecting a specific movement (gesture) of the operating object. The operating object is detected, for example, from an image of the external world. This function may use various known techniques (for example, hand tracking technology). For example, by linking a movement such as tapping a button on the GUI with a finger to a process that assumes pressing the button, gesture operation (operation by gesture) of the button becomes possible.
[0019] Furthermore, the external world imaging unit 20 has a function for detecting the distance from the display device 100 (user) to a real object existing in the external world (a function for acquiring information (distance information) relating to the distance). This function may use various known technologies. For example, a method may be used in which a time period from when a light wave (e.g., laser light or LED (Light-Emitting Diode) light) is emitted toward an object until the light wave is reflected by the object and returns may be measured. Sound waves or radio waves may be used instead of light waves. A distance measuring sensor different from the above-mentioned imaging sensor (e.g., a LiDAR (Light Detection And Ranging) sensor using laser light, or a TOF (Time Of Flight) sensor using LED light) may be used. A method may be used in which the distance is calculated from an image capturing the external world. For example, a stereo camera that measures distance using two imaging sensors may be used. A method of measuring distance using a plurality of different pixels (a plurality of photoelectric conversion elements) in the imaging surface of an image sensor (phase difference AF (AutoFocus)) may also be used.
[0020] A SPAD (Single Photon Avalanche Diode) sensor, a type of TOF sensor, can measure distances at high speed and with high sensitivity. With a SPAD sensor, the detection error of distance is less than a few percent in real-time distance measurement up to a few meters ahead. By using phase-difference AF, it is possible to measure the distance to a real object and detect the operating body using the same image sensor. Since the number of sensors used is reduced, costs can be reduced.
[0021] Note that there is a correlation between the distance to a real object and the convergence angle of the user when viewing the real object, and these can be detected with high accuracy within a range of about 1 m from the display device 100 (user).
[0022] 2 is a block diagram showing an electrical configuration of the display device 100. The CPU 2 is a central processing unit of a microcomputer built into the display device 100, and controls the entire display device 100. The display device 11, the light source driving circuit 12, the line of sight detection circuit 15, the external world imaging unit 20, the distance detection circuit 201, and the memory unit 3 are connected to the CPU 2.
[0023] The memory unit 3 has a function of storing the image signal from the eye imaging element 17, a function of storing a line of sight correction coefficient (a parameter for correcting individual differences in the line of sight), and the like.
[0024] The gaze detection circuit 15 A / D converts the output of the eye imaging element 17 (an image of the eye) when an optical image of the eye is formed on the eye imaging element 17, and transmits the result to the CPU 2. The CPU 2 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm, and detects the user's gaze from the positions of the feature points.
[0025] The distance detection circuit 201 A / D converts signals (voltages) from a plurality of pixels (a plurality of photoelectric conversion elements) for phase difference detection included in the image sensor in the external world imaging unit 20, and sends the converted signals to the CPU 2. The CPU 2 uses the signals from the plurality of pixels to calculate the distance to a real object (subject) corresponding to each distance detection point.
[0026] In the above explanation, a head-mounted display device (HMD) has been given as an example, but the present invention is not limited to this and can be applied to any optical see-through XR device that detects gaze information from both of the user's eyes and displays a virtual object based on the gaze position of both eyes.
[0027] FIG. 3 shows an external view of a vehicle equipped with a head-up display (HUD) as another example of the display device 100 to which the present invention can be applied. The HUD has a configuration similar to that of the display device 100 described in FIG. 1, and can display a virtual object on the windshield of the vehicle. The same components as those in FIGS. 1(A) and 1(B) are given the same reference numerals and will not be described. The display device 11 of the HUD displays a virtual object by superimposing it on the field of view of a user who sees the outside world through a front window (optical member) functioning as a lens 10. In the example of FIG. 3, the HUD displays a frame 30 (virtual object) by superimposing it on a person 40 (an object in the outside world) that the user is looking at.
[0028] <Virtual object display processing> The display device 100 is an optical see-through type XR device that has a function of detecting the line of sight direction, and displays a virtual object relative to the object that the user is looking at. When the user transitions from a state in which the object is viewed with both eyes to a state in which the object is viewed with one eye, the display device 100 corrects the display position of the virtual object so that the relationship between the position of the object perceived by the user and the position of the virtual object does not change. By correcting the display position of the virtual object, the position of the virtual object shifts relative to the position of the object. This reduces the user's misconception that the
[0029] 4 is a flowchart illustrating a virtual object display process according to the first embodiment. In the virtual object display process, when the user transitions from a state in which the user views the object with both eyes to a state in which the user views the object with one eye, the display device 100 corrects the display position of the virtual object. The virtual object display process is started when the user starts using the display device 100 (for example, when the user wears the display device 100, when the user turns on the power of the display device 100, etc.). The process shown in FIG. 4 is executed for each frame captured by the external world imaging unit 20.
[0030] In step S101, CPU 2 acquires an image of the outside world captured by outside world imaging unit 20. In step S102, CPU 2 acquires an image of the user. Specifically, light sources 13a and 13b radiate infrared light toward the user's eyes. An image of the user's eyes illuminated by the radiated infrared light is formed on eye imaging element 17 through light receiving lens 16. The image of the user's eyes is photoelectrically converted by eye imaging element 17 and can be processed as an electrical signal.
[0031] In step S103, the CPU 2 acquires images of the left and right eyeballs from the user's eye image captured in step S102. When the display device 100 is a head-mounted display device, the display device 100 includes left and right eye imaging elements 17, and the CPU 2 can acquire the images captured by the left and right eye imaging elements 17 in step S102 as images of the left and right eyeballs, respectively.
[0032] If the display device 100 is a HUD, in step S102, the CPU 2 acquires one image of the user captured so as to include both eyes by the eye imaging element 17 as shown in Fig. 5(A). The CPU 2 can acquire a right eye image 501 and a left eye image 502 from the image of the user acquired in step S102.
[0033] In step S104, the CPU 2 executes gaze detection processing for each of the images of the left and right eyeballs acquired in step S103, and acquires gaze information such as the eyeball angles of the left and right eyeballs. The gaze detection processing will be described later with reference to Figs. 9 to 11.
[0034] In step S105, CPU 2 judges whether or not at least one eye is open based on the images of the left and right eyeballs of the user obtained in step S103. When the user has both eyes open or one of the eyes is closed, CPU 2 judges that at least one eye is open and proceeds to step S106. When the user has both eyes closed or the eyeballs of both eyes are not captured within the shooting range of eye imaging element 17, or when the line of sight of both eyes is not detected, CPU 2 returns to step S101.
[0035] In step S106, the CPU 2 determines whether or not both eyes are open. The CPU 2 can determine whether or not both eyes are open based on the processing result of step S105. If both eyes are open, the process proceeds to step S110. If the line of sight of one eye is not detected, such as when only one eye is open, the process proceeds to step S120.
[0036] In step S110, the CPU 2 determines the position (gaze position) at which the user is looking in the outside world based on the eyeball angles of both eyes acquired in step S104. When both the left and right eyeball angles have been acquired, the CPU 2 can acquire the user's gaze position by averaging the left and right gaze positions or by weighting the average using a preset weighting coefficient.
[0037] In step S111, the CPU 2 detects an object present at the gaze position from the outside world image acquired in step S101, using the gaze position determined in step S110.
[0038] In step S112, the CPU 2 acquires the distance to the object detected in step S111 by the distance detection circuit 201. The distance detection circuit 201 can measure the distance to the object in the external world by using, for example, pixels included in the external world imaging unit 20, through a known technology known as image plane phase difference AF by a camera.
[0039] Note that the CPU 2 does not need to measure the distance to the object every frame. For example, when the object detected in step S111 is the same as the object detected in the previous frame, or when there is no change in the eyeball angle from the previous frame, the CPU 2 may use the distance acquired in the previous frame as is without measuring the distance to the object in the external world. When the distance from the user to the object in the external world does not change, the CPU 2 can reduce the processing load for displaying the virtual object by not measuring the distance from the user to the object in the external world.
[0040] In step S113, the CPU 2 determines the display position of a virtual object to be displayed relative to the object detected in step S111. The CPU 2 determines the display position of the virtual object based on the line of sight position determined in step S110 and the distance to the object measured in step S112.
[0041] When proceeding from step S113 to step S140, the CPU 2 displays a virtual object at the display position determined in step S113. In the example of Fig. 5(B), the virtual object is a frame 30 indicating the user's line of sight. Note that when the display device 100 is a head-mounted display device, the virtual object is displayed on each of the left and right lenses 10 (display units), but for convenience of explanation, Fig. 5(B) shows an example in which the virtual object is displayed on one lens 10.
[0042] The display device 100 can detect the position of the user's line of sight and display a virtual object (frame 30) on the lens 10 by superimposing it on the user's field of vision. The user can view the virtual object superimposed on the lens 10 of the display device 100 while viewing an object in the outside world (real object) through a see-through optical member.
[0043] In the example of FIG. 5(B), the user can see an object in the outside world (e.g., a person) and a frame 30 indicating the gaze position. In FIG. 5(B), the object in the outside world and the frame indicating the gaze position are arranged to overlap. When the user activates a function of selecting or capturing an object in the outside world by gazing at the object, the user can confirm that the object he or she wants to select has been selected because the frame 30 indicating the gaze position overlaps with the object. For example, when the gaze position of the user is at the position of the object for a predetermined time, the display device 100 can determine that the user is gazing at the object.
[0044] The following are examples of use of the function of selecting or capturing an external object by gaze. A user driving can select or capture an object whose behavior the user wants to pay attention to, such as a child who may dangerously run out into the street, by gazing at it, and the CPU2 of the display device 100 can monitor the selected object and warn the driver if the object exhibits dangerous behavior. When the user gazes at an object such as a signboard or a road sign, the CPU2 can enlarge and display an image of the object being gazed at. When the user gazes at a building, the CPU2 can display information related to the building being gazed at.
[0045] When the user directs his / her gaze at an object in the outside world to select or capture it, if the state of looking with both eyes is changed to a state of looking with one eye, the positional relationship between the object in the outside world and the virtual object displayed at the gaze position appears to the user to have changed. In the processes up to step S123, the display position of the virtual object is corrected so that the relationship between the positions of the external objects perceived by the user and the positions of the virtual object does not change.
[0046] In step S120, CPU 2 judges whether or not the state has changed from both eyes open to one eye open within a predetermined time. In step S120, when one eye is open and there is a frame in which both eyes are open among the frames up to the predetermined time before the current frame, CPU 2 can judge that the state has changed from both eyes open to one eye open within the predetermined time.
[0047] If the state transitions from both eyes open to one eye open within a predetermined time, i.e., if the state transitions from looking with both eyes to looking with one eye within a predetermined time, the process proceeds to step S121. If the state transitions from both eyes open to one eye open within a predetermined time, i.e., if the state of looking with one eye continues for a predetermined time or longer, the process proceeds to step S130.
[0048] Here, the appearance of a virtual object when viewed with both eyes and when viewed with one eye will be described with reference to Figures 6(A), 6(B) and 7(A) to 7(C). When transitioning from a state of viewing with both eyes to a state of viewing with one eye, the positions of objects in the outside world and the positions of the virtual object appear to be shifted relative to each other.
[0049] Fig. 6(A) illustrates a scene seen in the user's field of vision through the lens 10 when the user is looking with both eyes. Note that the display device 100 illustrated in Fig. 1 has left and right lenses 10, but for simplicity, the following description will be given with one lens 10 illustrated. The user can view the outside world through the optically see-through lens 10. The display device 11 displays a frame 30, which is a virtual object, on the lens 10, superimposed on a person 40 (an object in the outside world) that the user is looking at.
[0050] FIG. 6(B) shows a bird's-eye view of a state in which a user is looking with both eyes. The user is looking at a virtual object (frame 30) displayed on the lens 10 and an object in the outside world (person 40). Since both eyes are open, the user recognizes both the right eye field of view 50 surrounded by a dotted line and the left eye field of view 51 surrounded by a two-dot chain line, and recognizes both fields in an integrated manner as shown in FIG. 6(C). The left and right fields of view are combined in the brain with the gaze position as the center. The frame 30 displayed on the lens 10 is placed in front of the user together with the person 40, and the user recognizes that the frame 30 and the person 40 are placed without any misalignment. When the user activates the function of selecting or capturing an object in the outside world by gaze, the user can select the object in the outside world to be selected without making a mistake by looking at it with both eyes.
[0051] 7(A) to 7(C) are diagrams for explaining the change in appearance when a state of viewing with both eyes is transitioned to a state of viewing with one eye by closing one eye. Fig. 7(A) shows a bird's-eye view when transitioning to a state of viewing with one eye. The user continues from the state of viewing with both eyes and views a virtual object (frame 30) and an external object (person 40) displayed on the lens 10.
[0052] However, when a user closes one eye, the left and right fields of view are not integrated and the user only sees the right eye field of view 50, which leads to the user perceiving a shift in the position between the frame 30 and the person 40 as shown in Fig. 7(B). This is because when the user visually recognizes the frame 30 at the center of the right eye field of view 50, a shift amount ΔH occurs in the position of the person 40 from the center of the right eye field of view 50.
[0053] As described above, even if there appears to be no misalignment between the positions of virtual objects and external objects when viewing with both eyes, when the user closes one eye, The perception is that the relationship with the position has changed.
[0054] At the moment of transition from a state of viewing with both eyes to a state of viewing with one eye, the virtual object and the object in the external world appear to be misaligned, so if the user closes one eye while selecting an object based on the gaze position, the user may mistakenly believe that the selection of the object has failed. Transitions from a state of viewing with both eyes to a state of viewing with one eye are expected to occur, for example, when one eye blinks, when dust gets in and one eye is closed, when one eye is closed due to glare, etc.
[0055] In order to prevent the user from mistakingly recognizing that the virtual object and the object in the outside world are misaligned when the user closes one eye, the display device 100 corrects the display position of the virtual object in steps S121 to S123.
[0056] In step S121, the CPU 2 detects an object where the gaze position was when both eyes were open, from the image captured by the external imaging unit 20. The CPU 2 compares a frame image captured with both eyes open and a frame image captured next, among the images captured by the external imaging unit 20, and can track the object that the user was looking at with both eyes by a pattern matching technique such as a block matching method.
[0057] CPU2 performs pattern matching for each frame up to the frame when the state transitions to the state of viewing with one eye, thereby making it possible to detect the object viewed with both eyes in the frame when the state transitions to the state of viewing with one eye. Note that CPU2 only needs to be able to detect the object viewed with both eyes, and may perform pattern matching every few frames, not just every frame, to track the object.
[0058] 8(A) to 8(C) are diagrams for explaining the block matching method. n The image 61 captured at time t n A time t later than n+1 FIG. 8C shows an image 64 captured at time t n Image 61 taken at time t n+11 is a diagram showing a vector 69 that indicates the amount of movement of a subject (object) that a user is looking at with respect to an image 68 in which an image 64 acquired in the first embodiment is displayed superimposed on the image 68.
[0059] 8(A), the CPU 2 focuses on a region of interest 63 including a part of the subject 62 and tracks the subject 62. The size of the region of interest 63 can be set arbitrarily, but can be set to, for example, 8×8 pixels.
[0060] The CPU 2 can search for where the region of interest 63 has moved to in the image 64 shown in FIG. 8B, using the sum of absolute difference (SAD) or the like as a criterion.
[0061] Specifically, the CPU 2 calculates the SAD between the region 65 in the image 64, which corresponds to the region of interest 63 in the image 61, while shifting the region 65 in the image 64 in the direction indicated by the arrow 66. The CPU 2 acquires the position at which the SAD is minimum as the region of interest 67 in the image 64, which corresponds to the region of interest 63. As shown in FIG. 8C, the region of interest 63 is n From time t n+1 During this time, the object moves along vector 69 to the position of region of interest 67. In this manner, CPU 2 can track the subject (external object) captured by external world imaging unit 20 by the block matching method.
[0062] In step S122, the CPU 2 acquires the distance in the Z direction to the object detected in step S121 by the distance detection circuit 201. The distance detection circuit 201 can measure the distance to the object by using, for example, a known technique known as image plane phase difference AF in a camera, using pixels included in the external world imaging unit 20. As in step S112, the distance to an external object need not be measured every frame, but the distance obtained in the previous frame may be used as is.
[0063] In step S123, CPU 2 corrects the display position of the virtual object displayed on lens 10. CPU 2 determines the corrected display position of the virtual object so that the relationship between the position of the object perceived by the user when viewing with both eyes and the position of the virtual object does not change even when viewing with one eye.
[0064] An example of a method for determining a display position after correction of a virtual object will be described with reference to Figs. 7(A) to 7(C). Fig. 7(A) shows, in a bird's-eye view, a state of a user immediately after transition from a state of viewing with both eyes to a state of viewing with one eye. When a state of viewing a frame 30 (virtual object) located in front of the user transitions from a state of viewing with both eyes to a state of viewing with one eye, the user continues to view the position of the same frame 30. Therefore, even immediately after transition to a state of viewing with one eye, the user continues to view the position of the frame 30 that was viewed in the front direction with both eyes with one eye. Therefore, the gaze position acquired from the eyeball angle of one eye is the same position as the original position of the frame 30.
[0065] When the user closes his / her left eye and only the right eye field of view 50 is visible, if frame 30 continues to be displayed in the same position, the position of frame 30 and the position of person 40 appear to be misaligned, as shown in Fig. 7(B). Since the user intends to continue looking at the same position even when closing one eye from a state in which both eyes are looking, the user may become confused when frame 30 and person 40 suddenly appear to be misaligned.
[0066] Therefore, the CPU 2 corrects the display position of the frame 30 so that the user can recognize that the position of the frame 30 is not misaligned with the position of the person 40, as shown in Fig. 7(C). Specifically, in Fig. 7(A), if the center of the open eyeball is point R and the center of the person 40 is point O, the position of the frame 30 may be corrected to a position on a straight line OR connecting point O of the person 40 and point R of the right eyeball. When a state transitions from a state of viewing with both eyes to a state of viewing with one eye, the CPU 2 may correct the display position of the frame 30 so that the frame 30 is displayed at the position of the intersection WR of the straight line OR connecting points O and R and the display surface of the lens 10.
[0067] The three-dimensional coordinates of point O, which is the position of person 40, can be calculated from the image of the external world imaging unit 20. Moreover, the three-dimensional coordinates of point R, which is the position of the eyeball, can be calculated from the image of the eye imaging element 17. Note that point O is not limited to the center of person 40 (external world object) and may be the center of gravity of person 40 or a position on the boundary of person 40. Furthermore, point R is not limited to the center of the eyeball and may be another position on the eyeball, such as the center of the pupil.
[0068] 7(A) to 7(C) explain a method of correcting the display position when a frame 30, which is a virtual object, is displayed superimposed on an external object present at the user's line of sight, but the virtual object does not have to be displayed superimposed on an external object. The virtual object may be text information related to an external object (for example, a facility or a store) and may be displayed alongside the object.
[0069] When the virtual object does not overlap with an external object, the CPU 2 can correct the display position of the virtual object to, for example, a position on a line passing through the position of the user's open eyes and the position where the virtual object was displayed when the user was looking with both eyes (a position corresponding to the position of the external object). Also, when the virtual object does not overlap with an external object, the CPU 2 may correct the display position of the virtual object based on the displacement amount ΔH of the external object (person 40) described in Fig. 7(B) and the relative position of the virtual object (frame 30) with respect to the external object.
[0070] When the process proceeds from step S123 to step S140, the CPU 2 In response, a virtual object (frame 30) is displayed at the display position corrected in step 123. In this way, when a state transitions from viewing with both eyes to a state viewing with one eye, CPU 2 displays frame 30 as a virtual object at the display position corrected in step S123, instead of at the gaze position determined from the detected eyeball angle of one eye. This allows display device 100 to perform control so that the relationship between the position of frame 30 (virtual object) perceived by the user and the position of person 40 (external object) does not change.
[0071] In step S120, if the predetermined time or more has elapsed with one eye open, CPU 2 determines that there has been no transition from both eyes open to one eye open within the predetermined time, and proceeds to step S130. In step S130, CPU 2 determines the gaze position from the detected eyeball angle of one eye.
[0072] In step S131, the CPU 2 determines the display position of the frame 30, which is a virtual object, based on the gaze position determined in step S130. This is because if a predetermined time or more has passed with one eye open, the user will not have the illusion that the position of the virtual object is misaligned with the position of an object in the outside world. When proceeding from step S131 to step S140, the CPU 2 displays the virtual object (frame 30) in the display position determined in step S131 in the display area of the lens 10.
[0073] In step S141, the CPU 2 waits for a predetermined time and then proceeds to step S142. In step S142, the CPU 2 determines whether or not the user has finished using the display device 100 and turned off the power. If the display device 100 has been turned off, the process shown in Fig. 4 ends. If the display device 100 has not been turned off, the process returns to step S101.
[0074] According to the virtual object display process of Fig. 4, when the user transitions from a state where the user is viewing with both eyes to a state where the user is viewing with one eye, the display device 100 corrects the display position of the virtual object so that the relative position of the virtual object with respect to the position of the object in the external world as seen by the user does not change. Therefore, when the user transitions from a state where the user is viewing with both eyes to a state where the user is viewing with one eye, the display device 100 can suppress the positional deviation perceived by the user with respect to the virtual object displayed with respect to the object in the external world.
[0075] <Gaze detection processing> 9 to 11, the gaze detection process (gaze detection method) in step S104 in Fig. 4 will be described. The gaze of the right eye and the gaze of the left eye can be detected by the following gaze detection method.
[0076] In step S103 of Fig. 4, as illustrated in Fig. 5(A), the CPU 2 acquires a right eye image 501 and a left eye image 502 from the image of the user acquired in step S102. In step S104, the CPU 2 executes gaze detection processing on the acquired eye images. The CPU 2 calculates the eyeball angles of both the left and right eyes, and can determine the gaze position based on the calculated left and right eyeball angles as described below.
[0077] FIG. 9 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for detecting the gaze. As shown in FIG. 9, light sources 13a and 13b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the user. Light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of light-receiving lens 16, and illuminate user's eyeball 14. A part of the light emitted from light sources 13a and 13b and reflected by the eyeball is focused on eye imaging element 17 by light-receiving lens 16. When display device 100 is a head-mounted gaze detection device, display device 100 has left and right eye imaging elements 17 as shown in FIG. 1, and acquires left and right eye images.
[0078] Fig. 10(A) is a schematic diagram of an eye image captured by eye imaging element 17 (eye image projected onto eye imaging element 17). Fig. 10(B) is a diagram showing the output intensity of the CCD in eye imaging element 17.
[0079] 11 is a flow chart illustrating the gaze detection process. When the gaze detection process starts, in step S201, light sources 13a and 13b irradiate infrared light toward user's eyeball 14. An optical image of the user's eye illuminated by the infrared light is formed on eye imaging element 17 through light receiving lens 16 and photoelectrically converted by eye imaging element 17. This results in a processable electrical signal of the eye image. In step S202, CPU 2 acquires an eye image (electrical signal of the eye image; image data of the eye image) from eye imaging element 17 via gaze detection circuit 15.
[0080] In step S203, the CPU 2 obtains the coordinates of the points corresponding to the corneal reflection images Pd, Pe of the light sources 13a, 13b and the pupil center c shown in Fig. 9 from the eye image obtained in step S202. The infrared light emitted from the light sources 13a, 13b illuminates the cornea 142 of the user's eyeball 14. The corneal reflection images Pd, Pe formed by a part of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eye imaging element 17 to become the corneal reflection images Pd', Pe' in the eye image. Similarly, the light beams from the ends a, b of the pupil 141 are imaged on the eye imaging element 17 to become the pupil end images a', b' in the eye image.
[0081] Fig. 10(A) shows an example of an eye image of a reflected image obtained from eye imaging element 17. Fig. 10(B) shows luminance information (luminance distribution) of area α1 in the eye image of Fig. 10(A). In Fig. 10(B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. The coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe. The coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb.
[0082] 10B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the area from coordinates Xb to Xa, which corresponds to the area of the pupil 141 (the area of the pupil image obtained by focusing the light beam from the pupil 141 on the eye imaging element 17), an extremely low level of luminance is obtained except for the coordinates Xd and Xe.
[0083] In contrast, a luminance intermediate between the above two types of luminance is obtained in the region of iris 143 outside pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from iris 143). Specifically, the luminance in the region where the X coordinate is smaller than Xb and the region where the X coordinate is larger than Xa is approximately intermediate between the above two types of luminance.
[0084] From the luminance distribution as shown in Fig. 10(B), the CPU 2 can obtain the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b'. Specifically, the CPU 2 can obtain the coordinates where the luminance is extremely high as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates of the boundary where the luminance is extremely low as the coordinates of the pupil edge images a' and b'.
[0085] When the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is equal to or smaller than a predetermined angle, the coordinate Xc of the pupil center image c' (center of the pupil image) obtained by forming an image of the light beam from the pupil center c on the eye imaging element 17 can be expressed as Xc≒(Xa+Xb) / 2. The predetermined angle can be determined, for example, so that the corneal reflection images Pd', Pe' are formed on the eye imaging element 17. The coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the CPU 2 can estimate the coordinates of the corneal reflection images Pd', Pe' and the coordinate of the pupil center image c'.
[0086] In step S204, the CPU 2 obtains the imaging magnification β of the eye image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be obtained as a function of the distance (Xd-Xe) between the corneal reflection images Pd', Pe'.
[0087] In step S205, the CPU 2 acquires the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16. The X coordinate of the midpoint between the corneal reflection images Pd' and Pe' approximately coincides with the X coordinate of the center of curvature C of the cornea 142. If the standard distance from the center of curvature C of the cornea 142 to the center c of the pupil 141 is Cc, then the rotation angle θ of the optical axis of the eyeball 14 in the ZX plane (plane perpendicular to the Y axis) X can be calculated by the following formula 1. The rotation angle θy of the eyeball 14 in the ZY plane (plane perpendicular to the X-axis) can be calculated in the same manner as the rotation angle θx. β×Cc×SINθ X ≒{(Xd+Xe) / 2}- Xc (Formula 1)
[0088] In step S206, the CPU 2 uses the rotation angles θx, θy calculated in step S205 to acquire the user's line of sight (hereinafter also referred to as the viewpoint) on the lens 10. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the center c of the pupil 141 on the lens 10, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)
[0089] Parameter m in Equations 2 and 3 is a constant determined by the configuration of the optical system (light receiving lens 16, etc.), and is a conversion coefficient that converts rotation angles θx, θy into coordinates corresponding to the center c of the pupil 141 on the lens 10. Parameter m is determined in advance and stored in the memory unit 3. Gaze correction coefficients Ax, Bx, Ay, and By are parameters that correct individual differences in the user's gaze, and are obtained by performing a calibration operation. Gaze correction coefficients Ax, Bx, Ay, and By are stored in the memory unit 3 before the gaze detection process starts.
[0090] In step S207, CPU 2 stores the coordinates (Hx, Hy) of the viewpoint in memory unit 3, and ends the gaze detection process. Note that, although the process in Fig. 11 shows an example in which the rotation angle of the eyeball is obtained by using the corneal reflection images of light sources 13a and 13b, and the coordinates of the viewpoint on lens 10 are obtained, the present invention is not limited to this. A method of obtaining the rotation angle of the eyeball from the eyeball image may be a method of measuring the gaze from the pupil center position.
[0091] <Calibration work> As described in Fig. 11, the display device 100 can estimate the viewpoint by acquiring the rotation angles θx, θy of the eyeball 14 from the eye image in the gaze detection process and performing coordinate conversion of the position of the pupil center c to a position on the lens 10. Fig. 12(A) is a diagram showing the user's visual field 400 (the range that the user can see through the lens 10, the imaging range of the external world imaging unit 20) and shows a state in which the display device 11 is operating. As shown in Fig. 12(A), the display device 11 displays a frame or the like at the estimated viewpoint P.
[0092] However, due to factors such as individual differences in the shape of the human eyeball, it may be difficult to estimate the viewpoint with high accuracy. Specifically, unless the line-of-sight correction coefficients Ax, Ay, Bx, and By are adjusted to appropriate values according to the user, the position of the estimated viewpoint R may deviate from the user's actual viewpoint Q, as shown in Fig. 12(B). In Fig. 12(B), the user is looking at a person at the position of the actual viewpoint Q, but the display device 100 mistakenly estimates the position of viewpoint R, which is included in the background, as the position of the user's viewpoint.
[0093] Therefore, the user performs a calibration operation of the line of sight detection function before normal use of the display device 100, and the display device 100 sets line of sight correction coefficients Ax, Ay, Bx, and B suitable for the user. y is determined and stored in the memory unit 3.
[0094] The calibration work is performed, for example, by displaying multiple indices at different positions as shown in Fig. 12(C) before normal use of the display device 100 and having the user look at the indices. A technology is known that performs a gaze detection operation when gazing at each target and determines gaze correction coefficients Ax, Ay, Bx, and By suitable for the user from the calculated multiple viewpoints (estimated positions) and the coordinates of each index.
[0095] <Detecting eye open state> An example of a method for detecting the open state of the left and right eyes in step S105 of Fig. 4 will be described with reference to Fig. 13(A) and 13(B). Fig. 13(A) is a schematic diagram of an eye image captured by eye imaging element 17 (eye image projected onto eye imaging element 17). Fig. 13(B) is a diagram showing the output intensity of the CCD in eye imaging element 17.
[0096] The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. Corneal reflection images Pd and Pe formed by a part of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and focused on the eye imaging element 17 to become corneal reflection images Pd' and Pe' in the eye image.
[0097] Fig. 13(A) shows an example of a reflected image obtained from the eye imaging element 17. Fig. 13(B) shows luminance information (luminance distribution) of area α2 in the eye image of Fig. 13(A). In Fig. 13(B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. The Y-axis (vertical) coordinate of the corneal reflection image Pd' is Yd. The Y coordinates of the pupil edge image (images of both ends of the pupil in the Y-axis direction) formed on the eye imaging element 17 by light beams from the upper and lower ends of the pupil 141 are Ya and Yb.
[0098] 13B, an extremely high level of luminance is obtained at the coordinate Yd of the corneal reflection image Pd'. In the area from coordinate Ya to Yb, which corresponds to the area of the pupil 141 (the area of the pupil image obtained by focusing the light beam from the pupil 141 on the eye imaging element 17), an extremely low level of luminance is obtained except for the coordinate Yd.
[0099] In contrast, the luminance is intermediate between the above two types of luminance in the region of the iris 143 outside the pupil 141 and in the regions outside the upper eyelid 145f and the lower eyelid 145g. Specifically, the luminance in the region where the Y coordinate is greater than Ya and the region where the Y coordinate is less than Yb is intermediate between the above two types of luminance.
[0100] The brightness values are also divided into two levels in the region where the Y coordinate is greater than Ya and the region where the Y coordinate is less than Yb, where intermediate brightness values are obtained. The region corresponding to the iris 143 from the Y coordinate Ya of the upper end of the pupil 141 to the Y coordinate Yf of the upper eyelid 145f has a brightness value of K1. The region outside the upper eyelid 145f, where the Y coordinate is greater than Yf, has a brightness value of K2.
[0101] Similarly, in the direction of the lower eyelid, the area corresponding to the iris 143 from the Y coordinate Yb of the lower end of the pupil 141 to the Y coordinate Yg of the lower eyelid 145g has a brightness value of K1. The area outside the lower eyelid 145g whose Y coordinate is smaller than Yg has a brightness value of K2.
[0102] The display device 100 can obtain the Y coordinate Yd of the corneal reflection image Pd', the Y coordinates Ya, Yb of the pupil edge image, the Y coordinate Yf of the upper eyelid 145f, and the Y coordinate Yg of the lower eyelid 145g from the variation information of the luminance value with respect to the Y coordinate. The difference between the Y coordinate Yf of the upper eyelid 145f and the Y coordinate Yg of the lower eyelid 145g represents the degree to which the user's eyelids are open or closed. The display device 100 can determine whether the eyes are open or closed by evaluating the degree to which the eyelids are open or closed. For example, the display device 100 can obtain the Y coordinate Yd of the upper eyelid 145f, When the difference between the Y coordinate Yf of the eyelid 145 and the Y coordinate Yg of the lower eyelid 145g is greater than a predetermined threshold, it can be determined that the eye is open. The predetermined threshold may be a standard distance between the upper eyelid and the lower eyelid in the open eye state, or may be set based on the distance between the upper eyelid and the lower eyelid measured for each user.
[0103] <Determining the gaze position with both eyes> A method for determining the position where the user is looking in the outside world based on the left and right eyeball angles, which are the gaze detection results of the left and right eyes, in step S140 in Fig. 4 will be described with reference to Figs. 14(A) to 14(C). Fig. 14(A) illustrates a scene seen in the user's field of view through the display area of the lens 10 when the user is looking with both eyes. Fig. 14(B) illustrates a bird's-eye view when the user is looking with both eyes.
[0104] Based on the left and right eyeball angles, which are the results of gaze detection for both the left and right eyes, the display device 100 can calculate the positions of the left eye viewpoint 61L and the right eye viewpoint 61R in the display area of the lens 10. The left eye viewpoint 61L and the right eye viewpoint 61R basically indicate approximately the same position in the display area of the lens 10, but under certain conditions they do not intersect at a single point and are separated from each other as shown in FIG.
[0105] Hereinafter, a method for determining the gaze position when viewing with both eyes will be described when the display device 100 is a HUD. When the user is paying attention to an external object that is located behind the front window (lens 10), the left and right gaze directions do not intersect on the front window. In this case, the left eye viewpoint 61L and the right eye viewpoint 61R do not intersect at a single point, but are separated from each other.
[0106] Therefore, the display device 100 determines the gaze position of both eyes by integrating the left and right viewpoints using the coordinates of the left and right viewpoints 61L, 61R. The display device 100 can display the frame 30, which is a virtual object, at the determined gaze position of both eyes. The gaze position 61C of both eyes by integrating the left and right viewpoints can be determined, for example, as follows.
[0107] The coordinates of gaze position 61C are (Xc, Yc, Zc), the coordinates of left eye viewpoint 61L are (Xl, Yl, Zl), and the coordinates of right eye viewpoint 61R are (Xr, Yr, Zr). Display device 100 can calculate gaze position 61C as the midpoint of a line segment connecting left eye viewpoint 61L and right eye viewpoint 61R, for example, as (Xc, Yc, Zc)=((Xl+Xr) / 2, (Yl+Yr) / 2, (Zl+Zr) / 2).
[0108] The display device 100 may calculate the coordinates of the gaze position 61C by multiplying the left and right eye by different weighting coefficients, not limited to the midpoint between the viewpoints 61L and 61R. This is because the viewpoint of the dominant eye is considered to be more reliable than the viewpoint of the non-dominant eye. For example, when the right eye is the dominant eye, if the weighting ratio of the right eye to the left eye is 0.6:0.4, the coordinates of the gaze position 61C are calculated as follows: (Xc, Yc, Zc)=(0.6×Xr+0.4×Xl, 0.6×Yr+0.4×Yl, 0.6×Zr+0.4×Zl). The weighting ratio of the right eye to the left eye may be set for each user. In this way, the display device 100 only needs to determine the gaze position 61C of both eyes based on the viewpoint 61L of the left eye and the viewpoint 61R of the right eye.
[0109] In the above-mentioned embodiment 1, when the user transitions from a state where they are viewing with both eyes to a state where they are viewing with one eye, the display device 100 corrects the display position of the virtual object being displayed relative to the object being viewed by the user. Specifically, when the user transitions from a state where they are viewing with both eyes to a state where they are viewing with one eye, the display device 100 corrects the display position of the virtual object so that the relationship between the position of the object perceived by the user and the position of the virtual object does not change when the user closes one eye. In this way, the display device 100 can display the virtual object in a manner that is consistent with the state where the user transitions from a state where they are viewing with both eyes to a state where they are viewing with one eye. When closed, it is possible to reduce the risk of mistakingly recognizing that the relationship between the position of an external object (or scene) and the position of a virtual object has changed.
[0110] In addition, the above-mentioned embodiment 1 describes an example in which the user transitions from a state in which the user is looking with both eyes to a state in which the user is looking with one eye, but is not limited to this. The embodiment 1 can also be applied to a case in which the user transitions from a state in which the user is looking with one eye to a state in which the user is looking with both eyes. In this case, when the user transitions from a state in which the user is looking with one eye to a state in which the user is looking with both eyes, the display device 100 may correct the display position of the virtual object to a position on a line passing through the gaze position in the state in which the user is looking with both eyes and the position of the object in the external world. In this way, the display device 100 corrects the display position of the virtual object when the user transitions from one of the state in which the user is looking with both eyes and the state in which the user is looking with one eye to the other, thereby reducing the possibility that the user will mistakenly recognize that the relationship between the object in the external world and the position of the virtual object has changed.
[0111] <Embodiment 2> In the first embodiment, when it is assumed that the distance from the user to the object in the external world does not change much, the display device 100 determines the display position of the virtual object using the distance used in the previous frame. In contrast, the second embodiment is an embodiment in which the distance from the user to the object in the external world is repeatedly detected, and the display position of the virtual object is corrected based on the detected distance from the user to the object in the external world. In the second embodiment, the display position of the virtual object is corrected taking into account the distance from the user to the object in the external world. The appearance and configuration of the display device 100 according to the second embodiment are the same as those of the first embodiment, and therefore will not be described.
[0112] Correction of the display position of a virtual object displayed relative to external objects at different distances from the user will be described with reference to Figs. 15(A) to 15(C). In the example of Fig. 15(A), a person 41 (external object) and a person 42 (external object) at different distances from the user are present in front of the user. A straight line O1R connecting a point O1 at the center of the person 41 and a point R at the center of the right eyeball in an open eye state intersects with the lens 10 at the intersection point WR1. A straight line O2R connecting a point O1 of the person 41 and a point R of the right eyeball in an open eye state intersects with the lens 10 at the intersection point WR2.
[0113] The intersections WR1 and WR2 are located at different positions on the lens 10. As shown in FIG. 15B, the amount of deviation between the person 41 and the frame 30 is ΔH1. As shown in FIG. 15C, the amount of deviation between the person 42 and the frame 30 is ΔH2. The amount of deviation ΔH1 between the person 41 and the frame 30 is larger than the amount of deviation ΔH2 between the person 42 and the frame 30. In this way, when the user is looking at the person 41 who is farther away than the person 42, the amount of deviation ΔH1 of the frame 30 with respect to the person 41 is larger than the amount of deviation ΔH2 of the frame 30 with respect to the person 42. Therefore, the amount of correction of the display position of the frame 30, which is a virtual object, differs between when the user is looking at the person 41 and when the user is looking at the person 42 who is closer than the person 41.
[0114] Therefore, in the second embodiment, the display device 100 repeatedly detects the distance from the user to the external object, and in the process of step S123 in Fig. 4, the display device 100 corrects the display position of the virtual object by taking into account the distance from the user to the external object. The display device 100 may detect the distance from the user to the external object for each frame, or may detect it at a predetermined time interval. Furthermore, the display device 100 may re-detect the distance from the user to the external object when the amount of movement of the external object from the previous frame is equal to or greater than a predetermined threshold.
[0115] According to the above-mentioned embodiment 2, even if an object in the outside world moves and the distance from the user to the object in the outside world changes, the display device 100 can appropriately correct the display position of the virtual object according to the amount of deviation between the object in the outside world and the virtual object as perceived by the user.
[0116] <Embodiment 3> In the first embodiment, the display device 100 determines the display position of the virtual object on the assumption that the distance from the user to the lens 10 that displays the virtual object is substantially constant. That is, when it is assumed that the distance from the user to an object in the external world does not change much, the display device 100 determines the display position of the virtual object using the distance to the object used in the previous frame. In contrast, the third embodiment is an embodiment in which the distance from the user to the lens 10 that displays the virtual object is repeatedly detected, and the display position of the virtual object is corrected based on the detected distance from the user to the lens 10. In the third embodiment, the display position of the virtual object is corrected taking into account the distance from the user to the lens 10 that displays the virtual object. The appearance and configuration of the display device 100 according to the third embodiment are the same as those of the first embodiment, and therefore will not be described.
[0117] 16(A) and 16(B), the correction of the display position of a virtual object when the distance from the user to the lens 10 that displays the virtual object changes will be described. FIG. 16(A) shows an example in which the eyeball of user U1 is at the position of Z coordinate Zr1, and the eyeball of user U2 is at the position of Z coordinate Zr2. A straight line O3R1 connecting the center point O3 of person 40 (external object) and the center point R1 of the right eyeball of user U1 intersects with the lens 10 at the intersection point WR1'. Also, a straight line O3R2 connecting the point O3 of person 40 and the point R2 of the right eyeball of user U2 intersects with the lens 10 at the intersection point WR2'.
[0118] Intersection WR1' and intersection WR2' are located at different positions on lens 10. When user U1 is looking at person 40, the amount of deviation between frame 30 (virtual object) and person 40 is greater than when user U2 is looking at person 40. Therefore, the amount of correction of the display position of frame 30, which is a virtual object, differs between when user U1 is looking at person 40 and when user U2 is looking at person 40.
[0119] Therefore, in the third embodiment, the display device 100 repeatedly detects the distance from the user to the lens 10 that displays the virtual object, and in the process of step S123 in FIG. 4, the display device 100 corrects the display position of the virtual object by taking into account the distance from the user to the lens 10. The display device 100 may detect the distance from the user to the lens 10 at a predetermined time interval. Since the distance from the user to the lens 10 is expected to change less and change less frequently than the distance from the user to an object in the outside world described in the second embodiment, the distance may be detected at a longer time interval. In addition, the display device 100 may redetect the distance from the user to the lens 10 when the amount of movement due to a change in the user's posture or the like from the previous frame is equal to or greater than a predetermined threshold value.
[0120] A method for calculating the distance from the user's eyeball to the lens 10 that displays a virtual object will be described with reference to Fig. 16(B). The display device 100 can calculate the distance from the user's eyeball to the lens 10 by using the distance between the two Purkinje images in the eyeball image described with reference to Fig. 10(A).
[0121] FIG. 16(B) is a graph showing the relationship between image interval ΔP and eyeball distance Z. Image interval ΔP is the distance between corneal reflection image Pd' and corneal reflection image Pe' formed by light sources 13a and 13b. Eyeball distance Z is the distance from eye imaging element 17 to the eyeball. Eyeball distance Z monotonically decreases nonlinearly as image interval ΔP increases. When image interval ΔP is determined, display device 100 can uniquely calculate eyeball distance Z from eye imaging element 17 to the eyeball.
[0122] 16B, when the image distance between the corneal reflection images Pd1' and Pe1' of user U1 is ΔP1, the eyeball distance of user U1 is Z1. When the image distance between the corneal reflection images Pd2' and Pe2' of user U2 is ΔP2, the eyeball distance of user U2 is Z2. The wider the image distance ΔP, the greater the eyeball distance Z.
[0123] The correspondence relationship between the image interval ΔP of the corneal reflection image and the eyeball distance Z is stored in the memory unit 3, for example. The distance Z can be set in advance in the form of a correlation table or the like that displays the measured image interval ΔP. The display device 100 can obtain the eyeball distance Z from the measured image interval ΔP by reading out the correlation table. The display device 100 can determine the display position of the virtual object by taking the obtained eyeball distance Z as the distance from the user to the lens 10.
[0124] The method of calculating the distance from the user to the lens 10 that displays the virtual object is not limited to the calculation method using the image interval ΔP of the Purkinje images described in Fig. 16(B). The display device 100 only needs to have a means capable of measuring the distance from the user to the lens 10, and may calculate the distance to the eyeball by a distance detection circuit incorporated in the eye imaging element 17, for example.
[0125] According to the above-mentioned embodiment 3, even if there is a change in the distance from the user to the lens 10 that displays the virtual object, the display device 100 can appropriately correct the amount of deviation between the virtual object and an external object perceived by the user in accordance with the change in distance. For example, if the display device 100 is a HUD, even if there is a change in the distance to the front window of the HUD due to the user changing his / her posture, the HUD can appropriately correct the display position of the virtual object.
[0126] <Embodiment 4> In the fourth embodiment, when the display position of a virtual object after correction is included in a restricted area where the display of the virtual object is restricted, the display device 100 controls to display the virtual object at the display position before correction or not to display the virtual object. Specifically, the display device 100 corrects the display position of the virtual object and then determines whether the display position of the virtual object after correction is included in the restricted area. When the display position of the virtual object after correction is included in the restricted area, the display device 100 controls to display the virtual object at the display position before correction or not to display the virtual object. The appearance and configuration of the display device 100 according to the fourth embodiment are the same as those of the first embodiment, and therefore description thereof will be omitted.
[0127] Fig. 17 is a diagram illustrating a restricted area in which the display of a virtual object is restricted. Fig. 17 illustrates a scene seen in the user's field of view through the lens 10. The user can view the outside world through the optically see-through lens 10. The display device 11 displays a frame 30, which is a virtual object, in the display area of the lens 10. A part of the lens 10 is designated as a restricted area 600, in which the display of a virtual object is restricted.
[0128] The restricted area 600 is an area in the display area of the lens 10 where the display of virtual objects is restricted. The restricted area 600 is an area where, for example, important information is displayed and the display of other information is restricted. A HUD, which is an example of the display device 100, displays, for example, important road signs, vehicle failure conditions, and the like in the restricted area 600. The restricted area 600 may be disposed at a fixed position in the display area of the lens 10, or the position may be changed depending on the external situation, such as the position of a road sign.
[0129] 17 is a virtual object indicating the gaze position detected when the user is looking with both eyes. Frame 31 indicates the position of frame 30 after correction when the user transitions to a state where one eye is closed and the user is looking with one eye. When the corrected frame 31 is included in the restricted area 600, the display device 100 performs control so as not to display the frame 31 within the restricted area 600.
[0130] Specifically, after acquiring the corrected display position of the virtual object (frame 30), display device 100 determines whether or not the corrected display position is included in restriction area 600. If the corrected display position of the virtual object is included in restriction area 600, display device 100 displays the virtual object at the pre-correction display position. In the example of Fig. 17, frame 31 is included in restriction area 600, so display device 100 displays the frame at the pre-correction display position (the position of frame 30) without correcting the display position. Furthermore, when the corrected display position of the virtual object is included in the restricted area 600, the display device 100 may not display the virtual object itself.
[0131] When the display position of the virtual object is not corrected, the display device 100 performs control (notification control) so as to perform a predetermined notification to inform the user of the fact. When the display position of the virtual object is not corrected, the user perceives a deviation between the virtual object and the object in the outside world and feels uncomfortable. In order to prevent a hindrance to the operation based on the user's gaze position, the display device 100 displays a message in the notification area 601 as shown in FIG. 17 to notify the user that the correction of the display position (frame position) of the virtual object has been restricted. Instead of displaying the message, the display device 100 may notify the user that the display position of the virtual object has not been corrected by changing at least one of the color and the shape of the virtual object (frame 30) or by displaying the virtual object in a blinking manner.
[0132] Furthermore, even when the corrected display position of the virtual object is included in the restricted area 600 and the virtual object itself is not displayed, the display device 100 performs control (notification control) to perform a predetermined notification to inform the user of this fact. When the virtual object is not displayed, the user feels uncomfortable because the virtual object cannot be seen. In order to prevent interference with operations based on the user's gaze position, the display device 100 displays a message in the notification area 601 to notify the user that the display of the virtual object itself has been restricted.
[0133] Fig. 18 is a flowchart illustrating a virtual object display process according to embodiment 4. The virtual object display process shown in Fig. 18 is a process in which steps S150 to S152 are added to the virtual object display process according to embodiment 1 shown in Fig. 4. The same processes as those in Fig. 4 are denoted by the same reference numerals and description thereof will be omitted.
[0134] The CPU 2 of the display device 100 determines the display position of the virtual object in steps S101 to S113, steps S120 to S123, and steps S130 to S131, in the same manner as in the first embodiment.
[0135] In step S150, the CPU 2 determines whether or not the display position of the determined virtual object is included in the restricted area 600. For example, when the center point of the virtual object is included in the restricted area 600, the CPU 2 can determine that the display position of the virtual object is included in the restricted area 600. Note that the CPU 2 may determine that the display position of the virtual object is included in the restricted area 600 when the entirety or a part of the virtual object is included in the restricted area 600. When the display position of the virtual object is not included in the restricted area 600, the process proceeds to step S140. When the display position of the virtual object is included in the restricted area 600, the process proceeds to step S151.
[0136] In step S151, the CPU 2 does not display a virtual object in the restricted area 600, but displays the virtual object at a display position before the correction. The CPU 2 may also hide the virtual object. It can be set in advance whether the virtual object is displayed at the display position before the correction or is hidden.
[0137] In step S152, the CPU 2 notifies the user that the virtual object has been displayed at the display position before correction, i.e., that the display position of the virtual object has not been corrected. The CPU 2 displays, for example, in a notification area 601 shown in FIG. 17, a message notifying that correction of the display position of the virtual object has been restricted. The CPU 2 may notify the user that the display position of the virtual object has not been corrected by changing at least one of the color and the shape of the virtual object (frame 30) displayed at the display position before correction, or by displaying the frame in a blinking manner. When the CPU 2 hides the virtual object, it may display a message notifying the user to that effect in the notification area 601.
[0138] In step S141, the CPU 2 waits for a predetermined time and then proceeds to step S142. In step S142, the CPU 2 determines whether or not the user has finished using the display device 100 and turned off the power. If the display device 100 has been turned off, the process shown in Fig. 18 ends. If the display device 100 has not been turned off, the process returns to step S101.
[0139] According to the above-described fourth embodiment, when the user transitions from a state in which the user is viewing with both eyes to a state in which the user is viewing with one eye, the display device 100 corrects the display position of the virtual object while restricting the display of the virtual object in the restricted area 600. Therefore, the display device 100 can control the display of the virtual object without interfering with the visibility of the main information displayed in the restricted area 600.
[0140] The above-described embodiment is illustrative of the present invention and is not restrictive, and the present invention can be embodied in various other modified and altered forms. In addition, various forms can be appropriately combined without departing from the scope of the present invention.
[0141] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.
[0142] The disclosure of the present embodiment includes the following configuration, method, program, and medium. (Configuration 1) A display control means for controlling the display of a virtual object relative to an object being viewed by a user; A detection means for detecting a gaze position of the user; having The display control means corrects a display position of the virtual object when the user transitions from one of a state where the virtual object is viewed with both eyes and a state where the virtual object is viewed with one eye to the other. A gaze detection device comprising: (Configuration 2) The display control means corrects a display position of the virtual object so that a relationship between a position of the object perceived by the user and a position of the virtual object does not change between a state where the object is viewed with both eyes and a state where the object is viewed with one eye. 2. The gaze detection device according to configuration 1, (Configuration 3) When the user transitions from a state where the virtual object is viewed with both eyes to a state where the virtual object is viewed with one eye, the display control means corrects the display position of the virtual object to a position on a straight line passing through the position of the user's open eye and a position corresponding to the position of the virtual object. 3. The gaze detection device according to configuration 1 or 2. (Configuration 4) A distance from the user to the object is repeatedly detected, and a display position of the virtual object is corrected based on the detected distance from the user to the object. 3. The gaze detection device according to configuration 1 or 2. (Configuration 5) A distance from the user to a display unit that displays the virtual object is repeatedly detected, and a display position of the virtual object is corrected based on the detected distance from the user to the display unit. 4. The gaze detection device according to any one of configurations 1 to 3. (Configuration 6) The display control means controls the display position of the virtual object after the correction so that the display of the virtual object is restricted. If the virtual object is included in the restricted area, the virtual object is displayed at the display position before the correction, or the virtual object is not displayed. 6. The gaze detection device according to any one of configurations 1 to 5. (Configuration 7) The display device further includes a notification control means for controlling to give a predetermined notification to the user when the display position of the virtual object after the correction is included in the restricted area in which the display of the virtual object is restricted. 7. The gaze detection device according to configuration 6, (Configuration 8) The detection means detects whether the user has both eyes open or one of the eyes closed. 8. The gaze detection device according to any one of configurations 1 to 7. (method) a display control step of controlling the display of a virtual object relative to an object being viewed by a user; A detection step of detecting a gaze position of the user; having In the display control step, when the user transitions from one of a state where the virtual object is viewed with both eyes and a state where the virtual object is viewed with one eye to the other, the display position of the virtual object is corrected. A control method for a gaze detection device comprising: (program) A program for causing a computer to function as each of the means of the gaze detection device according to any one of configurations 1 to 8. (medium) A computer-readable storage medium storing a program for causing a computer to function as each of the means of the gaze detection device according to any one of configurations 1 to 8. [Explanation of symbols]
[0143] 2: CPU, 10: lens, 17: eye image sensor, 100: display device (gaze detection device)
Claims
1. A display control means for controlling the display of a virtual object relative to an object being viewed by a user; A detection means for detecting a gaze position of the user; having The display control means corrects a display position of the virtual object when the user transitions from one of a state where the virtual object is viewed with both eyes and a state where the virtual object is viewed with one eye to the other. A gaze detection device comprising:
2. The display control means corrects a display position of the virtual object so that a relationship between a position of the object perceived by the user and a position of the virtual object does not change between a state where the object is viewed with both eyes and a state where the object is viewed with one eye.
2. The gaze detection device according to claim 1,
3. When the user transitions from a state where the virtual object is viewed with both eyes to a state where the virtual object is viewed with one eye, the display control means corrects the display position of the virtual object to a position on a straight line passing through the position of the user's open eye and a position corresponding to the position of the virtual object.
2. The gaze detection device according to claim 1,
4. The display control means repeatedly detects a distance from the user to the object, and corrects a display position of the virtual object based on the detected distance from the user to the object.
2. The gaze detection device according to claim 1,
5. The display control means repeatedly detects a distance from the user to a display unit that displays the virtual object, and corrects a display position of the virtual object based on the detected distance from the user to the display unit.
2. The gaze detection device according to claim 1,
6. When the display position of the virtual object after the correction is included in a restricted area in which the display of the virtual object is restricted, the display control means controls so as to display the virtual object at the display position before the correction or not to display the virtual object.
2. The gaze detection device according to claim 1,
7. The display control unit further includes a notification control unit that controls the display position of the virtual object after the correction to be included in the restricted area in which the display of the virtual object is restricted so as to give a predetermined notification to the user.
7. The gaze detection device according to claim 6.
8. The detection means detects whether the user has both eyes open or one of the eyes closed.
2. The gaze detection device according to claim 1,
9. a display control step of controlling the display of a virtual object relative to an object being viewed by a user; A detection step of detecting a gaze position of the user; having In the display control step, when the user transitions from one of a state where the virtual object is viewed with both eyes and a state where the virtual object is viewed with one eye to the other, the display position of the virtual object is corrected. A control method for a gaze detection device comprising:
10. A program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 8.
11. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the gaze detection device according to any one of claims 1 to 8.
Citation Information
Patent Citations
Visual axis-detecting device
JP2003230539A
Line-of-sight detection method, line-of-sight detection device, and control program
JP2021077265A
