Near-to-eye display device and gaze tracking method

Abstract

The invention provides near-to-eye display equipment. The near-to-eye display equipment comprises at least one sight tracking module and a control module, the at least one sight tracking module comprises a first light source, a second light source and an information acquisition module; the first light source is used for emitting first detection light towards human eyes to lighten the human eyes; the second light source is used for emitting second detection light towards the human eyes to lighten the human eyes; the information acquisition module is used for generating an eye image based on the first detection light and the second detection light, and the eye image comprises a first cornea reflection point formed by the first light source on the human eye and a second cornea reflection point formed by the second light source on the human eye; and the control module is used for receiving the eye image, acquiring a reflection reference point according to the first cornea reflection point and the second cornea reflection point, acquiring viewpoint information of the human eyes, and adjusting the display image according to the viewpoint information of the human eyes. The invention further provides a sight tracking method.

Description

Technical Field

[0001] This application relates to the field of near-eye display technology, and more particularly to a near-eye display device and a gaze tracking method. Background Technology

[0002] With technological advancements, Augmented Reality (AR) and Mixed Reality (MR) technologies are being applied across various fields, including gaming, education, healthcare, and retail. Eye-tracking technology is a core technology for enabling user-device interaction in AR display devices, providing richer and more personalized interactive experiences. Currently, most AR display devices on the market use Pupil Center Corneal Reflection (PCCR) to capture the user's gaze point. This involves reflecting infrared light through the cornea to create a Pupil-Chin spot on the eye's surface, thus achieving gaze tracking.

[0003] However, existing AR display devices often only have a single infrared light source. A single infrared light source can only form a single Pulcim spot on the surface of the eyeball. A single Pulcim spot is easily affected by the user's head movement, which can easily cause serious deviations and thus affect the stability of the AR display device's eye tracking. Summary of the Invention

[0004] In view of the above, this application provides a near-eye display device and an eye-tracking method, which can reduce the error caused by the deviation of the Puerchin spot due to the user's head shaking, and is beneficial to improving the stability of eye-tracking.

[0005] The first aspect of this application provides a near-eye display device, comprising: At least one gaze-tracking module includes a first light source, a second light source, and an information acquisition module; the first light source emits a first detection light toward a human eye; the second light source emits a second detection light toward the human eye; the information acquisition module receives the first detection light and the second detection light reflected by the human eye, and generates an eye image based on the first detection light and the second detection light, the eye image including a first corneal reflection point formed on the human eye by the first detection light emitted from the first light source and a second corneal reflection point formed on the human eye by the second light source; and A control module, electrically connected to the gaze tracking module, is used to receive the eye image and to obtain a reflection reference point based on the first corneal reflection point and the second corneal reflection point, and to obtain the center point of the human eye pupil based on the eye image. The reflection reference point and the center point of the human eye pupil are used to obtain the viewpoint information of the human eye. The control module is also used to adjust the displayed image based on the viewpoint information of the human eye.

[0006] The near-eye display device provided in this application embodiment uses at least one gaze tracking module and a control module. The information acquisition module generates an eye image, which includes a first corneal reflection point formed by a first light source on the human eye and a second corneal reflection point formed by a second light source on the human eye. The control module obtains a reflection reference point based on the first and second corneal reflection points and obtains the center point of the human eye's pupil based on the eye image. The reflection reference point and the center point of the human eye's pupil can obtain the human eye's viewpoint information, enabling stable gaze tracking. Compared to using a single infrared light source to obtain a single Pullchin spot to obtain the human eye's viewpoint information, the near-eye display device in this application embodiment, by using a first and a second light source, can obtain both a first and a second corneal reflection point. This helps reduce errors caused by Pullchin spot deviation due to user head movement, improves gaze tracking stability, and thus enhances the stability of the near-eye display device, ultimately improving the user experience.

[0007] In some embodiments, the near-eye display device further includes a frame, the frame including a lens frame, the lens frame including a first lens frame portion and a second lens frame portion connected to each other, the first lens frame portion and the second lens frame portion being used to engage lenses, and the at least one gaze tracking module being disposed on the first lens frame portion or the second lens frame portion.

[0008] In some embodiments, the first light source, the second light source, and the information acquisition module are all disposed on the first lens frame portion. The information acquisition module has a photosensitive surface for receiving the first detection light and the second detection light. The central axis of the photosensitive surface passes through the center point of the pupil of the human eye, and the central axis of the photosensitive surface is parallel to a first direction.

[0009] In some embodiments, the information acquisition module is located between the first light source and the second light source; The first light source and the information acquisition module are at a distance S1 in the second direction, and the second light source and the information acquisition module are at a distance S2 in the second direction. The ratio of the distance S1 to the distance S2 is in the range of 1:0.3 to 1:0.8, and the second direction is perpendicular to the first direction.

[0010] In some embodiments, the first light source is located between the information acquisition module and the second light source; the arrangement direction of the first light source and the second light source is parallel to the first direction; The first light source and the second light source are disposed on the side of the information acquisition module that is close to or away from the second frame portion.

[0011] In some embodiments, the ratio of the distance S1 between the first light source and the information acquisition module in the second direction and the distance S3 between the second light source and the first light source in the first direction is in the range of 1:0.7 to 1:1, and the second direction is perpendicular to the first direction.

[0012] A second aspect of this application provides a gaze tracking method, comprising: Control the first light source to emit the first probe light and the second light source to emit the second probe light; The control information acquisition module receives the first detection light and the second detection light reflected by the human eye, and acquires an eye image of the human eye based on the first detection light and the second detection light; Based on the eye image, the first corneal reflection point formed by the first light source and the second corneal reflection point formed by the second light source are obtained, and the reflection reference point of the human eye and the center point of the human eye pupil are determined. Based on the reflection reference point and the center point of the human eye's pupil, the viewpoint information of the human eye is obtained.

[0013] The gaze tracking method provided in this application embodiment controls a first light source to emit a first probe light and a second light source to emit a second probe light. Based on the first and second probe lights, an eye image of the human eye is acquired. A first corneal reflection point formed by the first light source and a second corneal reflection point formed by the second light source are obtained from the eye image to determine the reflection reference point of the human eye and the center point of the pupil. Finally, the viewpoint information of the human eye is obtained based on the reflection reference point and the center point of the pupil. This method can stably track the human eye's gaze. Compared to using a single infrared light source to acquire a single Pullchin spot to obtain the viewpoint information, the gaze tracking method in this application embodiment, by controlling the first and second light sources to illuminate the human eye, can obtain both the first and second corneal reflection points. This helps reduce errors caused by Pullchin spot deviation due to head movement, improves the stability of gaze tracking, and thus enhances the user experience.

[0014] In some embodiments, the step of obtaining a first corneal reflection point formed by the first light source and a second corneal reflection point formed by the second light source based on the eye image, and determining the reflection reference point of the human eye and the center point of the human eye pupil includes: A perspective transformation is performed on the image of the human eye to obtain a frontal view of the eye; Based on the frontal view of the eye, the first corneal reflection point formed by the first light source, the second corneal reflection point formed by the second light source, the first canthus point of the human eye, and the second canthus point are obtained. The reflection reference point of the human eye is calculated based on the first corneal reflection point and the second corneal reflection point; The center point of the human eye pupil is calculated based on the first and second corner points of the eye.

[0015] In some embodiments, the step of calculating the center point of the human eye pupil based on the first canthus point and the second canthus point includes: Vector normalization is performed based on the first corneal reflection point and the second corneal reflection point.

[0016] In some embodiments, the step of obtaining the viewpoint information of the human eye based on the reflection reference point and the center point of the human eye pupil includes: Vector calculation is performed based on the reflection reference point of the human eye and the center point; Based on a preset homography matrix, coordinate mapping is performed to obtain the viewpoint information of the human eye. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of a near-eye display device according to an embodiment of this application.

[0018] Figure 2 This is a schematic diagram showing the positions of the first light source and the second light source according to an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of an eye image obtained by the information acquisition module according to an embodiment of this application.

[0020] Figure 4 This is a schematic diagram showing the positions of the first and second light sources according to another embodiment of this application.

[0021] Figure 5 This is a schematic diagram showing the positions of the first and second light sources in yet another embodiment of this application.

[0022] Figure 6 This is a schematic diagram of an eye image obtained by the information acquisition module according to another embodiment of this application.

[0023] Figure 7 Histograms showing the number of viewpoint errors for embodiments of this application and a pair of proportions.

[0024] Figure 8 This is a schematic flowchart of an embodiment of the gaze tracking method of this application.

[0025] Figure 9 This is a flowchart illustrating step S300 of a gaze tracking method according to an embodiment of this application.

[0026] Figure 10 This is a flowchart illustrating step S400 of a gaze tracking method according to an embodiment of this application.

[0027] Explanation of key component symbols: Steps S100, S200, S300, S400, S31, S32, S33, S34, S41, S42 Near-eye display device 100 Eyeglasses frame 1 Frame 10 First frame section 101 Second frame section 102 Temples 12 Optical systems 11a, 11b Eye Tracking Module 2 First Light Source 21 First probe light L1 First corneal reflex point R1 Second light source 22 Second probe light L2 Second corneal reflex point R2 Reflection reference point R0 Information Acquisition Module 23 Photosensitive surface 231 The central axis M of the photosensitive surface Geometric Center 231a First corner of the eye J1 Second corner of the eye J2 C1, the center point of the human pupil Control Module 5 Distance S1, S2 First direction X Second direction Y Human eye E Rectangular regions Q1 and Q2 The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0029] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0030] To further illustrate the technical means and effects adopted by this application in achieving its intended purpose, the following detailed description of this application is provided in conjunction with the accompanying drawings and preferred embodiments.

[0031] Please refer to the following: Figure 1 and Figure 2 The near-eye display device 100 of this application embodiment includes a frame 1, two gaze tracking modules 2a and 2b disposed on the frame 1, and a control module 5. The frame 1 includes a lens frame 10 and temples 12. The lens frame 10 also includes a first lens frame portion 101 and a second lens frame portion 102 connected to each other, both of which are used to engage lenses. The control module 5 is disposed on one temple 12.

[0032] The two eye-tracking modules 2a and 2b have essentially the same structure and function. One eye-tracking module 2a is disposed on the first frame portion 101 and is used to acquire the user's left eye's viewpoint information; the other eye-tracking module 2b is disposed on the second frame portion 102 and is used to acquire the user's right eye's viewpoint information. In other embodiments, the near-eye display device 100 may also include a single eye-tracking module 2 or multiple eye-tracking modules 2, which is not limited in this application. The near-eye display device 100 also includes two optical systems 11a and 11b disposed on the frame 1. The two optical systems 11a and 11b have essentially the same structure and function. One optical system 11a is disposed on the first frame portion 101 and is used to image for the user's left eye; the other optical system 11b is disposed on the second frame portion 102 and is used to image for the user's right eye.

[0033] The near-eye display device 100 in this embodiment is an AR display device. When a user wears the AR display device, the user's eyes can observe the projected image displayed by the two optical systems 11a and 11b. At the same time, ambient light reflected from objects in the user's real environment can also enter the user's eye E through the near-eye display device 100, allowing the user's eye E to observe images of the real world. Furthermore, the user's eye E can observe the projected image superimposed on the real world image. In other embodiments, the near-eye display device 100 can also be an MR display device or an Extended Reality (XR) display device; this application does not impose any limitations.

[0034] The structure and function of the near-eye display device 100 are described below using one side as an example.

[0035] Please refer to the following: Figure 1 , Figure 2 and Figure 3 The gaze tracking module 2a of the near-eye display device 100 in this application embodiment includes a first light source 21, a second light source 22, and an information acquisition module 23. The first light source 21 emits a first detection light L1 toward the human eye E to illuminate the human eye E. The second light source 22 emits a second detection light L2 toward the human eye E to illuminate the human eye E. The information acquisition module 23 receives the first detection light L1 and the second detection light L2 reflected by the human eye E, and generates an eye image based on the first detection light L1 and the second detection light L2. The eye image includes a first corneal reflection point R1 formed on the human eye E by the first detection light L1 emitted by the first light source 21, and a second corneal reflection point R2 formed on the human eye E by the second light source 22. The control module 5 is electrically connected to the eye-tracking module 2 and is used to receive eye images. It is also used to obtain a reflection reference point R0 based on the first corneal reflection point R1 and the second corneal reflection point R2, and to obtain the center point C1 of the human eye pupil based on the eye image. The reflection reference point R0 and the center point C1 of the human eye pupil are used to obtain the viewpoint information of the human eye E. The control module 5 is also used to adjust the displayed image based on the viewpoint information of the human eye E.

[0036] The near-eye display device 100 provided in this application embodiment uses at least one gaze tracking module 2 and a control module 5. An information acquisition module 23 generates an eye image, which includes a first corneal reflection point R1 formed by a first light source 21 on the human eye E and a second corneal reflection point R2 formed by a second light source 22 on the human eye E. The control module 5 is used to obtain a reflection reference point R0 based on the first and second corneal reflection points R1 and R2, and to obtain the center point C1 of the human pupil based on the eye image. The reflection reference point R0 and the center point C1 of the human pupil can be used to obtain the human eye's... The viewpoint information of eye E can stably track the human eye E's gaze. Compared to the method of obtaining the viewpoint information of human eye E by setting a single infrared light source to acquire a single Puerchin spot, the near-eye display device 100 of this application embodiment can obtain the first corneal reflection point R1 and the second corneal reflection point R2 by setting the first light source 21 and the second light source 22. This is beneficial to reducing the error caused by the deviation of the Puerchin spot due to the user's head shaking, and is beneficial to improving the stability of gaze tracking. This is beneficial to improving the stability of the near-eye display device 100, and thus beneficial to improving the user's experience.

[0037] Both the first light source 21 and the second light source 22 are disposed on the first lens frame portion 101. In some embodiments, the first light source 21 and the second light source 22 are both infrared light-emitting diodes, and the wavelength range of the first detection light L1 and the second detection light L2 is 750nm-1100nm. Specifically, the wavelengths of the first detection light L1 and the second detection light L2 can be 750nm, 780nm, 800nm, 820nm, 840nm, 860nm, 880nm, 910nm, 930nm, 950nm, 970nm, 1000nm, or 1100nm. By setting the wavelengths of the first detection light L1 and the second detection light L2 to the above values, when the sensing light L1 is incident on the human eye EE, not only can the first detection light L1 and the second detection light L2 form a bright Pulcim spot on the corneal surface of the human eye EE, but also, since the human eye EE has low sensitivity to infrared light above 750nm, it is beneficial to avoid stimulating pupil contraction, thereby improving the accuracy of the information acquisition module 23 in acquiring the gaze information of the human eye EE.

[0038] Information acquisition module 23 is disposed on the first lens frame portion 101 and electrically connected to control module 5. In some embodiments, information acquisition module 23 is a camera, which is used to acquire an eye image generated based on a first probe light L1 and a second probe light L2. The eye image includes a first corneal reflection point R1 formed on the human eye E by a first light source 21, a second corneal reflection point R2 formed on the human eye E by a second light source 22, a first canthus point J1 and a second canthus point J2 of the human eye E. After receiving the eye image, control module 5 is used to acquire a reflection reference point R0 based on the first corneal reflection point R1 and the second corneal reflection point R2. The reflection reference point R0 is the midpoint between the first corneal reflection point R1 and the second corneal reflection point R2, that is, the distance from the reflection reference point R0 to the first corneal reflection point R1 and the second corneal reflection point R2 is equal. The control module 5 is also used to calculate the center point C1 of the human eye pupil based on the first corner point J1 and the second corner point J2 of the human eye E, where the center point C1 of the human eye pupil is the midpoint of the first corner point J1 and the second corner point J2, that is, the distance from the center point C1 of the human eye pupil to the first corner point J1 and the second corner point J2 is equal.

[0039] The information acquisition module 23 has a photosensitive surface 231 for receiving a first probe light L1 and a second probe light L2. The photosensitive surface 231 is the photosensitive surface of a camera. The information acquisition module 23 is located directly below the center point C1 of the human eye's pupil. The photosensitive surface 231 is approximately rectangular. The axis of symmetry of the photosensitive surface 231 is defined as its central axis M. The geometric center 231a of the photosensitive surface 231 passes through the central axis M of the photosensitive surface 231, which passes through the center point C1 of the human eye's pupil. The central axis M of the photosensitive surface 231 is parallel to a first direction.

[0040] Please refer to the following: Figure 2 and Figure 3In some embodiments, the information acquisition module 23 is located between the first light source 21 and the second light source 22. Specifically, the first light source 21, the second light source 22, and the information acquisition module 23 are all located below the human eye E; in other embodiments, the first light source 21, the second light source 22, and the information acquisition module 23 may also be located above the human eye E, and this application does not impose any restrictions. Furthermore, through a limited number of experiments, it was found that when the information acquisition module 23 is located between the first light source 21 and the second light source 22, and the first light source 21, the second light source 22, and the information acquisition module 23 are all located below the human eye E, the error value of the viewpoint information of the human eye E obtained when the first corneal reflection point R1 formed by the first light source 21 on the human eye E and the second corneal reflection point R2 formed by the second light source 22 on the human eye E fall within a rectangular area Q1 of 14 pixels × 10 pixels around the center point C1 of the human pupil is relatively small. By placing the first light source 21, the second light source 22, and the information acquisition module 23 below the human eye E, the influence of the eyelashes of the human eye E on the eye image acquired by the information acquisition module 23 can be reduced, which is beneficial to further reduce the error of gaze tracking and further improve the stability of gaze tracking.

[0041] In some embodiments, the ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y and the distance S2 between the second light source 22 and the information acquisition module 23 in the second direction Y is in the range of 1:0.3 to 1:0.8, that is, the second light source 22 is closer to the information acquisition module 23 than the first light source 21, and the second direction Y is perpendicular to the first direction X. For example, the ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y, and the distance S2 between the second light source 22 and the information acquisition module 23 in the second direction Y, can be 1:0.3, 1:0.4, 1:0.5, 1:0.6, 1:0.7, or 1:0.8. By setting the ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y, and the distance S2 between the second light source 22 and the information acquisition module 23 in the second direction Y, the distances S1 between the first light source 21 and the information acquisition module 23, and S2 between the second light source 22 and the information acquisition module 23, can be different. That is, the second light source 22 is closer to the information acquisition module 23 than the first light source 21. This helps to reduce the influence of the first detection light L1 emitted by the first light source 21 on the second detection light L2 emitted by the second light source 22, which helps to further reduce the error of eye tracking and further improve the stability of eye tracking. In other embodiments, the first light source 21 can also be closer to the information acquisition module 23 than the second light source 22, which is not a limitation of this application.

[0042] Please refer to the following: Figure 4 , Figure 5 and Figure 6In some embodiments, the first light source 21 is located between the information acquisition module 23 and the second light source 22; the arrangement direction of the first light source 21 and the second light source 22 is parallel to the first direction X. The first light source 21 and the second light source 22 are disposed on the side of the information acquisition module 23 near the second lens frame portion 102; in other embodiments, the first light source 21 and the second light source 22 may also be disposed on the side of the information acquisition module 23 away from the second lens frame portion 102, and this application does not impose any limitations. In addition, through a limited number of experiments, it was found that when the first light source 21 is located between the information acquisition module 23 and the second light source 22; and the first light source 21 and the second light source 22 are disposed on the side of the information acquisition module 23 near or away from the second lens frame portion 102, the error value of the viewpoint information of the human eye E obtained when the first corneal reflection point R1 formed by the first light source 21 on the human eye E and the second corneal reflection point R2 formed by the second light source 22 on the human eye E fall on two 10-pixel × 10-pixel rectangular regions Q2 symmetrically disposed on both sides of the center point C1 of the human pupil is small.

[0043] The ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y, and the distance S3 between the second light source 22 and the first light source 21 in the first direction X, is in the range of 1:0.7 to 1:1, where the second direction Y is perpendicular to the first direction X. For example, the ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y, and the distance S3 between the second light source 22 and the first light source 21 in the first direction X, can be 1:0.7, 1:0.8, 1:0.9, or 1:1. By setting the ratio of the distance S1 between the first light source 21 and the information acquisition module 23 in the second direction Y, and the distance S3 between the second light source 22 and the first light source 21 in the first direction X, to the above-mentioned range, the second light source 22 can be further away from the information acquisition module 23 than the first light source 21. This helps to reduce the influence of the first detection light L1 emitted by the first light source 21 on the second detection light L2 emitted by the second light source 22, further reducing the error of eye tracking and further improving the stability of eye tracking. In other embodiments, the first light source 21 may be further away from the information acquisition module 23 than the second light source 22, and this application does not impose any restrictions.

[0044] Please refer to the following: Figure 1 , Figure 2 and Figure 3The control module 5 includes a memory (not shown) and a processor (not shown). The memory stores the computer program and the basic processing parameters corresponding to the preset processing mode. The processor is used to execute the computer program. The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0045] The near-eye display device 100 provided in this application embodiment uses at least one gaze tracking module 2 and a control module 5. The gaze tracking module 2 includes a first light source 21 and a second light source 22 to illuminate the human eye E. An information acquisition module 23 generates an eye image, which includes a first corneal reflection point R1 formed by the first light source 21 on the human eye E and a second corneal reflection point R2 formed by the second light source 22 on the human eye E. The control module 5 obtains a reflection reference point R0 based on the first and second corneal reflection points R1 and R2, and obtains the center point C1 of the human eye's pupil based on the eye image. The reflection reference point R0 and the center point C1 of the human eye's pupil can acquire the viewpoint information of the human eye E, enabling stable gaze tracking of the human eye E. Please refer to [further details omitted]. Figure 2 and Figure 7 , Figure 7 This is a histogram of the number of viewpoint errors in the embodiments of this application and a comparative example. The comparative example uses a method of obtaining a single Puerchin spot by setting a single infrared light source to obtain the human eye's E viewpoint information. Compared with the method of obtaining a single Puerchin spot by setting a single infrared light source to obtain the human eye's E viewpoint information, the near-eye display device 100 provided in the embodiments of this application has significantly fewer viewpoint errors. The near-eye display device 100 in the embodiments of this application can obtain a first corneal reflection point R1 and a second corneal reflection point R2 by setting a first light source 21 and a second light source 22. This helps to reduce the error caused by the deviation of the Puerchin spot due to the user's head shaking, and helps to improve the stability of eye tracking, thereby improving the stability of the near-eye display device 100 and thus improving the user's experience.

[0046] Please refer to the following: Figure 1 and Figure 8 This application embodiment also provides a gaze tracking method, including the following steps S100-S400. The control module in the near-eye display device of this application embodiment is used to perform the following steps S100-S400: Step S100: Control the first light source to emit the first probe light and the second light source to emit the second probe light.

[0047] Step S200: The control information acquisition module receives the first detection light and the second detection light reflected by the human eye, and acquires an eye image of the human eye based on the first detection light and the second detection light.

[0048] Step S300: Based on the eye image, obtain the first corneal reflection point formed by the first light source and the second corneal reflection point formed by the second light source, and determine the reflection reference point of the human eye and the center point of the human eye pupil.

[0049] Step S400: Obtain the viewpoint information of the human eye based on the reflection reference point and the center point of the human eye pupil.

[0050] The gaze tracking method provided in this application embodiment can stably track the human eye. Compared with the method of setting a single infrared light source to obtain a single Puerchin spot to obtain the human eye's viewpoint information, the gaze tracking method in this application embodiment can obtain a first corneal reflection point and a second corneal reflection point by controlling the first light source and the second light source to illuminate the human eye. This helps to reduce the error caused by the Puerchin spot deviation due to the user's head shaking, and helps to improve the stability of gaze tracking, thereby improving the user's experience.

[0051] Please refer to the following: Figure 1 , Figure 2 and Figure 8 In step S100, both the first light source 21 and the second light source 22 are infrared light-emitting diodes, and both the first light source 21 and the second light source 22 are disposed on the first mirror frame portion 101.

[0052] Specifically, in step S200, control module 5 acquires an eye image of the human eye E through control information acquisition module 23. Please refer to [the relevant documentation / reference]. Figure 2 and Figure 3 The information acquisition module 23 is located between the first light source 21 and the second light source 22; or the first light source 21 is located between the information acquisition module 23 and the second light source 22; the arrangement direction of the first light source 21 and the second light source 22 is parallel to the first direction X; the first light source 21 and the second light source 22 are disposed on the side of the information acquisition module 23 that is close to or away from the second frame portion 102.

[0053] Please refer to the following: Figure 1 , Figure 2 and Figure 9 The steps in step S300 include: Step S31: Perform perspective transformation on the human eye image to obtain a frontal view of the eye.

[0054] Step S32: Based on the frontal view of the eye, obtain the first corneal reflection point formed by the first light source, the second corneal reflection point formed by the second light source, the first canthus point of the human eye, and the second canthus point.

[0055] Step S33: Calculate the human eye's reflection reference point based on the first corneal reflection point and the second corneal reflection point.

[0056] Step S34: Calculate the center point of the human eye pupil based on the first and second canthus points.

[0057] Specifically, in step S31, in order to avoid the user turning their head or shaking their head from affecting the eye image, it is necessary to first perform perspective transformation on the eye image of the human eye E through image processing to obtain a frontal view of the eye. If the obtained eye image is not significantly skewed, step S31 may not be performed, and this application does not impose any restrictions.

[0058] After perspective transformation, the frontal view of the eye in step S32 includes the first corneal reflection point R1 formed by the first light source 21 on the human eye E, the second corneal reflection point R2 formed by the second light source 22 on the human eye E, the first corner point J1 and the second corner point J2 of the human eye E. It is necessary to establish a rectangular coordinate system with any point in the frontal view of the eye as the origin, and obtain the coordinates (X1, Y1) of the first corneal reflection point R1 formed by the first light source 21 on the human eye E, the coordinates (X2, Y2) of the second corneal reflection point R2 formed by the second light source 22 on the human eye E, the coordinates (X3, Y3) of the first corner point J1 of the human eye E, and the coordinates (X4, Y4) of the second corner point J2 of the human eye E.

[0059] In step S33, based on the coordinates (X1, Y1) of the first corneal reflection point R1 and the coordinates (X2, Y2) of the second corneal reflection point R2, the coordinates (Xr, Yr) of the human eye E's reflection reference point R0 are calculated using the formulas Xr=(X1+X2)÷2 and Yr=(Y1+Y2)÷2. The reflection reference point R0 is the midpoint between the first corneal reflection point R1 and the second corneal reflection point R2, meaning that the distances from the reflection reference point R0 to the first corneal reflection point R1 and the second corneal reflection point R2 are equal.

[0060] In step S34, based on the coordinates (X3, Y3) of the first canthus point J1 and (X4, Y4) of the second canthus point J2 of the human eye E, the coordinates (Xc, Yc) of the center point C1 of the human eye pupil are calculated using the formulas Xc=(X3+X4)÷2 and Yc=(Y3+Y4)÷2. The center point C1 of the human eye pupil is the midpoint between the first canthus point J1 and the second canthus point J2, meaning that the distances from the center point C1 of the human eye pupil to the first canthus point J1 and the second canthus point J2 are equal.

[0061] The steps following step S34 include: Step S341: Perform vector normalization based on the first corneal reflection point R1 and the second corneal reflection point R2. This is done using the formula ((X1–X2)). 2 + (Y1–Y2) 2 ) 1 / 2 The normalized magnitudes of the coordinate vectors of the first corneal reflection point R1 and the second corneal reflection point R2 are calculated. Vector normalization helps to further reduce the error in gaze tracking and improve its stability.

[0062] Please refer to the following: Figure 2 and Figure 10 The steps in step S300 include: Step S41: Perform vector calculation based on the human eye's reflection reference point and center point.

[0063] Step S42: Perform coordinate mapping based on the preset homography matrix to obtain the viewpoint information of the human eye.

[0064] After obtaining the coordinates (Xr, Yr) of the reflection reference point R0 of the human eye E and the coordinates (Xc, Yc) of the center point C1 of the human eye pupil in step S41, the vector from the reflection reference point R0 coordinates (Xr, Yr) to the center point coordinates (Xc, Yc) can be obtained. Before executing the gaze tracking method, the control module 5 has already stored a preset homography matrix. By establishing a mapping model (such as multinomial regression or neural network) between the vector from the reflection reference point R0 coordinates (Xr, Yr) to the center point coordinates (Xc, Yc) and the preset homography matrix, the viewpoint information of the human eye E can be obtained.

[0065] The gaze tracking method provided in this application embodiment controls the first light source 21 and the second light source 22 to illuminate the human eye E, then acquires an eye image of the human eye E, and acquires the first corneal reflection point R1 formed by the first light source 21 and the second corneal reflection point R2 formed by the second light source 22 based on the eye image to determine the reflection reference point R0 of the human eye E and the center point C1 of the human eye pupil. Finally, based on the reflection reference point R0 and the center point C1 of the human eye pupil, the viewpoint information of the human eye E is obtained, which can stably track the human eye E. Compared with the method of setting a single infrared light source to acquire a single Puerchin spot to obtain the viewpoint information of the human eye E, the gaze tracking method of this application embodiment can obtain the first corneal reflection point R1 and the second corneal reflection point R2 by controlling the first light source 21 and the second light source 22 to illuminate the human eye E. This helps to reduce the error caused by the deviation of the Puerchin spot due to the user's head shaking, improves the stability of gaze tracking, and thus improves the user experience.

[0066] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A near-eye display device, characterized in that, include: At least one eye-tracking module includes a first light source, a second light source, and an information acquisition module; the first light source is used to emit a first detection light toward the human eye; The second light source is used to emit a second detection light toward the human eye; The information acquisition module is used to receive the first detection light and the second detection light reflected by the human eye, and to generate an eye image based on the first detection light and the second detection light. The eye image includes a first corneal reflection point formed on the human eye by the first detection light emitted from the first light source and a second corneal reflection point formed on the human eye by the second light source. as well as A control module, electrically connected to the gaze tracking module, is used to receive the eye image and to obtain a reflection reference point based on the first corneal reflection point and the second corneal reflection point, and to obtain the center point of the human eye pupil based on the eye image. The reflection reference point and the center point of the human eye pupil are used to obtain the viewpoint information of the human eye. The control module is also used to adjust the displayed image based on the viewpoint information of the human eye.

2. The near-eye display device as described in claim 1, characterized in that, The near-eye display device further includes a frame, which includes a lens frame and a first lens frame portion and a second lens frame portion connected to each other. Both the first lens frame portion and the second lens frame portion are used to engage lenses. The at least one gaze tracking module is disposed on the first lens frame portion or the second lens frame portion.

3. The near-eye display device as described in claim 2, characterized in that, The first light source, the second light source, and the information acquisition module are all disposed on the first lens frame. The information acquisition module has a photosensitive surface for receiving the first detection light and the second detection light. The central axis of the photosensitive surface passes through the center point of the pupil of the human eye, and the central axis of the photosensitive surface is parallel to the first direction.

4. The near-eye display device as described in claim 3, characterized in that, The information acquisition module is located between the first light source and the second light source; The first light source and the information acquisition module are at a distance S1 in the second direction, and the second light source and the information acquisition module are at a distance S2 in the second direction. The ratio of the distance S1 to the distance S2 is in the range of 1:0.3 to 1:0.8, and the second direction is perpendicular to the first direction.

5. The near-eye display device as described in claim 3, characterized in that, The first light source is located between the information acquisition module and the second light source; the arrangement direction of the first light source and the second light source is parallel to the first direction; The first light source and the second light source are disposed on the side of the information acquisition module that is close to or away from the second frame portion.

6. The near-eye display device as described in claim 5, characterized in that, The ratio of the distance S1 between the first light source and the information acquisition module in the second direction to the distance S3 between the second light source and the first light source in the first direction is in the range of 1:0.7 to 1:1, and the second direction is perpendicular to the first direction.

7. A gaze tracking method, characterized in that, include: Control the first light source to emit the first probe light and the second light source to emit the second probe light; The control information acquisition module receives the first detection light and the second detection light reflected by the human eye, and acquires an eye image of the human eye based on the first detection light and the second detection light; Based on the eye image, the first corneal reflection point formed by the first light source and the second corneal reflection point formed by the second light source are obtained, and the reflection reference point of the human eye and the center point of the human eye pupil are determined. Based on the reflection reference point and the center point of the human eye's pupil, the viewpoint information of the human eye is obtained.

8. The gaze tracking method as described in claim 7, characterized in that, The steps of obtaining a first corneal reflection point formed by the first light source and a second corneal reflection point formed by the second light source based on the eye image, and determining the reflection reference point of the human eye and the center point of the human eye pupil include: A perspective transformation is performed on the image of the human eye to obtain a frontal view of the eye; Based on the frontal view of the eye, the first corneal reflection point formed by the first light source, the second corneal reflection point formed by the second light source, the first canthus point of the human eye, and the second canthus point are obtained. The reflection reference point of the human eye is calculated based on the first corneal reflection point and the second corneal reflection point; The center point of the human eye pupil is calculated based on the first and second corner points of the eye.

9. The gaze tracking method as described in claim 8, characterized in that, The steps following the calculation of the center point of the human eye pupil based on the first and second canthal points include: Vector normalization is performed based on the first corneal reflection point and the second corneal reflection point.

10. The gaze tracking method as described in claim 8, characterized in that, The steps for obtaining the viewpoint information of the human eye based on the reflection reference point and the center point of the human eye pupil include: Vector calculation is performed based on the reflection reference point of the human eye and the center point; Based on a preset homography matrix, coordinate mapping is performed to obtain the viewpoint information of the human eye.

Images