Eye tracking method and near-eye display device

Abstract

An embodiment of the present invention provides an eye tracking method and a near-eye display device. The near-eye display device includes a light source assembly, an optical waveguide, an image acquisition unit, and a processing unit. The light source assembly includes a visible light source and a plurality of infrared light sources arranged at intervals. The image acquisition unit is used to capture an image of the eye. The eye tracking method includes: determining the position of a preset eye area based on the image of the eye; determining a target infrared light source from a plurality of infrared light sources based on the position of the preset eye area; controlling the target infrared light source to emit infrared light to the pupil area of ​​the eye; and determining the gaze direction of the eye based on the light spot formed by the infrared light in the image of the eye. The technical solution of the embodiment of the present invention is intended to enable the infrared light to be accurately transmitted to the pupil area when the pupil area of ​​the eye is in different positions, thereby achieving high-precision eye tracking.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to an eye tracking method and a near-eye display device. Background Art

[0002] Near-eye display devices usually track the movement state of the human eye and control the near-eye display device according to the movement state of the human eye to achieve human-computer interaction. Specifically, the near-eye display device can illuminate the pupil with infrared light and form a bright spot on the cornea, and then collect the bright spot image formed on the cornea, determine the movement state of the human eye based on the bright spot image, and achieve eye tracking. However, in the near-eye display device in the related art, when the human eye looks straight ahead, the infrared light shines on the cornea, forming a clear bright spot on the cornea, but when the human eye looks sideways, the infrared light shines on the sclera and is diffusely reflected, forming scattered speckles on the sclera, and the bright spot formed on the cornea is reduced or even disappears, resulting in the collected bright spot image being difficult to achieve high-precision eye tracking, and it may even be impossible to use the collected bright spot image to achieve eye tracking. Summary of the Invention

[0003] Embodiments of the present invention provide an eye tracking method and a near-eye display device, which aim to enable infrared light to be accurately transmitted to the pupil area when the pupil area of ​​the eye is in different positions, thereby achieving high-precision eye tracking.

[0004] In a first aspect, an embodiment of the present invention provides an eye tracking method based on a near-eye display device, comprising:

[0005] A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals;

[0006] an optical waveguide, into which the visible light can be coupled and then transmitted and then coupled out from the optical waveguide;

[0007] An image acquisition unit, configured to acquire images of the eye;

[0008] The eye tracking method comprises:

[0009] acquiring an image of the eye;

[0010] determining a position of a preset eye area according to the image;

[0011] Determining a target infrared light source from the plurality of infrared light sources according to the position of the preset eye area;

[0012] Controlling the target infrared light source to emit the infrared light to the pupil area of ​​the eye;

[0013] acquiring an image of the eye again;

[0014] The gaze direction of the eye is determined based on the light spot formed by the infrared light in the image acquired again.

[0015] Optionally, the preset eye area includes at least one of the following: the pupil area, the cornea area;

[0016] The distance between the position of the target infrared light source and the pupil area is smaller than the distance between other infrared light sources in the plurality of infrared light sources and the pupil area; and / or

[0017] The distance between the target infrared light source and the cornea region is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the cornea region.

[0018] Optionally, the preset eye area includes the pupil area, cornea area and sclera area;

[0019] The determining the position of the preset eye area according to the eye image includes:

[0020] determining a positional relationship among the pupil area, the cornea area, and the sclera area based on the image of the eye;

[0021] The step of determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area includes:

[0022] A target infrared light source is determined among the plurality of infrared light sources according to a positional relationship among the pupil area, the cornea area, and the sclera area.

[0023] Optionally, when determining a target infrared light source from the plurality of infrared light sources according to the position of the preset eye area, the method is used to:

[0024] Based on the preset correspondence between the plurality of infrared light sources and the positions of the preset eye area, the target infrared light source is determined among the plurality of infrared light sources according to the position of the preset eye area.

[0025] Optionally, a plurality of the infrared light sources are provided in the optical waveguide.

[0026] Optionally, the plurality of infrared light sources surround the outer periphery of the outcoupling structure of the optical waveguide, and the plurality of infrared light sources emit the infrared light toward a side away from the optical waveguide, and the transmission direction of the infrared light is inclined toward the side where the outcoupling structure is located.

[0027] Optionally, the near-eye display device also includes multiple phase modulators, which correspond one-to-one to the multiple infrared light sources. The phase modulators are arranged on the light-emitting side of the corresponding infrared light source, and the phase modulators are used to modulate the infrared light emitted by the corresponding infrared light source so that the transmission direction of the infrared light is inclined toward the side where the infrared light source is located.

[0028] Optionally, the phase modulator includes a lens or a liquid crystal phase modulator.

[0029] Optionally, the plurality of infrared light sources are arranged evenly; and / or the plurality of infrared light sources are arranged in an elliptical shape.

[0030] In a second aspect, an embodiment of the present invention further provides a near-eye display device, comprising:

[0031] a visible light source, for emitting visible light;

[0032] an optical waveguide, into which the visible light can be coupled and then transmitted, and then coupled out from the optical waveguide to the eye or a predetermined imaging component;

[0033] A plurality of infrared light sources, wherein the plurality of infrared light sources are arranged at intervals;

[0034] A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals;

[0035] An optical waveguide, into which the visible light emitted by the visible light source can be coupled and then transmitted and then coupled out from the optical waveguide;

[0036] An image acquisition unit, configured to acquire an image of the eye;

[0037] A processing unit is electrically connected to the image acquisition unit and the plurality of infrared light sources, and the processing unit is used to execute the steps of the eye tracking method as described in the first aspect.

[0038] An embodiment of the present invention provides an eye tracking method and a near-eye display device. The method obtains an image of the eye captured by an image acquisition unit, determines the position of a preset eye area based on the image of the eye, and determines a target infrared light source based on the position of the preset eye area, so that the target infrared light source emits infrared light to the pupil area of ​​the eye and forms a light spot in the cornea area of ​​the eye. Thus, the gaze direction of the eye can be determined based on the light spot formed by the infrared light in the image of the eye captured by the image acquisition unit again, thereby achieving eye tracking. Multiple infrared light sources are used to emit infrared light, thereby expanding the range that infrared light can cover and increasing the range of eye tracking. A target infrared light source is determined from the multiple infrared light sources based on the position of the preset eye area. The infrared light emitted by the target infrared light source can be accurately transmitted to the pupil area of ​​the eye, ensuring that a clear light spot can be formed in the cornea area. Other infrared light sources in the multiple infrared light sources do not emit infrared light, thereby avoiding interference and improving the accuracy of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 Schematic diagram of forming a light spot on the eye in the related technology provided by the present invention:

[0041] Figure 2 A structural block diagram of a near-eye display device provided by one embodiment of the present invention;

[0042] Figure 3 A flowchart of an eye tracking method provided by one embodiment of the present invention;

[0043] Figure 4 A schematic diagram of forming a light spot on the eye according to an embodiment of the present invention;

[0044] Figure 5 A schematic structural diagram of a near-eye display device provided by one embodiment of the present invention;

[0045] Figure 6 A diagram showing the states of multiple infrared light sources when the eyes are looking straight ahead, provided by one embodiment of the present invention;

[0046] Figure 7 A diagram showing the state of multiple infrared light sources when the eye is looking forward to the left, according to one embodiment of the present invention;

[0047] Figure 8 A diagram showing the state of multiple infrared light sources when the eye is looking right forward, according to one embodiment of the present invention;

[0048] Figure 9 A state diagram of a near-eye display device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0051] It should be understood that the terms used in this specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] The following embodiments of the present invention are described in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0053] like Figure 1 As shown, in the related art, when a user wears a near-eye display device and the user's eyes look straight ahead, infrared light directly enters the eyes, forming a clear light spot in the cornea area of ​​the eyes. When the user's eyes turn to look in other directions, for example, looking to the left front, infrared light enters the eyes, is diffusely reflected by the sclera of the eyes to form stray light, and forms speckles on the sclera of the eyes, while fewer or even no light spots are formed in the cornea area of ​​the eyes, resulting in poor eye tracking accuracy, or the inability to perform eye tracking, resulting in limited accuracy and range of eye tracking.

[0054] To address the above issues, embodiments of the present invention provide a near-eye display device, which may include augmented reality (AR) glasses, virtual reality (VR) glasses, AR helmets, VR helmets, and the like, without limitation.

[0055] Please refer to Figure 2 and Figure 5The near-eye display device 100 includes a light source assembly, an optical waveguide 20, an image acquisition unit 40, and a processing unit 50. The light source assembly includes a visible light source 10 and a plurality of infrared light sources 30 arranged at intervals. The image acquisition unit 40 is used to acquire an image of the eye.

[0056] Exemplarily, the plurality of infrared light sources 30 are arranged at intervals, and when the plurality of infrared light sources 30 all emit infrared light, the infrared light can cover the movement range of the pupil region 21 of the eye.

[0057] For example, the visible light may be light with a wavelength range of 400 nm to 760 nm, and the infrared light may be light with a wavelength range of 780 nm to 940 nm.

[0058] like Figure 3 As shown, the eye tracking method based on the above-mentioned near-eye display device includes steps S101 to S106.

[0059] Step S101: Acquire an image of the eye.

[0060] Exemplarily, the image of the eye captured by the image acquisition unit 40 is acquired.

[0061] Step S102: Determine the position of the preset eye area according to the image.

[0062] For example, the position of the preset eye area can be determined by performing image recognition on the image.

[0063] Step S103: determining a target infrared light source from a plurality of infrared light sources according to the position of the preset eye area.

[0064] It is understood that the target infrared light source may include one or more infrared light sources 30. For example, the near-eye display device includes six infrared light sources 30, and the target infrared light source may be one, two, or three infrared light sources 30 determined according to the location of the preset eye area.

[0065] Step S104: controlling the target infrared light source to emit infrared light to the pupil area of ​​the eye.

[0066] As can be understood, the image acquisition unit 40 acquires an image of the eye again. Since the target infrared light source emits infrared light to the pupil region 21 of the eye, the infrared light can form a light spot in the cornea region 22 of the eye. Therefore, the image of the eye acquired again includes the light spot formed by the infrared light in the cornea region 22 of the eye.

[0067] Step S105: Acquire the eye image again.

[0068] Step S106: Determine the gaze direction of the eye based on the light spot formed by the infrared light in the re-acquired image.

[0069] Exemplarily, the image acquisition unit 40 may include an infrared camera, which acquires an image of the eye through the infrared camera, determines the position of a preset area of ​​the eye based on the image of the eye, and acquires the image of the eye again through the infrared camera to determine the gaze direction of the eye based on the light spot formed by the infrared light in the again acquired image of the eye.

[0070] Exemplarily, the image acquisition unit 40 also includes a visible light camera and an infrared camera. The visible light camera includes a CCD (Charge Coupled Device) camera and a CMOS (Complementary Metal Oxide Semiconductor) camera. The image of the eye is acquired by the visible light camera, and the position of the preset area of ​​the eye is determined based on the image of the eye. The image of the eye is acquired again by the infrared camera to determine the gaze direction of the eye based on the light spot formed by the infrared light in the again acquired image of the eye.

[0071] For example, when a user wears the near-eye display device 100, after the image acquisition unit 40 captures an image of the eye, it can determine the location of a predetermined eye region based on the acquired eye image, such as the location of the pupil region 21 or cornea region 22. Then, based on the location of the predetermined eye region, a target infrared light source is determined from among the multiple infrared light sources 30, and the target infrared light source is controlled to emit infrared light toward the user's eye. This infrared light is able to fall on the user's pupil region 21 and form a clear light spot on the cornea region 22. It will be understood that the target infrared light source determined based on the location of the predetermined eye region is located in a manner associated with the location of the pupil region 21. The infrared light emitted by the target infrared light source is able to fall on the pupil region 21 and form a clear light spot on the cornea region 22. After the image acquisition unit 40 captures another image of the eye, it can determine the eye's gaze direction based on the light spot formed by the infrared light in the newly captured eye image, thereby achieving eye tracking.

[0072] The near-eye display device 100 of an embodiment of the present invention arranges multiple infrared light sources 30 at intervals to ensure that when the pupil region 21 of the eye is at any position, infrared light emitted by one or more infrared light sources 30 can fall on the pupil region 21, forming a light spot in the cornea region 22, thereby achieving eye tracking and increasing the range of eye tracking. Furthermore, after determining the position of the preset eye region based on the image of the eye, one or more infrared light sources 30 are determined as target infrared light sources based on the position of the preset eye region. The target infrared light sources emit infrared light, improving the position transmission accuracy of the emitted infrared light, allowing the infrared light to be more accurately transmitted to the pupil region 21, forming a clear light spot in the cornea region 22. The other infrared light sources 30 do not emit infrared light, preventing the infrared light emitted by the other infrared light sources 30 from interfering with the light spot in the cornea region 22. This helps to make the light spot in the re-captured eye image clearer and achieve high-precision eye tracking.

[0073] Exemplarily, the position of the preset eye area is determined based on the image of the eye, where the position of the preset eye area can be the position of the preset eye area in the image of the eye, or the position of the preset eye area in the world coordinate system, which can be adjusted according to actual conditions.

[0074] Exemplarily, the eye gaze direction is determined, and the eye movement state can be determined based on the eye gaze direction. For example, by comparing the eye gaze directions in different time frames, it can be determined whether the eyes are in a gaze state, a movement state, or a saccade state. When the deviation between the eye gaze directions in at least two adjacent time frames is less than or equal to a first preset difference, it is determined that the user's eyes are in a gaze state; when the deviation between the eye gaze directions in at least two adjacent time frames is greater than the first preset difference and less than or equal to a second preset difference, it is determined that the user's eyes are in a moving tracking state; when the deviation between the eye gaze directions in at least two adjacent time frames is greater than the second preset difference, it is determined that the user's eyes are in a saccade state.

[0075] Figure 4 FIG2 shows a schematic diagram of the infrared light falling on the pupil area 21 of the eye and forming a light spot on the cornea area 22 of the eye when the near-eye display device 100 in the embodiment of the present invention performs eye tracking. Figure 4 As shown, when the eyes look straight ahead or to the left front, the infrared light emitted by the target infrared light source falls on the pupil area 21 of the eye, forming a clear light spot in the cornea area 22 of the eye to achieve high-precision eye tracking.

[0076] In some embodiments, the preset eye area includes at least one of the following: pupil area 21 and cornea area 22. The distance between the position of the target infrared light source and pupil area 21 is smaller than the distance between other infrared light sources 30 in the plurality of infrared light sources 30 and pupil area 21, and / or the distance between the position of the target infrared light source and cornea area 22 is smaller than the distance between other infrared light sources 30 in the plurality of infrared light sources 30 and cornea area 22. It will be understood that when the eye looks in different directions, the areas in the eye image corresponding to the pupil area 21 and the cornea area 22, respectively, have a corresponding positional relationship. Therefore, after determining the position of the cornea area 22 based on the eye image, the position of the pupil area 21 is also determined, and the target infrared light source can be directly determined based on the pupil area 21 and / or the target infrared light source can be determined based on the cornea area 22.

[0077] For example, the distance between the infrared light source 30 and the pupil area 21 can be represented by the distance between the geometric center of the infrared light source 30 and the center of the pupil area 21. The distance between the infrared light source 30 and the cornea area 22 can be represented by the distance between the geometric center of the infrared light source 30 and the center of the cornea area 22.

[0078] For example, Figure 6 、 Figure 7 and Figure 8 The diagram shows the on and off states of multiple infrared light sources 30 when the eyes are observed in different directions while the eyes are looking in the direction facing the eyes. The dotted lines in the diagram represent the transmission paths of the infrared light. Figure 6 The diagram shows an image of the eye captured by the image acquisition unit 40 when the eye is looking straight ahead. The pupil area 21 and the cornea area 22 of the eye are located in the relative middle of the image of the eye. Five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit. Figure 7 The diagram shows an image of the eye captured by the image acquisition unit 40 when the eye is looking forward to the left. The pupil area 21 and the cornea area 22 of the eye are located on the relatively right side of the image of the eye. The five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit. Figure 8 The diagram shows an image of the eye captured by the image acquisition unit 40 when the eye is looking to the right front. The pupil area 21 and the cornea area 22 of the eye are located on the relatively left side of the image of the eye. The five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit.

[0079] For example, Figure 9When a user wears the near-eye display device 100, a top view of the user's eyes and the near-eye display device 100 is provided. Figure 9 The dotted line represents the transmission path of infrared light, and the dotted line represents the transmission path of visible light. Figure 9 As shown in Figure (A), when the eye is looking to the right and front, the distance between the infrared light source 30 on the right side of the eye and the pupil area 21 is smaller than the distance between the infrared light source 30 on the left side and the pupil area 21. The infrared light source 30 on the right side is lit as the target infrared light source and is controlled to emit infrared light toward the pupil area 21. Figure 9 As shown in Figure (B), when the eye is looking to the left front, the distance between the infrared light source 30 on the left side of the eye and the pupil area 21 is smaller than the distance between the infrared light source 30 on the right side and the pupil area 21. The infrared light source 30 on the left side is lit as the target infrared light source and is controlled to emit infrared light to the pupil area 21.

[0080] In some embodiments, the plurality of infrared light sources 30 may be located on the same plane, and the target infrared light source may be determined based on the position of the projection of the preset eye area on the plane.

[0081] Exemplarily, the distance between the position of the target infrared light source and the projection of the pupil area 21 of the eye on the plane is smaller than the distance between the other infrared light sources 30 and the projection of the pupil area 21 of the eye on the plane.

[0082] In some embodiments, the preset eye region includes a pupil region 21, a cornea region 22, and a sclera region 23. When determining the position of the preset eye region based on an image of the eye, the method is used to: determine the positional relationship between the pupil region 21, the cornea region 22, and the sclera region 23 based on the image of the eye. When determining a target infrared light source from among the plurality of infrared light sources 30 based on the position of the preset eye region, the method is used to: determine the target infrared light source from among the plurality of infrared light sources 30 based on the positional relationship between the pupil region 21, the cornea region 22, and the sclera region 23. It can be understood that the position of the image acquisition unit 40 in the near-eye display device 100 is fixed. When the eye looks in different directions, the relative relationship between the pupil area 21, the cornea area 22 and the sclera area 23 in the image of the eye captured by the image acquisition unit 40 is different. Therefore, based on the positional relationship between the pupil area 21, the cornea area 22 and the sclera area 23, the gaze direction of the eye can be preliminarily determined, that is, the position of the pupil area 21 to which the infrared light needs to be transmitted is determined, and then the target infrared light source is determined according to the preliminarily determined gaze direction of the eye, so that the infrared light emitted by the target infrared light source can be accurately transmitted to the pupil area 21 of the eye, further determining the gaze direction of the eye, and achieving high-precision eye tracking.

[0083] For example, Figure 6An image of the eye is shown when the eye is looking straight ahead, in which the pupil region 21 and the cornea region 22 are located relatively centrally with respect to the sclera region 23 . Figure 7 An image of the eye is shown when the eye is looking diagonally toward the left front. In this image, the pupil region 21 and the cornea region 22 are located to the right of the sclera region 23 . Figure 8 An image of the eye is shown when the eye is looking diagonally toward the right front. In this image, the pupil region 21 and the cornea region 22 are located to the left of the sclera region 23 .

[0084] In some embodiments, when determining a target infrared light source from among the plurality of infrared light sources 30 based on the location of a preset eye area, the method is as follows: based on the correspondence between the plurality of preset infrared light sources 30 and the location of the preset eye area, the target infrared light source is determined from among the plurality of infrared light sources 30 based on the location of the preset eye area. It will be appreciated that, depending on the position of the preset eye area, one or more corresponding infrared light sources 30 may be pre-set. After the location of the preset eye area is determined based on the eye image, the one or more pre-set infrared light sources 30 may be illuminated as the target infrared light source.

[0085] For example, if Figure 9 As shown, the infrared light source 30 on the left side of the preset eye corresponds to the position of the pupil area 21 when it is located on the left side in the width direction of the eye, and the infrared light source 30 on the right side corresponds to the position of the pupil area 21 when it is located on the right side in the width direction of the eye. After the position of the pupil area 21 is determined according to the image of the eye, if the pupil area 21 is located on the left side in the width direction of the eye, the infrared light source 30 on the left side can be lit as the target infrared light source; if the pupil area 21 is located on the right side in the width direction of the eye, the infrared light source 30 on the right side can be lit as the target infrared light source.

[0086] like Figure 5 and Figure 9 As shown, in some embodiments, multiple infrared light sources 30 are provided in the optical waveguide 20. It can be understood that the optical waveguide 20 is usually located in front of the user's eyes, and the optical waveguide 20 is used as a carrier to carry the infrared light source 30 to facilitate the installation of the infrared light source 30.

[0087] Exemplarily, the optical waveguide 20 has a display side. When a user wears the near-eye display device 100, the user's eyes are located on the display side of the optical waveguide 20, allowing the optical waveguide 20 to couple visible light to the user's eyes. The infrared light source 30 can be located on the display side of the optical waveguide 20 to emit infrared light directly toward the pupil area 21 of the user's eyes, reducing light loss. Alternatively, the optical waveguide 20 has a non-display side disposed opposite the display side, and multiple infrared light sources 30 are disposed on the non-display side. The multiple infrared light sources 30 can correspond to the position of the coupling structure, and the infrared light can pass through the coupling structure and be emitted to the user's eyes.

[0088] Exemplarily, the optical waveguide 20 has an optical waveguide and a coupling structure and a coupling structure provided on the optical waveguide. Visible light is coupled into the optical waveguide through the coupling module and is totally reflected inside the optical waveguide and transmitted to the coupling structure. The visible light is then coupled out through the coupling structure to the user's eyes or a predetermined imaging component, such as imaging glass, so that the user can see the image.

[0089] Exemplarily, the optical waveguide 20 further includes a turning grating structure, and the turning grating structure is provided on the optical waveguide.

[0090] For example, the infrared light source 30 may be disposed on an optical waveguide.

[0091] Exemplarily, the infrared light source 30 is disposed at the peripheral edge of the optical waveguide, so that the infrared light source 30 can be connected to a power source while reducing obstruction to the user's sight.

[0092] For example, taking the near-eye display device 100 using AR / VR glasses as an example, the shell of the AR / VR glasses includes a frame and temples connected to the frame. The optical waveguide 20 and the infrared light source 30 can be arranged on the frame so that the infrared light and visible light can be emitted toward the user's eyes, and the visible light source 10 can be arranged on the temples.

[0093] In some embodiments, multiple infrared light sources 30 surround the periphery of the outcoupling structure of the optical waveguide 20. The multiple infrared light sources 30 emit infrared light toward the side away from the optical waveguide 20, and the transmission direction of the infrared light is tilted toward the side where the infrared light sources 30 are located. It is understandable that to prevent the infrared light sources 30 from blocking the visible light coupled out by the outcoupling structure, the infrared light sources 30 can be positioned at the periphery of the outcoupling structure of the optical waveguide 20. Typically, when viewed from a direction directly facing the eyes, the outcoupling structure of the optical waveguide 20 is positioned corresponding to the user's eyes, and the infrared light sources 30 are positioned at the periphery of the outcoupling structure, that is, the infrared light sources 30 are positioned at the periphery of the user's eyes. By tilting the transmission direction of the infrared light toward the side where the infrared light sources 30 are located, it is ensured that the infrared light falls within the active range of the pupil region 21 of the eye.

[0094] For example, the internal structure of the infrared light source 30 can be adjusted and designed so that the infrared light source 30 emits light at an angle, with the transmission direction of the infrared light tilted toward the side where the infrared light source 30 is located and falling within the active range of the pupil area 21. Alternatively, the mounting structure of the infrared light source 30 can be configured so that when the infrared light source 30 is installed in the optical waveguide, the normal light emission direction of the infrared light source 30 is tilted toward the side where the infrared light source 30 is located and toward the active range of the pupil area 21. Alternatively, the optical path of the infrared light emitted by the infrared light source 30 can be adjusted to change the transmission direction of the infrared light, so that the transmission direction of the infrared light is tilted toward the side where the infrared light source 30 is located.

[0095] In some embodiments, the near-eye display device 100 further includes a plurality of phase modulators (not shown), each corresponding to a plurality of infrared light sources 30. The phase modulators are disposed on the light-emitting side of the corresponding infrared light source 30 and are used to modulate the infrared light emitted by the corresponding infrared light source 30 so that the transmission direction of the infrared light is tilted toward the side where the infrared light source 30 is located. By disposing the phase modulators on the light-emitting side of the infrared light source 30, the infrared light changes its transmission direction after being emitted from the infrared light source 30 to the phase modulators, thereby simplifying the design of the near-eye display device 100 and facilitating its installation.

[0096] Furthermore, the phase modulator includes a lens or a liquid crystal phase modulator.

[0097] Exemplarily, the phase modulator utilizes a liquid crystal phase modulator. By adjusting the voltage applied to the liquid crystal phase modulator, the transmission direction of the infrared light emitted by the corresponding infrared light source 30 is controlled. Specifically, after determining the target infrared light source based on the location of the preset eye area, the target transmission angle of the infrared light can be further determined. Based on the target transmission angle, the voltage applied to the liquid crystal phase modulator is adjusted so that the infrared light, after entering the phase modulator, is emitted toward the pupil area 21 at the target transmission angle.

[0098] In some embodiments, the plurality of infrared light sources 30 are evenly arranged.

[0099] In some embodiments, the plurality of infrared light sources 30 are arranged in an elliptical shape to ensure that when the pupil area 21 moves to any position, the infrared light emitted by at least one infrared light source 30 can fall on the pupil area 21 .

[0100] Exemplarily, the near-eye display device 100 includes at least four infrared light sources 30 , and the four infrared light sources 30 are evenly spaced and arranged in a ring shape.

[0101] like Figure 2 and Figure 6As shown, an embodiment of the present invention further provides a near-eye display device. The near-eye display device may be AR glasses, VR glasses, an AR helmet, a VR helmet, etc. Specifically, the near-eye display device includes a light source assembly, an optical waveguide 20, an image acquisition unit 40, and a processing unit 50. The light source assembly includes a visible light source 10 and a plurality of infrared light sources 30 arranged at intervals, and the image acquisition unit 40 is used to acquire an image of the eye. The processing unit 50 is electrically connected to the image acquisition unit 40 and the plurality of infrared light sources 30, and the processing unit 50 is used to: determine the position of a preset area of ​​the eye according to the image of the eye; determine a target infrared light source among the plurality of infrared light sources 30 according to the position of the preset area of ​​the eye; control the target infrared light source to emit infrared light to the pupil area 21 of the eye; and determine the gaze direction of the eye according to the light spot formed by the infrared light in the image of the eye.

[0102] By spacing the multiple infrared light sources 30, it is ensured that when the pupil area 21 of the eye is at any position, infrared light emitted by one or more infrared light sources 30 can fall on the pupil area 21 of the eye, forming a light spot on the cornea area 22 of the eye, thereby achieving eye tracking and increasing the range of eye tracking. Furthermore, after determining the position of the preset eye area based on the image of the eye, one or more infrared light sources 30 are determined as target infrared light sources 30 based on the position of the preset eye area. The target infrared light sources 30 emit infrared light, improving the position transmission accuracy of the emitted infrared light, allowing the infrared light to be more accurately transmitted to the pupil area 21 of the eye, thereby forming a clear light spot on the cornea area 22 of the eye. The other infrared light sources 30 do not emit infrared light, preventing the infrared light emitted by the other infrared light sources 30 from interfering with the light spot in the cornea area. This makes the light spot in the re-collected eye image clearer, thereby achieving high-precision eye tracking.

[0103] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0104] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the eye tracking method for a near-eye display device of the present application.

[0105] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the near-eye display device.

[0106] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0107] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system that includes the element.

[0108] The serial numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope of protection of the claims.

Claims

1. An eye tracking method based on a near-eye display device, characterized in that: The near-eye display device comprises: A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals; an optical waveguide, into which the visible light can be coupled and then transmitted and then coupled out from the optical waveguide; An image acquisition unit, configured to acquire images of the eye; The eye tracking method comprises: acquiring an image of the eye; determining a position of a preset eye area according to the image; Determining a target infrared light source from the plurality of infrared light sources according to the position of the preset eye area; Controlling the target infrared light source to emit the infrared light to the pupil area of ​​the eye; acquiring an image of the eye again; The gaze direction of the eye is determined according to the light spot formed by the infrared light in the re-acquired image.

2. The eye tracking method according to claim 1, wherein: The preset eye area includes at least one of the following: the pupil area and the cornea area; The distance between the position of the target infrared light source and the pupil area is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the pupil area; and / or The distance between the target infrared light source and the cornea region is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the cornea region.

3. The eye tracking method according to claim 1, wherein: The preset eye area includes the pupil area, cornea area and sclera area; The determining the position of the preset eye area according to the eye image includes: determining a positional relationship among the pupil area, the cornea area, and the sclera area based on the image of the eye; The step of determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area includes: A target infrared light source is determined among the plurality of infrared light sources according to a positional relationship among the pupil area, the cornea area, and the sclera area.

4. The eye tracking method according to claim 1, wherein: When determining a target infrared light source from a plurality of infrared light sources according to the position of the preset eye area, it is used to: Based on the preset correspondence between the plurality of infrared light sources and the positions of the preset eye area, the target infrared light source is determined among the plurality of infrared light sources according to the position of the preset eye area.

5. The eye tracking method according to claim 1, wherein: The plurality of infrared light sources are arranged on the optical waveguide.

6. The eye tracking method according to claim 5, wherein: The plurality of infrared light sources surround the outer periphery of the outcoupling structure of the optical waveguide, and the plurality of infrared light sources emit the infrared light toward a side away from the optical waveguide, and the transmission direction of the infrared light is inclined toward the side where the outcoupling structure is located.

7. The eye tracking method according to claim 6, wherein: The near-eye display device also includes multiple phase modulators, which correspond one-to-one to the multiple infrared light sources. The phase modulators are arranged on the light-emitting side of the corresponding infrared light source. The phase modulators are used to modulate the infrared light emitted by the corresponding infrared light source so that the transmission direction of the infrared light is inclined toward the side where the coupling structure is located.

8. The eye tracking method according to claim 7, wherein: The phase modulator includes a lens or a liquid crystal phase modulator.

9. The eye tracking method according to claim 1, wherein: The plurality of infrared light sources are arranged evenly; and / or the plurality of infrared light sources are arranged in an elliptical shape.

10. A near-eye display device, characterized in that: include: A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals; An optical waveguide, into which the visible light emitted by the visible light source can be coupled and then transmitted and then coupled out from the optical waveguide; An image acquisition unit, configured to acquire an image of the eye; A processing unit is electrically connected to the image acquisition unit and the plurality of infrared light sources, and the processing unit is used to execute the steps of the eye tracking method according to any one of claims 1 to 9.

Images