Eye movement tracking method and near-eye display equipment

Abstract

The embodiment of the invention provides an eye movement tracking method and near-to-eye display equipment. The near-to-eye display device comprises a light source assembly, an optical waveguide, an image acquisition unit and a processing unit. The light source assembly comprises a visible light source and a plurality of infrared light sources arranged at intervals. The image acquisition unit is used for acquiring images of eyes. The eye movement tracking method comprises the following steps: determining the position of a preset eye region according to an eye image; determining a target infrared light source in the plurality of infrared light sources according to the position of the eye preset area; controlling the target infrared light source to emit infrared light to the pupil area of the eyes; and determining the gazing direction of the eye according to the light spot formed by the infrared light in the image of the eye. According to the technical scheme, when the pupil areas of the eyes are located at different positions, the infrared light can be accurately transmitted to the pupil areas, and high-precision eye movement tracking is achieved.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to an eye tracking method and a near-eye display device. Background Art

[0002] Near-eye display devices usually track the movement state of the human eye, and control the near-eye display devices according to the movement state of the human eye to achieve human-computer interaction. Specifically, the near-eye display device can illuminate the pupil with infrared light and form a bright spot on the cornea, and then collect the bright spot image formed on the cornea, determine the movement state of the human eye based on the bright spot image, and achieve eye tracking. However, in the near-eye display device in the related art, when the human eye looks straight ahead, the infrared light irradiates the cornea, forming a clear bright spot on the cornea, but when the human eye looks sideways, the infrared light irradiates the sclera and is diffusely reflected, forming scattered spots on the sclera, and the bright spot formed on the cornea is reduced or even disappears, resulting in the collected bright spot image being difficult to achieve high-precision eye tracking, and it may even be impossible to use the collected bright spot image to achieve eye tracking. Summary of the invention

[0003] Embodiments of the present invention provide an eye tracking method and a near-eye display device, which aim to enable infrared light to be accurately transmitted to the pupil area when the pupil area of ​​the eye is in different positions, thereby achieving high-precision eye tracking.

[0004] In a first aspect, an embodiment of the present invention provides an eye tracking method based on a near-eye display device, comprising:

[0005] A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals;

[0006] An optical waveguide, into which the visible light can be coupled for transmission and then coupled out from the optical waveguide;

[0007] An image acquisition unit, used for acquiring images of the eyes;

[0008] The eye tracking method comprises:

[0009] acquiring an image of the eye;

[0010] Determine the position of a preset eye area according to the image;

[0011] Determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area;

[0012] Controlling the target infrared light source to emit the infrared light to the pupil area of ​​the eye;

[0013] acquiring the image of the eye again;

[0014] The gaze direction of the eye is determined according to the light spot formed by the infrared light in the image acquired again.

[0015] Optionally, the preset eye area includes at least one of the following: the pupil area, the cornea area;

[0016] The distance between the position of the target infrared light source and the pupil area is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the pupil area; and / or

[0017] The distance between the position of the target infrared light source and the corneal area is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the corneal area.

[0018] Optionally, the preset eye area includes the pupil area, cornea area and sclera area;

[0019] The step of determining the position of the preset eye area according to the eye image comprises:

[0020] determining a positional relationship among the pupil region, the cornea region, and the sclera region according to the image of the eye;

[0021] The step of determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area comprises:

[0022] A target infrared light source is determined among the plurality of infrared light sources according to a positional relationship among the pupil area, the cornea area, and the sclera area.

[0023] Optionally, when determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area, it is used to:

[0024] Based on the correspondence between the preset plurality of infrared light sources and the positions of the preset eye areas, the target infrared light source is determined from among the plurality of infrared light sources according to the position of the preset eye area.

[0025] Optionally, a plurality of the infrared light sources are arranged on the optical waveguide.

[0026] Optionally, the plurality of infrared light sources surround the outer periphery of the outcoupling structure of the optical waveguide, and the plurality of infrared light sources emit the infrared light toward a side away from the optical waveguide, and a transmission direction of the infrared light is inclined toward a side where the outcoupling structure is located.

[0027] Optionally, the near-eye display device also includes multiple phase modulators, which correspond one-to-one to the multiple infrared light sources. The phase modulators are arranged on the light-emitting side of the corresponding infrared light source, and are used to modulate the infrared light emitted by the corresponding infrared light source so that the transmission direction of the infrared light is inclined toward the side where the infrared light source is located.

[0028] Optionally, the phase modulator comprises a lens or a liquid crystal phase modulator.

[0029] Optionally, the plurality of infrared light sources are arranged uniformly; and / or the plurality of infrared light sources are arranged in an elliptical shape.

[0030] In a second aspect, an embodiment of the present invention further provides a near-eye display device, including:

[0031] A visible light source, used for emitting visible light;

[0032] An optical waveguide, into which the visible light can be coupled for transmission and then coupled out from the optical waveguide to the eye or a predetermined imaging component;

[0033] A plurality of infrared light sources, wherein the plurality of infrared light sources are arranged at intervals;

[0034] A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals;

[0035] An optical waveguide, wherein the visible light emitted by the visible light source can be coupled into the optical waveguide for transmission and then coupled out from the optical waveguide;

[0036] An image acquisition unit, used for acquiring an image of the eye;

[0037] A processing unit is electrically connected to the image acquisition unit and the plurality of infrared light sources, and the processing unit is used to execute the steps of the eye tracking method as described in the first aspect.

[0038] The embodiment of the present invention provides an eye tracking method and a near-eye display device, which obtains an image of the eye collected by an image acquisition unit, determines the position of a preset area of ​​the eye according to the image of the eye, and determines a target infrared light source according to the position of the preset area of ​​the eye, so that the target infrared light source emits infrared light to the pupil area of ​​the eye and forms a light spot in the cornea area of ​​the eye, so that the gaze direction of the eye can be determined according to the light spot formed by the infrared light in the image of the eye collected by the image acquisition unit again, and eye tracking is achieved. Among them, multiple infrared light sources are used to emit infrared light, which expands the range that infrared light can cover and increases the range of eye tracking. The target infrared light source is determined from the multiple infrared light sources according to the position of the preset area of ​​the eye, and the infrared light emitted by the target infrared light source can be accurately transmitted to the pupil area of ​​the eye, ensuring that a clear light spot can be formed in the cornea area, while other infrared light sources among the multiple infrared light sources do not emit infrared light, avoiding interference and improving the accuracy of eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 Schematic diagram of forming a light spot on the eye in the related technology provided by the present invention:

[0041] Figure 2 A structural block diagram of a near-eye display device provided by an embodiment of the present invention;

[0042] Figure 3 A flow chart of an eye tracking method provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of forming a light spot on the eye according to an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of a near-eye display device provided by an embodiment of the present invention;

[0045] Figure 6 A state diagram of multiple infrared light sources when the eyes are looking straight ahead provided by an embodiment of the present invention;

[0046] Figure 7 A state diagram of multiple infrared light sources when the eye looks forward to the left provided by an embodiment of the present invention;

[0047] Figure 8 A state diagram of multiple infrared light sources when the eye looks to the right front provided by one embodiment of the present invention;

[0048] Fig. 9 A state diagram of a near-eye display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0051] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0052] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0053] like Figure 1 As shown, in the related art, when a user wears a near-eye display device and the user's eyes look straight ahead, infrared light directly enters the eyes, forming a clear light spot in the cornea area of ​​the eyes, and when the user's eyes turn to look in other directions, for example, looking to the left front, infrared light enters the eyes, is diffusely reflected by the sclera of the eyes to form stray light, and forms speckles on the sclera of the eyes, while fewer or even no light spots are formed in the cornea area of ​​the eyes, resulting in poor eye tracking accuracy, or the inability to perform eye tracking, resulting in limited accuracy and range of eye tracking.

[0054] In order to solve the above problems, an embodiment of the present invention provides a near-eye display device. The near-eye display device may include augmented reality (AR) glasses, virtual reality (VR) glasses, AR helmets, VR helmets, etc., without limitation.

[0055] Please refer to Figure 2 and Figure 5The near-eye display device 100 includes a light source assembly, an optical waveguide 20, an image acquisition unit 40 and a processing unit 50. The light source assembly includes a visible light source 10 and a plurality of infrared light sources 30 arranged at intervals. The image acquisition unit 40 is used to acquire an image of the eye.

[0056] Exemplarily, the plurality of infrared light sources 30 are arranged at intervals, and when the plurality of infrared light sources 30 emit infrared light, the infrared light can cover the movement range of the pupil region 21 of the eye.

[0057] Exemplarily, the visible light may be light with a wavelength range of 400nm-760nm, and the infrared light may be light with a wavelength range of 780nm-940nm.

[0058] like Figure 3 As shown, the eye tracking method based on the above-mentioned near-eye display device includes steps S101 to S106.

[0059] Step S101, acquiring an eye image.

[0060] Exemplarily, an image of the eye acquired by the image acquisition unit 40 is acquired.

[0061] Step S102: determining the position of the preset eye area according to the image.

[0062] Exemplarily, the position of the preset eye area may be determined by performing image recognition on the image.

[0063] Step S103: determining a target infrared light source from a plurality of infrared light sources according to the position of the preset eye area.

[0064] It can be understood that the target infrared light source may include one or more infrared light sources 30. For example, the near-eye display device includes six infrared light sources 30, and the target infrared light source may be one, two or three infrared light sources 30 determined according to the position of the preset eye area.

[0065] Step S104: controlling the target infrared light source to emit infrared light to the pupil area of ​​the eye.

[0066] It can be understood that the image acquisition unit 40 acquires an image of the eye again. Since the target infrared light source emits infrared light to the pupil area 21 of the eye, the infrared light can form a light spot in the cornea area 22 of the eye, so the image of the eye acquired again includes the light spot formed by the infrared light in the cornea area 22 of the eye.

[0067] Step S105, acquiring the eye image again.

[0068] Step S106: determine the gaze direction of the eye according to the light spot formed by the infrared light in the image acquired again.

[0069] Exemplarily, the image acquisition unit 40 may include an infrared camera, which acquires an image of the eye through the infrared camera, determines the position of a preset area of ​​the eye based on the image of the eye, and acquires the image of the eye again through the infrared camera to determine the gaze direction of the eye based on a light spot formed by infrared light in the re-acquired image of the eye.

[0070] Exemplarily, the image acquisition unit 40 also includes a visible light camera and an infrared camera. The visible light camera includes a CCD (Charge Coupled Device) camera and a CMOS (Complementary Metal Oxide Semiconductor) camera. The image of the eye is acquired by the visible light camera, and the position of a preset area of ​​the eye is determined based on the image of the eye. The image of the eye is acquired again by the infrared camera to determine the gaze direction of the eye based on the light spot formed by the infrared light in the again acquired image of the eye.

[0071] For example, when a user wears the near-eye display device 100, after the image acquisition unit 40 acquires an image of the eye, the position of the preset area of ​​the eye can be determined according to the acquired image of the eye, for example, the position of the pupil area 21 or the cornea area 22 of the eye can be determined, and then a target infrared light source is determined among the multiple infrared light sources 30 according to the position of the preset area of ​​the eye, and the target infrared light source is controlled to emit infrared light toward the user's eye, so that the infrared light can fall on the pupil area 21 of the user's eye and form a clear light spot in the cornea area 22 of the eye. It can be understood that the position of the target infrared light source determined according to the position of the preset area of ​​the eye is associated with the position of the pupil area 21 of the eye, and the infrared light emitted by the target infrared light source can fall on the pupil area 21 and form a clear light spot in the cornea area 22 of the eye. After the image acquisition unit 40 acquires the image of the eye again, the gaze direction of the eye can be determined according to the light spot formed by the infrared light in the image of the eye acquired again, so as to realize eye tracking.

[0072] The near-eye display device 100 of the embodiment of the present invention ensures that when the pupil area 21 of the eye is at any position, the infrared light emitted by one or more infrared light sources 30 can fall on the pupil area 21 to form a light spot in the corneal area 22, thereby realizing eye tracking and increasing the scope of eye tracking. After determining the position of the preset eye area according to the image of the eye, one or more infrared light sources 30 are determined as target infrared light sources according to the position of the preset eye area, so that the target infrared light source emits infrared light, improves the position transmission accuracy of the emitted infrared light, and makes the infrared light more accurately transmitted to the pupil area 21 to form a clear light spot in the corneal area 22, while other infrared light sources 30 do not emit infrared light, so as to avoid interference of the infrared light emitted by other infrared light sources 30 on the light spot in the corneal area 22, which is conducive to making the light spot in the image of the eye collected again clearer and realizing high-precision eye tracking.

[0073] Exemplarily, the position of the preset eye area is determined based on an image of the eye, wherein the position of the preset eye area may be the position of the preset eye area in the image of the eye, or the position of the preset eye area in the world coordinate system, which may be adjusted according to actual conditions.

[0074] Exemplarily, the gaze direction of the eye is determined, and the movement state of the eye can be determined according to the gaze direction of the eye. For example, by comparing the gaze directions of the eyes in different time frames, it is determined that the eyes are in a gaze state, a movement state, or a scanning state. When the deviation between the gaze directions of the eyes in at least two adjacent time frames is less than or equal to a first preset difference, it is determined that the user's eyes are in a gaze state; when the deviation between the gaze directions of the eyes in at least two adjacent time frames is greater than the first preset difference and less than or equal to a second preset difference, it is determined that the user's eyes are in a moving tracking state; when the deviation between the gaze directions of the eyes in at least two adjacent time frames is greater than the second preset difference, it is determined that the user's eyes are in a scanning state.

[0075] Figure 4 FIG. 1 is a schematic diagram showing that when the near-eye display device 100 in an embodiment of the present invention performs eye tracking, infrared light falls on the pupil area 21 of the eye and forms a light spot in the cornea area 22 of the eye. Figure 4 As shown, when the eye looks straight ahead or to the left front, the infrared light emitted by the target infrared light source falls on the pupil area 21 of the eye, and can form a clear light spot in the cornea area 22 of the eye to achieve high-precision eye tracking.

[0076] In some embodiments, the preset eye area includes at least one of the following: pupil area 21, cornea area 22. The distance between the position of the target infrared light source and the pupil area 21 is smaller than the distance between other infrared light sources 30 in the plurality of infrared light sources 30 and the pupil area 21, and / or, the distance between the position of the target infrared light source and the cornea area 22 is smaller than the distance between other infrared light sources 30 in the plurality of infrared light sources 30 and the cornea area 22. It can be understood that, when the eye looks in different directions, there is a corresponding positional relationship between the areas in the image of the eye corresponding to the pupil area 21 and the cornea area 22, respectively. Therefore, after determining the position of the cornea area 22 according to the image of the eye, the position of the pupil area 21 is also determined, so that the target infrared light source can be directly determined according to the pupil area 21, and / or, the target infrared light source can be determined according to the cornea area 22.

[0077] Exemplarily, the distance between the infrared light source 30 and the pupil area 21 can be embodied as the distance between the geometric center of the infrared light source 30 and the center of the pupil area 21. The distance between the infrared light source 30 and the cornea area 22 can be embodied as the distance between the geometric center of the infrared light source 30 and the center of the cornea area 22.

[0078] For example, Figure 6 , Figure 7 and Figure 8 It shows the on and off states of multiple infrared light sources 30 when the eyes are observed in a direction facing the eyes and the eyes are looking in different directions. The dotted lines in the figure represent the transmission path of the infrared light. Figure 6 The figure shows an image of the eye captured by the image acquisition unit 40 when the eye is looking straight ahead, in which the pupil area 21 and the cornea area 22 of the eye are located in the relatively middle part of the image of the eye, and five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit. Figure 7 The figure shows an image of the eye captured by the image acquisition unit 40 when the eye is looking forward to the left. The pupil area 21 and the cornea area 22 of the eye are located on the relatively right side of the image of the eye. Five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit. Figure 8 The figure shows an image of the eye captured by the image acquisition unit 40 when the eye is looking to the right front. The pupil area 21 and the cornea area 22 of the eye are located on the relatively left side of the image of the eye. Five infrared light sources 30 that are closer to the pupil area 21 and the cornea area 22 can be controlled to be lit as target infrared light sources, while the other three infrared light sources 30 are not lit.

[0079] For example, Fig. 9A top view of the user's eyes and the near-eye display device 100 when the user wears the near-eye display device 100, Fig. 9 The dotted line represents the transmission path of infrared light, and the dotted line represents the transmission path of visible light. Fig. 9 As shown in the figure (A) in FIG, when the eye is looking to the right front, the distance between the infrared light source 30 located on the right side of the eye and the pupil area 21 is smaller than the distance between the infrared light source 30 on the left side and the pupil area 21, and the infrared light source 30 on the right side is lit as the target infrared light source, and is controlled to emit infrared light to the pupil area 21. Fig. 9 As shown in the figure (B), when the eye looks forward to the left, the distance between the infrared light source 30 on the left side of the eye and the pupil area 21 is smaller than the distance between the infrared light source 30 on the right side and the pupil area 21. The infrared light source 30 on the left side is lit as the target infrared light source and controlled to emit infrared light to the pupil area 21.

[0080] In some embodiments, a plurality of infrared light sources 30 may be located on the same plane, and the target infrared light source may be determined according to the position of the projection of the preset eye area on the plane.

[0081] Exemplarily, the distance between the position of the target infrared light source and the projection of the pupil area 21 of the eye on the plane is smaller than the distance between the other infrared light sources 30 and the projection of the pupil area 21 of the eye on the plane.

[0082] In some embodiments, the preset eye area includes a pupil area 21, a cornea area 22, and a sclera area 23. When determining the position of the preset eye area according to the image of the eye, it is used to: determine the positional relationship between the pupil area 21, the cornea area 22, and the sclera area 23 according to the image of the eye. When determining a target infrared light source among the multiple infrared light sources 30 according to the position of the preset eye area, it is used to: determine the target infrared light source among the multiple infrared light sources 30 according to the positional relationship between the pupil area 21, the cornea area 22, and the sclera area 23. It can be understood that the position of the image acquisition unit 40 in the near-eye display device 100 is fixed. When the eye looks in different directions, the relative relationship between the pupil area 21, the cornea area 22 and the sclera area 23 in the image of the eye acquired by the image acquisition unit 40 is different. Therefore, according to the positional relationship between the pupil area 21, the cornea area 22 and the sclera area 23, the gaze direction of the eye can be preliminarily determined, that is, the position of the pupil area 21 to which the infrared light needs to be transmitted is determined, and then the target infrared light source is determined according to the preliminarily determined gaze direction of the eye, so that the infrared light emitted by the target infrared light source can be accurately transmitted to the pupil area 21 of the eye, and the gaze direction of the eye is further determined, thereby achieving high-precision eye tracking.

[0083] For example, Figure 6An image of the eye when the eye is looking straight ahead is shown, in which the pupil region 21 and the cornea region 22 are located in relatively central portions of the sclera region 23 . Figure 7 An image of the eye is shown when the eye is looking diagonally toward the left front. In this image, the pupil region 21 and the cornea region 22 are located on the relatively right side of the sclera region 23 . Figure 8 An image of the eye is shown when the eye is looking diagonally toward the right front. In this image, the pupil region 21 and the cornea region 22 are located on the relatively left side of the sclera region 23 .

[0084] In some embodiments, when determining a target infrared light source among a plurality of infrared light sources 30 according to the position of a preset eye area, it is used to: determine the target infrared light source among a plurality of infrared light sources 30 according to the position of the preset eye area based on the correspondence between the preset plurality of infrared light sources 30 and the position of the preset eye area. It can be understood that, according to any position that the preset eye area can be located at, one or more infrared light sources 30 corresponding thereto can be pre-set, and when the position of the preset eye area is determined according to the image of the eye, the preset one or more infrared light sources 30 can be lit as the target infrared light source.

[0085] For example, Fig. 9 As shown, the infrared light source 30 on the left side of the preset eye corresponds to the position when the pupil area 21 is located on the left side in the width direction of the eye, and the infrared light source 30 on the right side corresponds to the position when the pupil area 21 is located on the right side in the width direction of the eye. After the position of the pupil area 21 is determined according to the image of the eye, if the pupil area 21 is located on the left side in the width direction of the eye, the infrared light source 30 on the left side can be lit as the target infrared light source; if the pupil area 21 is located on the right side in the width direction of the eye, the infrared light source 30 on the right side can be lit as the target infrared light source.

[0086] like Figure 5 and Fig. 9 As shown, in some embodiments, a plurality of infrared light sources 30 are disposed on the optical waveguide 20. It can be understood that the optical waveguide 20 is usually located in front of the user's eyes, and the optical waveguide 20 is used as a carrier to carry the infrared light source 30, so as to facilitate the installation of the infrared light source 30.

[0087] Exemplarily, the optical waveguide 20 has a display side. When the user wears the near-eye display device 100, the user's eyes are located on the display side of the optical waveguide 20, so that the optical waveguide 20 can couple the visible light to the user's eyes. The infrared light source 30 can be located on the display side of the optical waveguide 20 to directly emit infrared light toward the pupil area 21 of the user's eyes to reduce the loss of light. Alternatively, the optical waveguide 20 has a non-display side that is arranged opposite to the display side, and multiple infrared light sources 30 are arranged on the non-display side. The multiple infrared light sources 30 can correspond to the position of the coupling structure, and the infrared light can pass through the coupling structure and be emitted to the user's eyes.

[0088] Exemplarily, the optical waveguide 20 has an optical waveguide and a coupling structure and a coupling structure arranged on the optical waveguide. Visible light is coupled into the optical waveguide through the coupling module and is totally reflected inside the optical waveguide and transmitted to the coupling structure. The visible light is then coupled out through the coupling structure to the user's eyes or a predetermined imaging component, such as imaging glass, so that the user can see the image.

[0089] Exemplarily, the optical waveguide 20 further includes a turning grating structure, and the turning grating structure is disposed on the optical waveguide.

[0090] Exemplarily, the infrared light source 30 may be disposed on an optical waveguide.

[0091] Exemplarily, the infrared light source 30 is disposed at the outer peripheral edge of the optical waveguide, so that the infrared light source 30 can be connected to electricity while reducing obstruction to the user's vision.

[0092] Exemplarily, taking the near-eye display device 100 using AR / VR glasses as an example, the shell of the AR / VR glasses includes a frame and temples connected to the frame, the optical waveguide 20 and the infrared light source 30 can be arranged on the frame so that infrared light and visible light can be emitted toward the user's eyes, and the visible light source 10 can be arranged on the temples.

[0093] In some embodiments, a plurality of infrared light sources 30 surround the periphery of the outcoupling structure of the optical waveguide 20, and the plurality of infrared light sources 30 emit infrared light to the side away from the optical waveguide 20, and the transmission direction of the infrared light is inclined toward the side where the infrared light source 30 is located. It can be understood that in order to prevent the infrared light source 30 from blocking the visible light coupled out by the outcoupling structure, the infrared light source 30 can be arranged at the periphery of the outcoupling structure of the optical waveguide 20. In general, when observing from the direction facing the eyes, the outcoupling structure of the optical waveguide 20 is arranged corresponding to the user's eyes, and the infrared light source 30 is located at the periphery of the outcoupling structure, that is, the infrared light source 30 is located at the periphery of the user's eyes. By inclining the transmission direction of the infrared light to the side where the infrared light source 30 is located, it is ensured that the infrared light can fall within the activity range of the pupil area 21 of the eye.

[0094] Exemplarily, the internal structure of the infrared light source 30 can be adjusted and designed so that the infrared light source 30 emits light obliquely, and the transmission direction of the infrared light is inclined toward the side where the infrared light source 30 is located, and falls within the activity range of the pupil area 21. Alternatively, the mounting structure of the infrared light source 30 can be set so that when the infrared light source 30 is installed in the optical waveguide, the positive light emission direction of the infrared light source 30 is inclined toward the side where the infrared light source 30 is located, and toward the activity range of the pupil area 21. Alternatively, the optical path of the infrared light emitted by the infrared light source 30 is regulated to change the transmission direction of the infrared light, so that the transmission direction of the infrared light is inclined toward the side where the infrared light source 30 is located.

[0095] In some embodiments, the near-eye display device 100 further includes a plurality of phase modulators (not shown), the plurality of phase modulators corresponding to the plurality of infrared light sources 30 one by one, the phase modulators being arranged on the light-emitting side of the corresponding infrared light source 30, and the phase modulators being used to modulate the infrared light emitted by the corresponding infrared light source 30 so that the transmission direction of the infrared light is tilted toward the side where the infrared light source 30 is located. By arranging the phase modulator on the light-emitting side of the infrared light source 30, the transmission direction of the infrared light after being emitted from the infrared light source 30 to the phase modulator is changed through the phase modulator, thereby simplifying the design of the near-eye display device 100 and facilitating the installation of the near-eye display device 100.

[0096] Further, the phase modulator includes a lens or a liquid crystal phase modulator.

[0097] Exemplarily, the phase modulator uses a liquid crystal phase modulator, and by adjusting the voltage connected to the liquid crystal phase modulator, the transmission direction of the infrared light emitted by the corresponding infrared light source 30 is adjusted. Specifically, after determining the target infrared light source according to the position of the preset eye area, the target transmission angle of the infrared light can be further determined, and the voltage connected to the liquid crystal phase modulator is adjusted according to the target transmission angle, so that the infrared light is emitted to the phase modulator and then emitted to the pupil area 21 at the target transmission angle.

[0098] In some embodiments, the plurality of infrared light sources 30 are evenly arranged.

[0099] In some embodiments, the plurality of infrared light sources 30 are arranged in an elliptical shape to ensure that when the pupil region 21 moves to any position, infrared light emitted by at least one infrared light source 30 can fall on the pupil region 21 .

[0100] Exemplarily, the near-eye display device 100 includes at least four infrared light sources 30 , and the four infrared light sources 30 are evenly spaced and arranged in a ring.

[0101] like Figure 2 and Figure 6As shown, an embodiment of the present invention further provides a near-eye display device. The near-eye display device may be AR glasses, VR glasses, an AR helmet, a VR helmet, etc. Specifically, the near-eye display device includes a light source assembly, an optical waveguide 20, an image acquisition unit 40, and a processing unit 50. The light source assembly includes a visible light source 10 and a plurality of infrared light sources 30 arranged at intervals, and the image acquisition unit 40 is used to acquire an image of the eye. The processing unit 50 is electrically connected to the image acquisition unit 40 and the plurality of infrared light sources 30, and the processing unit 50 is used to: determine the position of a preset area of ​​the eye according to the image of the eye; determine a target infrared light source from the plurality of infrared light sources 30 according to the position of the preset area of ​​the eye; control the target infrared light source to emit infrared light to the pupil area 21 of the eye; and determine the gaze direction of the eye according to the light spot formed by the infrared light in the image of the eye.

[0102] By setting multiple infrared light sources 30 at intervals, it is ensured that when the pupil area 21 of the eye is at any position, the infrared light emitted by one or more infrared light sources 30 can fall on the pupil area 21 of the eye, so as to form a light spot in the cornea area 22 of the eye, realize eye tracking, and increase the scope of eye tracking. After determining the position of the preset eye area according to the image of the eye, one or more infrared light sources 30 are determined as target infrared light sources 30 according to the position of the preset eye area, so that the target infrared light source 30 emits infrared light, improves the position transmission accuracy of the emitted infrared light, and makes the infrared light more accurately transmitted to the pupil area 21 of the eye, so as to form a clear light spot in the cornea area 22 of the eye, while other infrared light sources 30 do not emit infrared light, so as to avoid interference of the infrared light emitted by other infrared light sources 30 on the light spot in the cornea area, so that the light spot in the image of the eye collected again is clearer, and high-precision eye tracking is realized.

[0103] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0104] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the eye tracking method for a near-eye display device of the present application.

[0105] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0106] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0107] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0108] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An eye tracking method based on a near-eye display device, characterized in that: The near-eye display device comprises: A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals; An optical waveguide, into which the visible light can be coupled for transmission and then coupled out from the optical waveguide; An image acquisition unit, used for acquiring images of the eyes; The eye tracking method comprises: acquiring an image of the eye; Determine the position of a preset eye area according to the image; Determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area; Controlling the target infrared light source to emit the infrared light to the pupil area of ​​the eye; acquiring the image of the eye again; The gaze direction of the eye is determined according to the light spot formed by the infrared light in the image acquired again.

2. The eye tracking method according to claim 1, characterized in that: The preset eye area includes at least one of the following: the pupil area and the cornea area; The distance between the position of the target infrared light source and the pupil area is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the pupil area; and / or The distance between the position of the target infrared light source and the corneal area is smaller than the distance between other infrared light sources among the plurality of infrared light sources and the corneal area.

3. The eye tracking method according to claim 1, characterized in that: The preset eye area includes the pupil area, cornea area and sclera area; The step of determining the position of the preset eye area according to the eye image comprises: determining a positional relationship among the pupil region, the cornea region, and the sclera region according to the image of the eye; The step of determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area comprises: A target infrared light source is determined among the plurality of infrared light sources according to a positional relationship among the pupil area, the cornea area, and the sclera area.

4. The eye tracking method according to claim 1, characterized in that: When determining a target infrared light source from among the plurality of infrared light sources according to the position of the preset eye area, it is used to: Based on the correspondence between the preset plurality of infrared light sources and the positions of the preset eye areas, the target infrared light source is determined from among the plurality of infrared light sources according to the position of the preset eye area.

5. The eye tracking method according to claim 1, characterized in that: A plurality of infrared light sources are arranged on the optical waveguide.

6. The eye tracking method according to claim 5, characterized in that: The plurality of infrared light sources surround the outer periphery of the outcoupling structure of the optical waveguide, and the plurality of infrared light sources emit the infrared light toward a side away from the optical waveguide, and the transmission direction of the infrared light is inclined toward the side where the outcoupling structure is located.

7. The eye tracking method according to claim 6, characterized in that: The near-eye display device also includes multiple phase modulators, which correspond one-to-one to the multiple infrared light sources. The phase modulators are arranged on the light-emitting sides of the corresponding infrared light sources, and are used to modulate the infrared light emitted by the corresponding infrared light sources so that the transmission direction of the infrared light is inclined toward the side where the coupling-out structure is located.

8. The eye tracking method according to claim 7, characterized in that: The phase modulator includes a lens or a liquid crystal phase modulator.

9. The eye tracking method according to claim 1, characterized in that: The plurality of infrared light sources are arranged evenly; and / or the plurality of infrared light sources are arranged in an elliptical shape.

10. A near-eye display device, characterized in that: include: A light source assembly, comprising a visible light source and a plurality of infrared light sources arranged at intervals; An optical waveguide, wherein the visible light emitted by the visible light source can be coupled into the optical waveguide for transmission and then coupled out from the optical waveguide; An image acquisition unit, used for acquiring an image of the eye; A processing unit is electrically connected to the image acquisition unit and the plurality of infrared light sources, and the processing unit is used to execute the steps of the eye tracking method as described in any one of claims 1 to 9.

Images