Eye movement tracking method and near-eye display equipment

Abstract

The embodiment of the invention provides an eye movement tracking method and near-to-eye display equipment. The near-to-eye display device comprises a light source assembly, an optical waveguide, a driving assembly, a light path adjusting assembly, an image acquisition unit and a processing unit. The light source assembly is used for emitting visible light and infrared light. The infrared light can irradiate the light path adjusting assembly. The eye movement tracking method comprises the following steps: determining the position of a preset eye region according to an eye image; determining a target posture of the light path adjusting assembly according to the position of the eye preset area; the driving assembly is controlled to adjust the light path adjusting assembly to the target posture, so that the light path adjusting assembly emits the infrared light to the pupil area of the eye; and determining the gazing direction of the eye according to the light spot formed by the infrared light in the image of the eye. According to the technical scheme of the embodiment of the invention, the technical problem that the eye movement tracking precision is poor due to the fact that the infrared light is less or cannot irradiate the eye pupil area is solved.

Description

Technical Field

[0001] The present invention relates to the field of eye tracking technology, and in particular to an eye tracking method and a near-eye display device. Background Art

[0002] When a user uses a near-eye display device, the near-eye display device usually tracks the gaze direction of the human eye and controls the near-eye display device according to the gaze direction of the human eye to achieve the purpose of human-computer interaction. In the related art, the near-eye display device determines the gaze direction of the human eye based on the collected image of the human eye under infrared light. However, the infrared light irradiation range of the near-eye display device in the related art is fixed and has a range limit. When the human eye looks straight ahead, the infrared light can irradiate the pupil and form a clear bright spot on the cornea. When the human eye squints to the left or right front, the human eye rotates, the position of the cornea moves, and the infrared light irradiates the sclera. The infrared light is diffusely reflected by the sclera to form scattered spots on the sclera, and fewer or no bright spots are formed on the cornea, resulting in the inability to determine the gaze direction of the human eye based on the collected image of the human eye, or the accuracy of the gaze direction of the human eye determined based on the image is poor. Summary of the invention

[0003] The embodiments of the present invention provide an eye tracking method and a near-eye display device, which aim to solve the technical problem that insufficient infrared light or the inability to illuminate the pupil area of ​​the eye results in poor eye tracking accuracy.

[0004] In a first aspect, an embodiment of the present invention provides an eye tracking method based on a near-eye display device, wherein the near-eye display device comprises:

[0005] A light source assembly, used for emitting visible light and infrared light;

[0006] An optical waveguide, the optical waveguide having a display side, the visible light being able to be coupled into the optical waveguide and coupled out to the display side of the optical waveguide after being transmitted in the optical waveguide;

[0007] Drive components;

[0008] An optical path adjustment component, which is transmission-connected to the driving component, and the infrared light can irradiate the optical path adjustment component;

[0009] An image acquisition unit, used for acquiring an image of the eye;

[0010] The eye tracking method comprises:

[0011] acquiring an image of the eye;

[0012] Determine the position of a preset eye area according to the image;

[0013] Determining a target posture of the optical path adjustment component according to the position of the preset eye area;

[0014] Controlling the driving component to adjust the optical path adjustment component to the target posture, so that the optical path adjustment component emits the infrared light to the pupil area of ​​the eye;

[0015] acquiring the image of the eye again;

[0016] The gaze direction of the eye is determined according to the light spot formed by the infrared light in the image acquired again.

[0017] Optionally, the preset eye area includes a pupil area, a cornea area and a sclera area;

[0018] The step of determining the position of the preset eye area according to the eye image comprises:

[0019] determining a positional relationship among the pupil region, the cornea region, and the sclera region according to the image of the eye;

[0020] The step of determining the target posture of the optical path adjustment component according to the position of the preset eye area includes:

[0021] The target posture of the optical path adjustment component is determined according to the positional relationship among the pupil area, the cornea area and the sclera area.

[0022] Optionally, the optical path adjustment component includes a reflective element, the reflective element has a reflective surface facing the light source component and the display side, and the infrared light can be reflected to the pupil area through the reflective surface; or,

[0023] The optical path adjustment component includes a refractive element, which has an incident surface and an exit surface arranged opposite to each other, the incident surface faces the light source component, and the exit surface faces the display side. The infrared light can enter the refractive element through the incident surface and be refracted and emitted from the exit surface to the pupil area.

[0024] Optionally, the optical path adjustment component is located between the light source component and the optical waveguide, and the optical path adjustment component can transmit the visible light and reflect the infrared light.

[0025] Optionally, the optical path adjustment component includes a dichroic mirror.

[0026] Optionally, the optical path adjustment component includes a lens and a mounting seat, the lens is arranged on the mounting seat, the driving component is transmission-connected to the mounting seat, and the driving motor can adjust the lens to the target posture.

[0027] Optionally, the mounting seat comprises a first seat body and a second seat body, the second seat body is rotatably connected to the first seat body, and the lens is arranged on the second seat body;

[0028] The drive motor includes a first drive motor and a second drive motor, the first drive motor is transmission-connected to the first seat body, the second drive motor is transmission-connected to the second seat body, the first drive motor can drive the first seat body and the second seat body to rotate around a first axis, the second drive motor can drive the second seat body to rotate around a second axis relative to the first seat body, and the first axis is different from the second axis.

[0029] Optionally, the optical waveguide includes a first waveguide and a second waveguide spaced apart along a first direction, one side of the first waveguide and the second waveguide along the second direction is the display side, the first axis is perpendicular to the first direction and the second direction, and the second axis is perpendicular to the first axis.

[0030] Optionally, the light source assembly includes a visible light source and a plurality of infrared light sources, and the plurality of infrared light sources are used to emit infrared light toward different positions of the optical path adjustment assembly.

[0031] In a second aspect, an embodiment of the present invention further provides a near-eye display device, including:

[0032] A light source assembly, used for emitting visible light and infrared light;

[0033] An optical waveguide, the optical waveguide having a display side, the visible light being able to be coupled into the optical waveguide and coupled out to the display side of the optical waveguide after being transmitted in the optical waveguide;

[0034] Drive components;

[0035] An optical path adjustment component, which is transmission-connected to the driving component, and the infrared light can irradiate the optical path adjustment component;

[0036] An image acquisition unit, used for acquiring an image of the eye;

[0037] A processing unit is electrically connected to the driving component and the image acquisition unit, and the processing unit is used to execute the steps of the eye tracking method as described in the first aspect.

[0038] The embodiment of the present invention provides an eye tracking method and a near-eye display device, wherein an image acquisition unit acquires an image of the eye to determine the position of a preset eye area, and determines the target posture of an optical path adjustment component according to the position of the preset eye area, so that after the driving component adjusts the optical path adjustment component to the target posture, the optical path adjustment component can change the optical path of the infrared light emitted by the light source component, so that the infrared light is emitted to the pupil area of ​​the eye, and a clear light spot is formed on the cornea area of ​​the eye, so that the gaze direction of the eye can be determined according to the light spot formed by the infrared light in the image of the eye acquired again. Among them, the optical path adjustment component can be driven by the driving component to move, expand the range of infrared light transmission, and thus increase the range of eye tracking, and the driving of the driving component is controlled according to the position of the preset eye area, so that the infrared light can be transmitted to the pupil area more accurately, thereby achieving high-precision eye tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 Schematic diagram of the light spot formed on the eye in the related technology provided by the present invention:

[0041] Figure 2 A structural block diagram of a near-eye display device provided by one embodiment of the present invention;

[0042] Figure 3 A schematic diagram of a flow chart of an eye tracking method provided by an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of a light spot formed on an eye according to an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of the structure of a near-eye display device provided by one embodiment of the present invention;

[0045] Figure 6 A schematic structural diagram of another near-eye display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0048] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0049] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0050] like Figure 1 As shown, in the related art, when the eye looks straight ahead, infrared light directly enters the eye, forming a light spot on the cornea of ​​the eye, and when the eye looks diagonally to the left front, the cornea of ​​the eye rotates, and infrared light enters the eye, and is diffusely reflected by the sclera of the eye, forming speckles on the sclera of the eye, while less or even no light spot is formed on the cornea of ​​the eye, resulting in poor accuracy of eye tracking, or the inability to perform eye tracking.

[0051] In order to solve the above problems, an embodiment of the present invention provides an eye tracking method based on a near-eye display device. The near-eye display device may include augmented reality (AR) glasses, virtual reality (VR) glasses, AR helmets, VR helmets, etc., without specific limitation.

[0052] like Figure 2 , Figure 5 and Figure 6As shown, the near-eye display device includes a light source assembly 10, an optical waveguide 20, a driving assembly 30, an optical path adjustment assembly 40, and an image acquisition unit 50. The light source assembly 10 is used to emit visible light and infrared light. The optical waveguide 20 has a display side, and visible light can be coupled into the optical waveguide 20 and coupled out to the display side of the optical waveguide 20 after being transmitted in the optical waveguide 20. The optical path adjustment assembly 40 is transmission-connected to the driving assembly 30, and infrared light can be irradiated to the optical path adjustment assembly 40. The image acquisition unit 50 is used to capture images of the eye.

[0053] Figure 5 and Figure 6 The dotted line represents the transmission path of infrared light, and the dotted line represents the transmission path of visible light.

[0054] Exemplarily, the visible light may be light with a wavelength range of 400nm-760nm, and the infrared light may be light with a wavelength range of 780nm-940nm.

[0055] Exemplarily, when a user wears a near-eye display device, the user's eyes are located on the display side of the optical waveguide 20, so that the optical waveguide 20 can couple visible light to the user's eyes so that the user's eyes can see the image displayed by the near-eye display device.

[0056] like Figure 3 As shown, the eye tracking method based on the above-mentioned near-eye display device includes steps S101 to S106.

[0057] Step S101, acquiring an eye image.

[0058] It can be understood that the image acquisition unit 50 acquires an image of the eye.

[0059] Step S102: determining the position of the preset eye area according to the image.

[0060] Exemplarily, the position of the preset eye area may be determined by performing image recognition on the image.

[0061] Step S103: determining the target posture of the optical path adjustment component according to the position of the preset eye area.

[0062] Step S104: controlling the driving component to adjust the optical path adjustment component to a target posture, so that the optical path adjustment component emits infrared light to the pupil area of ​​the eye.

[0063] It can be understood that the optical path adjustment component 40 can adjust the transmission path of the infrared light, and the driving component 30 can adjust the posture of the optical path adjustment component to adjust the posture of the infrared light emitted by the light source component 40 when it enters the optical path adjustment component, thereby changing the regulating effect of the optical path adjustment component 40 on the infrared light path, that is, it can change the direction of transmission of the infrared light after adjustment by the optical path adjustment component 40.

[0064] Step S105, acquiring the eye image again.

[0065] It can be understood that the image acquisition unit 50 acquires an image of the eye again. Since the infrared light is emitted to the pupil area of ​​the eye via the optical path adjustment component 40, it can form a light spot in the cornea area of ​​the eye, so the image of the eye acquired again includes the light spot formed by the infrared light in the cornea area of ​​the eye.

[0066] Step S106: determine the gaze direction of the eye according to the light spot formed by the infrared light in the image acquired again.

[0067] Exemplarily, the image acquisition unit 50 may include an infrared camera, which acquires an image of the eye through the infrared camera, determines the position of a preset area of ​​the eye based on the image of the eye, and acquires the image of the eye again through the infrared camera to determine the gaze direction of the eye based on a light spot formed by infrared light in the re-acquired image of the eye.

[0068] Exemplarily, the image acquisition unit 50 may also include a visible light camera, such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera, to acquire an image of the eye through the visible light camera, determine the position of a preset area of ​​the eye based on the image of the eye, and acquire the image of the eye again through an infrared camera to determine the gaze direction of the eye based on a light spot formed by infrared light in the again acquired image of the eye.

[0069] It can be understood that after acquiring the image of the eye captured by the image acquisition unit 50, the features in the image can be extracted by image recognition to determine the position of the preset area of ​​the eye, so that the target posture of the optical path adjustment component 40 can be determined according to the position of the preset area of ​​the eye. After the control drive component 30 adjusts the optical path adjustment component 40 to the target posture, the optical path adjustment component 40 in the target posture can change the transmission path of the infrared light emitted by the light source component 10, so that the infrared light is emitted to the pupil area of ​​the eye after passing through the optical path adjustment component 40, and forms a light spot in the cornea area of ​​the eye. Then, the image of the eye captured by the image acquisition unit 50 is acquired again, and the gaze direction of the eye is determined according to the relative position of the light spot formed by the infrared light and the pupil area in the image of the eye captured again, so as to realize eye tracking. Among them, the optical path adjustment component 40 can be driven by the driving component 30 to expand the range of infrared light transmission, ensure that when the eyes are looking at any position, the infrared light can be transmitted to the pupil area, and increase the range of eye tracking. By controlling the driving component 30 to adjust the optical path adjustment component 40 according to the position of the preset eye area, the infrared light can be transmitted to the pupil area more accurately, avoiding the infrared light falling on the sclera area and interfering with eye tracking, thereby achieving high-precision eye tracking.

[0070] Figure 4 FIG. 1 is a schematic diagram showing a near-eye display device in an embodiment of the present invention performing eye tracking, where infrared light enters the eye and forms a light spot in the corneal region of the eye. Figure 4 As shown, when the eyes look in different directions, for example, looking straight ahead, or looking diagonally to the left front, the infrared light emitted by the light source assembly 10 can be transmitted to the pupil area and form a light spot in the cornea area after being adjusted by the optical path adjustment assembly 40, so as to determine the gaze direction of the eye according to the light spot formed by the infrared light in the image of the eye acquired again, thereby achieving high-precision eye tracking.

[0071] Exemplarily, after determining the gaze direction of the eyes, the near-eye display device can be controlled according to the gaze direction of the eyes. For example, when the user's eyes keep looking at a virtual button on the screen of the near-eye display device for a preset time, the virtual cursor can be controlled to click the virtual button and jump to the next interface. Alternatively, the rendering resolution of the non-gaze area is adjusted according to the gaze direction of the eyes. Alternatively, when the gaze direction of the user's eyes changes rapidly, some operations of the near-eye display device can be locked to avoid accidental touches. The specific control can be adjusted according to actual needs and is not limited here.

[0072] Exemplarily, the position of the preset eye area is determined based on an image of the eye, wherein the position of the preset eye area may be the position of the preset eye area in the image of the eye, or the position of the preset eye area in the world coordinate system, which may be adjusted according to actual conditions.

[0073] In some embodiments, the preset eye area includes a pupil area, and the coordinates of the pupil area in the image acquisition unit 50 are determined based on the image of the eye, or the position of the pupil area in real coordinates is determined based on the image of the eye, and the target posture of the optical path adjustment component 40 is determined based on the position of the pupil area.

[0074] In some embodiments, the preset eye area includes a pupil area, a corneal area, and a scleral area. Determining the position of the preset eye area according to the image of the eye includes: determining the positional relationship between the pupil area, the corneal area, and the scleral area according to the image of the eye. Determining the target posture of the optical path adjustment component according to the position of the preset eye area includes: determining the target posture of the optical path adjustment component according to the positional relationship between the pupil area, the corneal area, and the scleral area. It can be understood that when the direction of the eye's line of sight is different, the relative relationship between the pupil area, the corneal area, and the scleral area is different. Therefore, according to the positional relationship between the pupil area, the corneal area, and the scleral area, the gaze direction of the eye can be preliminarily determined, that is, the position of the pupil area to which the infrared light needs to be transmitted is determined, so that the target posture of the optical path adjustment component 40 is determined according to the preliminarily determined gaze direction of the eye, so that the infrared light emitted by the optical path adjustment component 40 can be accurately transmitted to the pupil area of ​​the eye, and the gaze direction of the eye is further determined, so as to achieve high-precision eye tracking.

[0075] For example, when the eye's line of sight is tilted toward the right front, the image of the eye captured by the image capture unit 50 may be as follows: Figure 5 As shown in the middle figure (A), in the image of the eye, the cornea region partially overlaps with the pupil region, and the pupil region and the cornea region are located on the left side of the sclera region. When the eye is looking straight ahead, the image of the eye captured by the image acquisition unit 50 can be as follows: Figure 5 As shown in the middle figure (B), in the image of the eye, the cornea region partially overlaps with the pupil region, and the pupil region and the cornea region are located in the middle of the sclera region. When the eye's line of sight is tilted to the left front, the image of the eye captured by the image acquisition unit 50 can be as follows: Figure 5 As shown in the middle figure (C), in the image of the eye, the cornea region partially overlaps with the pupil region, and the pupil region and the cornea region are located on the relative right side of the sclera region.

[0076] like Figure 6 As shown, in some embodiments, the light source assembly 10 includes a visible light source and multiple infrared light sources. The multiple infrared light sources are used to emit infrared light toward different positions of the optical path adjustment assembly 40. The optical path adjustment assembly 40 can emit infrared light emitted by the multiple infrared light sources to different positions of the pupil area, thereby improving the accuracy of eye tracking.

[0077] For example, the plurality of infrared light sources may be arranged in a ring shape, and the infrared light emitted by the plurality of infrared light sources forms a plurality of light spots in a ring shape in the pupil area, such as Figure 6 shown.

[0078] Exemplarily, at least one infrared light source is located in a central area of ​​a plurality of infrared light sources arranged in a ring.

[0079] It can be understood that the light source assembly 10 includes an infrared light source assembly and a visible light source assembly, and the visible light source assembly and the infrared light source assembly can be arranged separately, or the visible light source assembly and the infrared light source assembly can be arranged adjacent to each other. For example, taking the near-eye display device as an AR / VR glasses as an example, the housing of the near-eye display device includes a frame and temples, then when the visible light source assembly is arranged on the temples, the infrared light source assembly can be arranged on the frame, or the infrared light source assembly and the visible light source assembly can be arranged on the temples together.

[0080] In some embodiments, the optical path adjustment component 40 includes a reflective element, which has a reflective surface facing the light source component 10 and the display side, and the infrared light can be reflected to the pupil area of ​​the eye through the reflective surface. Alternatively, the optical path adjustment component 40 includes a refractive element, which has an incident surface and an exit surface that are arranged opposite to each other, the incident surface facing the light source component 10, and the exit surface facing the display side, and the infrared light can enter the refractive element through the incident surface, and be emitted from the exit surface to the pupil area of ​​the eye through refraction.

[0081] It can be understood that when the user wears a near-eye display device, the user's eyes are located on the display side of the optical waveguide 20. Then, when the optical path adjustment component 40 includes a reflective element, the reflective surface of the reflective element faces the light source component 10 and the user's eyes, and the infrared light can be reflected through the reflective surface to the pupil area of ​​the user's eyes. When the optical path adjustment component 40 includes a refractive element, the incident surface of the refractive element faces the light source component 10, and the exit surface faces the user's eyes, and the infrared light can be refracted through the refractive element to the pupil area of ​​the user's eyes.

[0082] In some embodiments, the optical path adjustment component 40 is located between the light source component 10 and the optical waveguide 20, and the optical path adjustment component 40 can transmit visible light and reflect infrared light. It can be understood that the optical path adjustment component 40 can reflect infrared light to the pupil area of ​​the eye, and the reflection adjustment method makes the optical path adjustment of the infrared light by the optical path adjustment component 40 more accurate, and the energy loss of the infrared light is smaller, and the optical path adjustment component 40 can transmit visible light, so that the visible light emitted by the light source component 10 is coupled into the optical waveguide 20 through the optical path adjustment component 40 and coupled out to the display side after being transmitted inside the optical waveguide 20. The optical path adjustment component 40 can transmit visible light, which makes the setting position of the optical path adjustment component 40 more flexible, and facilitates the compact arrangement of the devices inside the near-eye display device.

[0083] Exemplarily, the light source assembly 10 and the optical path adjustment assembly 40 are arranged on the temples, the optical waveguide 20 is arranged on the frame, and the optical path adjustment assembly 40 is located between the light source assembly 10 and the optical waveguide 20. There is a certain gap between the frame and the user's eyes, so that the infrared light reflected by the optical path adjustment assembly 40 has enough transmission space, ensuring that when the pupil area of ​​the user's eyes rotates to any position, the infrared light can be transmitted to the pupil area of ​​the eyes.

[0084] Furthermore, the optical path adjustment component 40 includes a dichroic mirror.

[0085] Exemplarily, the dichroic mirror includes a substrate, and a dichroic filter film layer and an anti-reflection film layer disposed on opposite sides of the substrate. When the infrared light and visible light emitted by the light source assembly 10 irradiate the dichroic mirror, the dichroic filter film layer will reflect the infrared light, generating interference enhancement on the infrared light, while the dichroic mirror is completely transparent to the visible light, and the visible light passes through the dichroic mirror and then emits to the optical waveguide 20. The dichroic mirror can effectively separate the infrared light and the visible light, and the dichroic mirror causes less light intensity loss to the infrared light and the visible light, which is conducive to ensuring that the infrared light is transmitted to the pupil area and forms a clear light spot in the cornea area, and the visible light forms a clear image in the eye after being transmitted through the optical waveguide 20.

[0086] In other embodiments, the optical path adjustment component 40 may also be located on the side of the optical waveguide 20 away from the light source component 10, that is, the optical waveguide 20 is located between the optical path adjustment component 40 and the light source component 10. The infrared light emitted by the light source component 10 can pass through the optical waveguide 20 and then be transmitted to the optical path adjustment component 40. After being emitted from the optical path adjustment component 40, it can pass through the optical waveguide 20 again and then be transmitted to the pupil area of ​​the user's eye.

[0087] In some embodiments, the optical path adjustment component 40 includes a lens and a mounting seat, the lens is arranged on the mounting seat, the drive component 30 is connected to the mounting seat, and the drive motor can adjust the lens to the target posture. It can be understood that the lens can reflect or refract infrared light to change the transmission path of infrared light, and the lens is arranged on the mounting seat and is connected to the drive component through the mounting seat to facilitate the installation and fixation of the lens.

[0088] Exemplarily, the lens can be fixed on the mounting seat by gluing or snapping.

[0089] Furthermore, the mounting seat includes a first seat body and a second seat body, the second seat body is rotatably connected to the first seat body, and the lens is arranged on the second seat body. The drive motor includes a first drive motor and a second drive motor, the first drive motor is transmission-connected to the first seat body, the second drive motor is transmission-connected to the second seat body, the first drive motor can drive the first seat body and the second seat body to rotate around a first axis, and the second drive motor can drive the second seat body to rotate around a second axis relative to the first seat body, and the first axis is different from the second axis. It can be understood that the first axis is inclined relative to the second axis, or the first axis and the second axis are perpendicular, so that the first seat body and the second seat body can drive the lens to achieve universal rotation, so that the infrared light can be accurately emitted to the pupil area of ​​the eye through reflection or refraction of the lens, thereby achieving high-precision eye tracking.

[0090] Furthermore, both the first seat body and the second seat body are annular structures, the first seat body is arranged around the outer periphery of the second seat body, the two opposite ends of the second seat body are respectively rotatably connected to the two opposite ends of the first seat body, the first drive motor is connected to the middle part of the first seat body, the second seat body is arranged around the outer periphery of the lens, and the first axis is perpendicular to the second axis. It can be understood that the two opposite ends of the second seat body are respectively connected to the two opposite ends of the first seat body, so as to ensure the connection stability between the first seat body and the second seat body. At the same time, the first seat body is arranged around the outer periphery of the second seat body, and the second seat body is arranged around the outer periphery of the lens. The first seat body and the second seat body can avoid the two opposite sides of the lens, so that the light can pass through the two opposite sides of the lens and emit, avoiding blocking the transmission of light.

[0091] Exemplarily, the first base body and the second base body may be an open-loop structure, or the first base body and the second base body may be a closed-loop structure.

[0092] Exemplarily, as described above, when the lens can transmit visible light and reflect infrared light, the first seat and the second seat are located at the periphery of the lens, so that visible light can pass through the opposite sides of the lens and be emitted to the optical waveguide 20 .

[0093] Specifically, the optical waveguide 20 includes a first waveguide and a second waveguide arranged at intervals along a first direction, one side of the first waveguide and the second waveguide along the second direction is a display side, the first axis is perpendicular to the first direction and the second direction, and the second axis is perpendicular to the first axis. It can be understood that the first waveguide and the second waveguide can be respectively arranged corresponding to the left and right eyes of the user, the direction from the optical waveguide 20 to the user's eyes is the second direction, the first axis is perpendicular to the first direction and the second direction, so that when the optical path adjustment component 40 rotates around the first axis, the infrared light reflected by the optical path adjustment component 40 can move in the width direction of the user's eyes, that is, move to the left and right sides of the user's eyes, and the second axis is perpendicular to the first axis, so that when the optical path adjustment component 40 rotates around the second axis, the infrared light reflected by the optical path adjustment component 40 can move in the height direction of the user's eyes, that is, move to the upper and lower sides of the user's eyes, ensuring that when the eyes are looking at any position, the infrared light can be emitted to the pupil area through the optical path adjustment component 40 to achieve high-precision eye tracking.

[0094] In other embodiments, the mounting seat may include a third seat body and a fourth seat body, the third seat body may be arranged on the shell, the third seat body and the fourth seat body are spherically hinged, and the driving motor can drive the fourth seat body to rotate universally relative to the third seat body, thereby adjusting the lens to the target posture.

[0095] The embodiment of the present invention further provides a near-eye display device. The near-eye display device may be AR glasses, VR glasses, AR helmets, VR helmets, etc. Specifically, Figure 2 , Figure 5 and Figure 6 As shown, the near-eye display device includes a light source assembly 10, an optical waveguide 20, a driving assembly 30, an optical path adjustment assembly 40, an image acquisition unit 50, and a processing unit 60. The light source assembly 10 is used to emit visible light and infrared light. The optical waveguide 20 has a display side, and visible light can be coupled into the optical waveguide 20 and coupled out to the display side of the optical waveguide 20 after being transmitted in the optical waveguide 20. The optical path adjustment assembly 40 is transmission-connected to the driving assembly 30, and infrared light can be irradiated to the optical path adjustment assembly 40. The image acquisition unit 50 is used to capture images of the eye. The processing unit 60 is electrically connected to the driving component 30 and the image acquisition unit 50, and the processing unit is used to: determine the position of a preset area of ​​the eye according to an image of the eye; determine the target posture of the optical path adjustment component 40 according to the position of the preset area of ​​the eye; control the driving component 30 to adjust the optical path adjustment component 40 to the target posture, so that the optical path adjustment component 40 emits infrared light to the pupil area of ​​the eye; determine the gaze direction of the eye according to the light spot formed by the infrared light in the image of the eye; and control the near-eye display device according to the gaze direction of the eye.

[0096] After the image of the eye is collected by the image acquisition unit, the features in the image of the eye can be extracted by image recognition to determine the position of the preset area of ​​the eye, so that the target posture of the optical path adjustment component can be determined according to the position of the preset area of ​​the eye. After the driving component adjusts the optical path adjustment component to the target posture, the optical path adjustment component in the target posture can change the optical path of the infrared light emitted by the light source component, so that the infrared light passes through the optical path adjustment component and is emitted to the pupil area of ​​the eye, forming a light spot in the cornea area of ​​the eye, and then the image of the eye is collected again by the image acquisition unit, and the processing unit can determine the gaze direction of the eye according to the relative position of the light spot formed by the infrared light and the pupil area in the image of the eye collected again. Among them, the optical path adjustment component can be driven by the driving component to expand the range of infrared light transmission, ensure that when the eye is looking at any position, the infrared light can be transmitted to the pupil area, and increase the range of eye tracking. According to the position of the preset area of ​​the eye, the driving component is controlled to adjust the optical path adjustment component, so that the infrared light can be transmitted to the pupil area more accurately, avoiding the infrared light falling on the sclera area and interfering with the eye tracking, and realizing high-precision eye tracking.

[0097] It is understandable that when the gaze direction of the eyes is determined and eye tracking is achieved, the near-eye display device can be controlled according to the user's eye state to achieve various interactive and perceptual functions. For example, when the user's eyes remain fixed on a virtual button on the screen of the near-eye display device for a preset time, the virtual cursor can be controlled to click the virtual button and jump to the next interface. Alternatively, the rendering resolution of the non-gaze area is adjusted according to the gaze direction of the eyes. Alternatively, when the gaze direction of the user's eyes changes rapidly, some operations of the near-eye display device can be locked to avoid accidental touches. The specific control can be adjusted according to actual needs and is not limited here.

[0098] The method of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0099] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The method implemented when the computer program is executed by a processor can refer to the various embodiments of the eye tracking method based on a near-eye display device of the present application.

[0100] The computer-readable storage medium may be an internal storage unit of the near-eye display device described in the aforementioned embodiment, such as a hard disk or memory of the near-eye display device. The computer-readable storage medium may also be an external storage device of the near-eye display device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the near-eye display device.

[0101] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0102] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0103] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An eye tracking method based on a near-eye display device, characterized in that: The near-eye display device comprises: A light source assembly, used for emitting visible light and infrared light; An optical waveguide, the optical waveguide having a display side, the visible light being able to be coupled into the optical waveguide and coupled out to the display side of the optical waveguide after being transmitted in the optical waveguide; Drive components; An optical path adjustment component, which is transmission-connected to the driving component, and the infrared light can irradiate the optical path adjustment component; An image acquisition unit, used for acquiring an image of the eye; The eye tracking method comprises: acquiring an image of the eye; Determine the position of a preset eye area according to the image; Determining a target posture of the optical path adjustment component according to the position of the preset eye area; Controlling the driving component to adjust the optical path adjustment component to the target posture, so that the optical path adjustment component emits the infrared light to the pupil area of ​​the eye; acquiring the image of the eye again; The gaze direction of the eye is determined according to the light spot formed by the infrared light in the image acquired again.

2. The eye tracking method according to claim 1, characterized in that: The preset eye area includes the pupil area, cornea area and sclera area; The step of determining the position of the preset eye area according to the image comprises: determining a positional relationship among the pupil region, the cornea region, and the sclera region according to the image of the eye; The step of determining the target posture of the optical path adjustment component according to the position of the preset eye area includes: The target posture of the optical path adjustment component is determined according to the positional relationship among the pupil area, the cornea area and the sclera area.

3. The eye tracking method according to claim 1, characterized in that: The optical path adjustment component includes a reflective element, the reflective element has a reflective surface facing the light source component and the display side, and the infrared light can be reflected to the pupil area through the reflective surface; or, The optical path adjustment component includes a refractive element, which has an incident surface and an exit surface arranged opposite to each other, the incident surface faces the light source component, and the exit surface faces the display side. The infrared light can enter the refractive element through the incident surface and be refracted and emitted from the exit surface to the pupil area.

4. The eye tracking method according to any one of claims 1 to 3, characterized in that: The optical path adjustment component is located between the light source component and the optical waveguide, and the optical path adjustment component can transmit the visible light and reflect the infrared light.

5. The eye tracking method according to claim 4, characterized in that: The optical path adjustment component includes a dichroic mirror.

6. The eye tracking method according to any one of claims 1 to 3, characterized in that: The optical path adjustment component includes a lens and a mounting seat, the lens is arranged on the mounting seat, the driving component is transmission-connected with the mounting seat, and the driving motor can adjust the lens to the target posture.

7. The eye tracking method according to claim 6, characterized in that: The mounting seat comprises a first seat body and a second seat body, the second seat body is rotatably connected to the first seat body, and the lens is arranged on the second seat body; The drive motor includes a first drive motor and a second drive motor, the first drive motor is transmission-connected to the first seat body, the second drive motor is transmission-connected to the second seat body, the first drive motor can drive the first seat body and the second seat body to rotate around a first axis, the second drive motor can drive the second seat body to rotate around a second axis relative to the first seat body, and the first axis is different from the second axis.

8. The eye tracking method according to claim 7, characterized in that: The optical waveguide includes a first waveguide and a second waveguide spaced apart in a first direction, one side of the first waveguide and the second waveguide in a second direction is the display side, the first axis is perpendicular to the first direction and the second direction, and the second axis is perpendicular to the first axis.

9. The eye tracking method according to any one of claims 1 to 3, characterized in that: The light source assembly includes a visible light source and a plurality of infrared light sources, and the plurality of infrared light sources are used to emit infrared light toward different positions of the optical path adjustment assembly.

10. A near-eye display device, characterized in that: include: A light source assembly, used for emitting visible light and infrared light; An optical waveguide, the optical waveguide having a display side, the visible light being able to be coupled into the optical waveguide and coupled out to the display side of the optical waveguide after being transmitted in the optical waveguide; Drive components; An optical path adjustment component, which is transmission-connected to the driving component, and the infrared light can irradiate the optical path adjustment component; An image acquisition unit, used for acquiring an image of the eye; A processing unit is electrically connected to the driving component and the image acquisition unit, and the processing unit is used to execute the steps of the eye tracking method as described in any one of claims 1 to 9.

Images