Eye movement detection device

Abstract

The invention discloses an eye movement detection device, and the device comprises an optical module which comprises a light source and a camera, the light source comprises a plurality of lamp beads, the lamp beads are used for emitting light to human eyes, and the camera is used for receiving the light reflected by the human eyes and collecting human eye images; the driving module is in communication connection with the optical module and is used for adjusting the light direction of the light emitted by the lamp beads; and the processing module is in communication connection with the driving module and the optical module, and is used for obtaining the number of effective light spots in the human eye image, and controlling the driving module to adjust the light direction under the condition that the number of the effective light spots is smaller than a preset number threshold value until the number of the effective light spots in the human eye image is larger than or equal to the preset number threshold value. According to the invention, the driving module adjusts the light direction to supplement the light spots, ensures that the number of effective light spots always meets the sight line estimation requirement, provides stable and sufficient light spot parameter support for sight line estimation, and guarantees the high-precision output of sight line estimation.

Description

Technical Field

[0001] This application belongs to the field of head-mounted display devices, specifically relating to an eye-tracking detection device. Background Technology

[0002] With the rapid development of human-computer interaction, VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality), eye-tracking technology has become a key support for improving product interaction and immersion. Existing eye-tracking technologies are often based on the pupil-corneal reflection method. This typically involves setting up an array of LEDs (Light Emitting Diodes) to illuminate the human eye. The LEDs reflect light onto the pupil, creating multiple reflective spots, with the number of spots matching the number of LEDs. By collecting the reflected light spots and pupil characteristics, the direction of gaze can be calculated.

[0003] However, in actual use, due to differences in human eye parameters, wearing fit, and the wearing of refractive lenses, some light spots are missing (for example, an LED light array has 6 LEDs, but only 4 light spots are detected), resulting in inaccurate line of sight calculation. Summary of the Invention

[0004] This application aims to provide an eye-tracking detection device that improves the accuracy of eye-tracking gaze.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows:

[0006] This application provides an eye-tracking detection device, comprising: an optical module including a light source and a camera, the light source comprising multiple LEDs for emitting light toward a human eye, and the camera for receiving light reflected from the human eye and acquiring an image of the human eye; a driving module, communicatively connected to the optical module, for adjusting the direction of the light emitted by the LEDs; and a processing module, communicatively connected to the driving module and the optical module, for acquiring the number of effective light spots in the human eye image, and controlling the driving module to adjust the direction of the light when the number of effective light spots is less than a preset threshold, until the number of effective light spots in the human eye image acquired by the optical module is greater than or equal to the preset threshold.

[0007] In the embodiments of this application, the eye-tracking detection device includes an optical module, a driving module, and a processing module. The optical module, driving module, and processing module form a collaborative closed loop. Light emitted from the light source in the optical module is reflected by the human eye, and the camera in the optical module captures the human eye image in real time and transmits it to the processing module. The processing module continuously monitors the number of effective light spots in the human eye image using a preset algorithm and compares it with a preset threshold in real time. When it determines that the number of effective light spots has not reached the preset threshold, the processing module can issue an adjustment command to the driving module. The driving module is located in the optical path between the light source and the human eye. After responding to the command, the driving module fine-tunes the direction of the light, allowing more light to accurately project onto the cornea of ​​the human eye and form effective reflection, until the number of effective light spots in the human eye image captured by the optical module meets or exceeds the preset threshold. This dynamic adjustment mechanism ensures that gaze estimation always has sufficient and stable effective light spot parameters, thereby improving the accuracy of gaze estimation in eye tracking.

[0008] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0009] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0010] Figure 1 This is one of the schematic diagrams of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0011] Figure 2 This is a second schematic diagram of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0012] Figure 3 This is an application diagram of adjusting the light source using a micro motor, as provided in this application;

[0013] Figure 4 This is an application diagram illustrating the emission of light from a light source to the human eye, based on the information provided in this application.

[0014] Figure 5 This is a third schematic diagram of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0015] Figure 6 This is the fourth schematic diagram of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0016] Figure 7 This is a schematic diagram illustrating the application of adjusting the direction of light using a liquid crystal adjustable optical element provided in this application;

[0017] Figure 8This is an application diagram illustrating the adjustment of light direction using a beam scanner provided in this application;

[0018] Figure 9 This is the fifth schematic diagram of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0019] Figure 10 This is a schematic diagram of the spot detection process provided in this application;

[0020] Figure 11 This is a schematic diagram of the frame structure of the eye-tracking detection device according to an embodiment of this application;

[0021] Figure 12 This is a schematic diagram of the eye diagram feature extraction process provided in this application;

[0022] Figure 13 This is a flowchart illustrating the pupil detection algorithm provided in this application;

[0023] Figure 14 This is a schematic diagram of the detection results of stray light spots on non-ocular refractive lenses provided in this application;

[0024] Figure 15 This is a flowchart illustrating the process of detecting stray light in refractive lenses provided in this application;

[0025] Figure 16 This is a flowchart of eye-tracking detection device estimation based on the application provided in this application;

[0026] Figure 17 This is a flowchart illustrating the eye-tracking detection process provided in this application.

[0027] Reference numerals: 100 Optical module; 110 Light source; 120 Camera; 111 Lamp bead; 200 Drive module; 210 Micro motor; 220 Optical adjustment element; 221 Liquid crystal adjustable optical element; 222 Beam scanner; 230 Power adjustment module; 300 Processing module; 410 Parameter calibration module; 420 Gaze estimation module; 430 Pupil feature detection module; 440 Blink recognition module; 450 Diopter lens recognition and stray light processing module; 10 Human eye; 20 Diopter lens. Detailed Implementation

[0028] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] The terms "first" and "second" in the specification and claims of this application may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise stated, "multiple" means two or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0031] Eye tracking is a technology that detects the position of a user's eyes, gaze, and eye movements (salivation, staring, blinking). It is widely used in interaction design, usability studies, rendering optimization (such as foveated rendering), and accessible input. Current technologies often rely on pupil-corneal reflection to perform eye tracking, calculating the direction of gaze by collecting reflected light spots from the eye and pupil features.

[0032] The following is combined Figures 1 to 17 This application describes an eye-tracking detection device according to an embodiment of the present application.

[0033] like Figure 1As shown, an eye-tracking detection device according to some embodiments of this application includes: an optical module 100, including a light source 110 and a camera 120, wherein the light source 110 includes a plurality of LEDs 111, the LEDs 111 being used to emit light to a human eye, and the camera 120 being used to receive light reflected by the human eye and acquire an image of the human eye; a driving module 200, communicatively connected to the optical module 100, being used to adjust the light direction of the light emitted by the LEDs 111; and a processing module 300, communicatively connected to the driving module 200 and the optical module 100, wherein the processing module 300 is used to acquire the number of effective light spots in the human eye image, and when the number of effective light spots is less than a preset threshold, controlling the driving module 200 to adjust the light direction until the number of effective light spots in the human eye image acquired by the optical module is greater than or equal to the preset threshold.

[0034] An eye-tracking detection device according to an embodiment of this application includes an optical module 100, a driving module 200, and a processing module 300. The optical module 100 includes a light source 110 and a camera 120. The light source 110 includes multiple LEDs 111, which emit light towards the human eye. The light is reflected off the corneal surface and captured by the camera 120 to form effective light spots. The camera 120 can acquire human eye images in real time and transmit them to the processing module 300. The processing module 300 continuously monitors the number of effective light spots in the human eye image using a preset algorithm and compares it in real time with a preset threshold. When the number of effective light spots is determined to be less than the preset threshold, the processing module 300 sends an adjustment command to the driving module 200. Upon responding to the command, the driving module 200 fine-tunes the light direction to ensure more light is accurately projected onto the cornea and forms effective reflections, until the number of effective light spots in the human eye image acquired by the optical module 100 meets or exceeds the preset threshold. This dynamic adjustment mechanism ensures that gaze estimation always has sufficient and stable effective light spot parameters, thereby improving the accuracy of gaze estimation in eye tracking.

[0035] It is understandable that the LED 111 is an independent light-emitting unit in the light source 110 and is the basic light-emitting carrier for forming an effective light spot; the light emitted by the LED 111 can be near-infrared light, short-wave infrared light, ultraviolet light, etc.

[0036] For example, the preset quantity threshold can be set to 3, 5 or 7, etc., to ensure that the number of effective light spots meets the requirements of the visual assessment.

[0037] like Figure 2 As shown, according to some embodiments of this application, the drive module 200 includes: a micro motor 210, which is connected to the light source 110 for transmission. The micro motor 210 is used to drive the light source 110 to rotate and adjust the light direction by adjusting the light emission angle.

[0038] In the above embodiment, the drive module 200 is a micro motor 210. The micro motor 210 can achieve rigid transmission of power to the light source 110 through reliable transmission connection methods such as gears and couplings, ensuring minimal power loss. When the processing module 300 issues an adjustment command, the micro motor 210 can perform high-precision angular displacement control according to the command, directly driving the light source 110 to rotate as a whole, thereby changing the emission angle of the light and causing the light direction to shift. This direct drive method reduces the response delay of intermediate transmission links and can quickly guide the shifted light back to the corneal reflective area of ​​the human eye, quickly filling in the missing effective light spot and ensuring the accuracy of line-of-sight estimation. At the same time, the micro motor 210 has the characteristics of small size, low cost, and stable operation, and is easy to integrate with the overall structure of the device, which can reduce the difficulty of mass production.

[0039] For example, Figure 3 This is a schematic diagram illustrating the application of a micro motor adjusting a light source, as provided in this application. Figure 3 As shown, the solid line represents the initial position of LED 111, and the dashed line represents the adjusted position. The position of LED 111 can be directly changed by the micro motor 210, thereby changing the angle at which light is emitted towards the human eye 10; LED 111 can also be finely adjusted to change the angle at which light is emitted towards the human eye 10.

[0040] According to some embodiments of this application, the processing module 300 is specifically used to: when the number of effective light spots is less than a preset number threshold, identify the exit pupil distance and eye rotation angle of the human eye from the human eye image, and based on the exit pupil distance and eye rotation angle, look up a preset first correspondence table to obtain the lamp head rotation angle of the light source 110, and control the micro motor 210 to drive the light source 110 to rotate according to the lamp head rotation angle, thereby completing the effective light spots, wherein the first correspondence table records the correspondence between the exit pupil distance, the eye rotation angle and the lamp head rotation angle.

[0041] In the above embodiments, the exit pupil distance is the straight-line distance from the human eye's pupil to the reference plane of the camera 120 lens or the reference plane of the light source 110. The human eye rotation angle is the deflection angle of the human eye around the horizontal axis (left-right rotation) or the vertical axis (up-down rotation). The lamp head is the core functional sub-unit integrating the LED beads 111 in the light source 110, and can be understood as the component that installs or carries the LED beads 111. The first correspondence table is a data table specifically recording the mapping relationship between the exit pupil distance, the human eye rotation angle, and the lamp head rotation angle, obtained in advance through multiple sets of experiments. The lamp head rotation angle is the specific angle value that the lamp head needs to rotate to fill in the missing effective light spot.

[0042] In the above embodiment, when the number of effective light spots is less than a preset threshold, the processing module 300 first extracts the pupil edge features and corneal reflection point position from the human eye image using an image analysis algorithm, and then calculates the exit pupil distance and the horizontal and vertical eye rotation angles. Based on these two key parameters, the processing module 300 quickly retrieves a preset first correspondence table. The first correspondence table has been calibrated through a large number of sample experiments and has established a precise mapping relationship between different exit pupil distances, eye rotation angles and lamp head rotation angles, which can directly match the optimal lamp head rotation angle parameters. Subsequently, the processing module 300 sends a control command for this lamp head rotation angle to the micro motor 210. The micro motor 210 drives the light source 110 to rotate precisely according to the command, so that the light direction is completely adapted to the current human eye position and posture, ensuring that the effective light spots are quickly filled. This control method based on the first correspondence table avoids complex real-time calculation processes, which shortens the adjustment response time, reduces the computing power consumption of the processing module 300, ensures the smooth operation of the device, and improves the accuracy of effective spot completion, thus optimizing both the real-time performance and accuracy of eye tracking.

[0043] For example, Figure 4 This is a schematic diagram illustrating an application scenario where a light source 110 emits light towards a human eye 10, as provided in this application. Figure 4 As shown, during eye tracking, multiple LED beads 111 in the light source 110 can emit light to the human eye 10 from multiple positions simultaneously. When the rotation angle of the lamp head is adjusted, the angles of multiple LED beads 111 can be adjusted simultaneously.

[0044] For example, during the initial simulation of the light source 110 structural layout scheme, the position of the light source 110 can be designed according to different ER (Eye Reflectivity) values ​​and eye rotation angles, establishing a first correspondence table. Specifically, using adjustable light sources 110, N positions of the light sources 110 are set. In system simulation, the disappearance sequence number of the light spot at different eye rotation positions is determined, and the corresponding position is set. Before use, the hardware parameters of the N positions of the light sources 110 are calibrated, establishing a hardware parameter table for different positions, i.e., the first correspondence table. In actual use, the hardware parameters of the corresponding positions of the LED beads 111 are directly obtained by looking up the table for line-of-sight estimation.

[0045] like Figure 5 As shown, according to some embodiments of this application, it further includes: a parameter calibration module 410, coupled to the processing module 300, used to calibrate the hardware parameters of the light source 110 based on the exit pupil distance and the human eye rotation angle; and a gaze estimation module 420, coupled to the parameter calibration module 410, specifically used to perform human eye gaze estimation based on the completed effective light spot and hardware parameters.

[0046] In the above embodiment, the parameter calibration module 410 is coupled to the processing module 300, receives the exit pupil distance and human eye rotation angle data transmitted by the processing module 300, and, in conjunction with the hardware parameters of the light source 110 (such as the spacing between the LED beads 111, the emission angle, etc.), corrects the hardware parameter deviation through a basic calibration algorithm (such as a simple fitting algorithm), and outputs the calibrated hardware parameters. At the same time, the processing module 300 synchronously transmits the supplemented effective light spot data to the gaze estimation module 420. The gaze estimation module 420 can perform human eye gaze estimation based on the supplemented effective light spot and hardware parameters according to the basic optical model. The effective light spot can provide accurate spatial coordinates of corneal reflection, and the calibrated hardware parameters of the light source 110 ensure the accuracy of the light propagation path calculation. The two work together to reduce the system error of gaze estimation and improve the accuracy and adaptability of gaze estimation in different scenarios.

[0047] like Figure 6 As shown, according to some embodiments of this application, the driving module 200 includes:

[0048] An optical adjustment element 220 is disposed on the light propagation path of the light source 110 and is used to change the propagation direction of the light to adjust the light direction.

[0049] In the above embodiments, the driving module 200 may include an optical adjustment element 220. The optical adjustment element 220 is deployed on the light propagation path of the light source 110. It does not require direct contact or rotation of the light source 110 body. It can directly change the propagation direction of the light by means of optical refraction, reflection and other principles, thereby adjusting the light direction. This non-mechanical contact adjustment method avoids wear and vibration caused by mechanical transmission, reduces mechanical loss of the device and extends the service life of the core components. At the same time, the response speed of the optical adjustment element 220 is not limited by mechanical inertia and can achieve microsecond-level fast adjustment, which can more accurately guide the light to the effective reflection area of ​​the human cornea and ensure the rapid filling of the effective light spot.

[0050] For example, the optical adjustment element 220 may be a liquid crystal adjustable optical element 221, a beam scanner 222, a microlens array, a mirror assembly, a prism assembly, an acousto-optic modulator, an electro-optic modulator, etc.

[0051] According to some embodiments of this application, the optical adjustment element 220 is a liquid crystal adjustable optical element 221. By applying a voltage to the liquid crystal adjustable optical element 221, the refractive index of the liquid crystal adjustable optical element 221 is adjusted, and the light is refracted to adjust the direction of the light.

[0052] In the above embodiment, the optical adjustment element 220 adopts a liquid crystal tunable optical element 221. Utilizing the electro-optic effect of liquid crystal materials, by applying voltages of different amplitudes to the liquid crystal tunable optical element 221, the alignment state of liquid crystal molecules can be precisely controlled, thereby changing the refractive index of the material. When light passes through the liquid crystal tunable optical element 221, the refraction angle is continuously adjusted according to the change in refractive index, thus achieving fine control of the light direction. This electro-tuning method has a response speed as fast as microseconds, which can meet the high-frequency effective light spot completion requirements. At the same time, the liquid crystal tunable optical element 221 can achieve continuous angle adjustment, rather than discrete step adjustment, which can more accurately adapt to the dynamic changes of the human eye and ensure the stability of the effective light spot.

[0053] For example, such as Figure 7 As shown, a liquid crystal adjustable optical element 221 is set in the light propagation path of the LED bead 111. By applying voltage to the liquid crystal adjustable optical element 221, the refractive index can be dynamically adjusted to produce different degrees of light deflection. During eye tracking, the algorithm detects the number of effective light spots in real time. If an effective light spot is missing, it immediately feeds back the applied voltage to change the refractive index of the optical element. When the required effective light spot appears in the eye diagram, the algorithm records and feeds back the current voltage, and adjusts the voltage in a timely manner according to the received information, so that the light emitted by the LED bead 111 caused by the liquid crystal adjustable optical element 221 is deflected to match the diopter lens 20 and the eyeball rotation angle of the human eye 10.

[0054] According to some embodiments of this application, the optical adjustment element 220 is a beam scanner 222, which has a light scanning plane and adjusts the direction of the light by reflecting the light through rotating the light scanning plane.

[0055] In the above embodiments, the optical adjustment element 220 can be a beam scanner 222. The light scanning plane of the beam scanner 222 is made of a high-reflectivity material, which can achieve efficient reflection of light and reduce light energy loss. When the processing module 300 issues an adjustment command, the light scanning plane of the beam scanner 222 rotates around a preset rotation axis. After the light is projected onto the light scanning plane, the reflection direction will change precisely with the change of the rotation angle of the light scanning plane, thereby realizing the adjustment of the light direction. The beam scanner 222 has high optical path turning efficiency and can maintain stable reflection performance over a wide wavelength range, ensuring the brightness consistency of the effective light spot. At the same time, the rotation angle control precision of the light scanning plane is high, which can achieve sub-arcsecond adjustment resolution, can quickly respond to the commands of the processing module 300, and promptly fill in the effective light spot during the movement of the human eye. Its strong anti-interference capability allows it to work stably in complex lighting environments, significantly improving the environmental adaptability and operational reliability of the device.

[0056] For example, such as Figure 8As shown, a beam scanner 222 is set on the light propagation path of the lamp bead 111. By rotating the beam scanner 222, the light emitted from the lamp bead 111 is turned until the desired effective light spot appears in the human image. The algorithm records and feeds back the current rotation angle information.

[0057] For example, the beam scanner 222 can sample scanning mirrors, MEMS (Micro-Electro-Mechanical Systems), etc.

[0058] like Figure 9 As shown, according to some embodiments of this application, the driving module 200 includes: a power adjustment module 230, which is communicatively connected to the light source 110 and is used to adjust the optical power of the light source 110; the processing module 300 is also used to detect the brightness of the effective light spot in real time, and when the brightness is lower than a first preset brightness threshold, the power adjustment module 230 increases the optical power of the light source 110; when the brightness is higher than a second preset brightness threshold, the power adjustment module 230 decreases the optical power of the light source 110, wherein the first preset brightness threshold is less than the second preset brightness threshold.

[0059] In the above embodiments, the driving module 200 includes a power adjustment module 230, enabling the driving module 200 to have dual adjustment capabilities for the number and brightness of effective light spots. While monitoring the number of effective light spots, the processing module 300 can detect the brightness value of each effective light spot in real time and compare it with a preset first and second preset brightness thresholds. When the brightness of an effective light spot is lower than the first preset brightness threshold, the processing module 300 sends a boost command to the power adjustment module 230, which increases the light power of the light source 110, enhancing the light intensity and raising the brightness of the effective light spots to a clearly identifiable range. When the brightness of an effective light spot is higher than the second preset brightness threshold, a deboost command is sent to reduce the light power of the light source 110, preventing excessively bright effective light spots from causing image saturation in the camera 120 or visual interference to the human eye. This dynamic brightness control can accurately compensate for the light energy loss caused by the reflection, absorption and refraction of light by the diopter lens, while also offsetting the light energy attenuation caused by factors such as light source rotation, the configuration of beam scanning devices and liquid crystal adjustable optical elements. This ensures that the effective light spot is always in the optimal detection brightness range, allowing the effective light spot to maintain high recognition and clear imaging in various scenarios, significantly improving the environmental adaptability of the device, and providing a reliable brightness level guarantee for the stable output of the processing module to perform human eye line estimation.

[0060] like Figure 4 As shown, according to some embodiments of this application, the lamp bead 111 is an infrared light-emitting diode lamp, and multiple lamp beads 111 are arranged in a ring.

[0061] In the above embodiments, the LED beads 111 can be infrared LEDs, and multiple LED beads 111 are arranged in a ring. The emission band of the infrared LEDs is in the infrared region invisible to the human eye, so it will not cause visual interference to the user. In addition, the infrared LEDs have low power consumption, fast start-up response, and long service life, which can reduce the overall energy consumption and maintenance cost of the device. Multiple infrared LEDs are evenly distributed in a ring, which can form a ring light source array 110 around the lens of the camera 120. After the emitted infrared light is projected onto the human eye, it will form a uniformly distributed array of effective light spots on the corneal surface. This uniformly distributed array of effective light spots can reflect the spatial position and orientation of the cornea from multiple angles, which makes it easy for the processing module 300 to quickly establish a one-to-one correspondence between the LED beads 111 and the effective light spots. When some effective light spots are missing, the missing area can be quickly located by the position information of adjacent effective light spots, improving the detection and completion efficiency of effective light spots. At the same time, the uniformly distributed effective light spots can provide a more comprehensive spatial coordinate reference for line of sight estimation, reducing the estimation deviation caused by uneven distribution of light spots.

[0062] According to some embodiments of this application, the processing module 300 is further specifically used to: use a spot detection model to detect candidate spots from the region of interest image corresponding to the human eye image, and identify the first position of the candidate spot in the region of interest image and the correspondence between the candidate spot and the LED bead 111; based on the positional relationship between the multiple LED beads 111 and the positional relationship between the multiple LED beads 111 and the human eye, map the multiple LED beads 111 to the region of interest image to obtain multiple target positions; remove candidate spots that do not conform to the target positions to obtain the detected valid spots; and based on the first position and the correspondence, map the valid spots back to the human eye image.

[0063] In the above embodiments, the spot detection model is an algorithmic model used to identify and locate candidate spots in an image, such as a deep learning model or a traditional image processing model. The region of interest image is a local image cropped from the human eye image, focusing on the core area of ​​the human eye, such as the pupil or the periphery of the iris. Candidate spots are image highlights identified by the spot detection model that may be valid spots. The first position is the coordinate position of the candidate spot in the region of interest image. The target position is mapped to the theoretical position of the valid spot in the region of interest image based on the positional relationship between the multiple LED beads 111 and the positional relationship between the multiple LED beads 111 and the human eye.

[0064] In the above embodiment, the processing module 300 first crops out the region of interest image focused on the pupil and iris from the complete human eye image, significantly reducing the detection range of the effective light spot, reducing the interference of background noise in other areas of the face on the detection results, and improving detection efficiency; then, it calls the light spot detection model to analyze the region of interest image, identifies all candidate light spots that meet the characteristics of light spot brightness and shape, and records the first position of each candidate light spot in the region of interest image, and establishes the correspondence between candidate light spots and LED beads 111 through feature matching algorithm; based on the pre-calibrated positional relationship between multiple LED beads 111 and the relative position parameters of multiple LED beads 111 and the human eye, the processing module 300 maps the positions of all LED beads 111 to the region of interest image, generating the target position that each effective light spot should have; by comparing the deviation between the first position of the candidate light spot and the target position, invalid candidate light spots with excessive deviation and that do not meet the target position are eliminated, and finally high-purity effective light spots are obtained, and the position information of the effective light spots is mapped back to the complete human eye image. This ensures the accuracy of effective spot quantity detection, provides a reliable basis for the processing module 300 to determine the preset quantity threshold and drive control, and significantly enhances the device's anti-interference capability.

[0065] For example, such as Figure 10 As shown, a keypoint detection / object detection model based on the ResNet+UNet structure (Residual Network + U-Net Structure) detects the index and coordinates of light spots in a 128×128 eye diagram. Candidate light spot localization is performed to facilitate tracking by the light spot module and to avoid overdetection in the light spot detection model (i.e., the number of light spots output by the model is greater than the actual number of light spots in the image). Light spot matching involves searching and matching the results of potential bright spot localization with the results directly predicted by the light spot model. Sub-pixel localization and original image coordinate mapping are then performed to obtain the light spot index, coordinates, and light spot morphology parameters (spot size, circularity, etc.).

[0066] like Figure 11 As shown, according to some embodiments of this application, it also includes:

[0067] The pupil feature detection module 430 is communicatively connected to the processing module 300 and the camera 120, and is used to detect pupil features from human eye images.

[0068] The gaze estimation module 420 is coupled to the processing module 300. Specifically, the gaze estimation module 420 is used to estimate the gaze of the human eye based on the completed effective light spot and pupil features.

[0069] In the above embodiment, the pupil feature detection module 430, processing module 300, and camera 120 form a collaborative detection mechanism. The human eye image acquired by camera 120 is synchronously transmitted to processing module 300 and pupil feature detection module 430. Pupil feature detection module 430 can accurately extract key pupil features such as the center coordinates, radius, and contour of the pupil from the human eye image through algorithms such as image segmentation and edge extraction. These pupil features directly reflect the spatial position and morphological changes of the pupil. Processing module 300 transmits the completed effective light spot data to gaze estimation module 420 and fuses it with pupil feature data. Thus, gaze estimation module 420 can estimate the human eye's gaze based on the completed effective light spot and pupil features, and calculate the direction of the human eye's gaze. Pupil features can comprehensively reflect the overall movement state of the eyeball, effectively offsetting estimation deviations caused by eyeball rotation and slight head movements, and significantly improving the accuracy and robustness of gaze estimation. At the same time, pupil feature extraction and effective light spot adjustment are processed synchronously and in parallel, avoiding the delay caused by serial processing and ensuring the real-time performance of gaze estimation.

[0070] like Figure 11 As shown, according to some embodiments of this application, it also includes:

[0071] The blink recognition module 440 is communicatively connected to the camera 120 and the gaze estimation module 420. The blink recognition module 440 is used to recognize blinks from the image of the region of interest corresponding to the human eye image.

[0072] The gaze estimation module 420 is used to exit human eye gaze estimation when blinking is detected.

[0073] In the above embodiment, the human eye image captured by camera 120 is cropped to obtain a region of interest image covering the eyelid area, and then transmitted to blink recognition module 440. Blink recognition module 440 receives the region of interest image transmitted by camera 120, recognizes the blinking action, and sends a command to gaze estimation module 420 to exit human eye gaze estimation until the eyelids reopen and the calculation restarts. This mechanism filters out invalid data during blinking, avoids outputting invalid gaze results, improves the reliability of gaze estimation results, conforms to the physiological activity patterns of the human eye, reduces unnecessary power consumption, and optimizes the user experience.

[0074] According to some embodiments of this application, the pupil feature detection module 430 is specifically used for: performing pixel reduction processing on a human eye image to obtain a pixel-reduced image; performing coarse segmentation on the pixel-reduced image based on a coarse segmentation model to obtain a coarse segmentation result; obtaining the region of interest image corresponding to the human eye image based on the coarse segmentation result; performing fine segmentation on the region of interest image based on a fine segmentation model to obtain a fine segmentation result; and performing post-processing on the fine segmentation result to obtain pupil features.

[0075] In the above embodiment, the pupil feature detection module 430 first performs pixel reduction processing on the high-resolution human eye image. While preserving the approximate outline features of the pupil, it significantly reduces the amount of image data, thereby reducing the computational load of subsequent algorithms. Then, the pixel-reduced image is input into a coarse segmentation model. Fast algorithms such as threshold segmentation and morphological operations can be used to initially locate the approximate region of the pupil, obtaining a coarse segmentation result containing the pupil and a small amount of background, thus narrowing the processing range for fine segmentation. Based on the coarse segmentation result, the corresponding region of interest image is cropped from the original human eye image and input into the fine segmentation model. The fine segmentation model can accurately extract the boundary contour of the pupil using algorithms such as deep learning or high-precision traditional algorithms, obtaining a fine segmentation result. Finally, post-processing operations such as median filtering and contour smoothing are used to remove noise points and edge burrs in the fine segmentation result, correct contour deviations, and output high-precision pupil features. This significantly improves processing speed while ensuring the accuracy of pupil feature extraction, avoiding the computational delay caused by directly performing fine segmentation on the high-resolution image. It allows pupil feature extraction to adapt to the rhythm of effective light spot adjustment in real time, providing accurate and timely input data for gaze estimation.

[0076] According to some embodiments of this application, based on the coarse segmentation result, obtaining the region of interest image corresponding to the human eye image includes: identifying the coarsely segmented pupil region from the coarse segmentation result; calculating the centroid of the coarsely segmented pupil region; mapping the centroid onto the human eye image; and cropping the human eye image with the mapped centroid as the center to obtain the region of interest image.

[0077] In the above embodiment, the pupil feature detection module 430 identifies the coarsely segmented pupil region from the reduced-pixel image using a coarse segmentation model. Although the coarsely segmented pupil region contains a small amount of background, it can accurately reflect the approximate position of the pupil. Subsequently, the centroid coordinates of the coarsely segmented pupil region are calculated, which can accurately represent the center position of the pupil. The centroid coordinates are then mapped back to the original human eye image according to the scaling ratio of the reduced-pixel processing to obtain the centroid position of the pupil in the original human eye image. Finally, using this centroid as the center, the original human eye image is cropped according to a preset size or ratio to obtain the region of interest image focused on the core area of ​​the pupil. This centroid-based cropping method ensures that the region of interest image is always centered on the pupil, maximizing the inclusion of the pupil and the surrounding effective light spot distribution area, while removing a large amount of irrelevant facial background and reducing the interference of background noise on subsequent fine segmentation and effective light spot detection. The data volume of the focused region of interest image is smaller, which can improve the processing speed and accuracy of subsequent algorithms.

[0078] For example, such as Figure 12 As shown, Figure 12This is a schematic diagram of the eye image feature extraction process provided in this application. The specific process includes S1 to S12. The specific steps are as follows: S1: Captured eye image: Obtain the captured eye image; S2: Pupil coarse segmentation model: Input the captured eye image into the pupil coarse segmentation model to obtain the human eye ROI map (Region of Interest). InterestImage (Image of the region of interest); S3: Human eye ROI map: Input the human eye ROI map into the spot detection model and the pupil segmentation model respectively; S4: Spot detection model: Input the human eye ROI map into the spot detection model; S5: Spot detection post-processing: Perform spot detection post-processing on the output of the spot detection model; S6: Spot number, coordinates and size: After spot detection post-processing, obtain the spot number, coordinates and size; S7: Pupil fine segmentation model: Input the human eye ROI map into the pupil segmentation model; S8: Pupil segmentation post-processing: Input the processed result of the pupil segmentation model into the blink judgment module; S9: Pupil fitting ellipse: Integrate the spot number, coordinates and size with the pupil fitting ellipse; S10: Spot + pupil parameter output: Output the integrated spot + pupil parameters; S11: Blink judgment (pupil coarse segmentation model result): Input the result of the pupil coarse segmentation model into the blink judgment module; S12: Early exit: The blink judgment module performs early exit.

[0079] For example, such as Figure 13 As shown, Figure 13 This is a flowchart illustrating the pupil detection algorithm provided in this application. The original image is scaled down to 128×128 to reduce the model's input size and parameters. Based on a coarse pupil segmentation model, such as a U-Net structure, the eye image features are segmented and located, outputting four segmentation results: pupil, iris, sclera, and background. The pupil region in the segmentation result image is identified, and its centroid or ellipse fitting center coordinates are roughly calculated and mapped to the original image. The original image is then cropped to obtain the eye's ROI image. The pupil ROI image is input into a fine pupil segmentation model, also a U-Net structure (but not limited to), which outputs classification results for the occluded pupil region, unoccluded pupil region, iris, sclera, and background. Post-processing is performed on the fine segmentation results, including pupil contour refinement and pupil ellipse fitting, to obtain sub-pixel precision contour parameters.

[0080] like Figure 11As shown, according to some embodiments of this application, a refractive lens identification and stray light processing module 450 is also included. The refractive lens identification and stray light processing module 450 is communicatively connected to the camera 120, the light source 110, and the processing module 300. The refractive lens identification and stray light processing module 450 is used to: identify that the user is wearing a refractive lens from the human eye image; when the user is identified as wearing a refractive lens, turn on the light source 110 and the camera 120, and capture multiple consecutive images of the user's face (excluding the eye area); cache the multiple consecutive images; identify stray light spots from the multiple consecutive images whose positional relationship with the light source 110 is mismatched and whose position is fixed in the multiple consecutive images; and remove the stray light spots from the human eye image.

[0081] In the above embodiment, the device further includes a refractive lens recognition and stray light processing module 450. The refractive lens recognition and stray light processing module 450 can identify whether the user is wearing a refractive lens 20 from the human eye image. When it is determined that the user is wearing a refractive lens 20, the module sends control commands to the light source 110 and the camera 120 to turn on the light source 110 and control the camera 120 to capture multiple consecutive images of the user's face (excluding the eye area). At the same time, these multiple consecutive images are cached in the local storage unit. By analyzing the position changes of the light spots in the multiple consecutive images, the module identifies light spots that do not match the position relationship of the light source 110 and whose positions remain fixed in the multiple consecutive images. These light spots are stray light spots generated by reflection or refraction of the refractive lens 20. Subsequently, the module transmits the position information of the stray light spots to the processing module 300. In the subsequent effective light spot detection process, the processing module 300 automatically removes the light spots located at these positions to avoid stray light spots being misjudged as effective light spots. This allows for the accurate identification and removal of stray light interference from the diopter lens 20, ensuring the accuracy and purity of the effective light spot detection results. At the same time, the stray light processing process is carried out simultaneously with the effective light spot adjustment and feature extraction process, without adding any additional processing delay, ensuring the real-time performance of the device, and improving the device's anti-interference capability and versatility.

[0082] According to some embodiments of this application, identifying a user wearing a diopter lens from a human eye image includes: identifying light spots from the human eye image; if the number of identified light spots is inconsistent with the number of multiple LED beads 111, or if there are light spots with irregular shapes, then identifying that the user is wearing a diopter lens 20.

[0083] In the above embodiment, the refractive lens recognition employs a dual-determination logic. The refractive lens recognition and stray light processing module 450 first extracts all light spots from the human eye image, counts the total number of light spots, and compares it with the total number of LED beads 111 of the light source 110. Simultaneously, it detects the shape characteristics of each light spot to determine whether there are irregularly shaped light spots that differ significantly from the standard circular or elliptical light spots projected by the LED beads 111. When either the condition of "the number of light spots is inconsistent with the number of LED beads 111" or "there are irregularly shaped light spots" is met, it can be quickly determined that the user is wearing a refractive lens 20. This dual-determination logic does not rely on additional sensors or complex biometric recognition; it can be achieved solely based on the human eye image acquired by the existing camera 120. It offers fast determination speed and high accuracy, providing timely trigger signals for the subsequent stray light spot removal process. Furthermore, the dual determination mechanisms complement each other, effectively avoiding misjudgments or omissions caused by a single determination logic, ensuring the reliability of refractive lens recognition, and laying the foundation for effective light spot detection and elimination of stray light interference.

[0084] For example, such as Figure 14 As shown, Figure 14 This is a schematic diagram of the detection results of stray light spots in non-ocular refractive lenses provided in this application.

[0085] For example, the refractive lens recognition and stray light processing module 450 can be divided into a refractive lens recognition module and a stray light processing module. Figure 15 As shown, Figure 15This is a flowchart illustrating the stray light detection process for refractive lenses provided in this application. Specifically, it includes steps S20 to S33. S20: Installing the refractive lens into the XR (Extended Reality) device: Installing the refractive lens into the extended reality device; S21: Refractive lens recognition module: Determining the installation status through the refractive lens recognition module. If installation is successful (Y), proceed to the next step; if not installed (N), trigger the IMU (Inertial Measurement Unit). Unit (Inertial Detection Unit) detection; S22: Turn on optical module 100: Turn on eye-tracking camera 120 and optical module 100; S23: Turn on image buffer module (excluding eye area): Turn on image buffer module and buffer N frames of data images; S24: Buffer N frames of data images: Complete buffering of N frames of data images; S25: Stray light processing module: Input the buffered images into stray light processing module; S26: Stray light spot detection result: Output stray light detection result; S27: IMU / gyroscope motion signal / foreign object intrusion detection system: Detect whether IMU / gyroscope motion signal / foreign object intrusion detection system, intrusion (Y S28: Start optical module 100: (Y branch of S27) Start eye-tracking camera 120 and optical module 100; S29: Turn off stray light processing module; S30: Start processing module (used to detect whether there is an eye region in the image): Determine whether the refractive power is valid. If valid (Y), enter the eye tracking process. If invalid (N), execute device screen off operation; S31: Device screen off operation: Execute device screen off operation; S32: Gaze estimation module 420: Input stray light detection results into eye tracking gaze estimation module 420; S33: Three-dimensional gaze data: Output three-dimensional gaze data.

[0086] For example, such as Figure 16 As shown, Figure 16This is a flowchart of eye-tracking detection device for gaze estimation provided in this application; specifically including S40 to S50. S40: Wearing Prescription Lenses: Determines whether prescription lenses are being worn. If Y (Yes), proceed to the stray light processing module; if N (No), proceed to the optical module 100. S41: Stray Light Processing Module: Proceed to the stray light processing module. S42: Optical Module 100 (LED & ET Camera): Proceed to the optical module 100 and acquire the human eye image. S43: Human Eye Image: Acquire the human eye image and input it into the pupil feature detection module 430. S44: Pupil Feature Detection: Perform pupil feature detection on the human eye image and input it into the light spot feature detection module. S45: Light Spot Feature Detection: Perform light spot feature detection on the pupil feature detection result. S46: Number and Size of Light Spots: Obtain the number and size of light spots and input them for light spot missing judgment. S47: Light Spot Missing OR (or) Few Light Spots: Determine whether there are missing or few light spots. If Y, proceed to the processing module 300; if N, proceed to the gaze estimation module. S48: Adjust the LED (Light-Emitting) via the actuator. LED (Light Emitting Diode) Position OR LED Brightness Adjustment: The processing module 300 adjusts the LED position or LED brightness through the actuator; S49: LED Position Ratio Calibration Parameter (Look-up Table Method): The optical module 100 is adjusted based on the LED position ratio calibration parameter (look-up table method); S50: Line of Sight Estimation: Line of sight estimation is completed. Here, LED refers to LED beads 111.

[0087] like Figure 17 As shown, according to some embodiments of this application, it includes:

[0088] S100: Emits infrared light to the human eye through multiple LED beads in the light source;

[0089] S110: Obtains the first human eye image by receiving infrared light reflected back from the human eye through a camera;

[0090] S120: The processing module confirms the number of valid light spots based on the first human eye image;

[0091] S130: If the number of effective light spots is less than the preset number threshold, it is confirmed that there are missing light spots in the first human eye image;

[0092] S140: If it is confirmed that there are missing light spots in the first human eye image, the control drive module adjusts the direction of the light, and then the optical module acquires the second human eye image. The number of effective light spots in the second human eye image is greater than the preset number threshold.

[0093] S150: The processing module estimates the human eye's line of sight based on the second human eye image.

[0094] In the above embodiment, multiple LEDs of the light source emit infrared light towards the human eye. The camera receives the infrared light reflected back from the human eye to obtain a first human eye image. The processing module determines the number of effective light spots based on the first human eye image. A preset threshold number is the minimum number of effective light spots required to achieve human eye gaze estimation. Effective light spots are formed by the reflection of infrared light emitted by the LEDs by the human eye. Each effective light spot corresponds to the infrared light reflection effect of one LED. When the number of effective light spots is less than the preset threshold number, it means that there are not enough LEDs to form effective light spots, which cannot provide sufficient optical feature support for subsequent human eye gaze estimation. Therefore, it can be confirmed that there are missing light spots in the first human eye image. When it is confirmed that there are missing light spots in the first human eye image, the processing module controls the drive module to adjust the direction of the infrared light emitted by the LEDs, changing the propagation path of the infrared light, so that the light spots that were not originally in the light spot are reflected back to the human eye. The infrared light emitted by the LED beads that form an effective light spot for the human eye can be successfully reflected by the human eye to form an effective light spot. Then, the optical module re-acquires a second human eye image. The number of effective light spots in the second human eye image is greater than the preset threshold, which can meet the requirements of the number of effective light spots for human eye gaze estimation. Finally, the processing module performs human eye gaze estimation based on the second human eye image. By supplementing the number of effective light spots, the problem of missing optical features caused by insufficient number of effective light spots corresponding to the LED beads is solved. This ensures that there are enough effective light spots as a basis for calculation in the human eye gaze estimation process, improving the accuracy of human eye gaze estimation. At the same time, the supplementation of effective light spots is achieved by actively adjusting the direction of the light, so that the eye-tracking detection device can still complete effective human eye gaze estimation in scenarios where the number of effective light spots formed by the LED beads is insufficient, thus expanding the applicable scenarios of the eye-tracking detection device.

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0096] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An eye-tracking detection device, characterized in that, include: An optical module includes a light source and a camera. The light source contains multiple LEDs for emitting light toward the human eye, and the camera is used to receive the light reflected by the human eye and capture an image of the human eye. The driving module is communicatively connected to the optical module and is used to adjust the direction of the light emitted by the lamp beads; The processing module is communicatively connected to the driving module and the optical module. The processing module is used to acquire the number of effective light spots in the human eye image. If the number of effective light spots is less than a preset threshold, the processing module controls the driving module to adjust the direction of the light until the number of effective light spots in the human eye image acquired by the optical module is greater than or equal to the preset threshold.

2. The eye movement detection device according to claim 1, characterized in that, The driving module includes: A miniature motor is connected to the light source for driving the light source to rotate, and the direction of the light is adjusted by adjusting the emission angle of the light.

3. The eye movement detection device according to claim 2, characterized in that, The processing module is specifically used for: When the number of effective light spots is less than a preset threshold, the exit pupil distance and eye rotation angle of the human eye are identified from the human eye image. Based on the exit pupil distance and the eye rotation angle, a preset first correspondence table is searched to obtain the lamp head rotation angle of the light source. The micro motor is then controlled to drive the light source to rotate according to the lamp head rotation angle, thereby completing the effective light spots. The first correspondence table records the correspondence between the exit pupil distance, the eye rotation angle, and the lamp head rotation angle.

4. The eye movement detection device according to claim 1, characterized in that, The driving module includes: An optical adjustment element is disposed on the light propagation path of the light source to change the propagation direction of the light to adjust the light direction.

5. The eye movement detection device according to claim 4, characterized in that, The optical adjustment element is a liquid crystal adjustable optical element. By applying a voltage to the liquid crystal adjustable optical element, the refractive index of the liquid crystal adjustable optical element is adjusted, thereby refracting the light and adjusting the direction of the light; or The optical adjustment element is a beam scanner with a light scanning plane. By rotating the light scanning plane, the light beam is reflected to adjust the direction of the light beam.

6. The eye movement detection device according to claim 1, characterized in that, The driving module includes: A power adjustment module, communicatively connected to the light source, is used to adjust the optical power of the light source; The processing module is also used to detect the brightness of the effective light spot in real time. When the brightness is lower than a first preset brightness threshold, the power adjustment module increases the light power of the light source. When the brightness is higher than a second preset brightness threshold, the power adjustment module decreases the light power of the light source. The first preset brightness threshold is less than the second preset brightness threshold.

7. The eye movement detection device according to claim 1, characterized in that, The processing module is also specifically used for: Using a spot detection model, candidate spots are detected from the region of interest image corresponding to the human eye image, and the first position of the candidate spot in the region of interest image and the correspondence between the candidate spot and the LED bead are identified. Based on the positional relationship between the multiple LED beads and the positional relationship between the multiple LED beads and the human eye, the multiple LED beads are mapped onto the image of the region of interest to obtain multiple target locations; Candidate light spots that do not conform to the target position are removed to obtain the detected effective light spots; Based on the first position and the corresponding relationship, the effective light spot is mapped back to the human eye image.

8. The eye movement detection device according to claim 1, characterized in that, Also includes: A pupil feature detection module, which is communicatively connected to the processing module and the camera, is used to detect pupil features from the human eye image; A gaze estimation module, coupled to the processing module, is specifically used to estimate the gaze of the human eye based on the completed effective light spot and the pupil features. The pupil feature detection module is specifically used for: The human eye image is subjected to pixel reduction processing to obtain a pixel-reduced image; Based on the coarse segmentation model, the reduced pixel image is coarsely segmented to obtain the coarse segmentation result; Based on the coarse segmentation result, the region of interest image corresponding to the human eye image is obtained; Based on the fine segmentation model, the image of the region of interest is finely segmented to obtain the fine segmentation result; The fine segmentation results are post-processed to obtain pupil features.

9. The eye movement detection device according to claim 1, characterized in that, It also includes a refractive lens identification and stray light processing module, which is communicatively connected to the camera, the light source, and the processing module. The refractive lens identification and stray light processing module is used for: From the image of the human eye, identify whether the user is wearing refractive lenses; If the user is found to be wearing diopter lenses, the light source and the camera are turned on, and multiple consecutive images of the user's face (excluding the eye area) are captured. Cache the multiple consecutive images; From the multi-frame continuous images, identify stray light spots whose positional relationship with the light source does not match and whose positions are fixed in the multi-frame continuous images; Remove the stray light spots from the human eye image; The step of identifying whether a user is wearing refractive lenses from the human eye image includes: Identify light spots from the human eye image; If the number of identified light spots is inconsistent with the number of multiple LED beads, or if there are light spots with irregular shapes, then it is determined that the user is wearing diopter lenses.

10. The eye movement detection device according to claim 1, characterized in that, include: Infrared light is emitted toward the human eye through the plurality of LED beads of the light source; A first human eye image is obtained by receiving infrared light reflected back from the human eye through the camera; The processing module determines the number of valid light spots based on the first human eye image; If the number of effective light spots is less than the preset number threshold, it is confirmed that there are missing light spots in the first human eye image; If it is confirmed that there are missing light spots in the first human eye image, the driving module is controlled to adjust the direction of the light, and then the second human eye image is obtained through the optical module. The number of effective light spots in the second human eye image is greater than the preset number threshold. The processing module estimates the line of sight of the human eye based on the second human eye image.

Images