A pupil center label-free estimation method based on line-of-sight intersection

Abstract

The application provides a pupil center label-free estimation method based on line-of-sight intersection, and relates to the technical field of computer vision and eye movement analysis. The method comprises the following steps: collecting a face image sequence through an RGB camera, and constructing a training sample by using labeled line-of-sight direction information. Face key point detection is performed on the image, and left and right eye regions are extracted, face and eye features are obtained through a feature extraction network, and pure line-of-sight related features are obtained through a attention mechanism. The model simultaneously predicts the pupil center position and the line-of-sight direction, combines the pupil position soft constraint, the line-of-sight direction constraint and the line-of-sight intersection geometric consistency constraint to construct a label-free training loss function, realizes end-to-end optimization training, and finally accurately estimates the pupil center position in a common RGB environment. The application solves the problem that it is difficult to realize label-free, robust and high-precision pupil center estimation in a common RGB environment.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and eye-tracking analysis technology, and in particular to a label-free method for pupil center estimation based on line-of-sight intersection. Background Technology

[0002] In real-world applications such as telemedicine, intelligent interaction, and assisted driving systems, high-precision pupil center localization is a crucial prerequisite for achieving gaze estimation, user intent recognition, and human-computer interaction. However, current mainstream pupil localization methods generally rely on large-scale labeled pupil center datasets for supervised training, which is hampered by difficulties in data acquisition and labeling errors. Since pupil centers are difficult to automatically calibrate non-contactly, manual labeling is not only costly and prone to errors, but also difficult to achieve under natural light, long distances, or non-cooperative conditions. This results in an extreme scarcity of high-quality training data, hindering the widespread application of supervised learning methods.

[0003] While numerous datasets with labeled gaze directions exist (such as MPIIFaceGaze and EyeDiap), these datasets lack corresponding pupil center coordinates, making them unsuitable for directly training pupil localization models. Furthermore, the gaze direction is jointly determined by the pupil center and the optical center of the eye, and the gazes of both eyes typically converge on the same point of interest in space, indicating a natural geometric coupling between the pupil center and gaze direction. However, existing methods generally treat gaze estimation and pupil center localization as independent tasks, failing to effectively utilize this geometric constraint of gaze intersection to provide inverse optimization support for pupil estimation under unlabeled conditions.

[0004] Several pupil localization schemes exist in the current technology, but all have significant limitations. The grayscale centroid-based method proposed in CN106203375A, while simple and computationally fast, is prone to centroid drift due to uneven lighting or eyeglass reflections, resulting in insufficient robustness. CN110598635A utilizes optical flow tracking and circle fitting correction to eliminate some jitter, but relies on clear iris edges and is prone to failure at low resolutions or during blinking. CN120563846A constructs a 3D eyeball model and performs precise projection through Purkinje reflection; although highly accurate, it requires active infrared illumination and multi-camera calibration, making the hardware complex and costly. In summary, existing methods either rely on high-quality data annotation, are severely affected by environmental interference, or are heavily dependent on specialized equipment, making it difficult to achieve low-cost, highly robust, and accurate pupil center localization in ordinary RGB camera environments. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a label-free pupil center estimation method based on line-of-sight intersection. This invention solves the problem that existing pupil localization methods rely on high-cost annotation or dedicated hardware, making it difficult to achieve label-free, robust, and high-precision pupil center estimation in ordinary RGB environments.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A label-free pupil center estimation method based on line-of-sight intersection includes:

[0008] The system acquires facial image sequences using an RGB camera and uses labeled gaze direction information to form input training samples, resulting in RGB facial image sequences and gaze direction labels.

[0009] Facial key points are detected in each frame of the RGB face image sequence and gaze direction label, and the left eye image and right eye image are cropped according to the eye corner coordinates to obtain the left eye image, right eye image and region location information;

[0010] The left eye image, right eye image, and eye images from the region location information, along with the entire face image, are input into the feature extraction network to obtain the face feature set and the eye feature set.

[0011] The correlation weights between the facial feature set and the eye feature set are calculated by using an attention mechanism, and the facial feature set is enhanced by a weighted method to obtain pure gaze-related features.

[0012] Input the pure gaze-related features into the corresponding gaze estimation branch, output the position of the pupil center of the left and right eyes in the image and output the gaze direction prediction results of both eyes, and obtain the preliminary prediction results of the pupil center and the gaze direction prediction results.

[0013] Based on the preliminary prediction results of the pupil center and the left eye image, right eye image and regional location information, a consistency constraint between the pupil center and the center of the eye region is constructed to obtain the soft constraint evaluation results of the pupil position.

[0014] Based on the prediction results of the line of sight direction, the deviation between the predicted line of sight direction and the labeled line of sight direction is calculated, and a direction loss term is generated to supervise the accuracy of the line of sight features, thus obtaining the line of sight direction constraint evaluation results.

[0015] Based on the preliminary prediction results of the pupil center and the prediction results of the line of sight, the line of sight of each eye is determined and the shortest distance in space is calculated. The shortest distance is used to measure the degree of focus of the line of sight in space, and the geometric consistency evaluation result of the line of sight intersection is obtained.

[0016] The evaluation results of soft constraints on pupil position, evaluation results of gaze direction constraints, and evaluation results of geometric consistency of gaze intersection are combined to form an unlabeled training loss function.

[0017] The model is jointly optimized end-to-end using an unlabeled training loss function until the loss converges. The trained model is then deployed on a computing device to output the pupil center coordinates of the input RGB face image, thus obtaining the unlabeled optimized pupil center estimation result.

[0018] The present invention discloses the following technical effects:

[0019] This invention provides a label-free pupil center estimation method based on gaze intersection, comprising: acquiring a sequence of face images using an RGB camera, and forming input training samples using labeled gaze direction information to obtain an RGB face image sequence and gaze direction labels; performing facial key point detection on each frame of the RGB face image sequence and gaze direction labels, and cropping the left and right eye images based on the eye corner coordinates to obtain the left eye image, right eye image, and region location information; inputting the eye images from the left eye image, right eye image, and region location information, along with the entire face image, into a feature extraction network to obtain a face feature set and an eye feature set; calculating the correlation weights between the face feature set and the eye feature set through an attention mechanism, and performing feature enhancement on the face feature set using a weighted method to obtain pure gaze-related features; inputting the pure gaze-related features into the corresponding gaze estimation branch, outputting the position of the pupil center of the left and right eyes in the image and the predicted gaze direction of both eyes, to obtain a preliminary prediction result of the pupil center and a predicted gaze direction. The evaluation results are as follows: Based on the preliminary prediction results of the pupil center and the left eye image, right eye image, and regional location information, a consistency constraint between the pupil center and the eye region center is constructed to obtain the soft constraint evaluation result of the pupil position; the deviation between the predicted gaze direction and the labeled gaze direction is calculated according to the gaze direction prediction results, and a direction loss term is generated to supervise the accuracy of gaze features to obtain the gaze direction constraint evaluation result; based on the preliminary prediction results of the pupil center and the gaze direction prediction results, the gaze lines of the left and right eyes are determined, and the shortest distance in space is calculated. The shortest distance is used to measure the degree of focus of the gaze in space to obtain the gaze intersection geometric consistency evaluation result; the soft constraint evaluation result of the pupil position, the gaze direction constraint evaluation result, and the gaze intersection geometric consistency evaluation result are combined to form an unlabeled training loss function; the unlabeled training loss function is used to perform end-to-end joint optimization training of the model until the loss converges, and the trained model is deployed on the computing device to output the pupil center coordinates to the input RGB face image to obtain the unlabeled optimized pupil center estimation result. This invention maintains high accuracy under extreme lighting conditions and large head postures by using line-of-sight labels, while manual marking of the pupil center is almost impossible in such scenarios. This invention uses the geometric consistency of "line of sight-pupil" as supervision and indirectly utilizes these highly reliable labels to significantly improve positioning accuracy in harsh environments.Meanwhile, the training set directly reuses publicly available large-scale cross-scene gaze data, and the model naturally inherits its robustness to changes in lighting, occlusion, and resolution, allowing it to run stably without additional augmentation. The entire training process relies solely on the "gaze direction" labels in the existing gaze dataset, and optimizes the pupil center in reverse through differentiable 3D geometric constraints, achieving zero pupil ground truth consumption. This completely eliminates the expensive and time-consuming manual frame-by-frame annotation process, directly reducing data preparation costs to near zero. If a small number of pupil center ground truths are subsequently obtained, an L1 loss term can be seamlessly added to the end of the original network for joint optimization with existing geometric constraints, without modifying the main network or redesigning the framework. Therefore, this invention can smoothly transition to "weakly supervised" or even "fully supervised" modes, accommodating both unlabeled startup and labeled fine-tuning needs, and adapting to the resource conditions of different application scenarios. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a labelless pupil center estimation method based on line-of-sight intersection is provided for an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of the feature extraction framework structure provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, this invention provides a label-free pupil center estimation method based on line-of-sight intersection, comprising:

[0026] Step 100: Collect face image sequences using an RGB camera, and use the labeled gaze direction information to form input training samples to obtain RGB face image sequences and gaze direction labels;

[0027] Specifically, the RGB face image sequence captured by the camera ,in Manually labeled gaze tags .

[0028] Step 200: Perform facial key point detection on each frame of the RGB face image sequence and gaze direction label, and crop the left eye image and right eye image according to the eye corner coordinates to obtain the left eye image, right eye image and region location information;

[0029] Specifically, Dlib was used to detect facial landmarks, including 68 key points, and the eye region image was obtained by cropping using the coordinates of the key points at the corners of the eyes. and and the coordinates of the left and right eye positions on the original image. and .

[0030] Step 300: Input the left eye image, right eye image, and eye image from the region location information, along with the entire face image, into the feature extraction network to obtain the face feature set and the eye feature set;

[0031] Specifically, such as Figure 2 As shown, the face image and eye image are fed into a feature extractor (ResNet18) to obtain face and eye features. The face features include all information such as head pose, gaze, and background. The eye features contain gaze-related information. The eye and face features are then fed into a self-attention layer to calculate the correlation between the face and eye features, thereby obtaining gaze-related similarity weights.

[0032] Step 400: Calculate the correlation weights between the face feature set and the eye feature set through the attention mechanism, and use a weighted method to enhance the face feature set to obtain pure gaze-related features;

[0033] Specifically, the similarity weights are used to obtain attention scores through a sigmoid layer, which are then multiplied by the original facial features to obtain the enhanced facial features. This step eliminates interference from background information by calculating the similarity between local and global regions.

[0034] Step 500: Input the pure gaze-related features into the corresponding gaze estimation branch, output the position of the pupil center of the left and right eyes in the image and output the gaze direction prediction results of both eyes, and obtain the preliminary prediction results of the pupil center and the gaze direction prediction results.

[0035] Specifically, the purified gaze-related features are concatenated with the original eye features and then fed into a multi-layer MLP to regress and predict the pixel coordinates of the pupil center in the face image. and , and These represent the predicted pixel coordinates of the pupil center for the left and right eyes, respectively. Since head pose information affects the morphological changes of the pupil in a face image, purified gaze-related features (including head pose information) are injected into the pupil center prediction branch.

[0036] The purified gaze-related features are also fed into the gaze estimation branch, where the gaze direction is obtained by regression from the feature space using an MLP with the same structure. and Since the model training is unlabeled at the pupil center, we supervise the model by using precise gaze direction.

[0037] Step 600: Based on the preliminary prediction results of the pupil center and the left eye image, right eye image and regional location information, construct the consistency constraint between the pupil center and the center of the eye region to obtain the soft constraint evaluation result of the pupil position;

[0038] Specifically, a soft constraint is applied to the pupil center. Since the pupil center must be located within the eye region, we define a position loss function. ,in, and They are and The mean of the coordinates of all angles in the center is the center of the eye.

[0039] Step 700: Calculate the deviation between the predicted line of sight and the labeled line of sight based on the line of sight prediction results, generate a direction loss term to supervise the accuracy of line of sight features, and obtain the line of sight constraint evaluation results;

[0040] Specifically, we calculate the L1 loss between the predicted gaze direction and the true label to supervise the accuracy of gaze-related features, i.e. + .

[0041] Step 800: Based on the preliminary prediction results of the pupil center and the prediction results of the line of sight, determine the line of sight for each of the left and right eyes, and calculate the shortest distance in space. Use the shortest distance to measure the degree of focus of the line of sight in space, and obtain the evaluation result of the geometric consistency of line of sight intersection.

[0042] Specifically, when a face gazes at an object, the line of sight originates from both eyes and focuses on the object in space. Therefore, we construct a focusing loss function, using the pupil center and the gaze vector to obtain a fixed straight line in space. The shortest skew distance between the gaze lines of the left and right eyes is calculated as the loss, i.e. .

[0043] Step 900: Combine the evaluation results of pupil position soft constraint, gaze direction constraint, and gaze intersection geometric consistency to form an unlabeled training loss function;

[0044] Specifically, the total losses from joint training Repeat the iterations until the loss converges, thus preserving the optimal model, i.e., [ ], = ,in, These represent images of faces. Cropped images of the left and right eyes. This represents the saved optimal model parameters.

[0045] Step 1000: Use the unlabeled training loss function to perform end-to-end joint optimization training on the model until the loss converges, and deploy the trained model on the computing device to output the pupil center coordinates of the input RGB face image, thus obtaining the unlabeled optimized pupil center estimation result.

[0046] Specifically, after deployment on the computer, the facial images captured by the camera are input, and the estimated pupil coordinates are printed and displayed on the application.

[0047] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0048] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A label-free pupil center estimation method based on line-of-sight intersection, characterized in that, include: The system acquires facial image sequences using an RGB camera and uses labeled gaze direction information to form input training samples, resulting in RGB facial image sequences and gaze direction labels. Facial key points are detected in each frame of the RGB face image sequence and gaze direction label, and the left eye image and right eye image are cropped according to the eye corner coordinates to obtain the left eye image, right eye image and region location information; The left eye image, right eye image, and eye images from the region location information, along with the entire face image, are input into the feature extraction network to obtain the face feature set and the eye feature set. The correlation weights between the facial feature set and the eye feature set are calculated by using an attention mechanism, and the facial feature set is enhanced by a weighted method to obtain pure gaze-related features. Input the pure gaze-related features into the corresponding gaze estimation branch, output the position of the pupil center of the left and right eyes in the image and output the gaze direction prediction results of both eyes, and obtain the preliminary prediction results of the pupil center and the gaze direction prediction results. Based on the preliminary prediction results of the pupil center and the left eye image, right eye image and regional location information, a consistency constraint between the pupil center and the center of the eye region is constructed to obtain the soft constraint evaluation results of the pupil position. Based on the prediction results of the line of sight direction, the deviation between the predicted line of sight direction and the labeled line of sight direction is calculated, and a direction loss term is generated to supervise the accuracy of the line of sight features, thus obtaining the line of sight direction constraint evaluation results. Based on the preliminary prediction results of the pupil center and the prediction results of the line of sight, the line of sight of each eye is determined and the shortest distance in space is calculated. The shortest distance is used to measure the degree of focus of the line of sight in space, and the geometric consistency evaluation result of the line of sight intersection is obtained. The evaluation results of soft constraints on pupil position, evaluation results of gaze direction constraints, and evaluation results of geometric consistency of gaze intersection are combined to form an unlabeled training loss function. The model is jointly optimized end-to-end using an unlabeled training loss function until the loss converges. The trained model is then deployed on a computing device to output the pupil center coordinates of the input RGB face image, thus obtaining the unlabeled optimized pupil center estimation result.

2. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 1, characterized in that, The facial landmark detection uses the Dlib algorithm and collects 68 landmarks.

3. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 1, characterized in that, The feature extraction network is ResNet18.

4. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 1, characterized in that, The loss function corresponding to the soft constraint evaluation result of the pupil position is: ； in, and They are and The mean of the coordinates of all angles in the eye area is found at the center of the eye. For the location loss function, and These represent the predicted pixel coordinates of the pupil center for the left and right eyes, respectively.

5. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 4, characterized in that, The loss function corresponding to the evaluation result of the line-of-sight direction constraint is: + ； in, This represents the line-of-sight direction constraint function. and These represent the true values ​​of the gaze direction annotations originating from the left and right eyes, respectively. and These represent the predicted directions of vision for the left and right eyes, respectively.

6. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 5, characterized in that, The loss function corresponding to the evaluation result of the geometric consistency of the line-of-sight intersection is: ； in, This is the line-of-sight intersection loss function, used to represent the skew distance between two spatial line-of-sight vectors.

7. The label-free pupil center estimation method based on line-of-sight intersection as described in claim 6, characterized in that, The expression for the unlabeled training loss function is: 。

Images