Pupil occlusion sight line estimation method based on image representation enhancement

Abstract

The invention provides a pupil occlusion sight line estimation method based on image representation enhancement, and relates to the field of sight line estimation. The method mainly comprises the following steps: carrying out region segmentation on an eye image obtained by detection, identifying a pupil region, a typoscope region and a background region, judging the position of a shielded region according to the position relationship between the pupil region and the typoscope region, and obtaining a pupil region image with the typoscope region being blank; complementing the pupil region image with the blank typoscope region based on a diffusion model to obtain a pupil restored image; and learning a mapping relation between the pupil feature point position and the sight line direction by adopting a deep learning algorithm, and realizing accurate sight line estimation based on the pupil restored image and the mapping relation. According to the method, the problems of light reflection and shielding caused by a typoscope can be effectively solved mainly through region segmentation, pupil shielding judgment, pupil complementation and sight line estimation based on a deep learning algorithm, and the accuracy and robustness of sight line tracking are improved.

Description

Technical Field

[0001] The present invention mainly belongs to the field of computer science and technology, especially the field of gaze tracking. It is applied to gaze estimation in situations such as eye occlusion and mirror reflection that occur during eye movement behavior analysis. Specifically, a method for pupil occlusion gaze estimation based on image representation enhancement is disclosed. Background Art

[0002] In the field of eye tracking technology, the existing technology has some defects and shortcomings. Most of the common eye tracking methods currently available on the market are only for situations where people are not wearing glasses or wearing normal glasses. These methods have high accuracy when there is no major occlusion of the pupil. However, when these methods are applied to people wearing optical vision aids, since the vision aids may cause the pupil to be partially blocked, these methods are not usable in many cases. Even in the cases where they are partially usable, there are large errors, which poses a challenge to subsequent research. In addition, in the existing technology, the main form of optical vision aids is eyeglass-type vision aids, most of which have specially made lenses, and these specially made lenses are attached with additional prisms, spherical lenses and other devices that enhance the user's field of view. Users need to wear vision aids normally like wearing glasses. Due to the presence of special lenses, users usually have the problem of partial occlusion of the pupil when wearing optical vision aids. In this case, eye tracking and subsequent research are very difficult. Summary of the Invention

[0003] In view of the technical problem in existing technologies where pupil occlusion affects gaze estimation, the present invention provides a method for pupil occlusion gaze estimation based on enhanced image representation. This method effectively addresses the issues of glare and occlusion caused by visual aids through region segmentation, pupil occlusion determination, pupil completion, and gaze estimation based on a deep learning algorithm, thereby improving the accuracy and robustness of gaze tracking.

[0004] The technical means adopted in the present invention are as follows:

[0005] A pupil occlusion sight line estimation method based on image representation enhancement, comprising:

[0006] Collect face images with eye areas and perform eye area detection based on an improved multi-scale feature fusion network;

[0007] Performing regional segmentation on the detected eye image to identify the pupil area, the visual aid area, and the background area. The location of the occluded area is determined based on the positional relationship between the pupil area and the visual aid area. The area belonging to the visual aid is then eliminated from the pupil area to obtain a pupil area image with the visual aid area blank.

[0008] Completing the pupil area image where the visual aid area is blank based on the diffusion model, thereby obtaining a pupil restoration image;

[0009] A deep learning algorithm is used to learn the mapping relationship between the position of pupil feature points and the direction of sight, and accurate sight estimation is achieved based on the pupil restoration image and the mapping relationship.

[0010] Furthermore, the collected facial images are preprocessed, including:

[0011] Scaling the facial image to a set size and normalizing the pixel values ​​to a specific range;

[0012] The RGB image is converted into a grayscale image using the weighted average method;

[0013] First, Gaussian filtering is used to smooth the Gaussian noise caused by ambient light, and then median filtering is used to eliminate sensor noise.

[0014] Furthermore, preprocessing the collected facial images also includes:

[0015] The preprocessed image is labeled with pupil center coordinates, head posture and gaze direction vector

[0016] Dynamic data augmentation is performed on the annotated images through geometric transformation, photometric perturbation, and occlusion simulation.

[0017] Furthermore, the detected eye image is segmented to identify the pupil area, the visual aid area, and the background area, including:

[0018] A pre-trained semantic segmentation model is used to extract feature information at different levels from the image. The semantic segmentation model includes a parallel shallow network and a deep network. The shallow network is used to capture edge and texture features, and the deep network is used to capture and recognize semantic information.

[0019] Fuse the features output by the shallow network and the features output by the deep network to obtain a fused regional feature image;

[0020] The convolutional neural network is used to determine the region to which the pixels of the fused regional feature image belong.

[0021] Furthermore, the detected eye image is segmented to identify the pupil area, the visual aid area and the background area, and further includes:

[0022] The head posture features are extracted from the eye area image with the visual aid area being blank through a convolutional network.

[0023] Furthermore, the pupil area image where the visual aid area is blank is completed based on the diffusion model to obtain a pupil restoration image, including:

[0024] The diffusion model is used to generate the overall geometric structure of the pupil area through step-by-step denoising, and the circular outline and center position are restored, thereby achieving coarse restoration of the pupil area.

[0025] The diffusion model is used to supplement the iris texture and reflective spots in the roughly repaired pupil area, thereby achieving fine repair of the pupil area.

[0026] Furthermore, a deep learning algorithm is used to learn the mapping relationship between pupil feature point positions and gaze directions, including:

[0027] Extracting pupil feature points in the pupil image using a convolutional neural network, where the pupil feature points include pupil center, corneal reflection points, and eye feature points;

[0028] The pupil feature point position is used as the input data of the deep learning network, and the sight direction is used as the output data of the deep learning network to train the deep learning network, thereby learning the mapping relationship between the pupil feature point position and the sight direction.

[0029] Existing gaze estimation methods are often affected by lens reflections or occlusions when wearing a visual aid, resulting in reduced accuracy and making it difficult to meet practical application requirements. To address this issue, this paper proposes a pupil-occluded gaze estimation method based on enhanced image representation. This method first captures and preprocesses facial images, then removes occluded areas through multi-level feature extraction and fusion techniques. It then completes the eye image and finally combines visual aid feature information and head posture information to achieve accurate gaze estimation.

[0030] Each technical point of the present invention brings significant beneficial effects:

[0031] 1. This invention utilizes multi-level image feature extraction and fusion technology to eliminate occlusions. This allows for high-precision identification and localization of key image regions, generating a visual aid segmentation image and acquiring head posture information. This design not only clarifies the occlusion range but also provides important auxiliary information for gaze estimation, significantly improving accuracy.

[0032] 2. This invention completes the eye image through a combination of coarse and fine inpainting stages, and employs a diffusion model to restore missing eye image information due to occlusion by the visual aid. This process effectively eliminates the effects of reflections and occlusions, generating high-quality eye images and laying a solid foundation for subsequent gaze estimation.

[0033] 3. This invention divides gaze estimation into two stages: pupil feature point location and feature mapping. It then combines information about the features of the visual aid and head posture for precise estimation. This design fully utilizes the feature information in the image, maintaining good estimation results even under complex visual conditions, significantly improving the robustness and accuracy of the system.

[0034] This invention can be used to estimate line of sight for people wearing visual aids. It can quickly and accurately complete line of sight estimation tasks even in the presence of lens reflections or occlusions, with high accuracy and robustness. It has important application value and positive implications for research on the attention mechanisms of visually impaired people, human-computer interaction, and assisted driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0036] Figure 1 This is a flow chart of a remote sensing line of sight estimation method based on image representation enhancement in an embodiment of the present invention.

[0037] Figure 2 This is the network architecture for performing region segmentation on eye region images in an embodiment of the present invention.

[0038] Figure 3 Schematic diagram of the process of completing the through-hole area in an embodiment of the present invention.

[0039] Figure 4 This is the overall network architecture for pupil area restoration and gaze direction estimation in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0041] like Figure 1 As shown, an embodiment of the present invention discloses a remote sensing line of sight estimation method based on image representation enhancement, comprising:

[0042] S1. Collect face images with eye areas and perform eye area detection based on an improved multi-scale feature fusion network.

[0043] This step mainly includes two stages: face image acquisition and eye area detection.

[0044] To ensure the accuracy and robustness of pupil tracking, this application adopts a series of acquisition technology solutions. First of all, in terms of data acquisition equipment and environmental configuration, high-resolution binocular cameras, infrared cameras and mobile device cameras can be used to cover different lighting scenes (such as low light, strong reflections) and usage scenarios (laboratory, real environment). A polarizer is installed in front of the camera to effectively filter the mirror reflection of the optical vision aid and reduce light spot interference. Ring-shaped auxiliary light sources are deployed on both sides of the camera to support dynamic brightness adjustment to ensure uniform facial lighting, adapt to data collection needs under complex indoor and outdoor backgrounds, and ensure the continuity and detail capture capabilities of dynamic pupil tracking.

[0045] For real-time data acquisition and eye detection, the camera captures high-definition images containing faces in real time. To address the specific needs of users with vision impairments, a multi-scale feature fusion network is used for eye detection. This network extracts local details (such as eyelid contours) and global contextual features through parallel multi-scale convolutional layers. Combined with adaptive ROI pooling, it outputs eye bounding boxes, maintaining high detection accuracy even in scenarios with significant head deflection or occlusion. Preferably, the multi-scale feature fusion network can use one of YOLOv7, Faster R-CNN, or SSD.

[0046] In order to improve the detection accuracy of the network model, the training data must be preprocessed during the training phase. The preprocessing process optimization includes the following steps.

[0047] First, the original image is uniformly scaled to 256×256 pixels and the pixel values ​​are normalized to the range of [-1, 1] to adapt to the deep learning model input.

[0048] Secondly, the polarizer is used to filter the specular reflection at the hardware level to preserve the pupil texture, and the weighted averaging method is used to convert the RGB image into a grayscale image to reduce the computational load.

[0049] Next, this application uses two-stage denoising to process the image. The first stage uses Gaussian filtering to smooth the Gaussian noise caused by ambient light, and the second stage uses median filtering to eliminate sensor noise and protect the sharpness of the pupil edge.

[0050] Then, the Dlib feature point detection model is used to extract the eye area ROI, and the subsequent processing range is narrowed to the upper half of the face for subsequent segmentation processing.

[0051] In terms of data annotation and enhancement, pre-processed images are annotated with pupil center coordinates, head pose Euler angles, and gaze direction vectors (including 2D screen coordinates and 3D space vectors). The processed image data is then used to construct a training dataset. Dynamic data augmentation, including geometric transformations, photometric perturbations, and occlusion simulation, improves the model's adaptability to lighting fluctuations and occlusions.

[0052] In terms of dataset partitioning and domain adaptation, the data is divided into training set, validation set and test set in an 8:1:1 ratio. The training set contains strictly controlled lighting and posture data, while the test set covers complex backgrounds and dynamic interference. This dataset is used for subsequent training of the eye image region segmentation network and the pupil restoration network.

[0053] S2. Segment the detected eye image to identify the pupil area, the visual aid area, and the background area. Determine the location of the occluded area based on the positional relationship between the pupil area and the visual aid area. Then, eliminate the area belonging to the gaze device from the pupil area to obtain a pupil area image with the visual aid area blank.

[0054] like Figure 2 As shown in the figure, this step uses multi-level image feature extraction and fusion technology, combined with a refined region segmentation method, to achieve high-precision identification and positioning of key areas in the image. The goal of region segmentation is to accurately distinguish different objects or regions in the image. In eye images, it is necessary to accurately identify the pupil, visual aid, and background areas to subsequently analyze whether the pupil is occluded. The implementation steps are as follows:

[0055] First, a pre-trained transformer network is used as the core network to extract feature information at different levels from the image. The shallow network mainly captures detailed features such as edges and textures, while the deep network recognizes more complex semantic information.

[0056] The shallow network consists of an input layer, two shallow Transformer blocks, and an output layer. Specifically, the input layer processes a 224×224 pixel image through a 3×3 convolutional embedding (stride 2, 64 output channels); the first layer of Transformer blocks contains 4 self-attention heads (each head has a dimension of 16) and a 3×3 convolution with a dilation rate of 1. The second layer adjusts the dilation rate to 3. Both layers consist of a feedforward network with a hidden layer dimension of 256; the output layer has a feature dimension of 128. To expand the network's ability to perceive global image information, this application introduces dilated convolution operations in the shallow network, enabling the model to focus on both local details and overall structure, thereby gaining a more comprehensive understanding of image content.

[0057] The deep network consists of an input layer, 4-6 deep Transformer blocks, and an output layer. The input layer receives shallow features or generates 128-dimensional features through 16×16 block embedding. The first and second Transformer blocks use 8-head self-attention (each head has a dimension of 16), while the third and fourth layers are feedforward networks with each layer connected to a hidden layer of dimension 512. The output layer has a feature dimension of 128.

[0058] The feature information at different levels is then spliced ​​together, combining the high-level features extracted by the deep network with the low-level features extracted by the shallow network. This allows both the location of the target area and the refinement of the boundary. A multi-scale feature fusion method is used to optimize the feature transfer path. This allows the model to not only use high-level semantic information to determine the existence of the pupil when identifying it, but also to correct the edge position in combination with low-level detailed information to avoid misidentifying reflections as pupils. A dynamic weight allocation mechanism can also be introduced to allow the model to automatically focus on important areas (such as the center of the pupil) while ignoring irrelevant background (such as skin texture), thereby improving segmentation accuracy.

[0059] Based on the fused features, a simple convolutional layer and a classification probability calculation function are used to classify each pixel as belonging to the pupil, the visual aid, or the background. To further optimize the results, boundary smoothing techniques are used to mathematically eliminate isolated noise points in the segmentation results (such as small reflections that are misclassified) and smooth jagged edges to make the pupil contour more natural. During training, two loss functions are used simultaneously: one to supervise the overall classification accuracy, and the other to optimize the segmentation of small objects, ensuring that the model can handle large areas while capturing subtle structures.

[0060] Furthermore, in the occlusion area elimination stage, this application will also analyze the positional relationship between the eye image and the visual aid in the image to determine the location of the occluded area. This link is crucial for the line of sight tracking system. When the user wears a visual aid (such as special glasses), the reflection of the lens or the frame may partially block the eye area, resulting in errors in the subsequent calculation of the line of sight direction. By accurately judging the occlusion situation, the system can take timely repair measures (such as completing the occluded area) or adjust the algorithm strategy to ensure the accuracy of the line of sight estimation. The specific process includes preparing input data, calculating the occlusion area and dynamic judgment.

[0061] The dynamic threshold mechanism at this stage adapts to the pupil size and shape of different users' visual aids, avoiding the lack of sensitivity caused by fixed thresholds. The threshold determines whether the pupil is occluded during the region segmentation and occlusion removal stages. For example, consider two visually impaired patients: one with a pupil diameter of approximately 4 mm under normal lighting, and the other with a pupil diameter of approximately 6 mm. When both wear the same visual aid for gaze tracking testing, for the patient with a 4 mm pupil diameter, the system adjusts the threshold for distinguishing the pupil from the visual aid and background area to a value more appropriate for smaller pupils. During image analysis, if a pixel's characteristic value (such as color or brightness) falls within this adjusted threshold range, it is identified as possibly belonging to the pupil area. For patients with a 6 mm pupil diameter, the system automatically increases the threshold range to ensure accurate identification of larger pupil areas. This allows for more accurate pupil occlusion analysis regardless of pupil size. Pupil size varies among users, and the dynamic threshold mechanism adapts to individual pupil sizes. For example, for users with larger pupils, the threshold is adjusted accordingly to ensure that the pupil area can be accurately distinguished from the visual aid and background areas during image analysis. For users with smaller pupils, the threshold is also adapted to avoid misjudgments due to unreasonable thresholds, such as misidentifying part of the pupil as a visual aid or background, which in turn affects the accuracy of the pupil occlusion judgment. During the model training phase, pupil features are extracted for images of different pupil sizes, and a correspondence between pupil features and pupil size is established. During the model's use phase, when a new patient arrives, the trained model is first used to extract their pupil features. Then, based on the pupil features, the corresponding diameter length is determined according to the pre-stored correspondence. Morphological filtering can reduce the interference of image noise (such as reflections misidentified as visual aids) on the calculation. In addition, the system provides an occlusion ratio value to facilitate the user or medical staff to understand the basis for the system's judgment. In practice, when a user wears a vision aid and the lens's glare obscures the upper right corner of the pupil, the system uses semantic segmentation to mark the vision aid area and the glare area, calculating the specific coordinates of the vision aid area. This step provides more effective location information for eye image completion.

[0062] S3. Completing the pupil area image where the visual aid area is blank based on the diffusion model, thereby obtaining a pupil restoration image.

[0063] This step uses deep learning techniques to restore eye information lost due to occlusion by the visual aid. This process primarily utilizes a diffusion model, combining multiple input formats to achieve highly accurate restoration. The model input consists of three components: a preprocessed grayscale image of the eye, which uses grayscale conversion and two-stage filtering to remove noise while preserving key textures; a semantic segmentation mask, which accurately labels the pupil region and the area occluded by the visual aid, defining the target restoration area; and the coordinates of key eye points, which are located by a feature point detection model to locate anatomical landmarks such as the canthus and eyelids, providing spatial position priors.

[0064] The restoration process is divided into two stages: the first is the coarse restoration stage, in which the model generates the overall geometric structure of the pupil through gradual denoising, prioritizing the restoration of macro features such as the circular outline and center position. This stage utilizes the iterative nature of the diffusion model to gradually eliminate noise interference in occluded areas to ensure shape accuracy. When the frame obscures the edge of the pupil, the model prioritizes reconstructing the continuity of the edge to avoid generating a broken or distorted outline. The second is the refinement stage, which, based on the coarse restoration, increases the resolution to a higher pixel to supplement detailed features such as iris texture and reflective spots, and optimizes the generation quality of local areas through a spatial attention mechanism. For the upper half of the pupil obscured by lens reflections, the model prioritizes restoring the iris texture of the lower half, and then gradually completes the reflective details of the upper half to ensure a natural transition with the unobstructed area.

[0065] To accommodate diverse occlusion scenarios, training data is constructed through a combination of synthetic and real-world annotation. Synthetic data is generated by simulating scenarios such as lens reflections and frame occlusions on full pupil images to generate occluded-full image pairs. Real-world data, on the other hand, comes from actual images of eyes wearing optics, with precise regions manually annotated. During training, model performance is jointly optimized using a multi-objective loss function, including noise prediction loss, structural consistency loss, edge alignment loss, and perceptual loss. Furthermore, the model employs key technical innovations such as mask guidance and dynamic attention allocation to improve the accuracy and efficiency of restoration.

[0066] The generated eye completion model undergoes multi-dimensional quality assessment to ensure reliability, including structural similarity and pupil geometry errors. By integrating multimodal input with a progressive generation strategy, this technology achieves a balance between occlusion restoration accuracy, real-time performance, and fault tolerance, resulting in significantly superior occlusion resistance and dynamic adaptability compared to traditional image restoration methods.

[0067] S4. Use a deep learning algorithm to learn the mapping relationship between the position of pupil feature points and the direction of sight, and achieve accurate sight estimation based on the pupil restoration image and the mapping relationship.

[0068] In the process of estimating gaze from pupil-restored images, this paper uses a deep learning algorithm to accurately estimate gaze. The entire process is divided into two main stages: pupil feature point localization and feature mapping. These two stages work together to ensure the accuracy and efficiency of gaze estimation.

[0069] Pupil feature point localization is the first stage of gaze estimation, primarily implemented using convolutional neural networks. By learning from large amounts of annotated data, convolutional neural networks can automatically extract key feature points from images, such as pupil centers and corneal reflection points. These feature points provide the fundamental information necessary for subsequent gaze estimation and gaze direction calculation. The feature point localization module ensures the extraction of valid feature points from the image. Once feature point localization is complete, the system outputs key parameters such as pupil center coordinates, providing the foundational data for subsequent gaze direction calculations. The core purpose of this stage is to ensure the accuracy of feature points and prevent biased results in subsequent steps due to accumulated initial errors.

[0070] Feature mapping is the second stage of gaze estimation. This step estimates gaze based on the pupil's feature points and positions. This step uses a deep learning network to map the extracted features into gaze directions. This deep learning network can be any of Gaze360, Full-face, RT-Gene, or MPIIGaze. The feature mapping module converts feature point position information into gaze directions. This conversion requires the model to understand the complex relationship between feature points and gaze directions. At this stage, the model needs to process and parse the information obtained from the feature point localization stage and convert it into a format suitable for gaze estimation.

[0071] During the feature mapping process, the model not only learns the location information of pupil feature points, but also the complex relationship between these feature points and gaze direction. This includes changes in pupil size and shape, and how they change with changes in head posture and ambient lighting. In this way, the model can adapt to different individual differences and environmental conditions, providing more personalized and robust gaze estimation. The key to this stage lies in the model's generalization ability, that is, the model needs to be able to handle a variety of different image conditions and user characteristics to ensure accurate gaze estimation.

[0072] To improve feature mapping accuracy, models typically employ complex network structures and training strategies. Multi-task learning is employed to simultaneously learn feature point localization and gaze estimation to enhance feature expressiveness. Furthermore, data augmentation techniques, such as random rotation, scaling, and brightness adjustment, are employed to improve the model's adaptability to varying image conditions. These techniques contribute to improved model robustness, enabling it to provide stable gaze estimation results in practical applications.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pupil occlusion sight line estimation method based on image representation enhancement, characterized in that: include: Collect face images with eye areas and perform eye area detection based on an improved multi-scale feature fusion network; Performing regional segmentation on the detected eye image to identify the pupil area, the visual aid area, and the background area. The location of the occluded area is determined based on the positional relationship between the pupil area and the visual aid area. The area belonging to the visual aid is then eliminated from the pupil area to obtain a pupil area image with the visual aid area blank. Completing the pupil area image where the visual aid area is blank based on the diffusion model, thereby obtaining a pupil restoration image; A deep learning algorithm is used to learn the mapping relationship between the position of pupil feature points and the direction of sight, and accurate sight estimation is achieved based on the pupil restoration image and the mapping relationship.

2. The pupil occlusion sight line estimation method based on image representation enhancement according to claim 1, characterized in that: Preprocess the collected facial images, including: Scaling the facial image to a set size and normalizing the pixel values ​​to a specific range; The RGB image is converted into a grayscale image using the weighted average method; First, Gaussian filtering is used to smooth the Gaussian noise caused by ambient light, and then median filtering is used to eliminate sensor noise.

3. The pupil occlusion sight line estimation method based on image representation enhancement according to claim 2, characterized in that: Preprocessing of the collected facial images also includes: The preprocessed image is labeled with pupil center coordinates, head posture and gaze direction vector Dynamic data augmentation is performed on the annotated images through geometric transformation, photometric perturbation, and occlusion simulation.

4. The pupil occlusion sight line estimation method based on image representation enhancement according to claim 1, characterized in that: Perform regional segmentation on the detected eye image to identify the pupil area, the visual aid area, and the background area, including: A pre-trained semantic segmentation model is used to extract feature information at different levels from the image. The semantic segmentation model includes a parallel shallow network and a deep network. The shallow network is used to capture edge and texture features, and the deep network is used to capture and recognize semantic information. Fuse the features output by the shallow network and the features output by the deep network to obtain a fused regional feature image; The convolutional neural network is used to determine the region to which the pixels of the fused regional feature image belong.

5. The method for estimating pupil occlusion sight line based on image representation enhancement according to claim 1, characterized in that: Perform regional segmentation on the detected eye image to identify the pupil area, the visual aid area, and the background area. This also includes: The head posture features are extracted from the eye area image with the visual aid area being blank through a convolutional network.

6. The pupil occlusion sight line estimation method based on image representation enhancement according to claim 1, characterized in that: Completing the pupil area image where the visual aid area is blank based on the diffusion model to obtain a pupil restoration image includes: The diffusion model is used to generate the overall geometric structure of the pupil area through step-by-step denoising, and the circular outline and center position are restored, thereby achieving coarse restoration of the pupil area. The diffusion model is used to supplement the iris texture and reflective spots in the roughly repaired pupil area, thereby achieving fine repair of the pupil area.

7. The method for estimating pupil occlusion sight line based on image representation enhancement according to claim 1, characterized in that: A deep learning algorithm is used to learn the mapping relationship between pupil feature point positions and gaze directions, including: Extracting pupil feature points in the pupil image using a convolutional neural network, where the pupil feature points include pupil center, corneal reflection points, and eye feature points; The deep learning network is trained by using the pupil feature point positions as input data and the sight line directions as output data of the deep learning network, thereby learning the mapping relationship between the pupil feature point positions and the sight line directions.

Images