Sight line estimation method based on color difference guided eye feature enhancement

Abstract

The invention provides a sight line estimation method based on color difference guided eye feature enhancement, and the method comprises the steps: carrying out the image preprocessing of a face image, and obtaining a preprocessed image; performing region detection on the preprocessed image to obtain an eye image; extracting multi-layer features of the face image and the eye image; convolution similarity among the multiple layers of features is obtained through convolution operation; performing pupil region identification on the eye image through chromatic aberration calculation to obtain a pupil significant mask; performing color difference on the multi-layer features through a pupil significant mask, and performing cross-modal integration on the multi-layer features and convolution similarity to obtain fusion features; and carrying out regression training and loss optimization on the fusion features to obtain a sight line direction vector. According to the method, a local feature enhancement and chromatic aberration guide mechanism is introduced into the deep neural network, so that stable capture of eye and pupil features under the conditions of complex illumination, posture change and partial shielding is realized, and the sight estimation precision is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent human-computer interaction technology, and in particular to a gaze estimation method based on color difference-guided eye feature enhancement. Background Technology

[0002] Currently, traditional models typically use the entire face image as input, extracting global features through multiple convolutional layers. However, as the network depth increases, subtle features such as those around the eyes are gradually weakened, leading to insufficient accuracy in capturing pupil changes and gaze direction. In the fusion of facial and eye features, simple concatenation or weighted summation is often used, making it difficult to establish similarity guidance relationships between layers. This results in insufficient interaction between global and local information, affecting the stability of gaze prediction. Furthermore, under complex lighting and skin color conditions, the brightness and chromaticity features of the pupil region differ significantly from those of the periocular skin. This difference can serve as a stable visual guidance signal, but existing methods largely rely on the network's self-learning attention, lacking explicit utilization of optical differences.

[0003] In summary, existing technologies have made some progress in terms of accuracy, robustness, and computational efficiency for gaze estimation, but they still have the following shortcomings: 1. Insufficient utilization of global geometric or spatial constraints in single-frame images, making it difficult to guarantee stable estimation under complex poses or deflections; 2. Multimodal feature fusion or text-guided mechanisms increase system complexity and lead to higher training and deployment costs; 3. Some methods rely on image pairs or reference frames, lacking single-frame universality and having limited generalization ability; 4. Lack of explicit guidance on key visual information (such as chromatic aberration and local geometric changes), making it difficult to balance accuracy and lightweight design. Therefore, it is essential to design a gaze estimation method based on chromatic aberration-guided eye feature enhancement. Summary of the Invention

[0004] The purpose of this invention is to provide a gaze estimation method based on color difference-guided eye feature enhancement. By introducing local feature enhancement and color difference guidance mechanisms into a deep neural network, the method can stably capture eye and pupil features under complex lighting, pose changes, and partial occlusion conditions, thereby improving the accuracy of gaze estimation.

[0005] To achieve the above objectives, the present invention provides the following solution: A gaze estimation method based on color difference-guided eye feature enhancement includes the following steps: The acquired facial images are preprocessed to obtain preprocessed images; A lightweight face detection network is used to perform region detection on the preprocessed image to obtain the eye image; Multi-layer features of face and eye images are extracted using a convolutional neural network. Convolutional similarity between multiple layers of features is obtained through convolution operations; The pupil region is identified by color difference calculation, resulting in a significant pupil mask; Color difference enhancement features are obtained by enhancing the color difference of multi-layer features through a prominent pupil mask; By integrating color difference enhancement features with convolutional similarity across modalities, a fused feature is obtained. Regression training and loss optimization are performed on the fused features to obtain the gaze direction vector.

[0006] Optionally, the acquired facial images are preprocessed to obtain a preprocessed image, including: The facial image is subjected to distortion correction and normalization to obtain the initial image; A smoothed image is obtained by suppressing random noise components in the initial image using a Gaussian smoothing kernel; The smoothed image is subjected to brightness equalization and scale normalization to obtain a preprocessed image.

[0007] Optionally, the expression for extracting multi-layer features from face images and eye images separately using a convolutional neural network is as follows: ;in, For facial images, Image of the eye. For multi-layer features of a face image, The Backbone is used as the feature extractor for the multi-layer features of the eye image.

[0008] Optionally, the formula for calculating convolutional similarity is: ;in, For channel-by-channel feature interaction, This is a 3×3 convolution operation. For the Sigmoid function, For multi-layer features of a face image, This represents the multi-layer features of the eye image.

[0009] Optionally, pupil region identification is performed on the eye image through color difference calculation to obtain a salient pupil mask, including: The color space of the eye image is converted to obtain the converted image; The color difference distribution map is calculated based on the global mean of the transformed image; the expression for the color difference distribution map is: ;in, To convert the image, This represents the mean value of the Lab channel; Threshold segmentation of the eye image is performed based on the color difference distribution map to obtain a significant pupil mask.

[0010] Optionally, chromatic aberration enhancement is performed on the multi-layer features using a pupil salient mask to obtain chromatic aberration enhanced features. The specific steps are as follows: The pupil salient mask and the multi-layer features are fused using element-wise multiplication; the expression for the chromatic aberration enhanced features is: ;in, To demonstrate element-wise multiplication, For learnable reinforcement coefficients, For multi-layer features of eye images, This is a 3×3 convolution operation. It serves as a significant mask for the pupil.

[0011] Optionally, the color difference enhancement features and convolutional similarity are integrated across modalities to obtain fused features. Specifically, the color difference enhancement features and convolutional similarity are weighted and fused; the expression for the fused features is: ;in, For balance coefficient, For convolutional similarity, For color difference enhancement features, This refers to the multi-layer features of a face image.

[0012] Optionally, the functional expression for loss optimization is: ;in, Let the line of sight direction be the vector. This represents the actual direction of gaze.

[0013] According to specific embodiments provided by the present invention, the following technical effects are disclosed: The gaze estimation method based on color difference-guided eye feature enhancement provided by the present invention includes: preprocessing the acquired facial image to obtain a preprocessed image; performing region detection on the preprocessed image using a lightweight face detection network to obtain an eye image; extracting multi-layer features from the face image and eye image respectively using a convolutional neural network; obtaining convolutional similarity between multi-layer features through convolution operations; performing pupil region recognition on the eye image through color difference calculation to obtain a salient pupil mask; enhancing the multi-layer features through the salient pupil mask to obtain color difference enhancement features; integrating the color difference enhancement features and convolutional similarity across modalities to obtain fused features; and performing regression training and loss optimization on the fused features to obtain a gaze direction vector. This method, by introducing local feature enhancement and color difference guidance mechanisms into a deep neural network, achieves stable capture of eye and pupil features under complex lighting, pose changes, and partial occlusion conditions, thereby improving the accuracy of gaze estimation. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of the line-of-sight estimation method of the present invention; Figure 2 This is a schematic diagram of the line-of-sight estimation network model structure according to an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] like Figure 1 As shown, this invention provides a gaze estimation method based on color difference-guided eye feature enhancement, comprising the following steps: Step 100: Perform image preprocessing on the acquired facial images to obtain preprocessed images; Step 200: Perform region detection on the preprocessed image using a lightweight face detection network to obtain the eye image; Step 300: Extract multi-layer features from the face image and eye image respectively using a convolutional neural network; Step 400: Obtain the convolutional similarity between multiple layers of features through convolution operations; Step 500: Identify the pupil region in the eye image by calculating color difference to obtain a salient pupil mask; Step 600: Enhance the color difference of the multi-layer features using a pupil salient mask to obtain the color difference enhanced features; Step 700: Integrate the color difference enhancement features and convolutional similarity across modalities to obtain fused features; Step 800: Perform regression training and loss optimization on the fused features to obtain the gaze direction vector.

[0019] like Figure 2As shown, the present invention also provides a network model for implementing a gaze estimation method based on color difference-guided eye feature enhancement.

[0020] Preferably, the acquired facial images are preprocessed to obtain a preprocessed image, including: The facial image is subjected to distortion correction and normalization to obtain the initial image; A smoothed image is obtained by suppressing random noise components in the initial image using a Gaussian smoothing kernel; The smoothed image is subjected to brightness equalization and scale normalization to obtain a preprocessed image.

[0021] In the specific implementation process, step 100 involves acquiring facial images of the driver or subject using a monocular RGB camera. Then, the facial image was subjected to distortion correction and normalization, and a Gaussian smoothing kernel was used. Suppressing random noise components to obtain a smooth image The expression is: Then on Brightness equalization and scale standardization are performed to ensure consistent feature distribution under different individuals, lighting conditions, and skin color conditions, thereby providing stable input for subsequent eye area detection and color difference calculation.

[0022] In some embodiments, step 200 uses a lightweight face detection network, RetinaFace, to extract face regions from the preprocessed image. Based on the face region, key points are located to obtain the binocular regions. It will identify The eye images were cropped and uniformly scaled to a standard size. This ensures the consistency of local features across spatial scales.

[0023] In some embodiments, step 300 will use a face image With eye images The data is input into two ResNet18 convolutional neural networks with a shared structure, and multi-layer features of each image are extracted. The expression is as follows: ; in, For facial images, Image of the eye. For multi-layer features of a face image, The Backbone represents the feature extractor, i.e., ResNet18, for the multi-layer features of the eye image.

[0024] In some embodiments, since fine structures such as the pupil are easily smoothed during convolution stacking, a local similarity mapping module is constructed in step 400, and a local enhancement mechanism is introduced to recover highly sensitive features. Specifically, the similarity response between the multi-layer features of the face image and the eye image is calculated through convolution operations. (i.e., convolutional similarity), the expression is: ; in, For channel-by-channel feature interaction, This is a 3×3 convolution operation. The sigmoid function is used to obtain the similarity response. It describes the regional responses in the face feature map that are highly correlated with the spatial features of the eyes, and is used to guide the model to focus on spatial locations related to potential gaze directions in subsequent stages.

[0025] Preferably, pupil region identification is performed on the eye image through color difference calculation to obtain a salient pupil mask, including: The color space of the eye image is converted to obtain the converted image; The color difference distribution map is calculated based on the global mean of the converted image; Threshold segmentation of the eye image is performed based on the color difference distribution map to obtain a significant pupil mask.

[0026] In the specific implementation process, step 500 first generates the eye image. Converting to the Lab color space, the expression is: ; Where L represents the luminance component and (a,b) represents the chrominance component. Then, the mean value of the Lab channel of the transformed image is calculated, and the color difference of each pixel is calculated using Euclidean distance to obtain the color difference distribution map. The expression is: ; in, The converted image, This is the average value for the Lab channel. Finally, we set the global color difference average value to be... The standard deviation is And calculate the threshold. The expression is: ; in, is the threshold factor. The eye image is then segmented based on the threshold to obtain a salient pupil mask, expressed as: A prominent pupil mask visually corresponds to the low-brightness, high-chromatic-difference area of ​​the pupil.

[0027] In the specific implementation process, step 600, in order to transfer saliency information to the deep semantic space, fuses the pupil saliency mask with multi-layer features, as expressed in the following expression: ; in, To demonstrate element-wise multiplication, These are learnable enhancement coefficients. They amplify the feature responses of the pupil region while maintaining the original semantic distribution, thereby strengthening local features directly related to the gaze direction.

[0028] In the specific implementation process, step 700 performs a weighted fusion of color difference enhancement features and convolutional similarity, expressed as follows: ; in, This is the balance coefficient.

[0029] It should be noted that weighted fusion is used to achieve layer-by-layer attention focusing from the global face to the local pupil, so that the model can maintain high sensitivity to pupil changes during deep convolution.

[0030] In the specific implementation process, step 800 will integrate the features. Global average pooling and multilayer perceptron (MLP) mapping are performed to output the gaze direction vector. , and These represent the horizontal and vertical viewing angles, respectively. The model is trained using a Euclidean loss function, the expression of which is: ; in, This represents the true gaze direction. End-to-end optimization from the enhanced feature space to the gaze angle space was achieved.

[0031] It should be noted that during the inference phase, the model is input with a single frame of a face image. The system automatically detects the eye region and extracts features, then sequentially processes these features through a local similarity mapping module, color difference-guided enhancement processing, and regression processing, ultimately outputting the predicted gaze angle. Since color difference information calculation only involves local pixel operations and convolution, the model features fast inference speed, low computational overhead, and real-time operation capabilities.

[0032] The beneficial effects of this invention are as follows: 1) By explicitly guiding feature extraction using local color difference information of the eye, the iris color difference change is mapped into a key feature enhancement signal, which strengthens the fine-grained features of the eye in the deep network, thereby significantly improving the accuracy of gaze direction prediction and sensitivity to small offsets. 2) Combining facial features and fine-grained eye features, the model achieves information complementarity through the local similarity mapping module, making full use of the full-face context and local eye features, so that the model can still maintain robust gaze estimation ability in scenarios such as complex lighting, partial occlusion or slight head movement. 3) Compared with traditional methods that rely on infrared, depth cameras or multiple cameras, this invention only requires a single camera to complete high-precision line-of-sight estimation, which reduces hardware costs and system complexity, and facilitates rapid deployment in vehicle, office, medical and general indoor environments.

[0033] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0034] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for gaze estimation based on color difference guided eye feature enhancement, characterized in that, The method comprises the following steps: image preprocessing is performed on the collected face image to obtain a preprocessed image; region detection is performed on the preprocessed image by a lightweight face detection network to obtain an eye image; multi-layer features of the face image and the eye image are extracted by a convolutional neural network respectively; convolution similarity between the multi-layer features is obtained by convolution operation; pupil region recognition is performed on the eye image by color difference calculation to obtain a pupil saliency mask; color difference reinforcement is performed on the multi-layer features by the pupil saliency mask to obtain color difference reinforced features; the color difference reinforced features and the convolution similarity are integrated in a cross-modal manner to obtain fusion features; regression training and loss optimization are performed on the fusion features to obtain a gaze direction vector.

2. The eye feature enhanced line of sight estimation method based on chromatic aberration guidance according to claim 1, wherein, image preprocessing is performed on the collected face image to obtain a preprocessed image, comprising: distortion removal and normalization are performed on the face image to obtain an initial image; random noise components of the initial image are suppressed by a Gaussian smoothing kernel to obtain a smoothed image; brightness equalization and scale standardization are performed on the smoothed image to obtain the preprocessed image.

3. The eye-gaze estimation method based on color difference guided eye feature enhancement according to claim 1, characterized in that, Expressions of multi-layer features of the face image and the eye image are extracted by a convolutional neural network respectively, and the expressions are: ; wherein, is the face image, is the eye image, is the multi-layer feature of the face image, is the multi-layer feature of the eye image, and Backbone is a feature extractor.

4. The eye feature enhanced line of sight estimation method based on chromatic aberration guidance according to claim 1, wherein, The operation formula of the convolution similarity is: ; wherein, is a channel-by-channel feature interaction operation, is a 3*3 convolution operation, is a Sigmoid function, is a multi-layer feature of a face image, is a multi-layer feature of an eye image.

5. The eye feature enhanced line of sight estimation method based on chromatic aberration guidance according to claim 1, wherein, pupil region recognition is performed on the eye image by color difference calculation to obtain a pupil saliency mask, comprising: color space conversion is performed on the eye image to obtain a converted image; A color difference distribution map is calculated according to the global mean value of the converted image; the expression of the color difference distribution map is: ; wherein, is the converted image, is the mean value of the Lab channel. threshold segmentation is performed on the eye image according to the color difference distribution map to obtain the pupil saliency mask.

6. The eye feature enhanced line of sight estimation method based on chromatic aberration guidance according to claim 1, characterized in that, color difference reinforcement is performed on the multi-layer features by the pupil saliency mask to obtain color difference reinforced features, and the specific steps are as follows: feature fusion is performed on the pupil saliency mask and the multi-layer features by element-by-element multiplication. The expression of the color difference reinforcement feature is: ; wherein, is an element-wise multiplication, is a learnable reinforcement coefficient, is a multi-layer feature of the eye image, is a 3x3 convolution operation, is a pupil saliency mask.

7. The eye-gaze estimation method based on color difference guided eye feature enhancement according to claim 1, characterized in that, The chromaticity difference reinforced feature is cross-modally integrated with the convolution similarity to obtain a fusion feature, and the specific steps are as follows: the chromaticity difference reinforced feature is weightedly fused with the convolution similarity; and an expression of the fusion feature is as follows: ; wherein, is a balance coefficient, is the convolution similarity, is the chromaticity difference reinforced feature, is a multi-layer feature of the face image.

8. The eye feature enhanced line of sight estimation method based on chromatic aberration guidance according to claim 1, wherein, The function expression optimized for the loss is: ; wherein, is a line of sight direction vector, is a true gaze direction.

Images