Self-adaptive line-of-sight estimation algorithm model construction method

Abstract

The invention relates to a self-adaptive sight line estimation algorithm model construction method, which belongs to the field of sight line estimation, and is characterized in that a displacement parameter and a zoom parameter which have guiding significance on eye feature extraction are extracted from a face image and an eye mask, and the module can dynamically adjust the normalization process of an eye feature map, so that the visual line estimation accuracy is improved. Therefore, the feature distribution can adapt to the facial features of different individuals and the position information of the eyes and the faces in the image more adaptively. According to the adaptive line-of-sight estimation algorithm model construction method, by introducing an adaptive guidance module and a multi-scale attention module, the adaptability and expression ability of the model in processing different individual differences are effectively improved, the main function of the adaptive guidance module is to dynamically adjust the normalization process of an eye feature map, and the adaptability and expression ability of the model in processing different individual differences are improved. Therefore, the feature distribution can adapt to the facial features of different individuals and the position information of the eyes and the faces in the image more adaptively, and the feature expression capability is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of line of sight estimation, and in particular to a method for constructing an adaptive line of sight estimation algorithm model. Background Art

[0002] Eyes are an important sense organ for humans to receive external information. Eye gaze estimation aims to determine the observer's three-dimensional line of sight direction or two-dimensional gaze point position by analyzing human eye or facial images. This technology occupies an important position in computer vision and has been widely used in human-computer interaction, virtual reality, and medical auxiliary diagnosis.

[0003] Gaze estimation algorithms are primarily categorized into two types: model-based and appearance-based. Model-based approaches are demanding on environmental conditions and struggle to adapt to complex and changing real-world scenarios. Appearance-based approaches, however, are not constrained by specific model assumptions and can directly extract features from images and map them to gaze directions. Especially with the integration of deep learning, appearance-based approaches have made significant progress, enabling more accurate gaze estimation.

[0004] However, existing convolutional neural network architectures often focus too much on local details when extracting features, neglecting global semantic information. This limitation can lead to poor performance in complex scenarios and an inability to fully exploit the complementary global and local information of facial features. The module designed in this paper addresses the shortcomings of traditional CNN methods by jointly capturing contextual information at different scales, significantly improving feature representation capabilities. Summary of the Invention

[0005] (1) Technical problems solved In response to the shortcomings of the existing technology, the present invention provides a method for constructing an adaptive line of sight estimation algorithm model, which has the advantages of improving efficiency and solving the problem that the appearance-based line of sight estimation method has limited effectiveness in dealing with personalized differences.

[0006] (2) Technical solution To achieve the above-mentioned purpose of improving efficiency, the present invention provides the following technical solution: a method for constructing an adaptive sight line estimation algorithm model, comprising the following steps: Step S1, extracting eyelid image and eyelid position information; In step S2, the facial image is input into a multi-scale attention module to extract facial features. The multi-scale attention module first divides the input feature map into G sub-feature groups. Each group extracts the attention weight descriptor of the grouped feature map through three parallel branches, including two 1×1 branches and a 3×3 branch based on depthwise separable convolution. The outputs of the three branches are aggregated through a dot product operation to generate preliminary attention weights. After nonlinear activation processing, the weights are element-wise multiplied with the original feature map to complete the fusion of multi-scale features. Step S3: The facial image is input into the multi-scale attention module to extract facial features, including displacement parameters and scaling parameters that are instructive for eye feature extraction. This process is achieved by feeding the concatenated vector of the facial features and the eye mask into a fully connected layer and then performing a nonlinear transformation. In step S4, the eye image is fed into the convolutional layer and the SE attention module to extract features, and the displacement and scaling parameters obtained in the previous step are added to the extraction process. In step S5, the eye features, facial features, and eyelid position information are input into the last fully connected layer to predict the preliminary gaze point position. The personalized bias parameters are added to adjust this preliminary gaze point position to achieve adaptive personalized fine-tuning, thereby determining the final gaze point position.

[0007] Preferably, in step S2, of the three parallel branches of the multi-scale attention module, two 1×1 branches are used to capture global spatial information and generate cross-channel global attention weights, and the 3×3 branch captures multi-scale spatial structural features in local areas through a depth-wise separable convolution kernel. The outputs of the three branches are aggregated through a dot product operation to generate preliminary attention weights. Finally, the weights are processed by nonlinear activation Sigmoid and multiplied element-by-element with the original feature map to complete the fusion of multi-scale features.

[0008] Preferably, in step S3, the adaptive guidance parameter is calculated by feeding the concatenated vector of facial features and eye mask into the fully connected layer and performing Gaussian error linear unit GELU nonlinear transformation; wherein, facial features and eyelid mask After processing, a concatenated vector is obtained, which is sent to a fully connected layer and then subjected to nonlinear transformation GELU to obtain the scaling parameter and offset parameters , the specific calculation is shown in formula 1, Represents the fully connected layer; the specific calculation is as shown in Formula 1, .

[0009] Preferably, in step S4, represents group normalization without offset and scaling, are the original input features, is the output feature, It is a learned bias term used to dynamically adjust the features of each channel, thereby enhancing the adaptability of the model to different inputs; the guidance process is as shown in Formula 2, .

[0010] Preferably, in step S5, the personalized bias parameter is initialized to a zero vector during model training and optimized by minimizing the estimation error; in the calibration stage, the personalized bias parameter is calculated using a small number of calibration samples, where the weight of the calibration sample is inversely proportional to the square of the sample prediction error.

[0011] Preferably, the calculation of the personalized bias parameter in the calibration stage also involves the variance of the prior distribution and the variance of the calibration sample line of sight estimation error.

[0012] Preferably, in step S5, specifically, during the network training phase, the personalized bias parameter is initialized to a zero vector. As the training progresses, the personalized bias parameter is optimized by minimizing the estimation error to gradually learn and capture the systematic deviation. The objective function calculation process is as shown in Formula 3, where is the actual gaze point position, is the predicted gaze position without adding personalized bias parameters, is the personalized bias parameter, is the L2 regularization parameter, which is set to 0.01 in this paper;

[0013] In the calibration phase, the personalized bias parameters are calculated using a small number of calibration samples. The calculation process is as shown in formula (4), where is the number of calibration samples, is the variance of the prior distribution, is the variance of the calibration sample line of sight estimation error, For the The weight of the calibration samples;

[0014]

[0015] Weight The calculation process of is shown in Formula 5. The weight is inversely proportional to the square of the sample prediction error. In this way, samples with smaller errors will have a greater impact on the model estimation.

[0016] (3) Beneficial effects Compared with the existing technology, the present invention provides a method for building an adaptive line of sight estimation algorithm model, which has the following beneficial effects: 1. This adaptive gaze estimation algorithm model construction method effectively improves the adaptability and expressiveness of the model when dealing with individual differences by introducing an adaptive guidance module and a multi-scale attention module. The main function of the adaptive guidance module is to dynamically adjust the normalization process of the eye feature map, so that the feature distribution can more adaptively adapt to the facial features of different individuals and the position information of the eyelids in the image. The main function of the multi-scale attention module is to make full use of the global and local complementary information of facial features to significantly improve the feature expression ability. At the same time, this method introduces personalized bias parameters, which can reduce the complexity of the fine-tuning process in the gaze estimation task of different individuals and improve the effectiveness of few-sample calibration.

[0017] 2. The adaptive gaze estimation algorithm model construction method performs well on small sample data sets: when conducting experiments on the public small sample data set MPIIFaceGaze, the gaze point error predicted by the method constructed by the present invention is only 3.79 cm, which shows that the method can maintain a high prediction accuracy even with a limited sample size. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of a method for constructing an adaptive line of sight estimation algorithm model proposed in the present invention; Figure 2 This is a schematic diagram of the structure of an adaptive sight line estimation algorithm model according to a method for constructing an adaptive sight line estimation algorithm model proposed in the present invention; Figure 3 This is a schematic diagram of the multi-scale attention module structure of the adaptive gaze estimation algorithm model construction method proposed in the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] The present invention designs a method for constructing an adaptive sight line estimation algorithm model. Figure 1 It mainly includes three parts: extracting eyelid images and eyelid position information, inputting eyelid position information and facial features to extract adaptive guidance parameters, and calculating personalized bias parameters to complete adaptive personalized fine-tuning.

[0021] Example 1: Step S1, extracting eyelid images and eyelid position information; first, obtain an image of a target person, who stares at the screen at a certain angle, obtain the target image through a camera, and segment the image into a facial image, a left eye image, a right eye image, and eyelid bounding box position information.

[0022] Step S2: Input the facial image into the multi-scale attention module to extract facial features; the multi-scale attention module structure is as follows: Figure 3 As shown in the figure, to enhance feature learning and noise compression capabilities, the multi-scale attention module first divides the input feature map into G sub-feature groups, each learning different semantics. Each group then extracts attention weight descriptors for the grouped feature map through three parallel branches: two 1×1 branches and a 3×3 branch based on depthwise separable convolution. A shared 1×1 convolution kernel reorganizes the sub-feature groups in the channel direction and employs a one-dimensional global average pooling operation to extract global channel information. This branch captures global spatial information, complementing local features and generating cross-channel global attention weights. Another 3×3 branch uses depthwise separable convolution kernels to capture multi-scale spatial structure in local regions. Depthwise convolution extracts local spatial features, while pointwise convolution fuses information across channels. The outputs of the three branches are aggregated through a dot product operation to generate preliminary attention weights. Finally, after a nonlinear activation (sigmoid), the weights are element-wise multiplied with the original feature map to complete the fusion of multi-scale features.

[0023] Step S3: Input the facial image into the multi-scale attention module to extract facial features, including displacement parameters and scaling parameters that are instructive for eye feature extraction. This process is achieved by feeding the concatenated vector of facial features and eye mask into the fully connected layer and performing nonlinear transformation. and eyelid mask After Figure 1 After the processing shown in the figure, a concatenated vector is obtained, which is fed into a fully connected layer and then subjected to nonlinear transformation (GELU) to obtain the scaling parameter and offset parameters , the specific calculation is shown in formula (1), represents the fully connected layer, .

[0024] Step S4: Send the eye image to the convolution layer and SE attention module to extract features, and add the displacement parameters and scaling parameters obtained in the previous step to the extraction process; Send the eye image to the convolution layer and SE attention module to extract features, and add the displacement parameters and scaling parameters obtained in the previous step to the extraction process, and add the position as shown in the figure. Figure 1As shown at "OUT", the specific guidance process is shown in formula (2). represents group normalization without offset and scaling, are the original input features, is the output feature, It is a learned bias term used to dynamically adjust the features of each channel to enhance the adaptability of the model to different inputs. .

[0025] In step S5, eye features, facial features, and eyelid position information are fed into the final fully connected layer to predict a preliminary gaze point location. A personalized bias parameter is then added to adjust this preliminary gaze point location, achieving adaptive, personalized fine-tuning to determine the final gaze point location. This parameter is updated using gradient descent during model training and recalculated based on a small number of calibration samples during fine-tuning.

[0026] Specifically, during the network training phase, the personalized bias parameters are initialized to zero vectors. As the training progresses, the personalized bias parameters are optimized by minimizing the estimation error to gradually learn and capture systematic biases. The objective function calculation process is as shown in formula (3), where is the actual gaze point position, is the predicted gaze position without adding personalized bias parameters, is the personalized bias parameter, is the L2 regularization parameter, which is set to 0.01 in this paper.

[0027]

[0028] In the calibration phase, the personalized bias parameters are calculated using a small number of calibration samples. The calculation process is as shown in formula (4), where is the number of calibration samples, is the variance of the prior distribution, is the variance of the calibration sample line of sight estimation error, For the The weights of the calibration samples.

[0029]

[0030]

[0031] Weight The calculation process of is shown in formula (5). The weight is inversely proportional to the square of the sample prediction error. In this way, samples with smaller errors will have a greater impact on the model estimation.

[0032] The beneficial effect of the present invention is that the adaptive gaze estimation algorithm model construction method effectively improves the adaptability and expressiveness of the model when dealing with differences between different individuals by introducing an adaptive guidance module and a multi-scale attention module. Among them, the main function of the adaptive guidance module is to dynamically adjust the normalization process of the eye feature map, so that the feature distribution can more adaptively adapt to the facial features of different individuals and the position information of the eyelids in the image. The main function of the multi-scale attention module is to make full use of the global and local complementary information of facial features and significantly improve the feature expression ability. At the same time, the method introduces personalized bias parameters, which can reduce the complexity of the fine-tuning process in the gaze estimation task of different individuals and improve the effectiveness of few-sample calibration.

[0033] 2. The adaptive gaze estimation algorithm model construction method performs well on small sample data sets: when conducting experiments on the public small sample data set MPIIFaceGaze, the gaze point error predicted by the method constructed by the present invention is only 3.79 cm, which shows that the method can maintain a high prediction accuracy even with a limited sample size.

[0034] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing an adaptive line of sight estimation algorithm model, characterized in that: The following steps are involved: Step S1, extracting eyelid image and eyelid position information; Step S2: input the facial image into the multi-scale attention module to extract facial features; The multi-scale attention module first divides the input feature map into G sub-feature groups. Each group extracts the attention weight descriptor of the grouped feature map through three parallel branches, including two 1×1 branches and a 3×3 branch based on depthwise separable convolution. The outputs of the three branches are aggregated through dot product operations to generate preliminary attention weights. After nonlinear activation processing, the weights are multiplied element-by-element with the original feature map to complete the fusion of multi-scale features. Step S3: The facial image is input into the multi-scale attention module to extract facial features, including displacement parameters and scaling parameters that are instructive for eye feature extraction. This process is achieved by feeding the concatenated vector of the facial features and the eye mask into a fully connected layer and then performing a nonlinear transformation. In step S4, the eye image is fed into the convolutional layer and the SE attention module to extract features, and the displacement and scaling parameters obtained in the previous step are added to the extraction process. In step S5, the eye features, facial features, and eyelid position information are input into the last fully connected layer to predict the preliminary gaze point position. The personalized bias parameters are added to adjust this preliminary gaze point position to achieve adaptive personalized fine-tuning, thereby determining the final gaze point position.

2. The method for constructing an adaptive sight line estimation algorithm model according to claim 1, wherein: In step S2, among the three parallel branches of the multi-scale attention module, the two 1×1 branches are used to capture global spatial information and generate cross-channel global attention weights, and the 3×3 branch captures multi-scale spatial structural features in local areas through depth-wise separable convolution kernels. The outputs of the three branches are aggregated through dot product operations to generate preliminary attention weights. Finally, the weights are processed by nonlinear activation Sigmoid and multiplied element-by-element with the original feature map to complete the fusion of multi-scale features.

3. The method for constructing an adaptive sight line estimation algorithm model according to claim 1, wherein: In step S3, the adaptive guidance parameters are calculated by feeding the concatenated vector of facial features and eye mask into the fully connected layer and then undergoing Gaussian error linear unit GELU nonlinear transformation; wherein, facial features and eyelid mask After processing, a concatenated vector is obtained, which is sent to a fully connected layer and then subjected to nonlinear transformation GELU to obtain the scaling parameter and offset parameters , the specific calculation is shown in formula 1, Represents the fully connected layer; the specific calculation is as shown in Formula 1, .

4. The method for constructing an adaptive sight line estimation algorithm model according to claim 1, wherein: In step S4, represents group normalization without offset and scaling, are the original input features, is the output feature, It is a learned bias term used to dynamically adjust the features of each channel, thereby enhancing the adaptability of the model to different inputs; the guidance process is as shown in Formula 2, .

5. The method for constructing an adaptive sight line estimation algorithm model according to claim 1, wherein: In step S5, the personalized bias parameter is initialized to a zero vector during model training and optimized by minimizing the estimation error; During the calibration phase, personalized bias parameters are calculated using a small number of calibration samples, where the weight of the calibration samples is inversely proportional to the square of the sample prediction error.

6. The method for constructing an adaptive sight line estimation algorithm model according to claim 5, characterized in that: The calculation of the personalized bias parameters during the calibration phase also involves the variance of the prior distribution and the variance of the calibration sample line of sight estimation error.

7. The method for constructing an adaptive sight line estimation algorithm model according to claim 5, wherein: In step S5, specifically, during the network training phase, the personalized bias parameters are initialized to a zero vector. As the training progresses, the personalized bias parameters are optimized by minimizing the estimation error to gradually learn and capture the systematic deviation. The objective function calculation process is as shown in Formula 3, where is the actual gaze point position, is the predicted gaze position without adding personalized bias parameters, is the personalized bias parameter, is the L2 regularization parameter, which is set to 0.01 in this paper; 8. In the calibration phase, the personalized bias parameters are calculated using a small number of calibration samples. The calculation process is as shown in formula (4), where is the number of calibration samples, is the variance of the prior distribution, is the variance of the calibration sample line of sight estimation error, For the The weight of the calibration samples; Weight The calculation process of is shown in Formula 5. The weight is inversely proportional to the square of the sample prediction error. In this way, samples with smaller errors will have a greater impact on the model estimation.

Images