A cross-view gaze estimation method based on feature decoupling and attention mechanism

Abstract

The application provides a cross-view gaze estimation method based on feature decoupling and attention mechanism, through double-camera acquisition of double-view face images, effectively eliminates the view blind area under single view, and alleviates the ambiguity problem caused by insufficient single view information; the face features are decoupled into personal appearance features, head posture features and gaze-related features, realizing accurate separation of different types of features; the feature decoupling strategy enhances the adaptability of the model in diversified scenes, ensuring that it still maintains high performance under different individuals and large head posture changes; then, cross attention mechanism is applied to the three types of features obtained by decoupling for adaptive weighting processing, through fusion of double-view gaze features, the gaze information is supplemented and enhanced; at the same time, combined with the processed appearance features and head posture features, the influence of personal factors and head movement is compensated, thereby significantly improving the accuracy and robustness of gaze estimation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a cross-view gaze estimation method based on feature decoupling and attention mechanism. Background Technology

[0002] With the development of technologies such as image processing and computer vision, gaze estimation has gradually become an important research direction in the field of human-computer interaction, and has been widely applied in virtual / augmented reality, assisted driving, and psychological research. The core goal of gaze estimation technology is to infer the user's gaze direction by analyzing images of the user's eyes or face, thereby enabling natural interaction between the user and the system.

[0003] Traditional gaze estimation methods are mainly divided into model-based methods and appearance-based methods. Model-based methods, by constructing and fitting an eye model, can provide high accuracy, but they usually require users to wear specific devices, which not only increases hardware costs but may also interfere with user experience, limiting their practical application. In contrast, appearance-based methods learn the relationship between the image and the gaze direction, and can perform gaze estimation using only a regular webcam, requiring no additional hardware configuration. This lower cost and wider application prospects have led to their gradual emergence as the mainstream method for gaze estimation.

[0004] Currently, most appearance-based gaze estimation methods rely on single-view images for gaze direction estimation. However, due to the limited field of view of single-view images, it is often difficult to fully capture complete facial information, which may lead to depth blur and ambiguity in gaze estimation. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a cross-view gaze estimation method based on feature decoupling and attention mechanisms. By combining the rich information provided by dual-view face images, feature decoupling effectively separates different types of features, and attention mechanisms are used to learn the relationships between features, thereby achieving adaptive feature fusion and significantly improving the accuracy of gaze estimation.

[0006] A cross-view gaze estimation method based on feature decoupling and attention mechanism is proposed. Two normalized face image sequences from two different viewpoints are input into a cross-view gaze estimation network, which outputs the gaze direction vector of the face. The cross-view gaze estimation network includes a face feature extraction backbone network, a feature decoupling module, a first to third cross-attention feature fusion module, a feature stitching module, and a multilayer perceptron.

[0007] The facial feature extraction backbone network is used to extract features from the normalized facial image sequence under dual-view conditions, obtaining the overall facial features f under both views. 1 and f2 ;

[0008] The feature decoupling module respectively performs f 1 and f 2 By performing feature decoupling, the appearance features under dual-view conditions are obtained. and Head posture characteristics and Visual characteristics and

[0009] The first cross-attention feature fusion module is used to integrate gaze features and Feature fusion is performed, and the fusion result is then compared with the original line-of-sight features. and The features are enhanced by addition to obtain the final fused gaze characteristics. and

[0010] The second cross-attention feature fusion module is used to combine gaze features and respectively on appearance features and Dynamic weighting is performed to obtain the weighted appearance features. and

[0011] The third cross-attention feature fusion module is used to combine gaze features and Head posture features and Dynamic weighting is performed to obtain the weighted head pose features. and

[0012] The feature stitching module is used to fuse the line-of-sight features from each camera's perspective. and Appearance features and Head posture characteristics and The features are concatenated separately to obtain the concatenated features. and

[0013] The multilayer perceptron is used to analyze splicing features. and Learn to predict the gaze direction vector of a face. and

[0014] Furthermore, the loss function L used when training the cross-viewpoint gaze estimation network model is... total for:

[0015] L total =α·L gt +β·L app_sim +ε·L head +γ·L ga

[0016] Among them, L gt Let α be the angle deviation loss function, and L be the angle deviation loss function. gt The corresponding weight, L app_sim Let β be the appearance similarity loss function, and L be the value of L. app_sim The corresponding weight, L head Let L be the head angle loss function, and ε be L. head The corresponding weight, L ga Let L be the cross-viewpoint gaze alignment loss function, and γ be L. ga The corresponding weights.

[0017] Furthermore, the appearance similarity loss function L app_sim The calculation method is as follows:

[0018]

[0019] Where B is the total number of faces used as training samples. Let i be the appearance features of the i-th face in two viewpoints. and The cosine similarity between them, and The calculation method is as follows:

[0020]

[0021] Where ||·|| represents the modulus.

[0022] Furthermore, the head angle loss function L head The calculation method is as follows:

[0023]

[0024] Where B is the total number of faces used as training samples. Let be the true value of the head orientation vector of the i-th face. Let ||·|| be the estimated value of the head orientation vector obtained by the multilayer perceptron of the head pose feature input for the i-th person's face pose feature, and ||·|| be the modulus.

[0025] Furthermore, the method for obtaining the true value of the head orientation vector of any face is as follows:

[0026] Face images were detected using Dlib, and 68 facial key points were extracted. Combined with a pre-trained 3D average head model, the EPnP algorithm was used to perform 3D fitting on the face images and calculate the rotation matrix of the head in the face image relative to the standard camera coordinate system. The rotation matrix was used to transform the three-axis direction vector of the head coordinate system to the standard camera coordinate system, and the head z-axis direction vector in the transformed standard camera coordinate system was normalized. The normalized head z-axis direction vector was used as the true value of the head direction vector of the face.

[0027] Furthermore, the cross-viewpoint gaze alignment loss function L ga The calculation method is as follows:

[0028]

[0029] Where B is the total number of faces used as training samples. Let f be the gaze feature of the i-th face in the first viewpoint. 1(i) gaze The initial 3D gaze direction is obtained from the input gaze feature multilayer perceptron. Let f be the gaze feature of the i-th face in the second viewpoint. 2(i) gaze The initial 3D gaze direction is obtained from the input gaze feature multilayer perceptron. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the first camera to the original coordinate system of the first camera. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the second camera to the original coordinate system of the second camera. Let be the rotation matrix of the first camera coordinate system relative to the global screen coordinate system. This is the rotation matrix of the second camera coordinate system relative to the global screen coordinate system.

[0030] Furthermore, the angle deviation loss function L gt The calculation method is as follows:

[0031]

[0032] Where B is the total number of faces used as training samples. Let be the true value of the gaze direction vector of the i-th face. f is the splicing feature corresponding to the i-th face. fused Input the gaze direction vector estimate obtained from the multilayer perceptron, where ||·|| is the modulus.

[0033] Furthermore, the first cross-attention feature fusion module is used to integrate gaze features and Feature fusion is performed, and the fusion result is then compared with the original line-of-sight features. and The features are enhanced by addition to obtain the final fused gaze characteristics. and The specific method is as follows:

[0034] View characteristics in the original coordinate system of the first camera Transform to the original coordinate system of the second camera to obtain the line-of-sight features after the viewpoint transformation.

[0035] Using multi-head cross-attention mechanism to capture gaze features With visual characteristics Perform adaptive fusion to obtain initial fused features.

[0036] Initial fusion features With visual characteristics Adding them together yields the final fused line-of-sight features.

[0037] View characteristics in the original coordinate system of the second camera Transform to the original coordinate system of the first camera to obtain the line-of-sight features after the viewpoint transformation.

[0038] Using multi-head cross-attention mechanism to capture gaze features With visual characteristics Perform adaptive fusion to obtain initial fused features.

[0039] Initial fusion features With visual characteristics Adding them together yields the final fused line-of-sight features.

[0040] Furthermore, the second cross-attention feature fusion module obtains the weighted appearance features. and The specific method is as follows:

[0041] Let k = 1, 2, and define the line-of-sight characteristics. Mapping to appearance features The same dimension yields the mapped gaze features. Using cross-attention components for appearance features and mapping gaze features Dynamic weighting is performed to obtain the weighted appearance features.

[0042] Furthermore, the third cross-attention feature fusion module obtains the weighted head pose features. and The specific method is as follows:

[0043] Let k = 1, 2, and define the line-of-sight characteristics. Mapping to head pose features The same dimension yields the mapped gaze features. Using cross-attention components to analyze head pose features and mapping gaze features Dynamic weighting is performed to obtain the weighted head pose features.

[0044] Beneficial effects:

[0045] This invention provides a cross-view gaze estimation method based on feature decoupling and attention mechanisms. By acquiring dual-view facial images using dual cameras, it effectively eliminates blind spots in single-view scenarios and alleviates ambiguity caused by insufficient information in single-view scenarios. By setting different loss functions, facial features are decoupled into personal appearance features, head pose features, and gaze-related features, achieving accurate separation of different feature types. The feature decoupling strategy enhances the model's adaptability to diverse scenarios, ensuring high performance even under varying individual and head pose conditions. Then, a cross-attention mechanism is applied to the three types of decoupled features for adaptive weighting, and the fusion of dual-view gaze features supplements and enhances gaze information. Simultaneously, by combining the processed appearance features and head pose features, the influence of personal factors and head movements is compensated, significantly improving the accuracy and robustness of gaze estimation. Attached Figure Description

[0046] Figure 1 This is a diagram of the overall network architecture of the present invention;

[0047] Figure 2 This is a schematic diagram of the DenseNet feature extraction module used in this invention;

[0048] Figure 3 This is a diagram showing the transformation relationships between the coordinate systems of this invention;

[0049] Figure 4(a) is a visualization of the distribution of the three types of features extracted by the model at the 100th iteration using t-SNE in an embodiment of the present invention.

[0050] Figure 4(b) is a visualization of the distribution of the three types of features extracted by the model at the 1500th iteration using t-SNE in an embodiment of the present invention.

[0051] Figure 5 This is a visualization diagram of the line-of-sight estimation results in an embodiment of the present invention. Detailed Implementation

[0052] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0053] With the continuous development of camera equipment and the gradual reduction in camera costs, dual-camera systems have gradually become a viable option. Compared to single-view images, dual-view gaze estimation systems can provide a wider field of view, effectively compensating for the information loss caused by facial occlusion in single-view images, thereby reducing estimation errors caused by viewpoint limitations. This dual-view gaze estimation method shows great potential in improving gaze estimation accuracy and robustness, becoming an important research direction in appearance-based gaze estimation methods.

[0054] With the increase in viewpoint information, effectively fusing features from different viewpoints has become a significant new challenge. Feature decoupling techniques, by decomposing complex feature vectors into multiple independent sub-features, can effectively reduce interference between features, thereby improving the clarity of feature representation. In the process of multi-viewpoint information fusion, feature decoupling helps to independently model each type of feature, avoiding information confusion and thus improving the accuracy and robustness of gaze estimation. Combining decoupled features with attention mechanisms can further strengthen the correlation between different features, enhancing the accuracy and robustness of gaze estimation.

[0055] Based on this, the present invention provides a cross-view gaze estimation method based on feature decoupling and attention mechanisms. Two normalized face image sequences from two different viewpoints are input into a cross-view gaze estimation network, which outputs the gaze direction vector of the face. The cross-view gaze estimation network includes a face feature extraction backbone network, a feature decoupling module, first to third cross-attention feature fusion modules, a feature stitching module, and a multilayer perceptron. Figure 1 As shown, the data processing procedure of the cross-viewpoint gaze estimation network is as follows:

[0056] S1: The facial feature extraction backbone network is used to extract features from the normalized facial image sequence under dual-view conditions, and obtain the overall facial features f under dual-view conditions respectively. 1 and f 2 ;

[0057] It should be noted that this invention can obtain normalized face videos V1 and V2 simultaneously captured by any two cameras from a publicly available gaze estimation dataset. Using the same frame rate, such as 10Hz, the videos from both perspectives are resampled, and the resulting image size is adjusted to 256×256 pixels, thereby generating a corresponding N-frame normalized face image sequence, denoted as . and The normalized face image sequence from dual perspectives is... and The data is sequentially input into the DenseNet face feature extraction backbone network to obtain face features f from both perspectives. 1 and f 2 .

[0058] For example, normalized RGB face image sequences from two different viewpoints are labeled with different tags, and then input into a DenseNet feature extraction network with a batch size of 128, resulting in input dimensions of (128, 3, 256, 256). After feature extraction by the backbone network, (128, 210) dimensional overall face features f are obtained for both viewpoints. 1 and f 2 .

[0059] The feature extraction backbone network uses the DenseNet dense connection network, and the network structure is as follows: Figure 2 As shown, DenseNetEncoder extracts low-level features through an initial convolutional layer, and then extracts higher-level features layer by layer through three dense blocks. Each dense block concatenates the output of the previous layer with the features of the current layer, with a growth rate of 32. LeakyReLU is used as the activation function, and InstanceNorm2d is used as the normalization method.

[0060] S2: The feature decoupling module respectively performs f 1 and f 2 By performing feature decoupling, the appearance features under dual-view conditions are obtained. and Head posture characteristics and Visual characteristics and

[0061] It should be noted that the facial feature vector f is assumed to consist of three parts: gaze-related features, head pose features, and personal appearance features. Let f[1:a] be the a-dimensional personal appearance feature f. app f[a+1:a+b] represents the b-dimensional head pose feature f head f[a+b+1:a+b+c] represents the c-dimensional line-of-sight related features. gaze For example, the first 42 dimensions of the feature vector f are set as the personal appearance features f. app The 36-dimensional features in the middle are set as head pose features f. head The latter 132 dimensions are set as line-of-sight related features f gaze .

[0062] S3: The first cross-attention feature fusion module is used to integrate gaze features and Feature fusion is performed, and the fusion result is then compared with the original line-of-sight features. and The features are enhanced by adding them separately to obtain the final fused gaze characteristics. and Specifically as follows:

[0063] S31: View characteristics in the original coordinate system of the first camera Transform to the original coordinate system of the second camera to obtain the line-of-sight features after the viewpoint transformation.

[0064]

[0065] In other words, this invention uses the line-of-sight features of the first camera in its original coordinate system C1. Multiply by the first camera's inverse normalized rotation matrix M1 in sequence. -1 First camera coordinate system x c1 -y c1 -z c1 To the global screen coordinate system x s -y s -z s Rotation matrix between Global screen coordinate system x s -y s -z s To the second camera coordinate system x c2 -y c2 -z c2 Rotation matrix between From the original coordinate system of the second camera to the normalized coordinate system of the second camera (x) m2 -y m2 -z m2 The normalized rotation matrix M2 is used to transform the image to the normalized coordinate system of the second camera C2. The overall transformation process is as follows: Figure 3 As shown. It should be noted that M1 and M2 are 3x3 normalization matrices obtained from the EVE dataset, which are transformed from the two camera coordinate systems to their corresponding normalized camera coordinate systems. This is a 3x3 rotation matrix that transforms the coordinates from the two camera coordinate systems to the global screen coordinate system.

[0066] S32: Utilizing multi-head cross-attention mechanism to capture visual features With visual characteristics Adaptive fusion is performed to obtain the initial fused features from the second perspective.

[0067] The multi-head attention mechanism is implemented through the nn.MultiheadAttention layer, using 132-dimensional feature input and four attention heads for computation. This module uses the gaze features after viewpoint transformation as the query and the gaze features of the target viewpoint as the key and value, respectively, to perform cross-viewpoint attention computation. The weighted gaze features f 2 gaze_att The implementation principle is as follows:

[0068]

[0069] In the above formula, Q1 represents the line-of-sight feature after the viewpoint transformation. K1 and V1 represent the line-of-sight features from the second camera's perspective. Feature dimension d k1 =132 / 4=33, which corresponds to the feature dimension of each attention head key vector.

[0070] S33: Initial fusion features With visual characteristics The features are added together to supplement and enhance the gaze characteristics, resulting in the final fused gaze characteristics from the second perspective.

[0071]

[0072] S34: View characteristics in the original coordinate system of the second camera Transform to the original coordinate system of the first camera to obtain the line-of-sight features after the viewpoint transformation.

[0073] S35: Utilizing a multi-head cross-attention mechanism to capture visual features With visual characteristics Perform adaptive fusion to obtain initial fused features.

[0074] S36: Initial fusion features With visual characteristics Adding them together yields the final fused line-of-sight features.

[0075] S4: The second cross-attention feature fusion module is used to combine gaze features and respectively on appearance features and Dynamic weighting is performed to obtain the weighted appearance features. and Specifically as follows:

[0076] Let k = 1, 2, and define the line-of-sight characteristics. Mapping to appearance features The same dimension yields the mapped gaze features. Utilizing a cross-attention component of two attention heads combined with mapped gaze features Appearance features Dynamic weighting is performed to obtain the weighted appearance features. The calculation formula is as follows:

[0077]

[0078] In the above formula, Q2 represents the mapped line-of-sight features. K2 and V2 are appearance features f app Feature dimension d k2 =42 / 2=21.

[0079] S5: The third cross-attention feature fusion module is used to combine gaze features and Head posture features and Dynamic weighting is performed to obtain the weighted head pose features. and Specifically as follows:

[0080] Let k = 1, 2, and define the line-of-sight characteristics. Mapping to head pose features The same dimension yields the mapped gaze features. Utilizing a cross-attention component of two attention heads combined with mapped gaze features Head posture characteristics Dynamic weighting is performed to obtain the weighted head pose features. The specific calculation formula is as follows:

[0081]

[0082] In the above formula, Q3 represents the mapped line-of-sight features. K3 and V3 are head posture features. Feature dimension d k3 =36 / 4=9.

[0083] S6: The feature stitching module is used to fuse gaze features and Appearance features and Head posture characteristics and Feature stitching is performed according to the viewpoint, and k = 1, 2, to obtain the stitched feature f. k fused :

[0084]

[0085] S7: The multilayer perceptron is used to analyze the splicing features f k fused Learning is performed to predict the gaze direction vectors of a face from two different perspectives.

[0086]

[0087] In other words, this invention utilizes a feature splicing method to fuse gaze features processed by an attention mechanism. and Appearance features and Head posture characteristics and By fusing the data according to the camera's perspective, a 210-dimensional fused view feature is obtained. The fused features are processed by a three-layer MLP with hidden layer sizes of 10⁵ and 5², respectively, to obtain the 3D gaze direction vector from the dual-camera perspective.

[0088] It should be noted that this invention uses the corresponding training and validation sets from the public multi-view gaze estimation dataset (EVE, ETH-XGaze) to train and test the cross-view gaze estimation network model, where the training and validation set sizes are 10:1. Furthermore, the loss function L used during the training of the cross-view gaze estimation network model... total for:

[0089] L total =α·L gt +β·L app_sim +ε·L head +γ·L ga

[0090] Among them, L gt Let α be the angle deviation loss function, and L be the angle deviation loss function. gt The corresponding weight, L app_sim Let β be the appearance similarity loss function, and L be the value of L. app_sim The corresponding weight, L head Let L be the head angle loss function, and ε be L. head The corresponding weight, L ga Let L be the cross-viewpoint gaze alignment loss function, and γ be L. ga The corresponding weights. It should be noted that, after multiple experimental verifications, the angle deviation loss L was set... gt The weight α = 1, and the appearance similarity loss L app_sim The weight β = 0.5, and the cross-viewpoint line alignment loss L gaThe weight γ = 0.3, and the head pose loss L head The weight ε = 0.5.

[0091] Furthermore, assuming that the same person has the same appearance features at the same time under different camera perspectives, then the appearance features from both perspectives should have a similar distribution in the feature space. The appearance similarity loss function is then used to constrain the appearance features from both perspectives. and To make them more similar. The appearance features of the i-th face in dual-view perspectives. and Cosine similarity between The calculation method is as follows:

[0092]

[0093] Where ||·|| represents the modulus.

[0094] Using the inverse cosine function to calculate cosine similarity Converting to angles, the average angle loss over all samples yields the appearance similarity loss function L. app_sim as follows:

[0095]

[0096] Where B is the total number of faces used as training samples.

[0097] Furthermore, the present invention incorporates head pose features and The input is fed into a three-layer multi-layer perceptron (MLP) with hidden layer sizes of 18 and 9. Each layer uses ReLU as the activation function, and the feature extraction results of the first two layers are normalized using BatchNorm1d to obtain the estimated three-dimensional orientation vector of the head. And use the true value h of the head orientation vector 3d Using the ground truth labels as supervision, we obtain the head angle loss function L. head as follows:

[0098]

[0099] Where B is the total number of faces used as training samples. Let be the true value of the head orientation vector of the i-th face. Let ||·|| be the estimated head orientation vector obtained from the multilayer perceptron of the head pose features input to the i-th face, and ||·|| be the modulus. The method for obtaining the true head orientation vector of any face is as follows:

[0100] Standardized face images were detected using Dlib, and 68 facial key points were extracted. Combined with a pre-trained 3D average head model, the EPnP algorithm was used to perform 3D fitting on the face images, and the rotation matrix of the head in the face image relative to the standard camera coordinate system was calculated. The rotation matrix was used to transform the three-axis direction vector of the head coordinate system to the standard camera coordinate system, and the head z-axis direction vector in the transformed standard camera coordinate system was normalized. The normalized head z-axis direction vector was used as the true value of the head direction vector of the face.

[0101] Furthermore, the cross-viewpoint gaze alignment loss function L ga The calculation method is as follows:

[0102]

[0103] Where B is the total number of faces used as training samples. Let i be the gaze features of the i-th face in the first viewpoint. The initial 3D gaze direction is obtained from a three-layer multilayer perceptron with hidden layer sizes of 66 and 33. The gaze features of the i-th face in the second perspective The initial 3D gaze direction is obtained from a three-layer multilayer perceptron with hidden layer sizes of 66 and 33. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the first camera to the original coordinate system of the first camera. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the second camera to the original coordinate system of the second camera. Let be the rotation matrix of the first camera coordinate system relative to the global screen coordinate system. This is the rotation matrix of the second camera coordinate system relative to the global screen coordinate system.

[0104] In other words, this invention uses the image normalization matrices M1 and M2 provided by the dataset to transform the initial estimated vectors under dual-view conditions. Multiply by M1 on the left respectively -1 M2 -1 Perform inverse normalization to transform to the original coordinate system x of both cameras. c1 -y c1 -z c1 x c2 -y c2 -z c2 Next, multiply both by the original camera coordinate system and the global screen coordinate system x, respectively. s -y s -z s Rotation matrix between It achieves the transformation of dual-view gaze direction to the same coordinate system.

[0105] Furthermore, this invention utilizes the ground-truth label values ​​of the gaze direction in the dataset to supervise the 3D gaze direction estimated by the network, with the angle bias loss function L... gt The calculation method is as follows:

[0106]

[0107] Where B is the total number of faces used as training samples. Let be the true value of the gaze direction vector of the i-th face. f is the splicing feature corresponding to the i-th face. fused Input the gaze direction vector estimate obtained from the multilayer perceptron, where ||·|| is the modulus.

[0108] It should be noted that the network model proposed in this invention is implemented based on the PyTorch framework, and the parameters are initialized using the Xavier method. During training, the batch size is set to 128, the optimizer is Adam, the initial learning rate is 0.001, and the weight decay value is 0.0005. The learning rate adopts the CosineAnnealingLR scheduling strategy, and the total number of training iterations is 4000. The model parameters are saved every 400 iterations for subsequent verification and testing.

[0109] This invention uses Mean Angular Error (MAE) to measure the model's estimation performance on the validation set. The formula for calculating MAE is as follows:

[0110]

[0111] Where N is the total number of samples in the test set. and g are the predicted gaze vector and the actual gaze vector, respectively. The smaller the MAE value, the higher the estimation accuracy of the model.

[0112] The performance of the trained cross-view gaze estimation model on the validation sets of two gaze estimation datasets is shown in Table 1. The gaze angle estimation errors of the proposed network model on the EVE and ETH-XGaze validation sets are 2.57° and 2.69°, respectively. Compared with multi-view gaze estimation models MMGE and MGT, and full-face gaze estimation models FullFace and GazeCLR, it has higher gaze estimation accuracy, proving that the proposed network model has good performance.

[0113] Table 1. Results of line-of-sight estimation angle error (unit: °)

[0114] Figures 4(a) and 4(b) use t-SNE to visualize the distributions of the three types of features extracted by the model at the 100th and 1500th training iterations, respectively. Among them, the personal appearance feature f... app Head posture features f head and gaze-related features f gaze The features are represented by red, yellow, and blue dots, respectively. By comparing the feature point distribution in Figure 4(a) and Figure 4(b), it can be seen that as training progresses, different types of features gradually achieve a clearer and more compact separation in the feature space, proving that the feature decoupling effect of this model is good and can effectively improve the expressive power of different types of features.

[0115] Figure 5 Figure 4(b) shows the convergence curves of the various loss functions of the network on the training set. As shown in Figure 4(b), the network training losses rapidly approach their minimum values ​​after 400 iterations on the training set, then slowly decrease and finally converge. Among them, the angle deviation loss L... gt Converging to 1.36°, appearance similarity loss L app_sim The convergence value was reduced to 0.057, the head pose loss was reduced to 2.05°, and the cross-viewpoint gaze alignment loss was reduced to L. ga The convergence to 1.11° fully validates the effectiveness of the network architecture and its good convergence performance.

[0116] To verify the effectiveness of the design of each module in the cross-viewpoint gaze estimation network, ablation experiments were conducted on the EVE and ETH-XGaze gaze estimation datasets. Table 2 shows the angle errors of six different ablation experiments. Experimental results show that each module in the network of this invention plays a positive role in improving model performance.

[0117] Table 2 Ablation Experiment Results (Unit: °)

[0118]

[0119] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A cross-view gaze estimation method based on feature decoupling and attention mechanism, characterized in that, Two normalized face image sequences from dual perspectives are input into a cross-view gaze estimation network, which outputs the gaze direction vector of the face. The cross-view gaze estimation network includes a face feature extraction backbone network, a feature decoupling module, a first to third cross-attention feature fusion module, a feature stitching module, and a multilayer perceptron. The facial feature extraction backbone network is used to extract features from the normalized facial image sequence under dual-view conditions, obtaining the overall facial features f under both views. 1 and f 2 ; The feature decoupling module respectively performs f 1 and f 2 By performing feature decoupling, the appearance features under dual-view conditions are obtained. and Head posture characteristics and Visual characteristics and The first cross-attention feature fusion module is used to integrate gaze features and Feature fusion is performed, and the fusion result is then compared with the original line-of-sight features. and The features are enhanced by addition to obtain the final fused gaze characteristics. and The second cross-attention feature fusion module is used to combine gaze features and respectively on appearance features and Dynamic weighting is performed to obtain the weighted appearance features. and The third cross-attention feature fusion module is used to combine gaze features and Head posture features and Dynamic weighting is performed to obtain the weighted head pose features. and The feature stitching module is used to fuse the line-of-sight features from each camera's perspective. and Appearance features and Head posture characteristics and The features are concatenated separately to obtain the concatenated features. and The multilayer perceptron is used to analyze splicing features. and Learn to predict the gaze direction vector of a face. and 2. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 1, characterized in that, The loss function L used during the training of the cross-view gaze estimation network model total for: L total =α·L gt +β·L app_sim +e·L head +γ·L ga Among them, L gt Let α be the angle deviation loss function, and L be the angle deviation loss function. gt The corresponding weight, L app_sim Let β be the appearance similarity loss function, and L be the value of L. app_sim The corresponding weight, L head Let L be the head angle loss function, and ε be L. head The corresponding weight, L ga Let L be the cross-viewpoint gaze alignment loss function, and γ be L. ga The corresponding weights.

3. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 2, characterized in that, Appearance similarity loss function L app_sim The calculation method is as follows: Where B is the total number of faces used as training samples. Let i be the appearance features of the i-th face in two viewpoints. and The cosine similarity between them, and The calculation method is as follows: Where ||·|| represents the modulus.

4. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 2, characterized in that, Head angle loss function L head The calculation method is as follows: Where B is the total number of faces used as training samples. Let be the true value of the head orientation vector of the i-th face. Let ||·|| be the estimated value of the head orientation vector obtained by the multilayer perceptron of the head pose feature input for the i-th person's face pose feature, and ||·|| be the modulus.

5. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 4, characterized in that, The method for obtaining the true value of the head orientation vector of any human face is as follows: The face image was detected by Dlib and 68 facial key points were extracted. Combined with a pre-trained 3D average head model, the face image was 3D fitted using the EPnP algorithm, and the rotation matrix of the head in the face image relative to the standard camera coordinate system was calculated. The three-axis direction vectors of the head coordinate system are transformed to the standard camera coordinate system using a rotation matrix. The head z-axis direction vector in the transformed standard camera coordinate system is then normalized, and the normalized head z-axis direction vector is taken as the true value of the head direction vector of the face.

6. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 2, characterized in that, Cross-view gaze alignment loss function L ga The calculation method is as follows: Where B is the total number of faces used as training samples. Let f be the gaze feature of the i-th face in the first viewpoint. 1 (i) gaze The initial 3D gaze direction is obtained from the input gaze feature multilayer perceptron. Let f be the gaze feature of the i-th face in the second viewpoint. 2(i) gaze The initial 3D gaze direction is obtained from the input gaze feature multilayer perceptron. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the first camera to the original coordinate system of the first camera. Let be the inverse normalized rotation matrix of the i-th face from the normalized coordinate system of the second camera to the original coordinate system of the second camera. Let be the rotation matrix of the first camera coordinate system relative to the global screen coordinate system. This is the rotation matrix of the second camera coordinate system relative to the global screen coordinate system.

7. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 2, characterized in that, Angular deviation loss function L gt The calculation method is as follows: Where B is the total number of faces used as training samples. Let be the true value of the gaze direction vector of the i-th face. f is the splicing feature corresponding to the i-th face. fused Input the gaze direction vector estimate obtained from the multilayer perceptron, where ||·|| is the modulus.

8. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 1, characterized in that, The first cross-attention feature fusion module is used to integrate gaze features and Feature fusion is performed, and the fusion result is then compared with the original line-of-sight features. and The features are enhanced by addition to obtain the final fused gaze characteristics. and The specific method is as follows: View characteristics in the original coordinate system of the first camera Transform to the original coordinate system of the second camera to obtain the line-of-sight features after the viewpoint transformation. Using multi-head cross-attention mechanism to capture gaze features With visual characteristics Perform adaptive fusion to obtain initial fused features. Initial fusion features With visual characteristics Adding them together yields the final fused line-of-sight features. View characteristics in the original coordinate system of the second camera Transform to the original coordinate system of the first camera to obtain the line-of-sight features after the viewpoint transformation. Using multi-head cross-attention mechanism to capture gaze features With visual characteristics Perform adaptive fusion to obtain initial fused features. Initial fusion features With visual characteristics Adding them together yields the final fused line-of-sight features.

9. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 1, characterized in that, The second cross-attention feature fusion module obtains the weighted appearance features. and The specific method is as follows: Let k = 1, 2, and define the line-of-sight characteristics. Mapping to appearance features The same dimension yields the mapped gaze features. Using cross-attention components for appearance features and mapping gaze features Dynamic weighting is performed to obtain the weighted appearance features.

10. The cross-view gaze estimation method based on feature decoupling and attention mechanism as described in claim 1, characterized in that, The third cross-attention feature fusion module obtains the weighted head posture features. and The specific method is as follows: Let k = 1, 2, and define the line-of-sight characteristics. Mapping to head pose features The same dimension yields the mapped gaze features. Using cross-attention components to analyze head pose features and mapping gaze features Dynamic weighting is performed to obtain the weighted head pose features.

Images