Multi-technology collaborative sight line estimation method and device, computer program and storage medium

Abstract

The invention discloses a multi-technology collaborative sight line estimation method and device, a computer program and a storage medium, belongs to the field of computer vision, and particularly relates to the technical field of sight line estimation. The problems of changeable head postures, illumination interference and insufficient model generalization in an appearance-based method are solved. The method comprises the following steps: acquiring an image; a multi-task convolutional neural network is utilized to detect a face frame and feature points, a face image is cut out, eye key points are obtained through Dlib, and an eye image is cut out and normalized; face and eye features are extracted by using MobileNet v2, and a head posture is obtained by using 3DMM and PnP algorithms; the face features are enhanced through a cross-layer dynamic convolution module, the head posture features and the face features are processed through GAT to obtain enhanced features, and the enhanced features, the enhanced face features and the eye features are spliced to obtain comprehensive features; and obtaining an estimated sight line vector by adopting CAS-ViT based on the comprehensive features. The method is suitable for the fields of man-machine interaction, intelligent transportation, medical health and the like.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to the technical field of line of sight estimation. Background Art

[0002] Gaze estimation technology is of great significance in many fields of modern society. It infers the direction of an individual's gaze by analyzing eye movement trajectories, thereby revealing the focus of attention and potential intentions. It is widely used in daily communication, medical health, marketing, human-computer interaction, intelligent transportation and other industries.

[0003] In the development of gaze estimation technology, two main methods have been formed: model-based and appearance-based:

[0004] Model-based methods are based on the eyeball structure and spatial geometric imaging model, and estimate the gaze direction by calibrating the invariant features of the eye and related geometric relationships. These methods include corneal reflection-based and shape-based methods. Among them, the corneal reflection-based method relies on an external infrared light source to detect eye features and calculates the pupil center and corneal reflection vector to achieve gaze estimation. Although it continues to develop, it is still limited by equipment, working distance, and range, and its performance degrades in complex environments. Shape-based methods analyze the geometric features of the eye to deduce the eyeball rotation angle to estimate the gaze. Although this method improves adaptability to a certain extent, the overall model-based method has high requirements for equipment and environment, making it difficult to be widely applied in complex real-world scenarios.

[0005] Appearance-based methods directly process images of the eye region and offer advantages in adaptability to complex scenes. Early traditional machine learning-based methods, such as neural networks and local linear interpolation, estimated gaze by fitting a nonlinear mapping function from the image to the gaze direction. However, these methods faced challenges such as requiring a large number of personalized samples, cumbersome calibration, and decreased accuracy in complex environments. Despite subsequent improvements such as semi-supervised Gaussian process regression and adaptive linear regression, these methods remain significantly affected by environmental variations.

[0006] With the rapid development of deep learning, deep learning-based gaze estimation methods have gradually emerged. These methods utilize models such as convolutional neural networks and Transformers to automatically extract features from massive amounts of eye images, significantly improving the accuracy and robustness of gaze estimation. However, these methods still face numerous challenges, such as the variability of head pose, individual differences, blinking, and lighting interference, which all affect gaze feature extraction and direction determination. In particular, existing methods struggle to accurately capture the changes in the relative position and angle of the eyes during head movement, failing to effectively model dynamic spatial relationships. This results in limited accuracy and poor robustness in gaze estimation.

[0007] Insufficient model generalization is also a common problem. Existing gaze estimation models typically rely on specific datasets and environments for training. When the scene or user group changes, the model performance often degrades significantly. These models lack structural and algorithmic flexibility, making it difficult to adapt to a variety of head poses, gaze directions, and facial features, thus failing to meet the needs of a wide range of practical applications. Summary of the Invention

[0008] In order to solve the problems existing in the prior art, the present invention proposes the following solutions:

[0009] A multi-technology collaborative line of sight estimation method, the method comprising:

[0010] S1, step of collecting images;

[0011] S2, using a multi-task convolutional neural network to detect the image to obtain a face frame and face feature points, cropping the face frame to obtain a face image, using a Dlib algorithm to obtain face key point coordinates based on the face feature points, the face key point coordinates include eye key point coordinates, cropping the eye key point coordinates to obtain an eye image, and normalizing the face image and the eye image;

[0012] S3. Use MobileNet v2 network to extract the facial feature map F from the normalized face image and eye image face and the eye feature vector F eye , using a three-dimensional deformable face model to obtain three-dimensional coordinates corresponding to the coordinates of key points of the face, and using a PnP algorithm to obtain head posture features based on the three-dimensional coordinates;

[0013] S4. Enhance the facial feature map F using cross-layer dynamic convolution modules face Get the face feature enhancement image F′ face , using the graph attention network based on the face feature map F face and head posture features to obtain enhanced feature vector H′, concatenate enhanced feature vector H′ and face feature enhanced map F′ face and the eye feature vector F eye , obtain the comprehensive feature vector F combined Steps;

[0014] S5, based on the comprehensive feature vector F combined , the steps of building a convolutional additive self-attention network to obtain the estimated gaze vector.

[0015] Furthermore, in step S3, the MobileNet v2 network is used to extract the facial feature map F from the normalized face image and eye image. face and the eye feature vector Feye The method is: use convolution layer and depth separable convolution to extract multi-level features layer by layer, and obtain a 128×28×28px face feature map F face and the 64-dimensional eye feature vector F eye .

[0016] Furthermore, the head posture feature in step S3 includes the yaw angle vector θ of the head posture x , pitch angle vector θ y and the roll angle vector θ z .

[0017] Furthermore, in step S4, the cross-layer dynamic convolution module is used to process the face feature map F face Get the face feature enhancement image F′ face The method is: the face feature map F face Perform multi-channel division, combine BiFormer block and residual connection processing to obtain new features, and then pass through a 1×1 convolution kernel to combine with the face feature map F face Fusion, obtain the face feature enhancement map F′ face .

[0018] Furthermore, the method for obtaining the enhanced feature vector H′ in step S4 is: by global average pooling the face feature map F face The dimension is reduced to a 128-dimensional vector F, which is weightedly concatenated with the head posture feature to obtain the fused feature vector F′. The feature vector ′ is split into 9 sub-vectors and normalized before being input into the graph attention network as the node to obtain 9 enhanced node feature vectors. The node feature vectors are aggregated and fused again with the feature vector F′ to obtain the enhanced feature vector H′.

[0019] Furthermore, in step S5, the method for obtaining the estimated gaze vector by constructing a convolutional additive self-attention network is as follows: the comprehensive feature vector F combined Expanded to feature map F′ combined The input is input to the CAS-ViT network, and after being processed by the Stem block, it enters the intermediate layer. The intermediate layer includes 4 stages, each of which consists of an integration layer, a CATM module and an MLP. After being processed by the 4 stages of the intermediate layer, it enters the output layer. The output layer predicts the position probability of the gaze point through a fully connected layer and 2 Softmax layers to obtain the final feature, and the final feature is mapped to the representation space of the gaze vector to obtain the estimated gaze vector.

[0020] A multi-technology coordinated line of sight estimation device, comprising:

[0021] Module for acquiring images;

[0022] A module for detecting the image using a multi-task convolutional neural network to obtain a face frame and face feature points, cropping the face frame to obtain a face image, obtaining face key point coordinates based on the face feature points using a Dlib algorithm, wherein the face key point coordinates include eye key point coordinates, cropping the eye key point coordinates to obtain an eye image, and normalizing the face image and the eye image;

[0023] Used to extract normalized face and eye images using the MobileNet v2 network to obtain the face feature map F face and the eye feature vector F eye , a module that uses a three-dimensional deformable face model to obtain the three-dimensional coordinates corresponding to the coordinates of the facial key points, and uses the PnP algorithm to obtain the head posture features based on the three-dimensional coordinates;

[0024] Used to enhance the face feature map F using cross-layer dynamic convolution modules face Get the face feature enhancement image F′ face , using the graph attention network based on the face feature map F face and head posture features to obtain enhanced feature vector H′, concatenate enhanced feature vector H′ and face feature enhanced map F′ face and the eye feature vector F eye , obtain the comprehensive feature vector F combined Modules;

[0025] For the comprehensive feature vector F combined The module for estimating the gaze vector is obtained by constructing a convolutional additive self-attention network.

[0026] Based on the same inventive concept, the present invention also proposes a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes any one of the methods described in the present invention.

[0027] Based on the same inventive concept, the present invention also proposes a computer, comprising a processor and a storage medium. When the processor reads the computer program stored in the storage medium, the computer executes any one of the methods described in the present invention.

[0028] Based on the same inventive concept, the present invention also proposes a computer program product, which is a computer program. When the computer program is read, it implements any one of the methods described in the present invention.

[0029] Compared with the prior art, the present invention has the following beneficial effects

[0030] 1. The present invention solves the problem that existing model-based methods have high requirements for equipment and environment and are difficult to adapt to complex scenes by combining a multi-task convolutional neural network, a three-dimensional deformable face model and a graph attention network, and has the effect of improving the accuracy and robustness of line of sight estimation in complex environments. Traditional model-based methods rely on external equipment and specific environmental conditions, and their performance drops significantly in complex scenes. The present invention uses deep learning technology to directly extract face and eye features from images, and combines a three-dimensional deformable face model to accurately capture head posture information, avoiding dependence on external equipment. At the same time, the introduction of the graph attention network enhances the spatial relationship modeling capability of features, so that the model can still maintain high accuracy under complex lighting, occlusion and variable head postures, significantly improving the adaptability and stability of line of sight estimation.

[0031] 2. The present invention solves the problem of inaccurate feature extraction of existing appearance-based methods under variable head posture, individual differences and lighting interference through cross-layer dynamic convolution modules and convolutional additive self-attention networks, which has the effect of enhancing feature expression capabilities and improving the accuracy of line of sight estimation. Appearance-based methods are easily affected by head posture, individual differences and lighting changes in complex environments, resulting in inaccurate feature extraction. The present invention performs multi-channel division and dynamic convolution processing on facial features through a cross-layer dynamic convolution module, and combines BiFormer blocks with residual connections to achieve efficient fusion of local details and global semantics, thereby enhancing the expression capabilities of features. In addition, the convolutional additive self-attention network further optimizes the feature fusion process, allowing the model to more accurately extract line of sight related features, thereby achieving high-precision line of sight estimation under variable conditions.

[0032] 3. This invention introduces a graph attention network, which is used to process the spatial relationship between head posture features and facial features. Features are treated as graph nodes, and the abstract associations between features are represented by a graph structure. After processing by the graph attention network, enhanced feature vectors are obtained. This technical approach reduces the impact of head posture changes on gaze estimation and accurately captures dynamic spatial relationships during head movement. This design enables the model to more accurately reflect the relationship between head posture and gaze direction, thereby significantly improving the accuracy of gaze estimation, particularly in scenarios with rapid head movement or large posture changes.

[0033] 4. The present invention solves the problems of insufficient generalization of existing models and dependence on specific data sets and environmental training through graph attention networks and global feature fusion, and has the effect of adapting to a variety of head postures, gaze directions and facial feature differences. Existing gaze estimation models usually rely on specific data sets for training, resulting in insufficient generalization ability and difficulty in adapting to new scenarios or user groups. The present invention uses a graph attention network to globally aggregate and fuse feature vectors, thereby enhancing the model's adaptability to different head postures, gaze directions and facial features. In addition, the global feature fusion mechanism enables the model to better capture individual differences among users, thereby maintaining high performance in a variety of practical application scenarios, significantly improving the generalization ability of the model.

[0034] 5. The present invention solves the problems of high computational complexity and high resource consumption of existing methods through a lightweight MobileNet v2 network and a cross-layer dynamic convolution module, significantly improving the efficiency of feature extraction. Traditional line of sight estimation methods suffer from excessive computational load due to complex model design, which limits the flexibility of actual deployment. The present invention uses a lightweight MobileNet v2 network to extract facial and eye features, and combines it with a cross-layer dynamic convolution module to optimize the feature extraction process, significantly reducing computational redundancy. At the same time, the design of the convolutional additive self-attention network further reduces invalid calculations and improves the computational efficiency of the model. The synergistic effect of the above technologies enables the present invention to significantly improve computational efficiency while ensuring high precision, providing feasibility support for the deployment of low-resource devices in complex scenarios.

[0035] 6. The present invention solves the problem of lack of flexibility in the structure and algorithm of existing methods through the collaboration of multiple technologies, and significantly improves the scalability and scene adaptability of the model. Existing line of sight estimation methods usually have a fixed structure and are difficult to adjust and optimize according to actual needs. The cross-layer dynamic convolution module is organically combined with technologies such as multi-task convolutional neural networks, three-dimensional deformable face models, graph attention networks, and convolutional additive self-attention networks, making the model structure flexible and easy to expand. For example, new feature extraction networks or attention mechanisms can be introduced according to specific application scenarios to further improve performance. This design enables the present invention to adapt to the needs of different fields and application scenarios, and has broad practical application value.

[0036] The present invention is applicable to fields such as human-computer interaction, intelligent transportation, medical health, marketing, education research, and security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of a multi-technology collaborative line of sight estimation method described in Embodiment 1 of the present invention;

[0038] Figure 2 Schematic diagram of the face rectangular frame and five feature points described in Embodiment 7 of the present invention;

[0039] Figure 3 Schematic diagram of 68 feature points of a human face described in Embodiment 7 of the present invention;

[0040] Figure 4 This is the human face schematic diagram with a size of 224×224 as described in the seventh embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of binoculars with a size of 64×32 as described in Embodiment 7 of the present invention, wherein the left side is the left eye and the right side is the right eye;

[0042] Figure 6 This is a schematic diagram of the head posture rotation direction described in the seventh embodiment of the present invention;

[0043] Figure 7 2 is a structural diagram of the cross-layer dynamic convolution module described in Embodiment 7 of the present invention;

[0044] Figure 8 This is a structural diagram of the BiFormer block described in the seventh embodiment of the present invention.

[0045] Figure 9 Schematic diagram of the segmentation feature input GAT described in Embodiment 7 of the present invention;

[0046] Figure 10 Schematic diagram of the graph attention mechanism described in Embodiment 7 of the present invention;

[0047] Figure 11 This is a diagram of the CAS-ViT network structure described in Embodiment 7 of the present invention;

[0048] Figure 12 This is the CATM module architecture diagram described in the seventh embodiment of the present invention

[0049] Figure 13 This is a block diagram of a multi-technology collaborative line of sight estimation method described in Embodiment 7 of the present invention. Implementation Method

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] Implementation Method 1

[0052] Combined with attachment Figure 1 This embodiment describes a multi-technology collaborative line of sight estimation method, the method comprising:

[0053] S1, step of collecting images;

[0054] S2, using a multi-task convolutional neural network to detect the image to obtain a face frame and face feature points, cropping the face frame to obtain a face image, using a Dlib algorithm to obtain face key point coordinates based on the face feature points, the face key point coordinates include eye key point coordinates, cropping the eye key point coordinates to obtain an eye image, and normalizing the face image and the eye image;

[0055] S3. Use MobileNet v2 network to extract the facial feature map F from the normalized face image and eye image face and the eye feature vector F eye , using a three-dimensional deformable face model to obtain three-dimensional coordinates corresponding to the coordinates of key points of the face, and using a PnP algorithm to obtain head posture features based on the three-dimensional coordinates;

[0056] S4. Enhance the facial feature map F using cross-layer dynamic convolution modules face Get the face feature enhancement image F′ face , using the graph attention network based on the face feature map F face and head posture features to obtain enhanced feature vector H′, concatenate enhanced feature vector H′ and face feature enhanced map F′ face and the eye feature vector F eye , obtain the comprehensive feature vector F combined Steps;

[0057] S5, based on the comprehensive feature vector F combined , the steps of building a convolutional additive self-attention network to obtain the estimated gaze vector.

[0058] This implementation addresses the low accuracy and poor robustness of existing gaze estimation methods in complex scenarios through the synergistic effects of a multi-task convolutional neural network, the Dlib algorithm, the MobileNet v2 network, a 3D deformable face model, a cross-layer dynamic convolution module, a graph attention network, and a convolutional additive self-attention network. By integrating these multiple technologies, this implementation accurately extracts facial and eye features and, combined with head pose information, significantly improves the accuracy and adaptability of gaze estimation, making it suitable for a variety of real-world scenarios.

[0059] Implementation Method 2

[0060] This embodiment is a further limitation of the first embodiment. Further, in step S3, the face feature map F in the normalized face image and eye image is extracted using the MobileNetv2 network. face and the eye feature vector F eyeThe method is: use convolution layer and depth separable convolution to extract multi-level features layer by layer, and obtain a 128×28×28px face feature map F face and the 64-dimensional eye feature vector F eye .

[0061] This implementation uses MobileNet v2's convolutional layers and depthwise separable convolutions to extract multi-level features layer by layer, addressing the inadequate feature extraction seen in traditional methods. By obtaining a 128×28×28px facial feature map and a 64-dimensional eye feature vector, this implementation more comprehensively captures facial and eye details, providing high-quality feature input for subsequent gaze estimation and improving the model's overall performance.

[0062] Implementation Method 3

[0063] This embodiment is a further limitation of the first embodiment. Further, the head posture feature in step S3 includes the yaw angle vector θ of the head posture. x , pitch angle vector θ y and the roll angle vector θ z .

[0064] This embodiment calculates the yaw, pitch, and roll angle vectors of the head posture through a three-dimensional deformable face model and the PnP algorithm, thereby improving the accuracy and robustness of the overall posture analysis and providing reliable input data support for subsequent posture feature-based analysis.

[0065] Implementation Method 4

[0066] This embodiment is a further limitation of the first embodiment. Furthermore, in step S4, the cross-layer dynamic convolution module is used to process the face feature map F. face Get the face feature enhancement image F′ face The method is: the face feature map F face Perform multi-channel division, combine BiFormer block and residual connection processing to obtain new features, and then pass through a 1×1 convolution kernel to combine with the face feature map F face Fusion, obtain the face feature enhancement map F′ face .

[0067] This implementation solves the problem of insufficient multi-scale feature expression capabilities in traditional methods by combining a cross-layer dynamic convolution module with a BiFormer block and a residual connection. Specifically, the input facial feature map is divided into multiple channels, and each sub-channel is subjected to deep convolution and double-layer routing attention processing through the BiFormer block to dynamically capture the correlation between local details and global semantics. Subsequently, the processed sub-features are concatenated and fused through a 1×1 convolution kernel, and then fused with the original input features to finally output the enhanced feature map. This feature enhancement process improves the robustness of the model, avoids the direct coupling of the feature extraction stage with the head posture, and thus focuses more on the expressive power of the underlying features.

[0068] Implementation Method Five

[0069] This embodiment is a further limitation of the first embodiment. Further, the method for obtaining the enhanced feature vector H′ in step S4 is: by global average pooling the face feature map F face The dimension is reduced to a 128-dimensional vector F, which is weightedly concatenated with the head posture feature to obtain the fused feature vector F′. The feature vector F′ is split into 9 sub-vectors and normalized before being input into the graph attention network as the node to obtain 9 enhanced node feature vectors. The node feature vectors are aggregated and fused again with the feature vector F′ to obtain the enhanced feature vector H′.

[0070] This embodiment solves the problem of insufficient modeling of dynamic relationships between features in existing methods through the optimized design of the graph attention network. Specifically, the fused feature vector is evenly divided into 9 sub-vectors, which are normalized and used as node inputs into the graph attention network. The graph attention network dynamically calculates the association weights between each sub-vector through the attention mechanism to generate 9 enhanced node feature vectors. Subsequently, these node features are aggregated and fused again with the original fused feature vector to form the final enhanced feature. The core of this process is to model the dynamic interaction relationship between sub-features through the graph attention network, so as to more accurately capture the complex correlation between head posture changes and line of sight direction, and significantly improve the generalization performance of the model for diverse user features and complex environments.

[0071] Implementation Method 6

[0072] This embodiment is a further limitation of the first embodiment. Further, in step S5, the method for obtaining the estimated sight line vector by constructing a convolutional additive self-attention network is as follows: the comprehensive feature vector F combined Expanded to feature map F′ cobmbinedThe input is to the convolutional additive self-attention (CAS-ViT) network, and after being processed by the starting module (Stem), it enters the intermediate layer, which includes 4 stages. Each stage consists of an integration layer, a convolutional additive label mixer (CATM) module and a multi-layer perceptron (MLP). After being processed by the 4 stages of the intermediate layer, it enters the output layer. The output layer predicts the position probability of the gaze point through a fully connected layer and 2 softmax layers to obtain the final feature, and the final feature is mapped to the representation space of the gaze vector to obtain the estimated gaze vector.

[0073] This implementation addresses the limited accuracy of gaze vector prediction in existing methods through a multi-stage encoding process using a convolutional additive self-attention (CAS-ViT) network. Through layer-by-layer optimization using integration layers, convolutional additive labeled mixer (CATM) modules, and multi-layer perceptron (MLP), this implementation more accurately predicts gaze position probabilities. Furthermore, using mean squared error loss training further enhances the accuracy and stability of gaze estimation, meeting the demands of high-precision applications.

[0074] Implementation Method Seven

[0075] This embodiment integrates the technical solutions described in the aforementioned multiple embodiments, combines the actual situation, and further verifies and explains the technical effects of the present invention through specific examples.

[0076] like Figure 1 As shown, the present invention proposes a multi-technology collaborative line of sight estimation method, which is implemented by the following steps:

[0077] Step 1: Input an image of size 256×256;

[0078] Step 2: Face detection and image preprocessing: Use a multi-task convolutional neural network and the Dlib algorithm to detect faces, determine facial key points, normalize the image, and crop the eye image.

[0079] Step 2.1, such as Figure 2 As shown in the figure, the multi-task convolutional neural network (MTCNN) is used to simultaneously detect the face area and facial key points, and finally outputs the xy coordinates of the starting point of the rectangular face frame and the length and width of the frame, as well as the five facial feature points: the center of the left eye, the center of the right eye, the tip of the nose, the left corner of the mouth, and the right corner of the mouth;

[0080] Preferably, MTCNN mainly includes three layers of networks. The first layer P-Net (Proposal Network) performs convolution and pooling on the input image, screens out possible face areas at multiple scales and outputs classification and regression information; the second layer R-Net (Refinement Network) receives the output results of the first layer, uses non-maximum suppression and new structure processing, filters out error boxes, corrects regression boxes, and outputs more accurate results; the third layer O-Net (Output Network) receives the suspected face area of the R-Net layer, and through complex structure and supervision mechanism, refines the processing to eliminate error boxes and outputs the classification result of whether it is a face, box information and the positions of 5 feature points.

[0081] Step 2.2: Select the 68 key points of the face detected by the Dlib algorithm:

[0082] The Dlib algorithm uses the five feature points detected by MTCNN as the starting guide, conducts a search operation within the face border area, extracts the coordinates of 68 key points, and can achieve sub-pixel accuracy during the detection process. Figure 3 As shown;

[0083] Step 2.3: Crop out the face and eye images and normalize them:

[0084] The original input image is cropped according to the face frame coordinates detected by MTCNN to obtain an image containing only the face area. The image size is unified to 224×224, as shown in Figure 4 As shown in the figure, according to the coordinates of the eye key points detected by the Dlib algorithm, the position area of the eye in the original image is determined, and then the eye image is cropped and the size is uniformly adjusted to 64×32, as shown in the figure. Figure 5 As shown; finally, the pixel value range (0-255) of all size-normalized face images and eye images is normalized to the interval [0,1].

[0085] The above steps are attached Figure 2-4 The facial images used are from the public database RT-Gene dataset.

[0086] Step 3: Extract facial and eye features and estimate head pose: Use the MobileNet v2 network to extract facial and eye features, and calculate head pose angles using key points based on 3D Morphable Models (3DMM) and the Perspective-n-Point (PnP) algorithm.

[0087] Step 3.1. Use the MobileNet v2 network to extract facial features and eye features. For facial images, a feature map of size 128 × 28 × 28 can be output, where 128 is the number of channels of the feature map, and 28 and 28 are the height and width of the feature map, respectively. For eye images, a 64-dimensional feature vector can be output.

[0088] Preferably, the preprocessed face image is input into the MobileNet v2 network. The network first extracts low-level features from the image through the convolution layer. As the network layer deepens, the depthwise separable convolution layer gradually extracts higher-level and more abstract features. These features can represent the overall structure, expression and other information of the face. Finally, after the global average pooling layer and the fully connected layer, the face image is converted into a 128×28×28 feature map, denoted as F. face .

[0089] Preferably, the preprocessed eye image is input into the MobileNet v2 network. The network first captures local features of the eye in the convolutional layer, such as iris texture, pupil shape, and eyelid contour; the depthwise separable convolutional layer further extracts more abstract features related to the eye state (eyes open, eyes closed, gaze direction); finally, the eye image is converted into a 64-dimensional feature vector, denoted as F eye .

[0090] Step 3.2: Use the PnP algorithm to estimate the head pose:

[0091] From the 68 key points, 7 key points are selected, including the inner and outer corners of the eyes, the tip of the nose, and the two corners of the mouth. The 3DMM model is then used to calculate the 3D coordinates of these 7 facial key points. The PnP algorithm is used to solve the position and posture of the camera relative to the head, and the yaw angle (Yaw), pitch angle (Pitch) and roll angle (Roll) of the head posture are calculated. Figure 6 shown.

[0092] Preferably, in the field of computer vision, the camera imaging model involves the camera coordinate system, the imaging plane coordinate system, and the world coordinate system, and the coordinate transformation formula is used to transform the world coordinate system and the imaging coordinate system. The coordinate transformation formula is shown in (2-1):

[0093]

[0094] Among them, (x, y) represents the imaging coordinates of any point, (u, v, w) represents the corresponding world coordinates; s is a scalar scaling factor; M is a matrix representing various internal parameters of the camera; R is the rotation matrix representing the world coordinate system to the camera coordinate system; t is a vector representing the translation of the world coordinate system relative to the camera coordinate system, and the head pose angle is only related to the rotation matrix R. Use 3 degrees of freedom to represent the head pose estimation, and use a 3×3 matrix to represent the rotation matrix R, as shown in formula (2-2):

[0095]

[0096] In the spatial coordinates, the x-axis, y-axis, and z-axis correspond to the degrees of freedom roll, pitch, and yaw in the three directions, respectively. Based on the rotation matrix R, the approximate deflection direction of the head posture can be estimated by formula (2-3), thereby obtaining the yaw angle θ of the head posture. x , pitch angle θ y and roll angle θ z .

[0097]

[0098] Step 4: Feature enhancement using a cross-layer dynamic convolution module and a graph attention network: The BiFormer block is introduced to optimize the cross-layer fusion convolution module to enhance facial features. At the same time, a graph attention network (GAT) is used to explore the complex relationship between head pose and eye gaze, thereby reducing the impact of head pose on gaze estimation.

[0099] Step 4.1: Improved cross-layer dynamic convolution module processes facial features: The facial feature map F extracted by MobileNet v2 network is converted to face As input, first perform 1×1 convolution, batch normalization, and ReLU activation function, and then evenly divide the obtained features into S=4 feature channels of equal size, respectively using x i (i=1,2,3,4) indicates that each feature channel has the same dimension and information capacity. Then, the four channels are processed by BiFormer block and residual connection, and the new features are concatenated and added to the input after passing through a 1×1 convolution kernel to obtain the final output F′. face , effectively enhancing facial features.

[0100] Preferably, the cross-layer dynamic convolution module can obtain receptive fields of different sizes on the input feature layer and fuse features of different scales to capture rich information from local details to global structures. Figure 7 As shown in , the structure mainly consists of a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function, a BiFormer block, and a residual structure, as shown in Figure 7 shown.

[0101] Preferably, when dividing features, the number of input channels is divided into multiple feature channels, so that the network can process different feature information separately on different channel branches, thereby realizing multi-scale feature extraction and fusion; each feature channel can be regarded as a specific perspective or representation of the original input features, which helps to capture richer feature patterns and details; in facial feature processing, different channels can focus on different facial area features such as eyes, nose, mouth, etc. or different feature levels.

[0102] Preferably, S is an important parameter for controlling the scale size. The larger S is, the stronger the multi-scale capability is. To ensure that each branch module processes different parts of the feature map evenly, the value of S should be 2. n ,Experimental verification shows that setting S to 4 ensures the consistency and symmetry of the feature map size after division, making the network more balanced and coordinated in processing on different branches, and avoiding information loss or processing deviation caused by uneven feature division.

[0103] Preferably, the input of the BiFormer block first passes through a 3×3 depthwise convolution (DWConv3×3), the output of which is connected to the original input for the first jump, and the result of the addition is sent to the layer normalization and the bi-level routing attention mechanism (Bi-level Routing Attention) in turn. The processed output is connected to the input of the previous layer normalization for the second jump, and then the layer normalization and multi-layer perceptron (MLP) processing are performed again. The output of the MLP is connected to the input of the second layer normalization for the third jump, and the result of the addition is used as the output of the module. The BiFormer block structure is shown in the figure below. Figure 8 shown.

[0104] The preferred dual-layer routing attention mechanism is the core of the BiFormer block. By dynamically focusing on the relationships between different regions, it selects the key-value pairs most relevant to the current query for attention calculation, significantly reducing the amount of computation while enhancing the feature representation relevant to the task. When processing facial features, it can better capture the relationship between the eye region and other facial regions, highlighting features related to gaze direction.

[0105] Optimally, the processing principle of the two-layer routing attention mechanism is:

[0106] First, linear projection is performed on the feature vectors in each region to obtain query (Q), key (K) and value (V) tensors; the region-level query (Q r ) and key (K r ), calculate the inter-region affinity matrix A through formula (2-4)r , and prune the affinity matrix to obtain the routing index matrix I r .

[0107] A r =Q r (K r ) T (2-4)

[0108] According to the routing index matrix I r , collect the relevant key-value pairs from the original key (K) and value (V) tensors to get the aggregated key (K g ) and value (V g ), apply the token-to-token attention mechanism on the aggregated key-value pairs, calculate the attention score A using formula (2-5), and obtain the output feature O using formula (2-6).

[0109] A=softmax(QK gT ) (2-5)

[0110] O=AV g (2-6)

[0111] Step 4.2: Graph Attention Network captures the dynamic spatial relationship between the head and eyes:

[0112] A global average pooling process is first performed on the 128×28×28 face feature map to achieve feature dimensionality reduction and obtain a 128-dimensional feature vector F; the face feature vector F and the head posture angle vector P obtained in step 3 are combined by weighted summation. x ,θ y ,θ z ] to perform feature splicing and obtain F′, as shown in formula (2-7):

[0113] F′=w1×F+w2×P (2-7)

[0114] Wherein, w1 and w2 are feature weights, and w1+w2=1.

[0115] like Figure 9As shown in the figure, the fused feature vector F′ is cut into 9 parts to obtain 9 sub-vectors F′=[f′1,f′2,f′3,f′4,f′5,f′6,f′7,f′8,f′9]. When slicing, try to ensure that the head posture features and facial features are evenly distributed to the 9 sub-vectors, so that each sub-vector can reasonably represent part of the information of the head posture and facial features; then each sub-vector is normalized so that the eigenvalue is distributed in a specific interval [0,1]; finally, the normalized 9 sub-vectors are input into the nodes in the graph attention network. After processing, the 9 enhanced node feature vectors output by GAT are aggregated and then fused with F′ into a new comprehensive feature vector H′=[h′1,h′2,h′3,h′4,h′5,h′6,h′7,h′8,h′9], which is the final output of the graph attention network.

[0116] Preferably, the graph attention network processes the input node features, and its core mechanism is to determine the importance weights between nodes by calculating the attention mechanism. When calculating the attention coefficient, the shared weight matrix W is first used to calculate the node feature f i ′(i=1,2,…,9) is linearly transformed to obtain Wf i ′, and then use the attention calculation formula (2-8) to calculate the attention coefficient,

[0117]

[0118] in, is the parameter vector of the attention function, LeakyReLU is the activation function (introducing the negative input slope α = 0.2 to enable the network to learn negative sample information), || represents the splicing operation, is the set of neighbor nodes of node i, and the mechanism of graph attention network is as follows Figure 10 shown.

[0119] Through the attention mechanism, GAT can aggregate the information of neighboring nodes in a targeted manner; for each node i, its output feature is the weighted sum of neighbor node features, as shown in Equation (2-9), where σ is the activation function.

[0120]

[0121] Step 4.3: Feature splicing of the three-branch output

[0122] The outputs of the three feature processing branches are: the enhanced feature vector H′ output by the graph attention network, the face feature F′ output by the cross-layer dynamic convolution module face , eye feature F output by MolileNet v2 network eye; Connect these three eigenvectors in sequence to form a new comprehensive eigenvector F combined =[H′,F′ face ,F eye ].

[0123] Step 5. Build the CAS-ViT gaze estimation network to output the gaze vector: Construct a Convolutional Additive Self-attention Vision Transformers (CAS-ViT) network, use the Convolutional Additive Label Mixer to effectively process features in the spatial and channel dimensions, and finally output the estimated gaze vector.

[0124] Preferably, the CAS-ViT network architecture mainly includes input, Stem block, 4-stage encoding layers and output, such as Figure 11 The input is an image of size C×H×W (C is the number of channels, H is the height, and W is the width). It first passes through the Stem module, which contains two consecutive convolutional layers with a stride of 2 to downsample the image to Then, the image passes through the encoding layer of four stages in sequence, and Patch Embedding is used to perform 2x downsampling between each two stages to obtain feature maps of different sizes, such as and Among them C i (i∈{1,2,3,4}) represents the number of channels of the feature map.

[0125] First, F combined After a series of processing, it is expanded into a feature map F′ of size C×H×W combined , input into the built CAS-ViT network for line of sight estimation;

[0126] F′ combined First, we enter the Stem block, which contains a convolutional layer with a kernel size of 3 and a stride of 2, as well as a batch normalization layer and a ReLU activation function. After processing by the Stem block, the resolution of the input features is reduced and the number of channels is increased.

[0127] The features processed by the Stem block enter the middle layer, which consists of four stages. Each stage consists of an integration layer, a Convolutional Additive Token Mixer (CATM), and a Multilayer Perceptron (MLP):

[0128] The integration layer contains a series of convolution and activation operations. The 1×1 convolution layer is used to adjust the number of channels to enhance the local feature representation capability.

[0129] The integrated features enter the Convolutional Additive Tagging Mixer (CATM) module, which projects the input features into three different spaces through a 1×1 convolutional layer to obtain Query (Q), Key (K), and Value (V). The CATM mechanism innovatively defines the similarity function as the sum of the context scores of Q and K:

[0130]

[0131] in, Represents the context mapping function. By concretizing it into Sigmoid-based channel attention C(·) and spatial attention S(·), the model can obtain global context information in multiple information interaction dimensions while avoiding the complex matrix multiplication and Softmax operation in traditional ViT. The final output is:

[0132]

[0133] Where Γ(·) is a linear transformation used to integrate context information, and the Convolutional Additive Label Mixer (CATM) module architecture is as follows: Figure 12 shown.

[0134] The features processed by the Convolutional Additive Label Mixer (CATM) enter the Multilayer Perceptron (MLP), which consists of multiple fully connected layers interspersed with activation functions and Dropout layers; it performs further nonlinear transformation on the input features to enhance the network's expressive power.

[0135] After four stages of processing in the intermediate layer, the features enter the output layer. The output layer predicts the classification probabilities of the horizontal and vertical positions of the gaze point through a fully connected layer and two softmax layers, respectively, and maps the final features to the representation space of the gaze vector.

[0136] Preferably, during the training process, the mean square error loss function is used to calculate the predicted sight vector V gaze With the real line of sight vector The difference between them is shown in formula (2-12), where n is the number of samples:

[0137]

[0138] After multiple training and parameter adjustment, the pitch angle and yaw angle are finally output. The network block diagram of the present invention is as follows: Figure 13 shown.

[0139] The technical solution provided by the present invention is further described in detail through the above specific embodiments in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the above specific embodiments are not intended to limit the present invention. Any reasonable modification and improvement of the present invention, combination of embodiments and equivalent replacement, etc. based on the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0140] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0141] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A multi-technology collaborative line of sight estimation method, characterized in that: The method comprises: S1, step of collecting images; S2, using a multi-task convolutional neural network to detect the image to obtain a face frame and face feature points, cropping the face frame to obtain a face image, using a Dlib algorithm to obtain face key point coordinates based on the face feature points, the face key point coordinates include eye key point coordinates, cropping the eye key point coordinates to obtain an eye image, and normalizing the face image and the eye image; S3. Use MobileNet v2 network to extract the facial feature map F from the normalized face image and eye image face and the eye feature vector F eye , using a three-dimensional deformable face model to obtain three-dimensional coordinates corresponding to the coordinates of key points of the face, and using a PnP algorithm to obtain head posture features based on the three-dimensional coordinates; S4. Enhance the facial feature map F using cross-layer dynamic convolution modules face Get the face feature enhancement image F′ faec , using the graph attention network based on the face feature map F face and head posture features to obtain enhanced feature vector H′, concatenate enhanced feature vector H′ and face feature enhanced map F′ face and the eye feature vector F eye , obtain the comprehensive feature vector F combined Steps; S5, based on the comprehensive feature vector F combined , the steps of building a convolutional additive self-attention network to obtain the estimated gaze vector.

2. The sight line estimation method according to claim 1, wherein: In step S3, the MobileNet v2 network is used to extract the facial feature map F from the normalized face image and eye image. face and the eye feature vector F eye The method is: use convolution layer and depth separable convolution to extract multi-level features layer by layer, and obtain a 128×28×28px face feature map F face and the 64-dimensional eye feature vector F eye .

3. The sight line estimation method according to claim 1, wherein: The head posture feature in step S3 includes the yaw angle vector θ of the head posture x , pitch angle vector θ y and the roll angle vector θ z .

4. The sight line estimation method according to claim 1, wherein: In step S4, the cross-layer dynamic convolution module is used to process the face feature map F face Get the face feature enhancement image F′ face The method is: the face feature map F face Perform multi-channel division, combine BiFormer block and residual connection processing to obtain new features, and then pass through a 1×1 convolution kernel to combine with the face feature map F face Fusion, obtain the face feature enhancement map F′ face .

5. The sight line estimation method according to claim 1, wherein: The method for obtaining the enhanced feature vector H′ in step S4 is: by global average pooling the face feature map F face The dimension is reduced to a 128-dimensional vector F, which is weightedly concatenated with the head posture feature to obtain the fused feature vector F′. The feature vector F′ is split into 9 sub-vectors and normalized before being input into the graph attention network as the node to obtain 9 enhanced node feature vectors. The node feature vectors are aggregated and fused again with the feature vector F′ to obtain the enhanced feature vector H′.

6. The sight line estimation method according to claim 1, wherein: In step S5, the method of obtaining the estimated sight vector by constructing the convolutional additive self-attention network is as follows: the comprehensive feature vector F combined Expanded to feature map F′ cobmbined The input is input to the CAS-ViT network, and after being processed by the Stem block, it enters the intermediate layer. The intermediate layer includes 4 stages, each of which consists of an integration layer, a CATM module and an MLP. After being processed by the 4 stages of the intermediate layer, it enters the output layer. The output layer predicts the position probability of the gaze point through a fully connected layer and 2 Softmax layers to obtain the final feature, and the final feature is mapped to the representation space of the gaze vector to obtain the estimated gaze vector.

7. A multi-technology collaborative sight line estimation device, characterized in that: The device comprises: Module for acquiring images; A module for detecting the image using a multi-task convolutional neural network to obtain a face frame and face feature points, cropping the face frame to obtain a face image, obtaining face key point coordinates based on the face feature points using a Dlib algorithm, wherein the face key point coordinates include eye key point coordinates, cropping the eye key point coordinates to obtain an eye image, and normalizing the face image and the eye image; Used to extract normalized face and eye images using the MobileNet v2 network to obtain the face feature map F face and the eye feature vector F eye , a module that uses a three-dimensional deformable face model to obtain the three-dimensional coordinates corresponding to the coordinates of the facial key points, and uses the PnP algorithm to obtain the head posture features based on the three-dimensional coordinates; Used to enhance the face feature map F using cross-layer dynamic convolution modules face Obtain facial feature enhancement image F′ face , using the graph attention network based on the face feature map F face And head posture features, obtain enhanced feature vector H′, splice enhanced feature vector H′, face feature enhanced map F′ face and the eye feature vector F eye , obtain the comprehensive feature vector F combined Modules; For the comprehensive feature vector F combined , a module for estimating the gaze vector is obtained by constructing a convolutional additive self-attention network.

8. A computer storage medium for storing a computer program, characterized in that When the computer program is read by a computer, the computer executes the method according to any one of claims 1 to 6.

9. A computer comprising a processor and a storage medium, characterized in that When the processor reads the computer program stored in the storage medium, the computer executes the method according to any one of claims 1 to 6.

10. A computer program product, being a computer program, characterized in that When the computer program is read, the method according to any one of claims 1 to 6 is implemented.

Images