High-precision gaze estimation method based on multimodal and Transformer attention mechanism

Abstract

The present invention discloses a high-precision line of sight estimation method based on multimodal and Transformer attention mechanism, including an eye image feature extraction part, a facial image feature extraction part, and a feature fusion part; for eye images, convolution processing is performed through an expanded convolution layer, and eye features are obtained through a fully connected layer. For facial images, a pre-trained VGG16 network is used as the basis, and the spatial attention mechanism is combined to further enhance the attention to the eyes, and finally the facial features are obtained through a fully connected layer. In the feature fusion part, the Transformer structure used independently enhances the internal representation of multimodal features. Finally, by fusing the attention mechanism, each feature is dynamically adjusted according to the importance of the multimodal features in line of sight estimation, so as to achieve more accurate line of sight estimation.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a high-precision line of sight estimation method based on multimodality and Transformer attention mechanism. Background Art

[0002] Gaze is a key indicator of human attention and is essential for human intention prediction. It plays an important role in multiple fields, such as fatigue driving detection and saliency detection, as well as in human-computer interaction, especially virtual reality. In the past, gaze estimation relied on traditional driven methods, which required complex equipment support, such as high-resolution cameras and infrared cameras. However, recent appearance-driven methods have made significant progress by using convolutional neural network (CNN)-based techniques to estimate gaze position directly from appearance information. These CNN methods are popular in gaze estimation, especially those that combine face and eye image inputs.

[0003] Early gaze estimation methods used monocular images as input and used convolutional neural network training to output the two-dimensional coordinate points of the gaze. Subsequently, binocular gaze estimation methods were proposed. This is because the monocular method failed to fully utilize the complementary information of the two eyes, and the binocular gaze estimation method successfully made up for this shortcoming. With the development of technology, full-face gaze estimation methods came into being. This method only needs to input a face image to directly output the final gaze estimation result. However, the existing gaze estimation methods have large errors. On the one hand, the existing technology mainly relies on normalized face images as the only input data source; on the other hand, the existing technology ignores the long-range dependency between eyes and facial images and the precise focus of key facial areas. Summary of the invention

[0004] The purpose of the present invention is to provide a high-precision line of sight estimation method based on multimodal and Transformer attention mechanism, which aims to improve the accuracy of line of sight estimation by introducing multimodal data fusion, Transformer encoder and spatial attention mechanism, as well as modular design strategy.

[0005] To achieve the above functions, the present invention designs a high-precision line of sight estimation method based on multimodality and Transformer attention mechanism. For the target face image, the following steps S1 to S8 are performed to complete the line of sight estimation of the target face:

[0006] Step S1: Use convolution operation to extract the features of the left eye image of the target face and obtain the left eye image feature F l1 ;

[0007] Step S2: The left eye image feature F l1Input dilated convolutional layer to capture the left eye image features F l1 The context information in the image is used to obtain the left eye image feature F after the dilated convolution. l2 ;

[0008] Step S3: Use the flattening operation to dilate the left eye image feature F after convolution l2 Convert it into a one-dimensional vector, perform linear transformation on the flattened one-dimensional vector through the fully connected layer, and finally obtain the fully connected representation F of the left eye image features l_fc ;

[0009] Step S4: Using the method from step S1 to step S3, the right eye image of the target face is processed to obtain the fully connected representation F of the right eye image features. r_fc ;

[0010] Step S5: For the facial image of the target face, the convolution module in the VGG-16 network structure is used to extract facial image features; then, a spatial attention module is used to enhance the attention to the eye image features; then, the facial image features are transformed based on the custom convolution layer, and finally a fully connected layer is used to obtain the fully connected representation F of the facial image features. f_fc ;

[0011] Step S6: The fully connected representation F of the left eye image features l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc The Transformer encoder uses a self-attention mechanism to enhance the internal representation of each feature and obtain the encoded left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en ;

[0012] Step S7: The left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en Input the fusion attention mechanism, dynamically adjust each feature, and finally get the fused feature representation F fusion ;

[0013] Step S8: The fused feature representation F fusion The fully connected layer is input to generate the prediction result of the gaze point of the target face. The MSELoss loss function is used to measure the error between the prediction result and the true value of the gaze point. The error is back-propagated to update the network parameters to complete the line of sight estimation of the target face.

[0014] Beneficial effects: Compared with the prior art, the advantages of the present invention include:

[0015] 1. The present invention introduces a spatial attention mechanism, which aims to enhance the focus on eye features in facial images to more effectively capture subtle changes in the eyes. This technology enables the present invention to generate facial features that are more focused on eye changes, thereby significantly improving the accuracy of gaze estimation.

[0016] 2. The present invention uses a Transformer encoder to process eye and facial features more efficiently to independently enhance the internal representation of each feature. This allows for more effective use of multimodal information when fusion of features, capturing the complex relationships between them. This innovation aims to make full use of the combined information of the face and eyes, thereby improving the accuracy of the estimation.

[0017] 3. The present invention proposes an effective solution to the problem of large errors caused by subtle changes in eye features in existing gaze estimation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a left eye image subtle change diagram provided according to an embodiment of the present invention;

[0019] Figure 2 is a general structural diagram of line of sight estimation provided according to an embodiment of the present invention;

[0020] Figure 3 is a network structure framework diagram of an existing sight line estimation method provided according to an embodiment of the present invention;

[0021] Figure 4 It is a network structure diagram of a high-precision sight line estimation method based on multimodality and Transformer attention mechanism provided according to an embodiment of the present invention;

[0022] Figure 5 is a flow chart comparing the existing sight line estimation method and the method designed by the present invention;

[0023] Figure 6 is a structural diagram of a spatial attention module provided according to an embodiment of the present invention;

[0024] Figure 7 is a structural diagram of a custom convolutional layer module provided according to an embodiment of the present invention;

[0025] Figure 8 is a structural diagram of an attention module fusing multimodal features provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0027] The appearance of people’s eyes changes subtly when they gaze at different locations, and these changes are important. Figure 1 are two left eye images taken from the MPIIFaceGaze gaze dataset, where Figure 1 In (a), the horizontal gaze angle of the left eye is 10°, and the vertical gaze angle is 0°. Figure 1 In (b), the horizontal gaze angle of the left eye is 20° and the vertical gaze angle is 0°. Although numerically, a 10° horizontal difference only results in a difference of a few pixels, these subtle appearance changes, such as the openness and shape of the eyes, actually contain a lot of information about the gaze direction.

[0028] In most convolutional neural network (CNN) research methods, eye images or face images containing eye information are usually selected as input data. The general structure diagram of gaze estimation is as follows: Figure 2 In this method, these input images are first processed by a convolutional neural network and then passed through a fully connected layer to finally obtain the two-dimensional coordinate points of the line of sight.

[0029] The network structure framework of an existing line of sight estimation method based on convolutional neural network is as follows: Figure 3 As shown, the flow chart is as follows Figure 5 As shown in (a), the line of sight estimation network of this method accepts normalized face images as input and completes the line of sight estimation task with the help of a convolutional neural network fused with a parameter-free attention mechanism. However, the limitation of this method is that it only relies on normalized face images as the input data source; in contrast, the present invention adopts a multimodal feature fusion method that can fuse information from multiple data sources, thereby providing a richer feature representation for the line of sight estimation task and greatly improving the accuracy.

[0030] The high-precision line of sight estimation method based on multimodality and Transformer attention mechanism provided by the embodiment of the present invention has a network structure as follows: Figure 4 As shown in the figure, a method named EFF-Net is proposed, which mainly consists of two parts: feature extraction part and feature fusion part. The feature extraction part includes eye feature extraction part and facial feature extraction part. For the eye feature extraction part, this part first performs feature extraction through the Conv convolution layer, then performs convolution processing through the DeConv expansion convolution layer, and finally obtains the eye feature description F through the fully connected layer Fc. l_fc For facial feature extraction, the present invention uses the pre-trained VGG16 network as the basis and combines the spatial attention mechanism. Figure 4The SA Module in the present invention further refines the focus on eye features, aiming to enhance the accuracy of eye feature description. Next, the present invention uses a series of custom convolutional layers, namely Figure 4 The SDD-Module module in the image is used to perform feature conversion. Finally, the present invention obtains the feature representation of the face through a fully connected layer. The core design of the feature fusion module adopts the Transformer structure, and its original intention is to independently enhance the internal representation ability of multimodal features. By fusing the attention mechanism, the method of the present invention can dynamically adjust these features according to the importance of multimodal features in line of sight estimation, thereby achieving more accurate line of sight estimation; the process comparison diagram of the existing line of sight estimation method and the method designed by the present invention is shown in Figure 5 ,in Figure 5 (a) is the flow chart of the existing line of sight estimation method. Figure 5 (b) A flow chart of the method designed for the present invention;

[0031] For the target face image, refer to Figure 5 (b) Execute the following steps S1 to S8 to complete the line of sight estimation of the target face:

[0032] Step S1: Use convolution operation to extract the features of the left eye image of the target face and obtain the left eye image feature F l ;

[0033] The specific method of step S1 is as follows:

[0034] For the left eye image of the target face, a 5×5 convolution kernel is used to capture the primary spatial structure information. The specific formula of the process is as follows:

[0035] F1=Conv(I,K 5×5 ,p=2)

[0036] Where F1 is the primary spatial structure information, I is the left eye image of the input target face, and K 5×5 represents a 5×5 convolution kernel, and p=2 represents an edge padding of 2; ensuring that the feature map size after convolution is consistent to ensure the spatial continuity of the feature map. Subsequently, batch normalization and ReLU Activation function is used to enhance the stability of feature extraction method and the nonlinear representation ability of features.

[0037] For the primary spatial structure information F1, in order to further reduce the spatial dimension of the feature and strengthen the expression of its local characteristics, a 2×2 maximum pooling layer is used to obtain the feature representation F2. For the feature representation F2, in order to deeply explore the more detailed and complex features of the image, two 3×3 convolution operations are applied continuously. The formula is expressed as follows:

[0038] F l1 =Conv(F2,K 3×3 ,p=1)

[0039] In the formula, F l1 is the left eye image feature obtained after the convolution operation, K 3×3 represents a 3×3 convolution kernel, and p=1 indicates that the edge padding is 1. Each convolution operation is followed by batch normalization and ReLU activation to ensure the quality of the features.

[0040] Step S2: The left eye image feature F l1 Input dilated convolutional layer to capture the left eye image features F l1 The context information in the image is used to obtain the left eye image feature F after the dilated convolution. l2 ;

[0041] The specific method of step S2 is as follows:

[0042] The left eye image feature F l1 The input dilated convolutional layer performs the convolution operation. First, a 1×1 convolutional layer is used to adjust the number of channels. l1 The number of channels is reduced from 128 to 64 to reduce computational complexity while retaining the main feature information. In order to capture spatial information of different scales, the F l1 Three 3×3 convolutional layers are input continuously, and the expansion rates of each layer are (2,2), (3,3), (4,5), respectively, so that the network can capture a wider range of contextual information without increasing the computational complexity; the i-th layer expansion convolution is as follows:

[0043] F l_i =ReLU(BN(W l_i * d F l_(i-1) +b l_i ))

[0044] In the formula, ReLU is the rectified linear activation function, BN represents the batch normalization operation, * d represents the dilated convolution operation, W l_i is the weight of the i-th layer dilated convolution, F l_i is the left eye image feature of the i-th layer, b l_i is the bias of the i-th layer dilated convolution;

[0045] In order to reduce the spatial dimension while maintaining feature richness, in F l_i A 2×2 maximum pooling layer is added at the end, and its formula is as follows:

[0046] F l2 =MaxPool(F l_3 )

[0047] Among them, F l2 is the left eye image feature after dilated convolution, F l_3 It is the left eye image feature after the third layer of dilated convolution processing.

[0048] Step S3: Use the flattening operation to dilate the left eye image feature F after convolution l2 Convert it into a one-dimensional vector, perform linear transformation on the flattened one-dimensional vector through the fully connected layer, and finally obtain the fully connected representation F of the left eye image features l_fc ;

[0049] The left eye image feature F after dilation convolution l2 Perform a flattening operation to convert it into a one-dimensional vector, and send the one-dimensional vector to a fully connected layer for linear transformation. In order to increase generalization ability and avoid overfitting, batch normalization operation and ReLU activation function are introduced at the same time; a Dropout layer with a dropout rate of 0.5 is used to further improve generalization, and finally the fully connected representation F of the left eye image features is output. l_fc .

[0050] Step S4: Using the method from step S1 to step S3, the right eye image of the target face is processed to obtain the fully connected representation F of the right eye image features. r_fc ;

[0051] Step S5: For the facial image of the target face, the convolution module in the VGG-16 network structure is used to extract facial image features; then, a spatial attention module is used to enhance the attention to the eye image features; then, the facial image features are transformed based on the custom convolution layer, and finally a fully connected layer is used to obtain the fully connected representation F of the facial image features. f_fc ;

[0052] The specific steps of step S5 are as follows:

[0053] Step S5.1: For the facial image of the target face, the convolution module in the VGG-16 network structure is used to extract facial image features. After each convolution layer, batch normalization and ReLU activation function are added. After the first and second convolution layers, a 2×2 maximum pooling layer is added to reduce the spatial dimension of the feature. At the same time, the number of output channels is 64, 128 and 256 respectively, and the final output feature F f_pre ;

[0054] Step S5.2: In order to further accurately focus on the area in the facial image that is critical for line of sight estimation and enhance the focus on eye features, the feature F output from step S5.1 is converted to f_pre Input spatial attention module and calculate feature F f_preThe average response and maximum response at each spatial location are given by:

[0055]

[0056] Among them, AVG(F f_pre ) represents the feature F f_pre The average response, MAX(F f_pre ) represents the feature F f_pre The maximum response of , C represents the number of feature channels, represents the response of the Cth channel;

[0057] The average response and the maximum response are concatenated along the channel direction to obtain a 2-channel feature map. In order to further refine the spatial information and calculate the final attention weight, a 3×3 convolution layer is used to process the feature map. The purpose is to capture the local information and relative relationship between the two responses and obtain a spatially weighted attention map. The formula is as follows:

[0058] F attention =σ(Conv([AVG,MAX]))

[0059] Among them, σ represents the Sigmoid activation function, Conv represents the 3×3 convolution layer operation, and F attention Represents the weighted attention map in space; the facial image feature F is obtained by enhancing the attention to the eye image features through the following formula l3 :

[0060] F l3 =F attention ⊙F f_pre

[0061] Among them, ⊙ represents element-by-element multiplication; the structure of the spatial attention module is as follows Figure 6 shown.

[0062] Step S5.3: Perform facial image feature conversion based on the custom convolution layer. The custom convolution layer first uses a 1x1 convolution kernel to reduce the dimension of 256 feature maps to 64, and performs a 2×2 maximum pooling process to reduce the spatial size of each feature map; two 3×3 convolution operations are performed continuously. During the two convolution operations, the output of 64 feature maps is maintained, and the size of the feature map is kept unchanged through appropriate padding; after each convolution operation, batch normalization and ReLU activation function are used to enhance the expression ability of the network and increase nonlinearity; finally, a 2×2 maximum pooling process is performed to further reduce the spatial size of each feature map; the module structure of the custom convolution layer is as follows: Figure 7 shown.

[0063] Step S5.4: Through the flattening operation, each feature map is converted into a one-dimensional vector, and then the one-dimensional vector is sent to a fully connected layer for linear transformation to obtain the fully connected representation F of the facial image features. f_fc .

[0064] Step S6: The fully connected representation F of the left eye image features l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc The Transformer encoder uses a self-attention mechanism to enhance the internal representation of each feature and obtain the encoded left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en ;

[0065] The specific steps of step S6 are as follows:

[0066] Step S6.1: Considering that the Transformer cannot capture the inherent order of the sequence by itself, the fully connected representation F of the left eye image features is transformed into l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc Input the Transformer encoder separately and add position encoding to each feature respectively. The formula is as follows:

[0067]

[0068] in, represents the feature after position encoding, PE is the position encoding function, and pos is the position; after this operation, the fully connected representation F of the left eye image feature l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc It not only contains the original feature information, but also adds location information;

[0069] Step S6.2: Use the self-attention mechanism to enhance the internal representation of multimodal features. The self-attention mechanism formula is defined as:

[0070]

[0071] For each feature after adding position encoding Perform self-attention operation and set and input feature dimension d k , where Q is the query, K is the key, and V is the value; the weight is calculated by the dot product of the query and all keys, then divided by Scale it, apply the Softmax function to get the weight, and multiply it with the value V to get the output. After the attention mechanism is processed, the left eye image feature F is obtained. l_en , right eye image feature F r_en and facial image features F f_en ;

[0072] Step S6.3: For each feature processed by the attention mechanism, a feedforward neural network is used to further transform the output of the self-attention module. The network is applied independently to each position. It consists of two linear transformations connected by a ReLU activation function, as shown in the following formula:

[0073] FFN(F i )=W2·ReLU(W1·F i +b1)+b2

[0074] where F i is the output of the self-attention module, W1 and W2 are the weight matrices of the two linear layers, and b1 and b2 are the bias terms of the two linear layers respectively;

[0075] Step S6.4: In order to enhance the training stability of the method and accelerate convergence, residual connection and layer normalization are added after the output of each sub-layer. The residual connection is as follows:

[0076] Residual(F i )=FFN(F i )+F i

[0077] Among them, FFN(F i ) is the output of the feedforward network, Residual(F i ) represents the output of the residual connection; the normalization is as follows to ensure that the network still has stable performance in deep layers:

[0078] Norm(F i )=LayerNorm(Residual(F i ))

[0079] Among them, LayerNorm represents the layer normalization operation, Norm(F i ) represents the normalized output.

[0080] Step S7: The left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en Input the fusion attention mechanism, dynamically adjust each feature, and finally get the fused feature representation F fusion; The structure of fusing multimodal features through the attention module is as follows Figure 8 As shown;

[0081] The specific steps of step S7 are as follows:

[0082] Step S7.1: Transform the left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en Concatenate to form a unified feature representation F contact , the formula is as follows:

[0083] F contact =(F l_en ,F r_en ,F f_en )

[0084] The concatenated features are represented as F contact It is transformed into three representations through three linear transformation layers: Query (Q), Key (K) and Value (V);

[0085] Step S7.2: Calculate the dot product of Query (Q) and Key (K) to form an attention score matrix, normalize it using the Softmax function, and get the weight of each feature for other features. Use the calculated weights to perform a weighted combination of the Value (V) representation to get the fused feature F. fusion_pre , which is defined as follows:

[0086] α=Softmax(QK T )

[0087] F fusion_pre =αV

[0088] Where Q is the query, K is the key, and α is the weight;

[0089] Step S7.3: fusion feature F fusion_pre Apply dropout, then perform adaptive average pooling on the features, perform batch normalization on the pooled features, and finally obtain the feature representation F after multimodal fusion fusion .

[0090] Step S8: The fused feature representation F fusion The fully connected layer is input to generate the prediction result of the gaze point of the target face. The MSELoss loss function is used to measure the error between the prediction result and the true value of the gaze point. The error is back-propagated to update the network parameters to complete the line of sight estimation of the target face.

[0091] The MSELoss loss function is as follows:

[0092]

[0093] Among them, L mse represents the mean square error loss, P i and Y i They represent the predicted value and true value of the i-th sample respectively, and n represents the total number of samples.

[0094] The following is an embodiment of the high-precision line of sight estimation method based on multimodality and Transformer attention mechanism designed by the present invention:

[0095] Embodiment 1:

[0096] First, the classic Gaze360 dataset and MPIIFaceGaze dataset need to be rectified, aiming to eliminate the interference of environmental factors and other factors through data preprocessing methods, thereby simplifying the gaze regression problem. The goal of this step is to improve the quality of the dataset so that subtle changes in eye features can be better extracted in subsequent experiments.

[0097] The Gaze360 dataset is a video data set collected from 238 subjects in the real world. The dataset is large in scale and combines 3D gaze annotations, a wide range of gaze and head poses, a variety of indoor and outdoor capture environments, and a diversity of subjects. This paper uses 84,902 images from the train group of the dataset as the test set and 11,318 images from the val group of the dataset as the test set.

[0098] The MPIIFaceGaze dataset contains a total of 45,000 images of 15 subjects. The present invention uses 3,000 images of subject P14 as the test set and the remaining 42,000 images as the training set.

[0099] Taking the processing of the MPIIFaceGaze dataset as an example, the specific steps are as follows:

[0100] Each image is processed as follows:

[0101] S1: Data path configuration:

[0102] The present invention defines the root path root of data, the root path sample_root of samples and the root path out_root of output to ensure that the present invention can read data from the correct location and save the result in the expected location. The present invention processes according to the data of each person. The sample list of each person is read from sample_root, and then for each person, the present invention reads each sample from the list and processes it.

[0103] S2: Processing of Personal Data:

[0104] S201: Read annotation information: extract information about gaze point, eye corner, head rotation and transfer vector, etc. from the annotation file.

[0105] S202: Read screen information: extract the width, height and ratio of the screen from the screen size file.

[0106] S203: Image preprocessing: Using the provided annotation information, the present invention determines the bounding boxes of the face, left eye and right eye.

[0107] Based on these bounding boxes, the present invention crops and resizes the image to the appropriate size. The present invention also creates a grid image that is 1 inside the face area and 0 outside the face area for further analysis.

[0108] S3: Save the processed image:

[0109] All processed images are saved in a predetermined directory. The image size of the dataset after preprocessing is 224×224.

[0110] S4: Extract and save information:

[0111] For each sample, the present invention extracts a series of information, such as the face, the coordinates of the left and right eyes, the rotation and translation vectors of the head, etc., and writes this information into an output file.

[0112] Then, after processing the dataset, the entire EFF-Net network is trained using the preprocessed MPIIFaceGaze and Gaze360 datasets. For the MPIIFaceGaze and Gaze360 datasets, the same training parameters are configured, bathsize is set to 64, epoch is set to 100, learning rate is set to 0.001, decay is set to 0.1, and decay step is set to 8000.

[0113] Finally, the trained EFF-Net is used for verification on the test set.

[0114] Evaluation indicators: At present, the mainstream evaluation indicators of line of sight estimation are mostly angle errors, that is, the deviation angle between the predicted value and the true value of the line of sight estimation. The smaller the indicator, the better the effect.

[0115] The comparison method uses advanced methods of line of sight estimation, such as Mnist, Gaze360, FullFace, iTracker, and Dilated-Net. In order to ensure the fairness of the experiment, the present invention selects several existing advanced methods as comparison, including the Mnist method based only on eye input, the Gaze360 and FullFace methods based on facial input, and the iTracker and Dilated methods that combine facial and eye input.

[0116] For the MPIIFaceGaze dataset, the bath size is set to 256, the epoch is set to 100, the learning rate is set to 0.001, the decay is set to 0.1, and the decay step is set to 5000.

[0117] Gaze360 method For the MPIIFaceGaze dataset, the bath size is set to 20, the epoch is set to 60, the learning rate is set to 0.0001, the decay is set to 1, and the decay step is set to 5000; for the Gaze360 dataset, the bath size is set to 60, the epoch is set to 100, the learning rate is set to 0.0001, the decay is set to 1, and the decay step is set to 5000.

[0118] For the MPIIFaceGaze and Gaze360 datasets, the FullFace method configures the same training parameters, with bathsize set to 64, epoch set to 20, learning rate set to 0.00001, decay set to 0.1, and decaystep set to 5000.

[0119] For the MPIIFaceGaze dataset, the bath size, epoch, learning rate, decay, and decaystep of the iTracker method are set to 64, 20, 0.00001, 0.1, and 5000.

[0120] For the MPIIFaceGaze and Gaze360 datasets, the Dilated method configures the same training parameters, with bath size set to 64, epoch set to 100, learning rate set to 0.001, decay set to 0.1, and decay step set to 8000.

[0121] The experimental results are shown in Table 1:

[0122] Table 1 Experimental results of the network proposed by this invention and other advanced networks

[0123]

[0124] It can be seen from the experimental data in Table 1 that, after experimental verification, the method of the present invention can effectively solve the problem of reduced accuracy of line of sight estimation caused by subtle changes in eye features, and has strong practical value.

[0125] Embodiment 2:

[0126] This embodiment will introduce an applicable scenario of the present invention:

[0127] Gaze estimation plays a vital role in advanced driver assistance systems, especially in detecting driver distraction and fatigue driving. The EFF-Net method of the present invention provides automobile manufacturers with a highly accurate and real-time gaze estimation strategy, thereby significantly improving vehicle safety.

[0128] step:

[0129] 1. Image capture: When the driver enters the vehicle and starts it, the vehicle camera begins to capture the driver's eye and facial images in real time.

[0130] 2. Sight feature extraction and encoding: Through the EFF-Net method, the driver's eye and facial images are quickly and accurately extracted and encoded.

[0131] 3. Line of sight estimation and judgment: The system estimates the driver’s line of sight in real time based on the extracted and encoded features to determine whether he is paying attention to the road.

[0132] 4. Safety intervention: If the system determines that the driver is distracted or shows signs of fatigue, it will automatically issue a warning, such as an audio prompt or seat vibration, to alert the driver.

[0133] By applying the EFF-Net method in advanced driver assistance systems, vehicles can promptly capture and respond to driver distraction or fatigue, thereby significantly improving driving safety.

[0134] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A high-precision line of sight estimation method based on multimodality and Transformer attention mechanism, characterized by: For the target face image, perform the following steps S1 to S8 to complete the line of sight estimation of the target face: Step S1: Use convolution operation to extract the features of the left eye image of the target face and obtain the left eye image feature F l1 ; Step S2: The left eye image feature F l1 Input dilated convolutional layer to capture the left eye image features F l1 The context information in the image is used to obtain the left eye image feature F after the dilated convolution. l2 ; Step S3: Use the flattening operation to dilate the left eye image feature F after convolution l2 Convert it into a one-dimensional vector, perform linear transformation on the flattened one-dimensional vector through the fully connected layer, and finally obtain the fully connected representation F of the left eye image features l_fc ; Step S4: Using the method from step S1 to step S3, the right eye image of the target face is processed to obtain the fully connected representation F of the right eye image features. r_fc ; Step S5: For the facial image of the target face, the convolution module in the VGG-16 network structure is used to extract facial image features; then, a spatial attention module is used to enhance the attention to the eye image features; then, the facial image features are transformed based on the custom convolution layer, and finally a fully connected layer is used to obtain the fully connected representation F of the facial image features. f_fc ; Step S6: The fully connected representation F of the left eye image features l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc The Transformer encoder uses a self-attention mechanism to enhance the internal representation of each feature and obtain the encoded left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en ; Step S7: The left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en Input the fusion attention mechanism, dynamically adjust each feature, and finally get the fused feature representation F fusion ; Step S8: The fused feature representation F fusion The fully connected layer is input to generate the prediction result of the gaze point of the target face. The MSELoss loss function is used to measure the error between the prediction result and the true value of the gaze point. The error is back-propagated to update the network parameters to complete the line of sight estimation of the target face.

2. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The specific method of step S1 is as follows: For the left eye image of the target face, a 5×5 convolution kernel is used to capture the primary spatial structure information. The specific formula of the process is as follows: F1=Conv(I,K 5×5 ,p=2) Where F1 is the primary spatial structure information, I is the left eye image of the input target face, and K 5×5 represents a convolution kernel of size 5×5, and p=2 represents the edge padding of 2; For the primary spatial structure information F1, a 2×2 maximum pooling layer is used to obtain the feature representation F2. For the feature representation F2, two 3×3 convolution operations are applied continuously. The formula is expressed as: F l1 =Conv(F2,K 3×3 ,p=1) In the formula, F l1 is the left eye image feature obtained after the convolution operation, K 3×3 represents a convolution kernel of size 3×3, and p=1 means that the edge is filled with 1.

3. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The specific method of step S2 is as follows: The left eye image feature F l1 The input dilated convolution layer performs the convolution operation. First, a 1×1 convolution layer is used to adjust the number of channels. l1 The number of channels is reduced from 128 to 64, and the F l1 Three 3×3 convolutional layers are input continuously, and the expansion rates of each layer are (2,2), (3,3), and (4,5) respectively; the expansion convolution for the i-th layer is as follows: F l_i =ReLU(BN(W l_i*d F l_(i-1) +b l_i )) In the formula, ReLU is the rectified linear activation function, BN represents the batch normalization operation, * d represents the dilated convolution operation, W l_i is the dilated convolution weight of the i-th layer, F l_i is the left eye image feature of the i-th layer, b l_i is the bias of the i-th layer dilated convolution; In F l_i A 2×2 maximum pooling layer is added at the end, and its formula is as follows: F l2 =MaxPool(F l_3 ) Among them, F l2 is the left eye image feature after dilated convolution, F l_3 It is the left eye image feature after the third layer of dilated convolution processing.

4. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The left eye image feature F after dilation convolution in step S3 l2 Perform a flattening operation to convert it into a one-dimensional vector, send the one-dimensional vector to a fully connected layer for linear transformation, and introduce batch normalization and ReLU activation function; use a dropout layer with a dropout rate of 0.5 to further improve generalization, and finally output the fully connected representation F of the left eye image features. l_fc .

5. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The specific steps of step S5 are as follows: Step S5.1: For the facial image of the target face, the convolution module in the VGG-16 network structure is used to extract facial image features. After each convolution layer, batch normalization and ReLU activation function are added. After the first and second convolution layers, a 2×2 maximum pooling layer is added to reduce the spatial dimension of the feature. At the same time, the number of output channels is 64, 128 and 256 respectively, and the final output feature F f_pre ; Step S5.2: The feature F output from step S5.1 f_pre Input spatial attention module and calculate feature F f_pre The average response and maximum response at each spatial location are given by: Among them, AVG(F f_pre ) represents the feature F f_pre The average response, MAX(F f_pre ) represents the feature F f_pre The maximum response, C represents the number of feature channels, F f c _pre represents the response of the Cth channel; The average response and the maximum response are concatenated along the channel direction to obtain a 2-channel feature map, which is processed using a 3×3 convolutional layer to obtain a spatially weighted attention map; the formula is as follows: F attention =σ(Conv([AVG,MAX])) Among them, σ represents the Sigmoid activation function, Conv represents the 3×3 convolution layer operation, and F attention Represents the weighted attention map in space; the facial image feature F is obtained by enhancing the attention to the eye image features through the following formula l3 : F l3 =F attention ⊙F f_pre Among them, ⊙ represents element-by-element multiplication; Step S5.3: Perform facial image feature conversion based on a custom convolutional layer. The custom convolutional layer first uses a 1x1 convolution kernel to reduce the dimension of 256 feature maps to 64, and performs a 2×2 maximum pooling process to reduce the spatial size of each feature map; two 3×3 convolution operations are performed continuously. During the two convolution operations, the output of 64 feature maps is maintained. After each convolution operation, batch normalization and ReLU activation function are used; finally, a 2×2 maximum pooling process is performed to further reduce the spatial size of each feature map; Step S5.4: Through the flattening operation, each feature map is converted into a one-dimensional vector, and then the one-dimensional vector is sent to a fully connected layer for linear transformation to obtain the fully connected representation F of the facial image features. f_fc .

6. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The specific steps of step S6 are as follows: Step S6.1: Represent the fully connected representation F of the left eye image features l_fc , the fully connected representation of the right eye image features F r_fc and the fully connected representation F of facial image features f_fc Input the Transformer encoder separately and add position encoding to each feature respectively. The formula is as follows: in, Represents the feature after position encoding, PE is the position encoding function, and pos is the position; Step S6.2: Use the self-attention mechanism to enhance the internal representation of multimodal features. The self-attention mechanism formula is defined as: For each feature after adding position encoding Perform self-attention operation and set and input feature dimension d k , where Q is the query, K is the key, and V is the value; the weight is calculated by the dot product of the query and all keys, then divided by Scale it, apply the Softmax function to get the weight, and multiply it with the value V to get the output. After the attention mechanism is processed, the left eye image feature F is obtained. l_en , right eye image feature F r_en and facial image features F f_en ; Step S6.3: For each feature processed by the attention mechanism, a feedforward neural network is used for processing, as shown in the following formula: <h2 style=";text-align:left;direction:ltr">FFN(F<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> )=W2·ReLU(W1·F<h2 style=";text-align:left;direction:ltr"> i <h2 style=";text-align:left;direction:ltr"> +b1)+b2 where F i is the output of the self-attention module, W1 and W2 are the weight matrices of the two linear layers, and b1 and b2 are the bias terms of the two linear layers respectively; Step S6.4: Add residual connection and layer normalization after the output of each sub-layer. The residual connection is as follows: Residual(F i )=FFN(F i )+F i Among them, FFN(F i ) is the output of the feedforward network, Residual(F i ) represents the output of the residual connection; the normalization is as follows: Norm(F i )=LayerNorm(Residual(F i )) Among them, LayerNorm represents the layer normalization operation, Norm(F i ) represents the normalized output.

7. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The specific steps of step S7 are as follows: Step S7.1: The left eye image feature F l_en , right eye image feature F r_en and facial image features F f_en Concatenate to form a unified feature representation F contact , the formula is as follows: F contact =(F l_en ,F r_en ,F f_en ) The concatenated features are represented as F contact It is transformed into three representations through three linear transformation layers: Query (Q), Key (K) and Value (V); Step S7.2: Calculate the dot product of Query (Q) and Key (K) to form an attention score matrix, normalize it using the Softmax function, and get the weight of each feature for other features. Use the calculated weights to perform a weighted combination of the Value (V) representation to get the fused feature F. fusion_pre , which is defined as follows: α=Softmax(QK T ) F fusion_pre =αV Where Q is the query, K is the key, and α is the weight; Step S7.3: fusion feature F fusion_pre Apply dropout, then perform adaptive average pooling on the features, perform batch normalization on the pooled features, and finally obtain the feature representation F after multimodal fusion fusion .

8. The high-precision line of sight estimation method based on multimodal and Transformer attention mechanism according to claim 1, characterized in that: The MSELoss loss function described in step S8 is as follows: Among them, L mse represents the mean square error loss, P i and Y i They represent the predicted value and true value of the i-th sample respectively, and n represents the total number of samples.

Images