Driver eye movement prediction method based on multi-scale attention

Abstract

The application discloses a driver eye movement prediction method based on multi-scale attention, mainly solves the problem that the prior art cannot effectively fuse multi-level features and is inaccurate in predicting driver eye movement points, and the scheme is: a driver eye movement dataset is constructed, and a training set and a test set are divided; an encoder is constructed to obtain multi-level features of a driving image sequence; a space-time attention feature fusion module is constructed to obtain multi-scale space-time attention features; a driver subjective consciousness guiding module is constructed to obtain multi-scale space features guided by driver subjective consciousness; a decoder is constructed to obtain a driver eye movement fixation point image; the encoder, the space-time attention feature fusion module, the driver subjective consciousness guiding module and the decoder are cascaded to form a driver eye movement prediction model; the model is trained by using the training set; and a predicted driver eye movement image is obtained by inputting the test set into the trained model. The application can effectively predict the driver eye movement fixation point, has high accuracy and robustness, and can be used for an intelligent driving assistance system.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, and further designs a driver eye movement prediction method that can be used in intelligent driving assistance systems. Background Technology

[0002] With the continuous development of technology, intelligent driving assistance systems are gradually becoming a hot topic in the automotive industry as an advanced traffic safety technology. By analyzing the driver's eye-tracking data and summarizing the patterns of eye-tracking changes, these systems can effectively understand the driver's vision and psychology, thereby taking corresponding driving assistance measures in advance to address abnormal driving behaviors and reduce traffic accidents. However, real driver eye-tracking data is difficult to obtain, and the collection process requires significant manpower and resources. Traditional video sequence eye-tracking prediction models evolved from image visual saliency prediction, manually extracting different low-level visual features such as color, contrast, and brightness. However, this model can only analyze low-level features and lacks high-level semantic features, failing to effectively extract inter-frame motion information in video sequences. Errors exist between the generated eye-tracking prediction map and the ground truth map recorded by the eye tracker, making it unable to handle dynamically changing real-world scenarios.

[0003] With the development of artificial neural networks, numerous research results have emerged, such as SALGAN, SALICON, ACLNet, TASED-Net, and UNISAL methods. These methods all combine deep neural networks for gaze detection. Although introducing long short-term memory networks can capture motion information between frames, they lose information with long temporal distances, resulting in incomplete global information. To address these issues, researchers have recently proposed models based on the Transformer architecture. This model employs an encoder-decoder network structure, where the encoder mainly consists of multiple identical structures (multi-head attention layers, normalization layers, and multi-layer perceptron layers), and residual structures from residual neural networks are used between the encoders. This model combines positional encoding and self-attention mechanisms to expand the receptive field, thereby enabling the acquisition of dependencies between spatially distant features and obtaining global and local features of the sequence.

[0004] Patent document CN202210853133.8 discloses a method for predicting driver gaze points based on a gaze point prediction model, which constructs a visual spatiotemporal attention layer to extract global and local features of the sequence. However, this method ignores the situation where the driver's gaze point changes due to subjective intention to change driving behavior, and cannot extract the temporal features hidden in the image sequence that contain the driver's subjective intention, thus failing to obtain an accurate driver eye movement prediction map.

[0005] Patent document with application number CN202310656871.8 discloses a training and estimation method, device, system and medium for a gaze prediction model. This method simply fuses the multi-level features obtained after feature extraction, ignoring the interaction of multi-level features, which leads to the masking or loss of some information and cannot well preserve global and local features.

[0006] Patent document CN202211000132.5 discloses a training method, device, and electronic device for a gaze prediction model. It designs a gaze hypergraph learning model to construct a gaze hypergraph and obtains the predicted gaze target based on the hypergraph. However, because this method only predicts gaze points through feature extraction networks and multilayer perceptron networks, it loses information with long temporal distances, resulting in incomplete global information. Furthermore, it fails to effectively fuse features from different levels, leading to a poor combination of semantic information from low-level and high-level features, thus affecting the accuracy of the model's gaze prediction results. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the prior art by proposing a driver eye movement prediction method based on multi-scale attention, which captures inter-frame motion information and global and local features of the sequence from the original image sequence, thereby achieving accurate prediction of the driver's gaze point.

[0008] The technical approach to achieve the objective of this invention is as follows: a multi-scale feature extraction module extracts multi-scale features from a driving image sequence; a spatiotemporal attention feature fusion module fuses high-level features representing semantic information and low-level features representing edge, color, and other information with contextual information; and a driver's subjective awareness guidance module decouples temporal and spatial features, enhances eye movement information in spatial features through temporal changes, and thus more accurately predicts the driver's eye gaze point.

[0009] Based on the above technical concept, the technical solution of the present invention includes the following steps:

[0010] (1) Collect driving videos of different drivers in different scenarios, times and weather, and at the same time use an eye tracker to collect video eye movement data, and correct and process the eye movement data to obtain one-to-one corresponding natural driving data D and eye movement truth map data E, and then divide them into training set and test set in a 4:1 ratio.

[0011] (2) Read the training set natural driving data and extract the T frame image sequence. Convert each frame image into tensor data and then stitch them together into T×H×W×3 four-dimensional tensor data, where H and W are the height and width of each frame image, respectively, and 3 is the number of RGB channels.

[0012] (3) Construct a driver eye-tracking prediction model encoder and use T×H×W×3 four-dimensional tensor data as the input of the encoder to obtain multi-level features F1, F2, F3, F4 of the driving image sequence.

[0013] (4) Construct a spatiotemporal attention feature fusion module:

[0014] 4a) Establish a feature fusion network including an upsampling module and a matrix addition module;

[0015] 4b) Establish a multi-scale channel temporal attention module including three parallel attention branches;

[0016] 4c) Establish a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function;

[0017] 4d) The feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network are connected in series to obtain the spatiotemporal attention feature fusion module;

[0018] (5) Using the spatiotemporal attention feature fusion module, multi-scale spatiotemporal attention features Y1, Y2, Y3, Y4 are obtained:

[0019] 5a) The i-th level feature F obtained in step (3) i With the adjacent (i+1)th level feature F i+1 The input is fed into the spatiotemporal attention feature fusion module, F i+1 Sampled to F by the upsampling module i At the same resolution, the upsampled feature F is obtained. i ' +1 Then with F i Adding them together yields the original fusion feature X. i X i The weights w of the fused features are obtained through a multi-scale channel temporal attention module. i ;

[0020] 5b) Weighted feature fusion network based on hierarchical features F i weight w i and hierarchical features F i ' +1 weight 1-w i , will F i With F i ' +1 The fusion process yields multi-scale spatiotemporal attention features Y. i ;

[0021] 5c) Repeat steps 5a)-5b) a total of 3 times to obtain three multi-scale spatiotemporal attention features Y1, Y2, and Y3. The fourth-level feature F4 remains unchanged, and the output is the fourth spatiotemporal attention feature Y4.

[0022] (6) Construct a module to guide the driver's subjective awareness:

[0023] 6a) Establish a multi-scale temporal attention module that includes two parallel attention branches;

[0024] 6b) Construct a feature fusion network including a matrix addition module, a sigmoid activation function, and a matrix multiplication module;

[0025] 6c) The multi-scale temporal attention module and the feature fusion network are connected in series to obtain the driver's subjective awareness guidance module;

[0026] (7) Using the driver's subjective awareness guidance module, the multi-scale spatial features Z1, Z2, Z3, Z4 of driver's subjective awareness guidance are obtained:

[0027] 7a) The i-th level spatiotemporal attention feature Y obtained in step (5) i Input into the driver's subjective awareness guidance module, for Y i Decoupling to obtain temporal features F containing driver consciousness information i T and the spatial characteristics F that cause changes in driver consciousness i S ;

[0028] 7b)F i T The driver's consciousness information F at the i-th layer is obtained through a multi-scale temporal attention module. i C Feature fusion network for F i C With F i S By integrating the data, the spatial characteristics Z guided by the driver's subjective consciousness can be obtained. i ;

[0029] 7c) Repeat steps 7a)-7b) a total of 4 times to obtain the multi-scale spatial features Z1, Z2, Z3, Z4 guided by the driver's subjective consciousness;

[0030] (8) Use the driver eye movement prediction model decoder to obtain the driver eye movement prediction image P;

[0031] (9) The encoder, spatiotemporal attention feature fusion module, driver subjective consciousness guidance module and decoder are cascaded in sequence to form a driver eye movement prediction model; all driving image sequences in the training set are input into the model and trained by gradient descent method to obtain a trained driver eye movement prediction model.

[0032] (10) Input the test set into the trained driver eye movement prediction model to obtain the predicted driver eye movement image.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] First, this invention constructs a spatiotemporal attention feature fusion module to address the different distributions and representation capabilities of features at different levels. By aggregating spatial information and temporal contextual information between features at different levels through multi-scale attention, it not only preserves both global and local features but also promotes the interaction of multi-scale contextual information between features at different levels in the network.

[0035] Second, this invention addresses the impact of the driver's subjective consciousness on changes in gaze point during driving by constructing a driver subjective consciousness guidance module. By leveraging the temporal characteristics of driver consciousness changes caused by multi-scale attention enhancement, the model can improve its understanding of the driver's subjective consciousness and cognitive patterns, thereby more accurately predicting the driver's eye movement gaze point during driving. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0037] Figure 2 This is a block diagram of the driver eye-tracking prediction model based on multi-scale attention in this invention;

[0038] Figure 3 This is a block diagram of the spatiotemporal attention feature fusion module in this invention;

[0039] Figure 4 This is a block diagram of the multi-scale channel temporal attention module in this invention;

[0040] Figure 5 This is a block diagram of the driver's subjective awareness guidance module in this invention;

[0041] Figure 6 This is a block diagram of the multi-scale temporal attention module in this invention;

[0042] Figure 7 This is a subjective comparison chart of the present invention and six existing methods on the collected dataset. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions and effects of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the present invention, and not all of the embodiments. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort should all fall within the protection scope of the present invention.

[0044] It should be noted that the step numbers in the specification and claims of this invention are only for the purpose of clearly describing the embodiments of this invention and facilitating understanding, and their order is not limited.

[0045] Reference Figure 1 The implementation steps of this embodiment are as follows:

[0046] Step 1: Construct a driver eye-tracking dataset by collecting driving videos of different drivers in different scenarios, times, and weather conditions, while also using an eye tracker to collect video eye-tracking data.

[0047] 1.1) For cases where some frames are missing eye movement points, the missing eye movement point positions in the missing frames are filled by using adjacent frames:

[0048] Due to prolonged operation of the eye tracker or excessively high light intensity, some frames may be missing eye movement points. For example, eye movement points may be present in frames t-1 and t+1, but missing in frame t. For such data, it is necessary to fill in the missing eye movement point positions by using adjacent frames. The specific implementation is as follows:

[0049] Assume the eye-tracking point coordinates of frame t-1 are (x t-1 ,y t-1 The eye-tracking point coordinates of frame t+1 are (x... t+1 ,y t+1 Fill the eye-tracking point coordinates of frame t with...

[0050] The video eye-tracking data was corrected using the above method to obtain video eye-tracking data that completely corresponds to the natural driving data D;

[0051] 1.2) Convert the eye movement points acquired by the eye tracker into a fixation point ground truth map:

[0052] An eye tracker records eye movement points as single pixels, while the human eye, due to its focusing characteristic, observes an area with a clear center and blurred surroundings. To simulate this characteristic, we generate scattered points around the eye movement points recorded by the eye tracker. The farther away from the eye movement point, the more dispersed the scattered points become. Then, we use Gaussian blur to generate a simulated human eye fixation area, obtaining a ground truth map of the fixation point, which serves as the driver's eye movement data E.

[0053] 1.3) Based on the different behaviors of the driver, the video data is divided into video segments containing only one type of driving behavior. The collected natural driving data D and the processed eye-tracking data E are packaged one-to-one and divided into training set and test set in a 4:1 ratio.

[0054] Step 2: Read the driver's eye-tracking dataset to obtain training tensor data and test tensor data.

[0055] Read the natural driving data from the training set and the test set respectively. Extract T frames of image sequence from each video segment. First, convert each frame of image into tensor data. Then, concatenate them into T×H×W×3 four-dimensional tensor data to obtain training tensor data and test tensor data, where H and W are the height and width of each frame of image, respectively, and 3 is the number of RGB channels.

[0056] Step 3: Construct a driver eye-tracking prediction model.

[0057] Reference Figure 2 The specific implementation of this step is as follows:

[0058] 3.1) Construct a driver eye-tracking prediction model encoder to obtain multi-level features F1, F2, F3, and F4 of the driving image sequence:

[0059] An encoder is constructed using an existing pre-trained Video Swin Transformer network and four parallel 3D convolutional layers. The encoder is input with a continuous T-frame H×W×3 driving image sequence from the training set to obtain multi-level features x1, x2, x3, x4. Then, the 3D convolutional layers are used to fine-tune x1, x2, x3, x4 to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence.

[0060] 3.2) Construct a spatiotemporal attention feature fusion module to obtain multi-scale spatiotemporal attention features Y1, Y2, Y3, Y4:

[0061] Reference Figure 3 The specific implementation of this step includes the following:

[0062] 3.2.1) Establish a feature fusion network including an upsampling module and a matrix addition module, used for processing high-level features F i+1 Upsampling and then combining with low-level features F i Add them together to obtain the fused feature X. i ;

[0063] 3.2.2) Establish a multi-scale channel temporal attention module including three parallel attention branches, such as... Figure 4 As shown:

[0064] The first attention branch is composed of a global average pooling layer, a channel-time pointwise convolutional layer, a batch normalization layer, a ReLU activation function, a channel-time pointwise convolutional layer, and a batch normalization layer cascaded together.

[0065] The second attention branch consists of a channel-by-channel convolutional layer, a batch normalization layer, a ReLU activation function, and another channel-by-channel convolutional layer and a batch normalization layer cascaded together.

[0066] The third attention branch is composed of a temporal pointwise convolutional layer, a batch normalization layer, a ReLU activation function, and a temporal pointwise convolutional layer and a batch normalization layer cascaded together.

[0067] 3.2.3) Establish a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function. This network is used to multiply different feature levels with their corresponding weights and then add them together. Finally, the Sigmoid activation function is used to obtain multi-scale spatiotemporal attention features.

[0068] 3.2.4) The feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network are connected in series to obtain the spatiotemporal attention feature fusion module;

[0069] 3.2.5) The i-th level feature F obtained in step 3.1) i With the adjacent (i+1)th level feature F i+1 The input is given to the spatiotemporal attention feature fusion module, F i+1 The upsampling module of the feature fusion network samples the data to F. i At the same resolution, the upsampled feature F is obtained. i ' +1 Then with F i Adding them together yields the original fusion feature X. i ;

[0070] 3.2.6) This fusion feature X i X is obtained by sequentially passing through the first attention branch of the multi-scale channel temporal attention module. i Global spatiotemporal context features g(X) i X is obtained through the second attention branch. i Local channel context features L C (X i X is obtained through the third attention branch. i Local temporal context features L r (X i These contextual features are added together and then combined with the fusion feature X. i Multiply the components and finally pass them through the Sigmoid activation function to obtain the weights w of the fused features. iThey are represented as follows:

[0071] g(X i )=BN(ρReLU(BN(ρ(A V (X i )))))

[0072] L C (X i )=BN(ρReLU(BN(ρ(X i ))))

[0073] L r (X i )=BN(ρReLU(BN(ρ(X i ))))

[0074]

[0075] Where BN represents batch normalization, A V (·) represents global average pooling, and ρ represents pointwise convolution;

[0076] 3.2.7) Weighted Feature Fusion Network based on hierarchical features F i weight w i and hierarchical features F i ' +1 weight 1-w i , will F i With F i ' +1 The fusion process yields multi-scale spatiotemporal attention features Y. i :

[0077]

[0078] 3.2.8) Repeat steps 3.2.5)-3.2.7) a total of 3 times to obtain three multi-scale spatiotemporal attention features Y1, Y2, and Y3. The fourth-level feature F4 remains unchanged, and the output is the fourth spatiotemporal attention feature Y4.

[0079] 3.3) Construct a driver's subjective awareness guidance module to obtain the multi-scale spatial features Z1, Z2, Z3, Z4 of driver's subjective awareness guidance:

[0080] Reference Figure 5 The specific implementation of this step includes the following:

[0081] 3.3.1) Establish a multi-scale temporal attention module including two parallel attention branches, such as... Figure 6 As shown:

[0082] The first attention branch is composed of a global average pooling layer, a temporal pointwise convolutional layer, a batch normalization layer, a ReLU activation function, a temporal pointwise convolutional layer, and a batch normalization layer cascaded together.

[0083] The second attention branch consists of a cascaded sequence of temporal pointwise convolutional layers, a batch normalization layer, a ReLU activation function, a temporal pointwise convolutional layer, and a batch normalization layer.

[0084] 3.3.2) Establish a feature fusion network including a matrix addition module, a sigmoid activation function, and a matrix multiplication module to incorporate the spatial features F that cause changes in driver consciousness. i S First, the driver's conscious information F i C Multiply by the product, add it to itself, and finally pass it through the Sigmoid activation function to obtain the spatial features Z guided by the driver's subjective consciousness. i ;

[0085] 3.3.3) The multi-scale temporal attention module and the feature fusion network are connected in series to obtain the driver's subjective awareness guidance module;

[0086] 3.3.4) The spatiotemporal attention feature Y of the i-th level obtained in step 3.2) i Input into the driver's subjective awareness guidance module for Y i To decouple, the channel dimension is first removed to obtain the temporal feature F containing driver consciousness information. i T Then, by removing the temporal dimension, we obtain the spatial features F that cause changes in driver consciousness. i S ;

[0087] 3.3.5) The temporal feature F containing driver consciousness information i T F is obtained sequentially through the first attention branch of the multi-scale temporal attention module. i T Global temporal context features g(F) i T F is obtained through the second attention branch. i T Local temporal context features L(F) i T These two contextual features are added together, then processed through a Sigmoid activation function, and finally combined with the temporal feature F containing driver consciousness information. i T Multiplying these together yields the consciousness information F of the driver at level i. i C They are represented as follows:

[0088] g(F i T )=BN(ρReLU(BN(ρ(A V (F i T )))))

[0089] L(F i T )=BN(ρReLU(BN(ρ(F i T ))))

[0090]

[0091] Where BN represents batch normalization, A V (·) represents global average pooling, and ρ represents pointwise convolution;

[0092] 3.3.6) Feature fusion network integrates the driver's consciousness information F i C Spatial characteristics F that cause changes in driver consciousness i S Combined, the spatial characteristics Z guided by the driver's subjective consciousness are obtained. i :

[0093]

[0094] 3.3.7) Repeat steps 3.3.4)-3.3.6) a total of 4 times to obtain the multi-scale spatial features Z1, Z2, Z3, Z4 guided by the driver's subjective consciousness;

[0095] 3.4) Construct a decoder for the driver's eye-tracking prediction model to obtain the final driver's eye-tracking prediction image P:

[0096] 3.4.1) Use a series of 2D convolutional layers, upsampling layers and sigmoid activation functions to form four parallel and identical branches, add these four branches together, and then use a 2D convolutional layer and a sigmoid activation function to form a decoder. In this example, but not limited to, there are four 2D convolutional layers and four upsampling layers.

[0097] 3.4.2) The spatial features Z1, Z2, Z3, and Z4 guided by the driver's subjective consciousness obtained in step 3.3) are used as the input of the decoder. Four different saliency images P1, P2, P3, and P4 are obtained through four branches. These four saliency images P1, P2, P3, and P4 are then added together and the driver's eye movement prediction image P is obtained through a 2D convolutional layer and a sigmoid activation function.

[0098] 3.5) The encoder, spatiotemporal attention feature fusion module, driver subjective awareness guidance module and decoder are cascaded in sequence to form a driver eye movement prediction model.

[0099] Step 4: Train the driver eye-tracking prediction model.

[0100] 4.1) Construct the loss function L(P,G) for the driver's eye-tracking prediction model:

[0101] The loss function L(P,G) for the driver eye-tracking prediction model is constructed using existing KL divergence and linear correlation coefficients:

[0102] L(P,G)=L KL (P,G)+L CC (P,G)

[0103] Where P represents the preliminary prediction result of the training set image sequence, and G represents the true value image of the driver's eye gaze point;

[0104] This represents the KL divergence loss, used to measure the distance between two probability distributions, where i represents the i-th pixel, and G... i P represents the pixel value of the i-th pixel in the true image G representing the driver's eye gaze point. i This represents the predicted pixel value of the i-th pixel in the eye-tracking image P;

[0105] Cov(·) represents the loss of the linear correlation coefficient CC, which is used to measure the linear relationship between two random variables. Cov(·) represents the covariance, and σ(·) represents the standard deviation.

[0106] 4.2) Set the video frame image size in the experimental dataset to 224×384, the batch size to 1, the optimizer to Adam, the number of training epochs to 200, and the initial learning rate to 10. -5 The momentum is set to 0.9;

[0107] 4.3) Divide all driving image sequences in the training set into different batches according to the batch size;

[0108] 4.4) Input the first 32 frames of all consecutive image sequences in a batch into the driver's eye movement prediction model to obtain the preliminary prediction result P of the eye movement point in the 32nd frame of all consecutive image sequences in the batch;

[0109] 4.5) Calculate the loss value using the preliminary prediction result P and the true value image of the eye-tracking fixation point G according to the loss function L(P,G), and update the network parameters using the Adam optimizer based on the loss value;

[0110] 4.6) Repeat steps 4.4)-4.5), reducing the learning rate by a factor of 10 every 20 iterations, until the maximum number of training rounds of 200 is reached or the loss function stops decreasing within 10 iterations, and the trained driver eye movement prediction model is obtained.

[0111] Step 5: Input the test set into the trained driver eye movement prediction model to obtain the predicted driver eye movement images.

[0112] The effects of the present invention will be further described in detail below with reference to simulation experiments.

[0113] 1. Simulation Experiment Conditions

[0114] The computer used has an Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz, 16GB of RAM, and two NVIDIA RTX 3090 GPUs with 24GB of video memory each.

[0115] The operating system is 64-bit Ubuntu 18.04, and the algorithm simulation uses Python, specifically version 2.1.0 of the deep learning framework PyTorch.

[0116] The evaluation metrics are the area under the Judd curve (AUC-J), the area under the Shuffled curve (s-AUC), the Pearson correlation coefficient (CC), the normalized scan path (NSS), and the similarity index (SIM). Their calculation formulas are as follows:

[0117]

[0118] Where TPR represents the true positive rate and FPR represents the false positive rate, TPR(FPR) is the TPR value corresponding to FPR on the Judd curve. shuffled FPR represents the true positive rate after randomly shuffling the location of the true fixation point. shuffled TPR represents the false positive rate after randomly shuffling the location of the true fixation point. shuffled (FPR shuffled ) is the FPR on the Shuffled curve shuffled The corresponding TPR shuffled Values; cov(·) represents the covariance, σ(·) represents the standard deviation; i represents the i-th pixel, N represents the total number of pixels, E(P) represents the pixel mean of the predicted eye tracker P, σ(P) represents the standard deviation of the predicted eye tracker P, G i P represents the pixel value of the i-th pixel in the true image G representing the driver's eye gaze point. i This represents the predicted pixel value of the i-th pixel in the eye-tracking image P; This represents the normalized value of the i-th pixel in the predicted eye-tracking image P. The normalized value of the i-th pixel.

[0119] 2. Simulation Experiment Content and Results

[0120] Experiment 1: Under the simulation conditions described above, the present invention and six existing eye-tracking fixation prediction methods were trained and tested on the dataset collected in step 1 to obtain eye-tracking prediction results. The above evaluation indicators were calculated to evaluate the driver's eye-tracking prediction results. The results are shown in Table 1.

[0121] Table 1. Eye movement prediction results of the present invention and six existing methods on the acquired dataset.

[0122] Method AUC-J↑ s-AUC↑ CC↑ NSS↑ SIM↑ TASED-Net 0.907 0.499 0.209 2.537 0.099 SalEMA 0.893 0.500 0.206 2.492 0.093 UNISAL 0.873 0.499 0.148 1.771 0.083 HD 2 S]]> 0.887 0.489 0.172 2.037 0.094 STSANet 0.803 0.473 0.211 2.785 0.117 ViNet 0.829 0.501 0.269 3.033 0.132 The method of the invention 0.935 0.511 0.331 4.094 0.217

[0123] The six existing methods in Table 1 are as follows:

[0124] TASED-Net: A temporal aggregation spatial encoder-decoder network proposed by Min et al. for video saliency prediction;

[0125] SalEMA: A video saliency prediction algorithm based on exponential moving average proposed by Linardos et al.

[0126] UNISAL: Droste et al. proposed a domain-adaptive lightweight encoder-RNN-decoder network for video saliency prediction;

[0127] HD 2 S: A video saliency prediction algorithm based on hierarchical domain adaptive feature learning proposed by Bellitto et al.;

[0128] STSANet: A video saliency prediction algorithm based on spatiotemporal self-attention mechanism proposed by Wang et al.

[0129] ViNet: A visual-inertial navigation system based on differentiable attitude cascade layers proposed by Jain et al. for line-of-sight prediction.

[0130] As can be seen from Table 1, in dynamically changing driving scenarios, the present invention outperforms existing models in five metrics: Judd area under curve (AUC-J), Shuffled area under curve (s-AUC), Pearson correlation coefficient (CC), normalized scan path (NSS), and similarity index (SIM). This indicates that the method of the present invention predicts the driver's gaze point more accurately and is closer to the true value.

[0131] Experiment 2: Under the simulation conditions described above, the present invention and six existing eye-tracking fixation prediction methods were trained on the training set collected in step 1. Image sequences showing left-turning behavior were then used for testing. The subjective results of the eye-tracking prediction were obtained as follows:Figure 7 As shown.

[0132] from Figure 7 As can be seen, the driver's gaze points predicted by the method of the present invention are relatively concentrated, and the eye movement point prediction results are more accurate than other methods. Moreover, the distribution of the gaze point prediction results does not change much during the entire left turn process and is relatively consistent, which is the same as the driver's attention in real situations. This indicates that the present invention has a good effect in eye movement prediction, and the driver's subjective awareness guidance module proposed can effectively use temporal features to predict the driver's gaze points.

[0133] The simulation results above demonstrate that the present invention has higher accuracy and robustness in predicting eye movement points in multi-target scenarios.

Claims

1. A driver eye-tracking prediction method based on multi-scale attention, characterized in that, Includes the following steps: (1) Collect driving videos of different drivers in different scenarios, times and weather, and collect video eye movement data using an eye tracker. Then, correct and process the eye movement data to obtain one-to-one corresponding natural driving data D and eye movement truth map data E. Then divide them into training set and test set in a 4:1 ratio. (2) Read the training set natural driving data and extract the T-frame image sequence. Convert each frame image into tensor data and then concatenate them into a single image. Four-dimensional tensor data, in which and These are the height and width of each frame, respectively, and 3 is the number of RGB channels; (3) Construct the driver's eye-tracking prediction model encoder, and then... Four-dimensional tensor data is used as input to the encoder to obtain multi-level features of the driving image sequence. ; (4) Construct a spatiotemporal attention feature fusion module: 4a) Establish a feature fusion network including an upsampling module and a matrix addition module; 4b) Establish a multi-scale channel temporal attention module including three parallel attention branches; 4c) Establish a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function; 4d) The feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network are connected in series to obtain the spatiotemporal attention feature fusion module; (5) Using the spatiotemporal attention feature fusion module, multi-scale spatiotemporal attention features are obtained. : 5a) The first one obtained in step (3) Features at each level With the adjacent first Features at each level The input is fed into the spatiotemporal attention feature fusion module. Sampled by the upsampling module to the same level At the same resolution, upsampled features are obtained. Again with Adding them together yields the original fusion characteristics. ,Should The weights of the fused features are obtained through a multi-scale channel temporal attention module. ; 5b) Weighted feature fusion network based on hierarchical features weight and hierarchical features weight ,Will and Multi-scale spatiotemporal attention features are obtained by fusion. ; 5c) Repeat steps 5a)-5b) a total of 3 times to obtain three multi-scale spatiotemporal attention features. The fourth level of features Keeping it unchanged, the output yields the fourth spatiotemporal attention feature. ; (6) Construct a module to guide the driver's subjective awareness: 6a) Establish a multi-scale temporal attention module that includes two parallel attention branches; 6b) Construct a feature fusion network including a matrix addition module, a sigmoid activation function, and a matrix multiplication module; 6c) The multi-scale temporal attention module and the feature fusion network are connected in series to obtain the driver's subjective awareness guidance module; (7) Using the driver's subjective awareness guidance module, the multi-scale spatial characteristics of driver's subjective awareness guidance are obtained. : 7a) The first one obtained in step (5) Spatiotemporal attention features at each level Input into the driver's subjective awareness guidance module, for Decoupling to obtain temporal features containing driver consciousness information and spatial features that cause changes in driver consciousness ; 7b) The first is obtained through a multi-scale temporal attention module. Driver's consciousness information Feature fusion network and By integrating these elements, spatial characteristics guided by the driver's subjective awareness can be obtained. ; 7c) Repeat steps 7a)-7b) a total of 4 times to obtain the multi-scale spatial characteristics guided by the driver's subjective consciousness. ; (8) Use the driver eye-tracking prediction model decoder to obtain the driver eye-tracking prediction image. ; (9) The encoder, spatiotemporal attention feature fusion module, driver subjective consciousness guidance module and decoder are cascaded in sequence to form a driver eye movement prediction model; all driving image sequences in the training set are input into the model and trained by gradient descent method to obtain a trained driver eye movement prediction model. (10) Input the test set into the trained driver eye movement prediction model to obtain the predicted driver eye movement image.

2. The method according to claim 1, characterized in that, In step (1), the eye-tracking data is corrected and processed as follows: 1a) For the first Frame and the The frame has eye-tracking points, while the first frame has eye-tracking points. In cases where eye movement points are missing in a frame, the missing eye movement point positions are filled by padding with adjacent frames: assuming the first frame... The eye-tracking coordinates of the frame are , No. The eye-tracking coordinates of the frame are Then the first The eye-tracking coordinates of the frame are filled with ; 1b) Generate scattered points around the eye movement point, and then use Gaussian blur to generate a simulated human eye fixation area to obtain eye movement truth map data E; In step (3), the driver's eye-tracking prediction model encoder is constructed as follows: 3a) Select an existing pre-trained Video Swin Transformer network and use T consecutive frames from the training set. The driving image sequence is input into this network to obtain multi-level features. ; 3b) Selecting existing 3D convolutional layers Fine-tuning was performed to obtain multi-level features of the driving image sequence. .

3. The method according to claim 1, characterized in that, Step 4a) Establish the structure and function of each module in the feature fusion network as follows: The upsampling module includes a bilinear interpolation upsampling layer, which uses bilinear interpolation to expand the spatial resolution of hierarchical features. The matrix addition module is used to perform matrix addition operations on feature layers with the same resolution.

4. The method according to claim 1, characterized in that, The structure and function of the three parallel attention branches in the multi-scale channel temporal attention module established in step 4b) are as follows; The first parallel attention branch consists of a global average pooling layer, a channel-time pointwise convolutional layer, a batch normalization layer, a ReLU activation function, another channel-time pointwise convolutional layer, and a batch normalization layer cascaded together, and is used to compute the fused features. Global spatiotemporal context features : ； Where BN represents batch normalization, Indicates global average pooling. This represents pointwise convolution; The second parallel attention branch, consisting of a cascaded sequence of channel-by-channel convolutional layers, batch normalization layers, ReLU activation functions, and another cascaded sequence of channel-by-channel convolutional layers and batch normalization layers, is used to compute the fused features. Local channel context features : ； Where BN represents batch normalization, This represents pointwise convolution; The third parallel attention branch consists of a cascaded sequence of temporal pointwise convolutional layers, a batch normalization layer, a ReLU activation function, and another cascaded sequence of pointwise convolutional layers and a batch normalization layer, used to compute the fused features. Local temporal context features : ； Where BN represents batch normalization, This represents pointwise convolution.

5. The method according to claim 1, characterized in that, Step 4c) Establish the functions of each module in the weighted feature fusion network and the Sigmoid activation function as follows: The matrix addition module is used to perform matrix addition operations on feature maps with the same resolution; The matrix multiplication module is used to perform matrix multiplication operations on the feature map and the weights; The Sigmoid activation function maps the input data to a non-linear range, enabling the network to process more complex non-linear relationships. It is expressed as follows: .

6. The method according to claim 1, characterized in that: The weights of the fusion features obtained in step 5a) , means as follows: ； in, express Global spatiotemporal context features, express Local channel context features, express Local temporal context features; The multi-scale spatiotemporal attention features obtained in step 5b) , means as follows: 。 7. The method according to claim 1, characterized in that: Step 6b) establishes the structure and function of two parallel attention branches in the multi-scale temporal attention module as follows: The first parallel attention branch, consisting of a global average pooling layer, a temporal pointwise convolutional layer, a batch normalization layer, a ReLU activation function, a temporal pointwise convolutional layer, and a batch normalization layer cascaded together, is used to compute temporal features containing driver awareness information. Global temporal context features : ； Where BN represents batch normalization, Indicates global average pooling. This represents pointwise convolution; The second parallel attention branch, consisting of a cascaded sequence of temporal pointwise convolutional layers, a batch normalization layer, a ReLU activation function, another temporal pointwise convolutional layer, and a batch normalization layer, is used to compute temporal features incorporating driver awareness information. Local temporal context features : ； Where BN represents batch normalization, This represents pointwise convolution; Step 6c) establishes the following functions for the matrix multiplication and matrix addition modules in the feature fusion network: This matrix multiplication module is used for... and Perform matrix multiplication. This matrix addition module is used to perform matrix addition on... and The result obtained after performing matrix multiplication is then combined with... Perform matrix addition.

8. The method according to claim 1, characterized in that: The driver consciousness information obtained in step 7b) Spatial characteristics guided by the driver's subjective consciousness They are represented as follows: ； ； in, express Global temporal context features, express Local temporal context features.

9. The method according to claim 1, characterized in that, In step (8), the driver's eye movement prediction model decoder is used to obtain the driver's eye movement prediction image. The implementation steps include the following: 8a) Use a series of 2D convolutional layers, upsampling layers and sigmoid activation functions to form four parallel and identical branches, add these four branches together, and then use a 2D convolutional layer and a sigmoid activation function to form a decoder; 8b) Spatial characteristics that guide the driver's subjective consciousness As input to the decoder, four distinct saliency images are first obtained through four branches. Then these four saliency images The images are summed and then processed through a 2D convolutional layer and a sigmoid activation function to obtain the driver's eye-tracking prediction image. .

10. The method according to claim 1, characterized in that, Step (9) uses gradient descent to train the driver's eye-tracking prediction model. The steps include the following: 9a) Constructing the loss function , means as follows: ； in, This represents the preliminary prediction results for the image sequences in the training set. Image representing the true value of the driver's eye fixation point; This represents the Kullback-Leibler (KL) divergence loss, used to measure the divergence between two probability distributions. The distance between them Indicates the first 1 pixel, Image representing the true value of the driver's eye fixation point No. The pixel value of each pixel. Indicates predictive eye movement. No. The pixel value of each pixel; The loss represents the linear correlation coefficient CC, used to measure the linear relationship between two random variables. Sexual relations Describing covariance, Indicates standard deviation; 9b) Set the video frame image size in the experimental dataset to 224×384, the batch size to 1, the optimizer to Adam, the number of training epochs to 200, and the initial learning rate to 10. -5 The momentum is set to 0.9; 9c) Input the first 32 consecutive image frames of a batch into the driver's eye movement prediction model to obtain the preliminary prediction result of the eye movement point of the 32nd frame of that batch. ; 9d) Using preliminary prediction results And the true value image G of the eye-tracking fixation point according to the loss function Calculate the loss value and use the Adam optimizer to update the network parameters based on the loss value; 9e) Repeat steps 9c)-9d), decreasing the learning rate by a factor of 10 every 20 iterations until the maximum number of training rounds of 200 is reached or the loss function no longer decreases within 10 iterations. Training then stops, and a well-trained driver eye movement prediction model is obtained.

Images