Driver eye movement prediction method based on multi-scale attention

Abstract

The invention discloses a driver eye movement prediction method based on multi-scale attention, which mainly solves the problems that multi-level features cannot be effectively fused and the prediction of driver eye movement points is inaccurate in the prior art, and the scheme is as follows: constructing a driver eye movement data set, and dividing a training set and a test set; constructing an encoder to obtain driving image sequence multi-level features; constructing a space-time attention feature fusion module to obtain multi-scale space-time attention features; constructing a driver subjective consciousness guiding module to obtain multi-scale spatial features of driver subjective consciousness guiding; constructing a decoder to obtain an eye movement fixation point diagram of the driver; an encoder, a space-time attention feature fusion module, a driver subjective consciousness guiding module and a decoder are cascaded to form a driver eye movement prediction model; training the model by using a training set; and inputting the test set into the trained model to obtain a predicted eye movement image of the driver. The method can effectively predict the eye movement fixation point of the driver, is high in accuracy and robustness, and can be used for an intelligent driving assistance system.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and further designs a driver eye movement prediction method which can be used in an intelligent driving assistance system. Background Art

[0002] With the continuous development of science and technology, intelligent driving assistance systems are gradually becoming a hot topic in the automotive industry as an advanced traffic safety technology. By analyzing the driver's eye movement data and summarizing the law of eye movement changes, the system can effectively recognize the driver's vision and psychology, and then take corresponding driving assistance measures for abnormal driving behavior in advance to reduce the occurrence of traffic accidents. However, real driver eye movement data is not easy to obtain, and its collection process requires a lot of manpower and material resources. The traditional video sequence eye movement point prediction model is developed from the image visual saliency prediction, and different underlying visual features such as color, contrast, brightness, etc. are manually extracted. However, this model can only analyze low-level features and lacks high-level semantic features. It cannot effectively extract inter-frame motion information in video sequences. There is an error between the generated eye movement prediction map and the true value map recorded by the eye tracker, which makes it impossible to handle dynamically changing real scenes.

[0003] With the development of artificial neural networks, a large number of research results have emerged, such as SALGAN, SALICON, ACLNet, TASED-Net, and UNISAL methods. These methods all combine deep neural networks for gaze point detection. Although the motion information between frames can be captured by introducing long short-term memory networks, the global information is incomplete because the information with a long temporal distance will be lost. In order to solve these problems, in recent years, researchers have proposed a model based on the Transformer structure. The model adopts an encoder-decoder network structure, in which the encoder is mainly composed of multiple identical structures (multi-head attention layer, normalization layer, multi-layer perceptron layer), and the residual structure in the residual neural network is used between the encoders. This model combines position encoding and self-attention mechanism to expand the receptive field, so that the dependency between features with a long spatial distance can be obtained, and the global and local features of the sequence can be obtained.

[0004] Patent document with application number CN202210853133.8 discloses a method for predicting the driver's gaze point based on a gaze point prediction model, which constructs a visual spatiotemporal attention layer to extract the global and local features of the sequence. However, this method ignores the situation where the driver's gaze point changes due to his subjective consciousness to change his driving behavior, and cannot extract the temporal features hidden in the image sequence that contain the driver's subjective consciousness, and cannot obtain an accurate driver eye movement prediction map.

[0005] The patent document with application number CN202310656871.8 discloses a training, estimation method, device, system and medium for a line of sight prediction model. This method simply fuses the multi-level features obtained after feature extraction, ignoring the interaction of the multi-level features, resulting in some information being masked or lost, and failing to retain global and local features well.

[0006] The patent document with application number CN202211000132.5 discloses a training method, device and electronic device for a fixation prediction model, which designs a fixation hypergraph learning model to construct a fixation hypergraph, and obtains a predicted fixation target based on the fixation hypergraph. However, since this method only predicts fixations through feature extraction networks and multi-layer perceptron networks, information with a long temporal distance will be lost, resulting in incomplete global information; there is no effective fusion of features at different levels, resulting in the inability to combine the semantic information of low-level features and high-level features well, affecting the accuracy of the model's prediction results for eye movement fixations. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a driver eye movement prediction method based on multi-scale attention, so as to capture the motion information between frames and the global and local features of the sequence from the original image sequence, so as to achieve accurate prediction of the driver's gaze point.

[0008] The technical ideas for achieving the purpose of the present invention are: extracting multi-scale features of driving image sequences through a multi-scale feature extraction module; using a spatiotemporal attention feature fusion module to fuse high-level features representing semantic information with low-level features representing edge, color and other information for contextual information; using a driver's subjective consciousness guidance module to decouple temporal features and spatial features, and enhancing eye movement information in spatial features through temporal changes, thereby more accurately predicting the driver's eye gaze point.

[0009] According to the above technical ideas, the technical solution of the present invention includes the following steps:

[0010] (1) Driving videos of different drivers in different scenarios, times, and weather conditions are collected, and eye movement data of the videos are collected using an eye tracker. The eye movement data are corrected and processed to obtain one-to-one corresponding natural driving data D and eye movement ground truth data E, which are then divided into a training set and a test set in a ratio of 4:1;

[0011] (2) Read the natural driving data of the training set and extract T frame image sequences, convert each frame image into tensor data, and then splice them into T×H×W×3 four-dimensional tensor data, where H and W are the height and width of each frame image, respectively, and 3 is the number of RGB channels;

[0012] (3) Construct a driver eye movement prediction model encoder and use the T×H×W×3 four-dimensional tensor data as the input of the encoder to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence;

[0013] (4) Constructing spatiotemporal attention feature fusion module:

[0014] 4a) establishing a feature fusion network including an upsampling module and a matrix addition module;

[0015] 4b) establishing a multi-scale channel temporal attention module consisting of three parallel attention branches;

[0016] 4c) establishing a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function;

[0017] 4d) connecting the feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network in series to obtain a spatiotemporal attention feature fusion module;

[0018] (5) Using the spatiotemporal attention feature fusion module, we obtain multi-scale spatiotemporal attention features Y1, Y2, Y3, and Y4:

[0019] 5a) The i-th level feature F obtained in step (3) i and the adjacent i+1th level feature F i+1 Input to the spatiotemporal attention feature fusion module, F i+1 Sampling to the same level as F through the upsampling module i The same resolution, get the upsampled feature F i ' +1 Then with F i Add together to get the original fusion feature X i , the X i The weight w of the fusion feature is obtained through the multi-scale channel temporal attention module i ;

[0020] 5b) Weighted feature fusion network based on hierarchical features F i The weight w i and the hierarchical features F i ' +1 The weight of 1-w i , F i With F i ' +1 Fusion obtains the multi-scale spatiotemporal attention feature Y i ;

[0021] 5c) Repeat steps 5a) to 5b) for a total of 3 times to obtain three multi-scale spatiotemporal attention features Y1, Y2, and Y3. The fourth level feature F4 remains unchanged, and the fourth spatiotemporal attention feature Y4 is output;

[0022] (6) Constructing the driver's subjective awareness guidance module:

[0023] 6a) building a multi-scale temporal attention module consisting of two parallel attention branches;

[0024] 6b) Establishing a feature fusion network including a matrix addition module, a Sigmoid activation function, and a matrix multiplication module;

[0025] 6c) Connecting the multi-scale temporal attention module and the feature fusion network in series obtains the driver's subjective consciousness guidance module;

[0026] (7) Using the driver's subjective consciousness guidance module, the multi-scale spatial features Z1, Z2, Z3, and Z4 of the driver's subjective consciousness guidance are obtained:

[0027] 7a) The i-th level spatiotemporal attention feature Y obtained in step (5) i Input to the driver's subjective consciousness guidance module, i Decoupling to obtain the time series feature F with driver awareness information i T and the spatial features F that cause the driver's awareness change i S ;

[0028] 7b)F i T The consciousness information F of the i-th layer driver is obtained through the multi-scale temporal attention module i C , the feature fusion network is used to i C With F i S Fusion is performed to obtain the spatial feature Z guided by the driver’s subjective consciousness i ;

[0029] 7c) Repeat step 7a) to step 7b) 4 times in total to obtain multi-scale spatial features Z1, Z2, Z3, and Z4 guided by the driver's subjective consciousness;

[0030] (8) Using the driver eye movement prediction model decoder, obtain the driver eye movement prediction image P;

[0031] (9) The encoder, the spatiotemporal attention feature fusion module, the driver's subjective consciousness guidance module, and the decoder are sequentially cascaded to form a driver's eye movement prediction model; all driving image sequences of the training set are input into the model, and the model is trained using the gradient descent method to obtain a trained driver's eye movement prediction model;

[0032] (10) The test set is input into the trained driver eye movement prediction model to obtain the predicted driver eye movement image.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] First, the present invention constructs a spatiotemporal attention feature fusion module based on the different distribution and representation capabilities of features at different levels. It aggregates the spatial information and temporal contextual information between features at different levels through multi-scale attention, which not only retains the global features and local features at the same time, but also promotes the network to interact with multi-scale contextual information between features at different levels.

[0035] Second, the present invention aims at the influence of the driver's subjective consciousness on the changes of gaze points during driving, and constructs a driver's subjective consciousness guidance module. By enhancing the temporal characteristics of the driver's consciousness changes through multi-scale attention, the model's understanding of the driver's subjective consciousness and cognitive patterns can be improved, thereby more accurately predicting the driver's eye gaze points during driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the implementation of the present invention;

[0037] Figure 2 It is a block diagram of the driver eye movement prediction model based on multi-scale attention in the present invention;

[0038] Figure 3 It is a block diagram of the spatiotemporal attention feature fusion module in the present invention;

[0039] Figure 4 It is a block diagram of the multi-scale channel temporal attention module in the present invention;

[0040] Figure 5 It is a block diagram of the driver's subjective consciousness guidance module in the present invention;

[0041] Figure 6 It is a block diagram of the multi-scale temporal attention module in the present invention;

[0042] Figure 7 This is a diagram of subjective comparison results between the present invention and six existing methods on the collected data sets. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme and effects in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the present invention, not all embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without creative work should all belong to the protection scope of the present invention.

[0044] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention to facilitate understanding, and the order of the step numbers is not limited.

[0045] Reference Figure 1 , the implementation steps of this embodiment are as follows:

[0046] Step 1: Build a driver eye movement dataset by collecting driving videos of different drivers in different scenarios, times, and weather conditions, and use an eye tracker to collect video eye movement data.

[0047] 1.1) When some frames are missing eye movement points, the eye movement point positions of the missing eye movement frames are filled by using adjacent frames:

[0048] Due to the long working time of the eye tracker or the high light intensity, some frames may be missing eye movement points. For example, there are eye movement points in the t-1th frame and the t+1th frame, but the eye movement points in the tth frame are missing. For such data, it is necessary to fill the eye movement point positions of the missing eye movement frames by using the adjacent frame filling method. The specific implementation is as follows:

[0049] Assume that the eye movement point coordinates of the t-1 frame are (x t-1 ,y t-1 ), the eye movement point coordinates of the t+1th frame are (x t+1 ,y t+1 ), fill the eye movement point coordinates of the tth frame as

[0050] The video eye movement data is corrected using the above method to obtain video eye movement data that completely corresponds to the natural driving data D;

[0051] 1.2) Convert the eye movement points collected by the eye tracker into the true value map of the gaze points:

[0052] The eye movement points recorded by the eye tracker are only single pixels, but in reality, due to the "focusing" characteristics of the human eye, what is observed is an area with a clear center and blurred surroundings. In order to simulate this characteristic, we generate scattered points around the eye movement points recorded by the eye tracker. The farther away from the eye movement point, the more scattered the scattered points are. Then, we use Gaussian blur to generate simulated human eye gaze areas and obtain the gaze point truth map as the driver's eye movement data E;

[0053] 1.3) According to the different behaviors of the driver, the video data is segmented into video clips containing only one driving behavior, the collected natural driving data D and the processed eye movement data E are packaged one by one, and divided into a training set and a test set in a ratio of 4:1.

[0054] Step 2: read the driver eye movement dataset to obtain training tensor data and test tensor data.

[0055] The natural driving data of the training set and the test set are read separately, and T frame image sequences are extracted from each video. Each frame image is first converted into tensor data; then they are spliced ​​into T×H×W×3 four-dimensional tensor data to obtain training tensor data and test tensor data, where H and W are the height and width of each frame image, respectively, and 3 is the number of RGB channels.

[0056] Step 3: Build a driver eye movement prediction model.

[0057] Reference Figure 2 , the specific implementation of this step is as follows:

[0058] 3.1) Construct the driver eye movement prediction model encoder to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence:

[0059] The existing pre-trained Video Swin Transformer network and four parallel 3D convolutional layers are used to form an encoder, and the training set continuous T frame H×W×3 driving image sequence is input into the encoder to obtain multi-level features x1, x2, x3, x4. Then, the 3D convolutional layer is used to fine-tune x1, x2, x3, x4 to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence.

[0060] 3.2) Construct a spatiotemporal attention feature fusion module to obtain multi-scale spatiotemporal attention features Y1, Y2, Y3, Y4:

[0061] Reference Figure 3 The specific implementation of this step includes the following:

[0062] 3.2.1) Establish a feature fusion network including an upsampling module and a matrix addition module to perform high-level feature fusion on the high-level features F i+1 After upsampling, it is combined with the low-level feature F i Add together to get the fusion feature X i ;

[0063] 3.2.2) Establish a multi-scale channel temporal attention module consisting of three parallel attention branches, such as Figure 4 As shown:

[0064] The first attention branch is composed of a global average pooling layer, a channel temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a channel temporal point-by-point convolution layer, and a batch normalization layer cascaded in sequence;

[0065] The second attention branch is composed of a channel point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a channel point-by-point convolution layer, and a batch normalization layer cascaded in sequence;

[0066] The third attention branch is composed of a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a temporal point-by-point convolution layer, and a batch normalization layer cascaded in sequence;

[0067] 3.2.3) Establish a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function, which is used to multiply different feature levels with corresponding weights and then add them together, and then obtain multi-scale spatiotemporal attention features through the Sigmoid activation function;

[0068] 3.2.4) Connecting the feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network in series to obtain a spatiotemporal attention feature fusion module;

[0069] 3.2.5) The i-th level feature F obtained in step 3.1) i and the adjacent i+1th level feature F i+1 Input into the spatiotemporal attention feature fusion module, F i+1 Through the upsampling module of the feature fusion network, the sampling is i The same resolution, get the upsampled feature F i ' +1 Then with F i Add together to get the original fusion feature X i ;

[0070] 3.2.6) The fusion feature X i Get X through the first attention branch of the multi-scale channel temporal attention module in sequence i The global spatiotemporal context feature g(X i ), and obtain X through the second attention branch i The local channel context feature L C (X i ), and obtain X through the third attention branch i The local temporal context feature L r (X i ), these context features are added together and then combined with the fusion feature X i Multiply them together, and finally use the Sigmoid activation function to get the weight w of the fusion feature i, which are respectively expressed as follows:

[0071] g(X i )=BN(ρReLU(BN(ρ(A V (X i )))))

[0072] L C (X i )=BN(ρReLU(BN(ρ(X i ))))

[0073] L r (X i )=BN(ρReLU(BN(ρ(X i ))))

[0074]

[0075] Among them, BN means batch normalization, A V (·) represents global average pooling, and ρ represents point-wise convolution;

[0076] 3.2.7) Weighted feature fusion network based on hierarchical features F i The weight w i and the hierarchical features F i ' +1 The weight of 1-w i , F i With F i ' +1 Fusion obtains the multi-scale spatiotemporal attention feature Y i :

[0077]

[0078] 3.2.8) Repeat steps 3.2.5) - 3.2.7) for a total of 3 times to obtain three multi-scale spatiotemporal attention features Y1, Y2, and Y3. The fourth level feature F4 remains unchanged, and the fourth spatiotemporal attention feature Y4 is output;

[0079] 3.3) Construct the driver's subjective consciousness guidance module and obtain the multi-scale spatial features Z1, Z2, Z3, and Z4 of the driver's subjective consciousness guidance:

[0080] Reference Figure 5 The specific implementation of this step includes the following:

[0081] 3.3.1) Establish a multi-scale temporal attention module consisting of two parallel attention branches, such as Figure 6 As shown:

[0082] The first attention branch is composed of a global average pooling layer, a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a temporal point-by-point convolution layer, and a batch normalization layer cascaded in sequence;

[0083] The second attention branch is composed of a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a temporal point-by-point convolution layer and a batch normalization layer cascaded in sequence;

[0084] 3.3.2) Establish a feature fusion network including a matrix addition module, a Sigmoid activation function, and a matrix multiplication module to transform the spatial features F that cause changes in driver awareness i S First, the driver's awareness information F i C Multiply, then add to itself, and finally use the Sigmoid activation function to get the spatial feature Z guided by the driver's subjective consciousness i ;

[0085] 3.3.3) Connect the multi-scale temporal attention module and the feature fusion network in series to obtain the driver's subjective consciousness guidance module;

[0086] 3.3.4) The i-th level spatiotemporal attention feature Y obtained in step 3.2) is i Input to the driver's subjective consciousness guidance module to Y i Decouple, that is, first delete the channel dimension to obtain the time series feature F with driver awareness information i T , and then delete the time series dimension to obtain the spatial feature F that causes the driver's awareness change i S ;

[0087] 3.3.5) The time series feature F with driver awareness information i T F is obtained by sequentially passing through the first attention branch of the multi-scale temporal attention module i T The global temporal context feature g(F i T ), and obtain F through the second attention branch i T The local temporal context feature L(F i T ), these two context features are added together and then activated by the Sigmoid function, and then combined with the temporal feature F containing the driver's awareness information. i T Multiply them together to get the consciousness information F of the i-th layer driver i C , which are respectively expressed as follows:

[0088] g(F i T )=BN(ρReLU(BN(ρ(A V (F i T )))))

[0089] L(F i T )=BN(ρReLU(BN(ρ(F i T ))))

[0090]

[0091] Among them, BN means batch normalization, A V (·) represents global average pooling, and ρ represents point-wise convolution;

[0092] 3.3.6) The feature fusion network combines the driver’s consciousness information F i C and the spatial features F that cause the driver's awareness change i S Combined with the spatial feature Z guided by the driver's subjective consciousness, i :

[0093]

[0094] 3.3.7) Repeat steps 3.3.4) to 3.3.6) for 4 times to obtain the multi-scale spatial features Z1, Z2, Z3, and Z4 guided by the driver's subjective consciousness;

[0095] 3.4) Construct the driver eye movement prediction model decoder to obtain the final driver eye movement prediction image P:

[0096] 3.4.1) Use a series of 2D convolutional layers, upsampling layers and Sigmoid activation functions to form four parallel and identical branches, add these four branches, and then form a decoder through a 2D convolutional layer and a Sigmoid activation function. This example takes but is not limited to four 2D convolutional layers and four upsampling layers;

[0097] 3.4.2) The spatial features Z1, Z2, Z3, and Z4 guided by the driver's subjective consciousness obtained in step 3.3) are used as the input of the decoder, and four different saliency images P1, P2, P3, and P4 are obtained through four branches. Then, these four saliency images P1, P2, P3, and P4 are added together, and the driver's eye movement prediction image P is obtained through a 2D convolution layer and a Sigmoid activation function;

[0098] 3.5) The encoder, spatiotemporal attention feature fusion module, driver subjective consciousness guidance module, and decoder are cascaded in sequence to form a driver eye movement prediction model.

[0099] Step 4: training the driver's eye movement prediction model.

[0100] 4.1) Construct the driver eye movement prediction model loss function L(P,G):

[0101] The existing KL divergence and linear correlation coefficient are used to construct the driver eye movement prediction model loss function L(P,G):

[0102] L(P,G)=L KL (P,G)+L CC (P,G)

[0103] Among them, P represents the preliminary prediction result of the training set image sequence, and G represents the real value image of the driver's eye gaze point;

[0104] represents the KL divergence loss, which is used to measure the distance between two probability distributions, i represents the i-th pixel, G i represents the pixel value of the i-th pixel in the driver’s eye gaze point true value image G, P i represents the pixel value of the i-th pixel of the predicted eye movement map P;

[0105] represents the loss of the linear correlation coefficient CC, which is used to measure the linear relationship between two random variables, cov(·) represents the covariance, and σ(·) represents the standard deviation;

[0106] 4.2) Set the video frame image size in the experimental dataset to 224×384, the batch size to 1, the optimizer to use Adam, the number of training rounds to 200, and the initial learning rate to 10 -5 , the momentum is set to 0.9;

[0107] 4.3) Divide all driving image sequences in the training set into different batches according to the batch size;

[0108] 4.4) Input the first 32 frames of all continuous image sequences of a batch into the driver's eye movement prediction model to obtain the preliminary prediction result P of the eye movement point of the 32nd frame of all continuous image sequences of the batch;

[0109] 4.5) Use the preliminary prediction result P and the true value image G of the eye movement fixation point to calculate the loss value according to the loss function L(P,G), and use the Adam optimizer to update the network parameters according to the loss value;

[0110] 4.6) Repeat steps 4.4)-4.5), and reduce the learning rate by 10 times every 20 iterations until the maximum number of training rounds reaches 200 or the loss function stops decreasing within 10 iterations, and the training is stopped to obtain a trained driver eye movement prediction model.

[0111] Step 5: Input the test set into the trained driver eye movement prediction model to obtain the predicted driver eye movement image.

[0112] The effects of the present invention are further described in detail below in conjunction with simulation experiments.

[0113] 1. Simulation experiment conditions

[0114] The computer processor used is Intel(R) Core(TM) i5-6600 CPU@3.30GHz, 16GB of memory, and the graphics card is two NVIDIA RTX 3090GPUs with 24GB of video memory.

[0115] The operating system is 64-bit Ubuntu 18.04, the algorithm simulation adopts Python language, and the deep learning framework PyTorch version 2.1.0 is used.

[0116] The evaluation indicators are the area under the Judd curve AUC-J, the area under the Shuffled curve s-AUC, the Pearson correlation coefficient CC, the normalized scan path NSS, and the similarity index SIM. The calculation formulas are as follows:

[0117]

[0118] Among them, TPR represents the true positive rate, FPR represents the false positive rate, TPR(FPR) is the TPR value corresponding to FPR on the Judd curve, TPR shuffled Represents the true positive rate after the position of the real gaze point is randomly shuffled, FPR shuffled Represents the false positive rate after the position of the true gaze point is randomly shuffled, TPR shuffled (FPR shuffled ) is the FPR on the Shuffled curve shuffled The corresponding TPR shuffled value; cov(·) represents covariance, σ(·) represents standard deviation; i represents the i-th pixel, N represents the total number of pixels, E(P) represents the pixel mean of the predicted eye movement map P, σ(P) represents the standard deviation of the predicted eye movement map P, G i represents the pixel value of the i-th pixel in the driver’s eye gaze point true value image G, P i represents the pixel value of the i-th pixel of the predicted eye movement map P; represents the normalized value of the i-th pixel of the predicted eye movement map P, The normalized value of the i-th pixel.

[0119] 2. Simulation experiment content and results

[0120] Experiment 1: Under the above simulation experimental conditions, the present invention and the existing six eye movement fixation point prediction methods are used to perform training and testing on the data set collected in step 1 to obtain eye movement prediction results, and the above evaluation indicators are calculated to evaluate the eye movement prediction results of the driver. The results are shown in Table 1:

[0121] Table 1 Eye movement prediction results of the present invention and six existing methods on the collected data set

[0122] method AUC-J↑ s-AUC↑ CC↑ NSS↑ SIM↑ TASED-Net 0.907 0.499 0.209 2.537 0.099 SalEMA 0.893 0.500 0.206 2.492 0.093 UNISAL 0.873 0.499 0.148 1.771 0.083 <![CDATA[HD 2 S]]> 0.887 0.489 0.172 2.037 0.094 STSANet 0.803 0.473 0.211 2.785 0.117 ViNet 0.829 0.501 0.269 3.033 0.132 Method of the present invention 0.935 0.511 0.331 4.094 0.217

[0123] The 6 existing methods in Table 1 are:

[0124] TASED-Net: A temporally aggregated spatial encoder-decoder network for video saliency prediction proposed by Min et al.

[0125] SalEMA: A video saliency prediction algorithm based on exponential moving average proposed by Linardos et al.

[0126] UNISAL: Droste et al. proposed a lightweight encoder-RNN-decoder network based on domain adaptation for video saliency prediction;

[0127] HD 2 S: A video saliency prediction algorithm based on hierarchical domain adaptive feature learning proposed by Bellitto et al.

[0128] STSANet: A video saliency prediction algorithm based on spatiotemporal self-attention mechanism proposed by Wang et al.

[0129] ViNet: A visual-inertial assisted navigation system based on a differentiable pose cascade layer for line of sight prediction proposed by Jain et al.

[0130] It can be seen from Table 1 that in a dynamically changing driving scenario, the present invention is superior to the existing model in five indicators: the area under the Judd curve AUC-J, the area under the Shuffled curve s-AUC, the Pearson correlation coefficient CC, the normalized scan path NSS, and the similarity index SIM, indicating that the method of the present invention is more accurate in predicting the driver's gaze point and is closer to the true value.

[0131] Experiment 2: Under the above simulation experimental conditions, the present invention and the existing six eye movement fixation point prediction methods were used to train on the training set collected in step 1, and the image sequence of left turn behavior was selected for testing. The subjective results of eye movement prediction were obtained as follows: Figure 7 shown.

[0132] from Figure 7 It can be seen that the driver's gaze points predicted by the method of the present invention are relatively concentrated, and the eye movement point prediction results are more accurate than other methods. In the entire left turn behavior process, the distribution of the gaze point prediction results does not change much and is relatively consistent, which is the same as the driver's attention in real situations. This shows that the present invention has a good effect in eye movement prediction, and the driver's subjective consciousness guidance module proposed by the present invention can effectively use time series features to predict the driver's gaze points.

[0133] The above simulation results show that the present invention has higher accuracy and robustness in predicting eye movement points in multi-target scenes.

Claims

1. A driver eye movement prediction method based on multi-scale attention, characterized in that: The steps include: (1) Driving videos of different drivers in different scenarios, times, and weather conditions are collected, and eye movement data of the videos are collected using an eye tracker. The eye movement data are corrected and processed to obtain one-to-one corresponding natural driving data D and eye movement ground truth data E, which are then divided into a training set and a test set in a ratio of 4:1; (2) Read the natural driving data of the training set and extract T frame image sequences, convert each frame image into tensor data, and then splice them into T×H×W×3 four-dimensional tensor data, where H and W are the height and width of each frame image, respectively, and 3 is the number of RGB channels; (3) Construct a driver eye movement prediction model encoder and use the T×H×W×3 four-dimensional tensor data as the input of the encoder to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence; (4) Constructing spatiotemporal attention feature fusion module: 4a) establishing a feature fusion network including an upsampling module and a matrix addition module; 4b) establishing a multi-scale channel temporal attention module consisting of three parallel attention branches; 4c) establishing a weighted feature fusion network including a matrix addition module, a matrix multiplication module, and a Sigmoid activation function; 4d) connecting the feature fusion network, the multi-scale channel temporal attention module, and the weighted feature fusion network in series to obtain a spatiotemporal attention feature fusion module; (5) Using the spatiotemporal attention feature fusion module, we obtain multi-scale spatiotemporal attention features Y1, Y2, Y3, and Y4: 5a) The i-th level feature F obtained in step (3) i and the adjacent i+1th level feature F i+1 Input to the spatiotemporal attention feature fusion module, F i+1 Sampling to the same level as F through the upsampling module i The same resolution, get the upsampled feature F′ i+1 Then with F i Add together to get the original fusion feature X i , the X i The weight w of the fusion feature is obtained through the multi-scale channel temporal attention module i ; 5b) Weighted feature fusion network based on hierarchical features F i The weight w i and level features F′ i+1 The weight of 1-w i , F i and F′ i+1 Fusion obtains the multi-scale spatiotemporal attention feature Y i ; 5c) Repeat steps 5a) to 5b) for a total of 3 times to obtain three multi-scale spatiotemporal attention features Y1, Y2, and Y3. The fourth level feature F4 remains unchanged, and the fourth spatiotemporal attention feature Y4 is output; (6) Constructing the driver's subjective awareness guidance module: 6a) building a multi-scale temporal attention module consisting of two parallel attention branches; 6b) Establishing a feature fusion network including a matrix addition module, a Sigmoid activation function, and a matrix multiplication module; 6c) Connecting the multi-scale temporal attention module and the feature fusion network in series obtains the driver's subjective consciousness guidance module; (7) Using the driver's subjective consciousness guidance module, the multi-scale spatial features Z1, Z2, Z3, and Z4 of the driver's subjective consciousness guidance are obtained: 7a) The i-th level spatiotemporal attention feature Y obtained in step (5) i Input to the driver's subjective consciousness guidance module, i Decoupling to obtain the time series feature F with driver awareness information i T and the spatial features F that cause the driver's awareness change i S ; 7b)F i T The consciousness information F of the i-th layer driver is obtained through the multi-scale temporal attention module i C , the feature fusion network is used to i C With F i S Fusion is performed to obtain the spatial feature Z guided by the driver’s subjective consciousness i ; 7c) Repeat step 7a) to step 7b) 4 times in total to obtain multi-scale spatial features Z1, Z2, Z3, and Z4 guided by the driver's subjective consciousness; (8) Using the driver eye movement prediction model decoder, obtain the driver eye movement prediction image P; (9) The encoder, the spatiotemporal attention feature fusion module, the driver's subjective consciousness guidance module, and the decoder are sequentially cascaded to form a driver's eye movement prediction model; all driving image sequences of the training set are input into the model, and the model is trained using the gradient descent method to obtain a trained driver's eye movement prediction model; (10) The test set is input into the trained driver eye movement prediction model to obtain the predicted driver eye movement image.

2. The method according to claim 1, characterized in that In step (1), the eye movement data is corrected and processed as follows: 1a) If there are eye movement points in the t-1th frame and the t+1th frame, but the eye movement points in the tth frame are missing, the eye movement point positions of the missing eye movement frames are filled in by using adjacent frames: Assume that the eye movement point coordinates of the t-1th frame are (x t-1 ,y t-1 ), the eye movement point coordinates of the t+1th frame are (x t+1 ,y t+1 ), then the eye movement point coordinates of the tth frame are filled in as 1b) Generate scattered points around the eye movement point, and then use Gaussian blur to generate a simulated human eye gaze area to obtain the eye movement true value map data E. In step (3), the driver eye movement prediction model encoder is constructed, which is implemented as follows: 3a) Select the existing pre-trained Video Swin Transformer network and input the training set of T consecutive frames H×W×3 driving image sequences into the network to obtain multi-level features x1, x2, x3, x4; 3b) Select the existing 3D convolutional layer to fine-tune x1, x2, x3, x4 to obtain the multi-level features F1, F2, F3, F4 of the driving image sequence.

3. The method according to claim 1, characterized in that Step 4a) The structure and functions of each module in the feature fusion network are as follows: The upsampling module includes a bilinear interpolation upsampling layer, which uses a bilinear interpolation method to expand the spatial resolution of hierarchical features; The matrix addition module is used to perform matrix addition operations on feature levels with the same resolution.

4. The method according to claim 1, characterized in that The structures and functions of the three parallel attention branches in the multi-scale channel temporal attention module established in step 4b) are as follows; The first parallel attention branch is composed of a global average pooling layer, a channel temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a channel temporal point-by-point convolution layer, and a batch normalization layer, which are cascaded in sequence to calculate the fusion feature X i The global spatiotemporal context feature g(X i ): g(X i )=BN(ρReLU(BN(ρ(A V (X i ))))) Among them, BN means batch normalization, A V (·) represents global average pooling, and ρ represents point-wise convolution; The second parallel attention branch is composed of a channel point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a channel point-by-point convolution layer, and a batch normalization layer cascaded in sequence to calculate the fusion feature X i The local channel context feature L C (X i ): L C (X i )=BN(ρReLU(BN(ρ(X i )))) Among them, BN means batch normalization, and ρ means point-by-point convolution; The third parallel attention branch is composed of a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, and a temporal point-by-point convolution layer and a batch normalization layer, which are cascaded in sequence to calculate the fusion feature X i The local temporal context feature L r (X i ): L r (X i )=BN(ρReLU(BN(ρ(X i )))) Among them, BN stands for batch normalization and ρ stands for point-by-point convolution.

5. The method according to claim 1, characterized in that Step 4c) Establish the functions of each module and the Sigmoid activation function in the weighted feature fusion network as follows: The matrix addition module is used to perform matrix addition operations on feature maps with the same resolution; The matrix multiplication module is used to perform matrix multiplication operations on feature maps and weights; The Sigmoid activation function is used to map the input data into a nonlinear range, so that the network can handle more complex nonlinear relationships, which is expressed as 6. The method according to claim 1, characterized in that: The weight w of the fusion feature obtained in step 5a) i , which is expressed as follows: Among them, g(X i ) indicates X i The global spatiotemporal context features, L C (X i ) indicates X i The local channel context features, L r (X i ) indicates X i Local temporal context features of The multi-scale spatiotemporal attention feature Y obtained in step 5b) i , which is expressed as follows:

7. The method according to claim 1, characterized in that: The structure and function of two parallel attention branches in step 6b) in establishing the multi-scale temporal attention module are as follows: The first parallel attention branch is composed of a global average pooling layer, a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a temporal point-by-point convolution layer, and a batch normalization layer, which are cascaded in sequence to calculate the temporal feature F with driver awareness information. i T The global temporal context feature g(F i T ): g(F i T )=BN(ρReLU(BN(ρ(A V (F i T ))))) Among them, BN means batch normalization, A V (·) represents global average pooling, and ρ represents point-wise convolution; The second parallel attention branch is composed of a temporal point-by-point convolution layer, a batch normalization layer, a ReLU activation function, a temporal point-by-point convolution layer, and a batch normalization layer, which are cascaded in sequence to calculate the temporal feature F with driver awareness information. i T The local temporal context feature L(F i T ): L(F i T )=BN(ρReLU(BN(ρ(F i T )))) Among them, BN means batch normalization, and ρ means point-by-point convolution; The functions of the matrix multiplication module and the matrix addition module in the feature fusion network established in step 6c) are as follows: The matrix multiplication module is used to i C and F i S Perform matrix multiplication operations; The matrix addition module is used to i C and F i S The result obtained after matrix multiplication is then added to F i S Perform matrix addition.

8. The method according to claim 1, characterized in that: The driver awareness information F obtained in step 7b) i C and the spatial feature Z guided by the driver's subjective consciousness i , respectively expressed as follows: Among them, g(F i T ) indicates F i T The global temporal context feature, L(F i T ) indicates F i T Local temporal context features.

9. The method according to claim 1, characterized in that: In step (8), the driver eye movement prediction model decoder is used to obtain the driver eye movement prediction image P, and the implementation steps include the following: 8a) Use a series of 2D convolutional layers, upsampling layers, and Sigmoid activation functions to form four parallel and identical branches, add these four branches together, and then form a decoder through a 2D convolutional layer and a Sigmoid activation function; 8b) The spatial features Z1, Z2, Z3, and Z4 guided by the driver’s subjective consciousness are used as the input of the decoder. First, four different saliency images P1, P2, P3, and P4 are obtained through four branches. Then, these four saliency images P1, P2, P3, and P4 are added together, and the driver’s eye movement prediction image P is obtained through a 2D convolutional layer and a Sigmoid activation function.

10. The method according to claim 1, characterized in that Step (9) uses the gradient descent method to train the driver eye movement prediction model, and the implementation steps include the following: 9a) Construct the loss function L(P,G), which is expressed as follows: L(P,G)=L KL (P,G)+L CC (P,G) Among them, P represents the preliminary prediction result of the training set image sequence, and G represents the real value image of the driver's eye gaze point; represents the Kullback-Leibler (KL) divergence loss, which is used to measure the distance between two probability distributions, i represents the i-th pixel, G i represents the pixel value of the i-th pixel in the driver’s eye gaze point true value image G, P i represents the pixel value of the i-th pixel of the predicted eye movement map P; represents the loss of the linear correlation coefficient CC, which is used to measure the linear relationship between two random variables, cov(·) represents the covariance, and σ(·) represents the standard deviation; 9b) Set the video frame image size in the experimental dataset to 224×384, the batch size to 1, the optimizer to use Adam, the number of training rounds to 200, and the initial learning rate to 10 -5 , the momentum is set to 0.9; 9c) inputting the first 32 frames of a batch of continuous image sequences into the driver's eye movement prediction model to obtain a preliminary prediction result P of the eye movement point of the 32nd frame of the batch; 9d) Using the preliminary prediction result P and the true value image G of the eye movement fixation point, the loss value is calculated according to the loss function L(P,G), and the Adam optimizer is used to update the network parameters according to the loss value; 9e) Repeat steps 9c)-9d), and reduce the learning rate by 10 times every 20 iterations until the maximum number of training rounds reaches 200 or the loss function no longer decreases within 10 iterations. The training stops and a trained driver eye movement prediction model is obtained.

Images