Lightweight driver attention prediction method and device based on multi-layer spatio-temporal fusion

Abstract

The application provides a lightweight driver attention prediction method and device based on multi-layer spatio-temporal fusion. The method comprises: acquiring video frames, extracting and storing multi-level spatial features between consecutive frames in the video frames; generating a driver attention saliency map by fusing the multi-level spatial features in the time dimension; and outputting a driver attention prediction result based on the driver attention saliency map. The speed of driver attention prediction in a driving scene is effectively improved through the design of a lightweight backbone and a lightweight 3D module. The application also uses a multi-scale strategy to extract scale information of a dynamic image at different levels, so that the network can fully utilize the time, space and scale information of the dynamic scene, thereby making the driver attention prediction more accurate, realizing fast and accurate perception of potential risks in the driving scene or providing key information required for decision-making, and enhancing the functional interpretability and robustness of intelligent vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a lightweight driver attention prediction method and device based on multi-layer spatio-temporal fusion. BACKGROUND

[0002] Human-like driving is one of the important ways to improve the intelligent degree of automobiles, and the driver attention prediction technology for identifying and locating the target and area of interest of the driver can quickly and accurately perceive the potential risks in the driving scene or provide key information required for decision-making, thereby effectively enhancing the functional understandability and robustness of intelligent automobiles. The distribution of the driver's attention will affect the safety and stability of vehicle driving, for example, the probability of traffic accidents caused by distracted driving behavior is much higher than that in the focused driving state, but skilled drivers can quickly identify various elements and their motion states in the traffic scene, thereby timely identifying direct or potential driving risk information. Therefore, by learning the visual characteristics of skilled drivers in focused driving, the variation law of the driver's attention in various traffic scenes is excavated, important targets and areas in the scene are detected and identified in real time, the efficiency of the driving assistance system is enhanced, the level of intelligent human-like driving system is improved, and the driving safety is improved.

[0003] The driving scene has high spatio-temporal characteristics, so the current driver attention prediction method focuses on how to utilize the spatio-temporal information in the scene. For example, a bypass network using optical flow method, semantic image, etc. is used, but these additional branches increase the computational load of the network; the method using LSTM to transmit time information between frames cannot well capture the deeper coupled spatio-temporal characteristics between continuous frames; using 3D convolution can extract deeper spatio-temporal coupled characteristics in the scene, but the full 3D convolution network has a large memory overhead and higher optimization difficulty.

[0004] It can be seen that it is very necessary to provide a driver attention prediction method that takes into account the prediction speed and accuracy. SUMMARY

[0005] To solve the above technical problems, the present application provides a lightweight driver attention prediction method and device based on multi-layer spatio-temporal fusion.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The first aspect of the present application provides a lightweight driver attention prediction method based on multi-layer spatio-temporal fusion, comprising: acquiring a video frame extracting and storing multi-level spatial features between continuous frames in the video frame ; generating a driver attention saliency map by fusing the multi-level spatial features in the time dimension; and outputting a driver attention prediction result based on the driver attention saliency map.

[0008] Further, in the obtaining video frames After that, further comprising: performing size adjustment processing and standardization processing on the image of the video frame ; the size adjustment processing is to scale the size of the RGB image of the video frame to 256x256; the standardization processing is to perform standardization processing on the RGB image after the size adjustment processing in R, G, and B color channels respectively using Z-Score standardization to generate standardized image data conforming to standard normal distribution, as shown in formula (1):

[0009] Wherein, x i is the input on the corresponding color channel; is the output after standardization on the corresponding color channel; μ i is the mean value of the image brightness value of the corresponding color channel on the training set of the data set, respectively {0.471, 0.448, 0.408}; σ i is the average value of the standard deviation of the corresponding image brightness value, respectively {0.234, 0.239, 0.242}.

[0010] Further, further comprising: establishing a multi-layer spatio-temporal fusion network model, and using the model to perform attention prediction on the standardized image data; the multi-layer spatio-temporal fusion network model comprises an encoder and a decoder, the encoder module is used to extract and store the multi-layer spatial features between the continuous frames in the video frame , and the decoder module is used to generate the driver's attention saliency map by fusing the multi-layer spatial features output by the encoder module in the time dimension.

[0011] Further, after the multi-layer spatio-temporal fusion network model is established, further comprising: training the multi-layer spatio-temporal fusion network model; training the multi-layer spatio-temporal fusion network model comprises using KL divergence as the loss function l KL (G,S) on the supervision layer to optimize the network; the calculation formula of the KL divergence is shown in formula (2): (2)

[0012] Wherein, G∈[0,1] is the true label of the driver's attention, S∈[0,1] is the saliency value predicted by the model, and i is the pixel value of each point; when the network is iteratively optimized to the loss value l KL (G,S) converges, the training is ended and the current network parameters are saved, and then the network and the network parameters are used to predict the driver's attention.

[0013] Further, the encoder module comprises a feature extraction backbone and a memory module; wherein the feature extraction backbone is configured to use full convolutional layers of a lightweight network MobileNetV2, and extract features of a current input frame from Level-1 to Level-4 four levels as the output of the backbone network; the memory module is configured to concatenate and aggregate features of the current frame I t in the time dimension and stored historical frames V t = {I t-T+1 ,…,I t-1} to obtain a feature tensor with a time length of T, and then transmit the feature tensor to the decoder module; and after the model at the current frame I t is executed, the memory module removes the features at the time I t-T+1 and retains the features at the time I t iteratively updated in sequence, so that only a feature tensor with a time length of T-1 is contained in the memory module before the next time calculation is performed, and finally acts on the time I t+1 .

[0014] Further, the decoder is a parallel multi-branch structure, each of the branch structures comprises a space-time fusion layer and a prediction layer; the space-time fusion layer comprises a 3D down-sampling layer, a lightweight 3D inverse bottleneck layer and a trilinear up-sampling layer; wherein the 3D down-sampling layer is a 3D convolution with a convolution kernel size of 3×1×1, a step size of 2×1×1 and a padding of 2×1×1, as shown in formula (3), the 3D down-sampling layer is configured to make the input space-time features only down-sampled in the time dimension to of the original, while the height and width remain unchanged;

[0015]

[0016] ​​In the formula, M is an input feature size, N is an output feature size, K is a convolution kernel size, P is a padding size, S is a step, and i represents a dimension of a feature, that is, a time dimension T, a height dimension H and a width dimension W processed by 3D convolution; the lightweight 3D inverse bottleneck layer comprises a first convolution layer, a second convolution layer and a third convolution layer; wherein the first convolution layer is a 3D separable convolution with a kernel size of 7*7*7, used to increase the window size of a receptive field; the second convolution layer is a standard 3D convolution with a kernel size of 1*1*1, used to expand the channel width, and the dilation ratio is set to 2; the third convolution layer is a standard 3D convolution with a kernel size of 1*1*1, used to gradually reduce the channel width in the spatio-temporal fusion process, and the dilation ratio is set to 0.5; the trilinear upsampling layer is cascaded after each lightweight 3D inverse bottleneck layer, used to expand the size of a feature map by 2 times while keeping the time dimension length unchanged; the prediction layer is a standard 2D convolution with a kernel size of 1*1*1 and with a sigmoid activation function, used to reduce the channel number to 1, and the Sigmoid function is shown in formula (4):

[0017] The output range of the Sigmoid function is (0, 1), so that the prediction layer can map the spatio-temporal fusion features of each layer to a probability distribution for output, obtain the saliency probability map of each feature layer, then splice and finally fuse them on the channel to generate a predicted saliency map, that is, the attention prediction result of the driver.

[0018] Further, batch normalization and a GELU activation function are added after each layer of convolution of the spatio-temporal fusion layer, used to increase the nonlinearity and optimization ability of the network; the GELU function is shown in formula (5), and the randomness is introduced in the training process of the multi-layer spatio-temporal fusion network model with the activation function, so as to improve the robustness of the training process.

[0019]

[0020] The second aspect of the present application provides a lightweight driver attention prediction device based on multi-layer spatio-temporal fusion, comprising: an input module, an encoder module and a decoder module; the input module is used to acquire video frames The encoder module is used to extract and store multi-level spatial features between consecutive frames in the video frames The decoder module is used to generate a saliency map of the driver's attention by fusing the multi-level spatial features in the time dimension.

[0021] The third aspect of the present application provides an electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the light-weight driver attention prediction method based on multi-layer spatio-temporal fusion as described above.

[0022] The fourth aspect of the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the light-weight driver attention prediction method based on multi-layer spatio-temporal fusion as described above.

[0023] Compared with the prior art, the technical solution provided by the present application has at least the following advantages:

[0024] The present application provides a light-weight driver attention prediction method based on multi-layer spatio-temporal fusion, device, electronic equipment and computer readable storage medium, the prediction method effectively improves the speed of driver attention prediction in driving scene through the design of light-weight backbone and light-weight 3D module. Moreover, the present application also uses a multi-scale strategy to extract the scale information of dynamic images at different levels, so that the network can fully utilize the time, space and scale information of dynamic scene, thereby making the driver attention prediction more accurate, realizing fast and accurate perception of potential risks in driving scene or providing key information required for decision-making, and effectively enhancing the functional understandability and robustness of intelligent vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0025] One or more embodiments are illustrated by way of example in the drawings that are for illustrative purposes only and that are not intended to limit the embodiments, unless otherwise specifically indicated, the drawings in the drawings do not constitute a proportional limit.

[0026] Figure 1 A flowchart of the light-weight driver attention prediction method based on multi-layer spatio-temporal fusion of the present application is shown in the figure.

[0027] Figure 2 An encoder-decoder network structure designed by the present application is shown in the figure.

[0028] Figure 3 A driver attention prediction example graph in various typical driving scenes provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0029] The inventors found that although the use of 3D convolution in the field of intelligent transportation can extract deeper spatio-temporal coupling features in the scene, the full 3D convolution network has the problems of huge memory overhead and higher optimization difficulty.

[0030] The present application aims at the deficiencies of the prior art in lightweight and real-time, designs a 2D-3D convolution spatio-temporal fusion network based on hierarchical encoder-decoder architecture, and proposes a lightweight driver attention prediction method. First, MobileNetV2 is selected as the backbone network of the encoder to extract multi-level spatial features of the current frame at four scales, which are stored in the memory module and superimposed with the multi-level features extracted from the historical frames in the time dimension to obtain the spatio-temporal features between consecutive frames, which are then transmitted to the decoder. Second, the decoder is designed based on the hierarchical decoding structure, and the inverse bottleneck 3D convolution module is used to design the spatio-temporal fusion layer to fuse the spatio-temporal features on each independent branch. Finally, the prediction results of different scales captured on the four independent branches are fused to obtain the driver attention prediction value as the output result of the model. Through the effective use of time, space and scale information between consecutive frames in dynamic driving scenes, the present application realizes the effective balance between speed and accuracy of the prediction method.

[0031] The present application will be further described below in conjunction with the drawings and examples.

[0032] Figure 1 The schematic block diagram shows that the process of the lightweight driver attention prediction method based on multi-layer spatio-temporal fusion is as follows:

[0033] Input video frame I t → image preprocessing → use the backbone network to extract spatial feature maps at four levels → use the memory module to stack the feature maps to obtain the spatio-temporal features of the consecutive 16 frames → use the spatio-temporal fusion layer in the decoder to fuse the spatio-temporal features and gradually fuse the time dimension into one → use the prediction layer on the four branches to obtain the saliency probability maps Sup1-Sup4 → fuse the saliency probability maps on the four branches to generate the final prediction saliency map Sup0 → in the training stage, calculate the loss of the input video frame, update the network parameters and the memory module; in the testing or deployment stage, only update the memory module → output the driver attention prediction result.

[0034] Example 1

[0035] As Figure 3 shown, eight typical driving scenes are selected in this embodiment, of which the mountain road is two consecutive frames of the same scene. The specific implementation steps of the present application are as follows:

[0036] First step, input video frame I t , preprocessing:

[0037] First, the image size of the video frame is scaled to 256x256x3, and then Z-score standardization is performed on the RGB color channel of the image, as shown in formula (6):

[0038]

[0039] where x i ∈ [0, 1] is the input pixel intensity value on the corresponding color channel, is the normalized output pixel value on the corresponding color channel. μ i is the mean of intensity values of RGB color channels, which are {0.471, 0.448, 0.408} respectively. And σ i is the standard deviation of intensity values of RGB color channels, which are {0.234, 0.239, 0.242} respectively.

[0040] Secondly, the preprocessed image is input into the established encoder-decoder network to generate a prediction result through spatio-temporal fusion as shown in the following formula: Figure 2

[0041] Firstly, the backbone network MobileNetV2 in the encoder module is used to extract feature maps after the 3rd, 4th, 5th and 6th down-sampling layers from Level-1 to Level-4.

[0042] Secondly, the four-level feature maps are input into the memory module in the encoder module in parallel to splice them with the feature maps of the previous 15 frames in the time dimension, and the feature vectors containing 16 continuous frames are obtained on each branch When it is at the starting time or the historical frames are not full 15 frames, the feature values of the memory module at this time are set to 0, indicating that there is no input at the previous time; when the historical frames are full 15 frames, the memory module will automatically splice and update.

[0043] Thirdly, the feature vectors output by the memory module are transmitted in parallel to the spatio-temporal fusion layers of each branch of the decoder to perform cascaded spatio-temporal fusion and up-sampling, and the fused feature vectors

[0044] Finally, the fused feature vectors are predicted by the prediction layer to generate the saliency probability map i.e. Figure 2 Sup1-Sup4; Sup1, Sup2, Sup3 and Sup4 are spliced in the channel dimension, and the results in four scales are fused by the prediction layer to generate the final saliency map Sup0, and the driver attention prediction value is output.

[0045] Thirdly, the loss for the input video frame I t is calculated:

[0046] After the above two steps, the loss for the input video frame I t ​the saliency map Sup0, in order to measure the difference with the ground truth label, the KL divergence is used as the loss function l KL (G,S), as shown in equation (7):

[0047]

[0048] In equation (7), G [0, 1] is the true label of the driver's attention, S [0, 1] is the saliency map Sup0, and i is the pixel value of each point.

[0049] The network parameters are continuously optimized using the stochastic gradient descent method to reduce the loss l KL (G,S) value to train the network. When l KL (G,S) converges, the network has been trained, and the training process can be stopped. The current network parameters are saved and used to perform the test or deployment steps.

[0050] The lightweight driver attention prediction process based on multi-layer spatio-temporal fusion is completed.

[0051] The prediction method proposed by the present application is verified by specific data as follows:

[0052] Step 1: Import the saved network parameters in step 3 into the network, use the training set of the public dataset DADA-2000 to train, then obtain the predicted saliency map on the test set and perform precision test.

[0053] Step 2: In order to verify the effectiveness of the method, qualitative analysis and quantitative index calculation are performed on the results. In qualitative analysis, the predicted gray image is superimposed on the original image to generate the corresponding heat map. The qualitative effect is shown in Figure 3 (a) and (b) of the drawings, the present application can accurately focus on the pedestrian in the mountain road Figure 3 (a); thanks to the full use of temporal context information, the present application can predict the gaze point to fall on the future trajectory of the pedestrian in the subsequent frame Figure 3 (b), indicating that the driver's intention is captured with the help of spatio-temporal information. For residential road Figure 3 (c), two-wheeled vehicle accident scene Figure 3 (d), crossroads Figure 3 (e), rural road Figure 3 (f), rainy road scene Figure 3 (g) and typical two-wheeled vehicle and car collision accident scene Figure 3 (h), the present application can more accurately locate the driver's attention area and target of interest, and can reflect the driver's driving intention to a certain extent.

[0054] For quantitative analysis, Area under the Curve by Judd (AUC-J), AUC-Borji (AUC-B), the Normalized Scanpath Saliency (NSS), Kullback-Leibler divergence (KL), Similarity (SIM) and Correlation Coefficient (CC) are used to evaluate the accuracy of the proposed method. These metrics can be divided into two categories, location-based methods including AUC-J, AUC-B and NSS, which use binary fixation maps as ground truth. Distribution-based methods including KL, SIM and CC, which use continuous saliency maps as ground truth. Where fixation maps are discrete, recording whether each pixel on the image receives human gaze, while continuous saliency maps can be obtained by blurring each fixation point using a Gaussian kernel.

[0055] The quantitative analysis results are shown in Table 1, where the higher the values of NSS, AUC-J, AUC-B, SIM and CC, the higher the accuracy of the prediction method, and the lower the value of KL, the higher the accuracy of the prediction method.

[0056] Table 1: Results of the proposed method on the test set of the DADA-2000 dataset

[0057]

[0058] The skilled person should note that in Table 1, ↑ indicates that the higher the value, the better, and ↓ indicates that the lower the value, the better.

[0059] In addition, as shown in Table 2, the network parameter amount of the method proposed in the present application is only 19M, and the running speed is fast enough to fully meet the real-time demand.

[0060] Table 2: Model parameter amount and inference time of the proposed method

[0061]

[0062] In the above embodiments, MobileNetV2, 3D convolution, 2D convolution, true value label, stochastic gradient descent method, 3D upsampling, network parameter amount, network training and testing are all known in the technical field.

[0063] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for realizing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make respective changes and modifications without departing from the spirit and scope of the present application, and therefore the protection scope of the present application should be limited by the scope defined by the claims.

Claims

1. A lightweight driver attention prediction method based on multi-layer spatiotemporal fusion, characterized in that, include: Acquire video frames ; Extract and store the video frames Multi-level spatial features between consecutive frames in the middle; A driver's attention saliency map is generated by fusing the multi-level spatial features in the time dimension; The driver attention prediction result is output based on the driver's attention saliency map. In the acquisition of video frames Following that, it also includes: For the video frame The image is resized and standardized. The resizing process involves scaling the RGB image of the video frame to a width and height of [value missing]. ; The standardization process involves using Z-Score standardization to standardize the RGB image after size adjustment in the R, G, and B color channels respectively, to generate standardized image data that conforms to a standard normal distribution, as shown in formula (1): in, For the input on the corresponding color channel; This is the output after normalization for the corresponding color channel; The mean values ​​of the image brightness for the corresponding color channels in the training set of the dataset are respectively ; The average of the standard deviations of the corresponding image brightness values ​​are respectively ; The method further includes: establishing a multi-layer spatiotemporal fusion network model, and using the model to perform attention prediction on the standardized image data; The multi-layer spatiotemporal fusion network model includes two modules: an encoder and a decoder. The encoder module is used to extract and store the video frames. The decoder module generates a driver attention saliency map by fusing the multi-level spatial features output by the encoder module in the temporal dimension, based on the multi-level spatial features between consecutive frames.

2. The lightweight driver attention prediction method based on multi-layer spatiotemporal fusion according to claim 1, characterized in that, After establishing the multi-layer spatiotemporal fusion network model, the method further includes: training the multi-layer spatiotemporal fusion network model; Training the multi-layer spatiotemporal fusion network model includes using KL divergence as the loss function on the supervision layer. Optimize the network; The formula for calculating the KL divergence is shown in equation (2): in, A true indicator of driver attention. The significance value predicted by the model. i For each pixel value; When the network is iteratively optimized to the loss value Upon convergence, training ends and the current network parameters are saved. Then, the network and network parameters are used to predict driver attention.

3. The lightweight driver attention prediction method based on multi-layer spatiotemporal fusion according to claim 1, characterized in that, The encoder module includes a feature extraction backbone and a memory module; The feature extraction backbone uses fully convolutional layers of the lightweight MobileNetV2 network to extract the current input frame. Features from Level-1 to Level-4 As the output of the backbone network; The memory module is used to splice and aggregate the current frame I in the time dimension. t Encoded features and stored historical frames Features The time length is obtained as T The feature tensor is then transmitted to the decoder module; and, in the current frame I t After the time-based model is completed, the memory module is removed. Characteristics of a moment ,reserve Characteristics of a moment Iterative updates are performed sequentially, ensuring that the memory module contains only data of time length 1 before the next calculation is performed. The feature tensor, and finally applied to time.

4. The lightweight driver attention prediction method based on multi-layer spatiotemporal fusion according to claim 1, characterized in that, The decoder is a parallel multi-branch structure, and each branch structure includes a spatiotemporal fusion layer and a prediction layer; The spatiotemporal fusion layer includes a 3D downsampling layer, a lightweight 3D inverse bottleneck layer, and a trilinear upsampling layer; The 3D downsampling layer has a convolution kernel size of Step size is Fill as The 3D convolution, as shown in Equation (3), uses a 3D downsampling layer to ensure that the spatiotemporal features of the input are downsampled only in the time dimension to the original value. While the height and width remain unchanged; In the formula, M For the input feature size, N Output feature size; K The kernel size is the convolution kernel size. P For fill size, S Step size; i The dimension representing the features, i.e., the time dimension processed by 3D convolution. T High-dimensional H and width dimension W ; The lightweight 3D inverse bottleneck layer includes a first convolutional layer, a second convolutional layer, and a third convolutional layer; Wherein, the kernel size of the first convolutional layer is The 3D separable convolution is used to increase the window size of the receptive field; the second convolutional layer expands the channel width, and the kernel size is... The standard 3D convolution is used, with a dilation ratio set to 2; the third convolutional layer has a kernel size that gradually reduces the channel width during the spatiotemporal fusion process. The standard 3D convolution with a dilation ratio set to 0.5; The trilinear upsampling layers are cascaded after each of the lightweight 3D inverse bottleneck layers to double the size of the feature map while keeping the time dimension length unchanged. The prediction layer is a convolutional kernel with a sigmoid activation function and a size of [missing value]. The standard 2D convolution is used to reduce the number of channels to 1, and the Sigmoid function is shown in equation (4): The output range of the Sigmoid function is (0,1), so that the prediction layer can map the spatiotemporal fusion features of each layer into a probability distribution for output, obtain the saliency probability map of each feature level, and then splice them on the channel and finally fuse them to generate the predicted saliency map, that is, the driver's attention prediction result.

5. The lightweight driver attention prediction method based on multi-layer spatiotemporal fusion according to claim 4, characterized in that, Each convolutional layer of the spatiotemporal fusion layer is followed by batch regularization and GELU activation function to increase the network's nonlinearity and optimization capability. The GELU function is shown in formula (5). It introduces randomness into the training process of the multi-layer spatiotemporal fusion network model by using the activation function to improve the robustness of the training process. 。 6. A lightweight driver attention prediction device based on multi-layer spatiotemporal fusion, the device being used to implement the lightweight driver attention prediction method based on multi-layer spatiotemporal fusion as described in any one of claims 1 to 5, characterized in that, The device includes: an input module, an encoder module, and a decoder module; The input module is used to acquire video frames. ; The encoder module is used to extract and store the video frames. Multi-level spatial features between consecutive frames in the middle; The decoder module is used to generate a driver attention saliency map by fusing the multi-level spatial features in the time dimension.

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the lightweight driver attention prediction method based on multi-layer spatiotemporal fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the lightweight driver attention prediction method based on multi-layer spatiotemporal fusion as described in any one of claims 1 to 5.

Images