A driver attention prediction method based on multi-scale fusion

Abstract

The application discloses a driver attention prediction method based on multi-scale fusion. The method first uses VGG16 to obtain the multi-scale features of the video frame, and constructs a multi-scale fusion module. Through the feature transmission in the upward and downward directions, the complementarity between the high and low layer features is fully utilized, and the multi-scale information is fused. Then, Conv-LSTM is used to further fuse the space-time information between the continuous video frames, capture the time dynamic relationship, generate a potential attention map, and obtain the final driver attention prediction result after decoding. Compared with the existing method, the application fully utilizes the multi-scale information, can obtain the complex information in the RGB image, avoids noise interference, and generates an attention prediction result with higher accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of traffic management, and particularly relates to a driver attention prediction method based on multi-scale fusion. BACKGROUND

[0002] In the process of driving a car, the driver needs to keep the vehicle within the lane line, observe the traffic light situation, and also needs to observe the nearby vehicles and pedestrians, and predict possible dangers. If the driver cannot reasonably handle these concurrent situations, it is likely to cause a traffic accident, causing unbearable consequences.

[0003] Studies have shown that the inattention and distraction of the driver will increase the likelihood of traffic accidents. This is because human visual attention can help the driver quickly identify and locate potential risks or important visual cues in the field of vision, such as suddenly appearing pedestrians, intruding non-motor vehicles, and changes in traffic lights. Some researchers point out that from the perspective of visual perception, 90% of the driving information obtained by the driver during driving is from the human visual system. Therefore, studying the visual attention focus of the driver in a sober state will provide great help for the development of assisted driving and autonomous driving, and reduce driving accidents caused by inattention, distraction and other behaviors.

[0004] With the rapid development of deep learning technology, more and more people are committed to the research of driver attention prediction technology based on deep learning. Compared with traditional information theory models and graph theory models, deep learning methods can achieve higher precision and robustness, and provide better safety in the driving process. Existing deep learning methods are mostly based on convolutional neural networks (CNN) for training, such as SCAFNet combining semantic images to assist in driver attention prediction, and VADD using conditional generative adversarial networks (GAN) to predict driver attention. Although these methods consider different modal information and adversarial ideas, they have improved accuracy, but there is a lack of utilization of multi-scale information, and the strong complementary effect between different layer features is not fully considered, resulting in the inability to well handle complex information and interference in driving images. Although SCAFNet attempts to directly fuse the semantic image and the RGB image as input to combine the two features, it needs to convert the RGB image into a semantic image in advance, which has a relatively large amount of calculation, and also does not achieve effective fusion of multi-scale features. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a driver attention prediction method based on multi-scale fusion, which uses VGG16 to obtain multi-scale features of an RGB image and effectively fuses the multi-scale features to obtain more global and local information.

[0006] A driver attention prediction method based on multi-scale fusion comprises the following steps:

[0007] Step 1, obtain continuous video frames of a driver in a driving scene, and decompose the frames as input data.

[0008] Step 2, use a VGG16 model comprising 5 convolution modules stacked together to extract features of the input data, and obtain four multi-scale feature maps T1-T4 from the outputs of the 2nd-5th convolution modules.

[0009] Step 3, perform feature enhancement processing on T1-T4, wherein A1=T1, and use the corresponding feature maps T1-T3 to assist in enhancing T2-T4, and output enhanced feature maps A2-A4. Then, input the enhanced feature maps A1-A4 into a bidirectional dense connection network, complete two transmission and fusion from bottom to top and from top to bottom, and output fusion features T c .

[0010] Step 4, perform spatial attention enhancement on the fusion features T c of the continuous video frames, and then input the enhanced results in a time sequence into a Conv-LSTM, which is used to capture key information and space-time relationship in the continuous T frame images and transfer them to subsequent image frames, and generate corresponding potential driver attention features T out for each frame of image.

[0011] Step 5, use a decoder to process the attention features T out output by the Conv-LSTM to obtain a driver attention prediction result.

[0012] The present application has the following beneficial effects:

[0013] This method uses VGG16 as the backbone network for feature extraction, extracting multi-scale features from the input data. Then, it fuses neighboring scale features using an attention-based mechanism. This improves high-dimensional features and filters noise by using low-dimensional features, resulting in a new set of multi-scale features. These multi-scale features are then fed into a bidirectional densely connected network, where convolution, upsampling, activation functions, and addition are performed to effectively extract multi-scale information. Subsequently, the combined use of Coordinate Attention (CoA) and Conv-LSTM better captures the temporal dynamics between consecutive frames, enhancing the network's ability to extract multi-scale contextual information, filtering out interference, and effectively obtaining the temporal relationships between consecutive frames. This helps model driver visual attention, and compared to traditional methods and existing deep learning models, the prediction performance is superior, making it suitable for assisting autonomous driving and driver assistance systems, and reducing driving accidents. Attached Figure Description

[0014] Figure 1 The flowchart shows a driver attention prediction method based on multi-scale fusion.

[0015] Figure 2 This is a schematic diagram of the network model structure constructed in the embodiment;

[0016] Figure 3 This is a schematic diagram of the adjacent scale attention module structure in the embodiment;

[0017] Figure 4 This is a schematic diagram of channel attention in the embodiment;

[0018] Figure 5 This is a schematic diagram of the bidirectional dense connection network structure in the embodiment;

[0019] Figure 6 This is a schematic diagram of the residual block structure;

[0020] Figure 7(a) shows the RGB image of the input network model in the embodiment, Figure 7(b) shows the corresponding network prediction result, and Figure 7(c) shows the actual attention labels. Detailed Implementation

[0021] The present invention will be further explained below with reference to the accompanying drawings;

[0022] like Figure 1 As shown, a driver attention prediction method based on multi-scale fusion includes the following steps:

[0023] Step 1: Acquire continuous video frames of the driver in the driving scenario. Perform data augmentation operations such as normalization on the video frames, and then scale the image resolution to 384×256 as input data.

[0024] Step 2, as shown, using a VGG16 model with 5 convolution modules, input the input data described in step 1, and obtain multi-scale feature maps T1-T4 from the outputs of the 2nd-5th convolution modules in turn. Figure 2

[0025] Step 3, fuse the multi-scale feature maps T1-T4, the specific steps are as follows:

[0026] Step 3.1, since the dimensions of the feature maps T1-T4 are different, the information contained is also different, in order to better fuse multi-scale information, respectively input the feature maps of adjacent two scales into 3 adjacent scale attention modules ASAM with the same structure, in order to enhance the representation ability of the features, suppress noise, and obtain new features A2-A4, A1=T1.

[0027] The structure of the ASAM is as shown in Figure 3 First, the input feature T i is processed to obtain R i :

[0028] R i =σ s (Conv3(Conv1(T i )+Conv3(CA(Conv3(LN(Conv1(T i ))))))) (1)

[0029] Where, Conv1 represents a convolution operation with a convolution kernel size of 1, Conv3 represents a convolution operation with a convolution kernel size of 3, σ s () represents a Sigmoid activation function, LN represents layer normalization, which is used to stabilize the training process. CA represents a channel attention operation, as shown in Figure 4 First, CA compresses spatial information into channels through a global average pooling layer, and then uses a 1x1 convolution to calculate channel attention:

[0030] CA(x) = x * Conv1(pool(x)) (2)

[0031] Where, x represents the input data of CA, and pool is global average pooling.

[0032] Then use the feature maps T i-1 of adjacent scales to enhance R i to obtain enhanced features A i :

[0033]

[0034] Where, ↓ represents a down-sampling operation,​ represents element-wise addition, represents element-wise multiplication.

[0035] Step 3.2, input A1-A4 into the bidirectional dense connection network as shown in Figure 5 , to fully extract and fuse multi-scale information. The bidirectional dense connection network first performs bottom-up transmission, and then performs top-down transmission.

[0036] Specifically, the bottom-up transmission first inputs A4 into the residual block RB as shown in Figure 6 , to obtain feature P4. P4 is convolved and upsampled to restore the same size as A3 and added to A3. After the activation function and convolution operation, P3 is obtained by inputting into the residual block RB:

[0037] P4=RB(A4) (4)

[0038] P3=RB(Conv3(σ sw (A3+↑Conv3(P4)))) (5)

[0039] where σ sw () represents the Swish activation function, ↑ represents the upsample operation, RB() represents the residual block, and the residual block performs repeated batch normalization, convolution and activation operation on the input feature, and then adds the processing result to the input feature, to improve the network performance and solve the gradient vanishing problem.

[0040] Similarly, P3 continues to be transmitted upwards to obtain P2 and P1 in turn.

[0041] The top-down transmission first performs convolution and down-sampling operation on A1 and O1 respectively, and then adds P2 of the same size. After the activation function and convolution operation, O2 is obtained:

[0042] O2=Conv3(σ sw (P2+↓Conv3(O1)+↓Conv3(A1))) (6)

[0043] where O1=P1. Similarly, O2 continues to be transmitted downwards to obtain O3 and O4 in turn, and finally O4 is taken as the output T c of the bidirectional dense connection network.

[0044] Step 4, obtain the multi-scale feature T c of each video frame through steps 2 and 3, and model the time sequence relationship between consecutive video frames based on T c . The specific steps are as follows:

[0045] Step 4.1, for the multi-scale feature Tc , using a plug-and-play CoA module, generates a more explicit spatial attention representation, cleaning up the potential spatial and temporal factors of dynamic attention, making it easier to capture temporal dynamic features later.

[0046] Step 4.2, input the spatial attention representation obtained in step 4.1 into Conv-LSTM to capture key information and spatio-temporal relationships in the continuous T-frame images, and transfer them to the subsequent image frames to generate corresponding latent driver attention features T out .

[0047] Step 5, input T out into the decoder, and after multiple convolution, batch normalization and Relu activation, and upsampling operations, output a feature map with a resolution of 384x256 as the driver attention prediction result.

[0048] After loss calculation between the attention prediction result output by the model and the labels of the dataset, the model parameters are trained. The loss function is composed of relative entropy, linear correlation coefficient and similarity measure, and each is assigned a corresponding weight of 1, -0.1 and -0.2 respectively. As shown in Fig. 7(a), input the RGB image into the trained model, and the prediction result is shown in Fig. 7(b), and the true label is shown in Fig. 7(c).

[0049] The BDD-A dataset is used to compare the proposed method with other common methods. Relative entropy, linear correlation coefficient and similarity measure are used as evaluation indicators to measure the performance of the model. The experimental results are shown in the following table:

[0050]

[0051] According to the values in the table, the prediction result of the present application is closer to the true value compared with other methods. The evaluation indicators are explained as follows:

[0052] (1) Relative entropy Kldiv, also known as KL divergence (Kullback-Leibler divergence, Kldiv), is a widely used metric for measuring the information loss between the true value and the predicted value. The smaller the value of relative entropy, the closer the predicted value is to the true value, and the smaller the information loss:

[0053]

[0054] Where Y represents the training label, and Y(i) represents the i-th pixel point in the label image. represents the output driver attention map, represents the i-th pixel point of all pixels in the predicted image.

[0055] (2) Linear correlation coefficient (Correlation, Coefficient, CC) is a statistical method to measure the degree of correlation between the true value and the predicted value, and the result is usually between -1 and 1. When the value of CC is 1 or -1, it means that the predicted value of the model is completely correlated with the true value:

[0056]

[0057] wherein Cov(Y, Y) represents the covariance between Y and Y, and σ() represents the standard deviation.

[0058] (3) Similarity measure (Similarity, SIM) is mainly used to evaluate the similarity between two distributions. The larger the value, the closer the predicted result is to the true value. The calculation formula is shown in (9):

[0059] ​​

Claims

1. A driver attention prediction method based on multi-scale fusion, using an RGB video of a driving scene to predict a driver's attention region, characterized in that: Specifically comprising the following steps: Step 1, decompose the continuous video of the driver in the driving scene into a series of RGB images frame by frame as input data; Step 2, use the VGG16 model including 5 convolution modules to extract features from the input data, and obtain four multi-scale feature maps T1-T4 from the outputs of the second to fifth convolution modules; Step 3, use the adjacent scale attention module to perform feature enhancement processing on T1-T4, wherein A1=T1; use the corresponding feature maps T1-T3 to assist in enhancing T2-T4, and output the enhanced feature maps A2-A4: R i = σ s (Conv3(Conv1(T i )+Conv3(CA(Conv3(LN(Conv1(T i ))))))) CA(x) = x * Conv1(pool(x)) wherein i = 2, 3, 4, Conv1 represents a convolution operation with a kernel size of 1, Conv3 represents a convolution operation with a kernel size of 3, ↓ represents a down-sampling operation, and ⊕ represents an element-wise addition, represents an element-wise multiplication, LN represents layer normalization, CA represents a channel attention operation, x represents input data of CA, and pool represents a global average pooling. Then the enhanced feature maps A1-A4 are input into the bidirectional dense connection network, and after two transmission and fusion from bottom to top and from top to bottom, the fusion features T are output c ; In the bidirectional dense connection network, the downward transmission and fusion process is: input A4 into the residual block RB to obtain feature P4, perform convolution and up-sampling operation on P4 to restore it to the same size as A3 and add it to A3, and then input it into the residual block RB to obtain P3: P4 = RB(A4) P3 = RB(Conv3(σ sw (A3+↑Conv3(P4)))) wherein σ sw () denotes a Swish activation function, ↑ denotes an up-sampling operation, RB() denotes a residual block, and the residual block performs repeated batch normalization, convolution, and activation operations on the input features, and then adds the processing result to the input features. Similarly, continue to transmit P3 upward to obtain P2 and P1 in turn; The upward transmission and fusion process is: respectively perform convolution and down-sampling operation on A1 and O1, then add them to P2 of the same size, and then perform activation function and convolution operation to obtain O2: O2 = Conv3(σ sw (P2 + ↓Conv3(O1) + ↓Conv3(A1)) Wherein, O1=P1; similarly, continue to pass down to O2, in turn get O3 and O4, finally O4 as the output T of the bidirectional dense connection network c ; Step 4, fusion features T for consecutive video frames c The spatial attention enhancement result is input into the Conv-LSTM in a time sequence, the Conv-LSTM is used to capture the key information and the space-time relationship in the consecutive T frame images, and transfer it to the subsequent image frames, and generate the corresponding latent driver attention features T for each image frame out ; Step 5, process the attention feature T of the Conv-LSTM output using the decoder out to obtain the driver attention prediction result.

2. The driver attention prediction method based on multi-scale fusion according to claim 1, characterized in that: Normalize the decomposed RGB images, and then adjust the resolution to a uniform size.

3. The driver attention prediction method based on multi-scale fusion according to claim 1, characterized in that: Fusing features T using coordinate attention CoA in step 4 c Spatial attention enhancement is performed to clean up the potential spatial and temporal factors of dynamic attention.

4. The driver attention prediction method based on multi-scale fusion according to claim 1, characterized in that: The decoder recovers the size of the latent driver attention feature T out to the same as the input data through multiple convolution, batch normalization and Relu activation and up-sampling operations, to obtain an attention prediction result.

5. The driver attention prediction method based on multi-scale fusion according to claim 1 or 4, characterized in that: After loss calculation between the attention prediction results output by the model and the labels of the data set, the model parameters are trained; set the loss function as the weighted sum of relative entropy, linear correlation coefficient and similarity measure.

6. The driver attention prediction method based on multi-scale fusion according to claim 5, characterized in that: The weights of relative entropy, linear correlation coefficient and similarity measure in the loss function are set to 1, -0.1 and -0.2 respectively.

Images