A Driver Intent Prediction Method Based on Multi-Dimensional Cross-Modal Information Interaction

Abstract

This invention provides a driver intent prediction method based on multi-dimensional cross-modal information interaction, applicable to the field of intelligent driving. The invention designs an efficient video classification network to extract channel-temporal features from in-vehicle and out-of-vehicle video sequences. During feature extraction, a cross-modal channel-spatial weighting mechanism is designed to achieve information interaction between in-vehicle and out-of-vehicle video sequence features. Furthermore, a contrastive learning module is proposed to force the two feature extraction networks to learn the structural knowledge of the vector representations of in-vehicle and out-of-vehicle video sequences. Then, a prediction module based on residual structure and a GRU classifier obtains preliminary prediction results, which are then corrected using GPS information to obtain the final prediction result.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving, and in particular to a method for predicting driver intentions based on multi-dimensional cross-modal information interaction. Background Technology

[0002] In recent years, Advanced Driver Assistance Systems (ADAS) have received significant attention for improving driving safety and preventing traffic accidents. As a key component of these systems, driver intent prediction allows drivers to perceive potential hazards in the shortest possible time, thus providing a potential solution for improving road safety.

[0003] In real-world scenarios, the human reaction time to any accident is 2-3 seconds. Therefore, driver intent prediction algorithms should be designed with sufficient anticipation to help drivers make real-time decisions. Jain et al. released an available natural driving dataset, including in-vehicle and vehicle video, GPS, and speed information. They used a deep learning sensory fusion architecture to achieve an anticipation time of 3.5 seconds and an accuracy of 86%. In recent years, many studies based on the Brain4cars dataset have achieved effective driver intent prediction, but some issues still warrant further investigation.

[0004] In recent years, deep learning has been widely developed and applied in many fields. Since deep learning utilizes a large number of neurons to simulate human visual and auditory perception and thinking activities, researchers have attempted to use it to solve the problem of driver intention prediction, achieving a series of positive results. In general, the advantages of deep learning-based driver intention prediction methods lie in their ability to automatically learn features, achieving an end-to-end learning process, resulting in more comprehensive features and better performance. However, most of these studies use 3D Conv, optical flow, or LSTM stacking to model the temporal information of video sequences, leading to problems such as large network parameters and high algorithm deployment costs. Furthermore, it is worth noting that most studies have not paid attention to or effectively utilized GPS information. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for predicting driver intent based on multi-dimensional cross-modal information interaction.

[0006] This invention proposes a driver intent prediction method based on multimodal information (in-vehicle and external video, and GPS information). In-vehicle and external video are key information in multimodal data; therefore, this invention designs an efficient video classification network to extract channel-temporal features from in-vehicle and external video sequences respectively. During feature extraction, this invention implements information interaction between in-vehicle and external video sequence features through a cross-modal channel-spatial weighting mechanism. Furthermore, this invention proposes a contrastive learning module to force the two feature extraction networks to learn the structural knowledge of the vector representations of in-vehicle and external video sequences. Then, a prediction module based on residual structure and a GRU classifier obtains preliminary prediction results, which are then corrected using GPS information to obtain the final prediction result.

[0007] A method for predicting driver intent based on multi-dimensional cross-modal information interaction, specifically including:

[0008] Step S1: Video frame sampling;

[0009] Divide the input video V into T video segments at equal intervals, i.e., V = (V1, V2, ... V2). T One frame is randomly extracted from each video segment to obtain T frames. After data augmentation, the image size is adjusted and used as the input to the video sequence feature extraction network. The input video includes in-vehicle and out-of-vehicle videos.

[0010] Step S2: The video sequence feature extraction network extracts features from the video frame sequences inside and outside the vehicle;

[0011] In the video sequence feature extraction network of step S1, a channel attention module, a long-term temporal module, and a short-term temporal module are inserted. In each Residual-Block, the Conv1 of the forward channel is replaced by the serial sum of the channel attention module and the short-term temporal module, and the long-term temporal module is inserted in the residual channel. Feature extraction is performed on the video frame sequence processed in step S1 to obtain the features of the in-vehicle and out-of-vehicle video sequences.

[0012] Step S3: The cross-modal channel-spatial weighting mechanism achieves effective multimodal feature information complementarity through cross-modal cross-weighting;

[0013] The CBAM module is used to extract the channel and spatial weights of the in-vehicle and out-of-vehicle video sequence features respectively. Then, the channel-spatial weights of the in-vehicle video sequence features are applied to the out-of-vehicle video sequence features, and the channel-spatial weights of the out-of-vehicle video sequence features are applied to the in-vehicle video sequence features. The cross-modal channel-spatial weighting mechanism is applied to the output features of each ResNet-Block of the in-vehicle and out-of-vehicle video sequence feature extraction network.

[0014] The cross-modal channel-spatial weighting mechanism is expressed as follows:

[0015]

[0016] in, and These represent the features of the in-vehicle and external video sequences to be weighted, respectively. and These represent the weighted features of the in-vehicle and out-of-vehicle video sequences, respectively. and These represent the channel and spatial weights, respectively, weighted onto the features of the in-vehicle video sequence; and These represent the channel and spatial weights, respectively, weighted onto the features of the external video sequence.

[0017] Step S4: The contrastive learning module narrows the distance between vector representations of different modalities of the same sample and widens the distance between vector representations of different samples;

[0018] The vector representations of in-vehicle and out-of-vehicle video sequences of the same sample are regarded as a pair of positive samples, while the vector representations between different samples are regarded as negative samples. First, the cosine distance is used as a standard to measure the distance between two vector representations. Then, the network is forced to spontaneously shorten the distance between positive samples and widen the distance between negative samples in space by learning the penalty factor through comparison.

[0019] Step S5: The prediction module fuses the vector representations of the video sequences inside and outside the vehicle to obtain preliminary prediction results, and finally uses GPS information to correct the preliminary prediction results to obtain the final prediction results;

[0020] The prediction module first uses a "stitching" operation to merge the in-vehicle and out-of-vehicle video sequence vectors processed in step S4, and then feeds them into a one-dimensional deep neural network based on residual structure to extract fused features. Finally, it uses a GRU classifier to model the temporal dependency of multi-frame features to obtain preliminary prediction results; and uses GPS information to perform logical correction on the preliminary prediction results.

[0021] Step S6: Use the fusion prediction result obtained from the prediction module as the main task and the single-modal prediction result of the single-modal network as the auxiliary task to predict the driver's intention based on the multi-task learning framework.

[0022] The loss function of the multi-task learning method is expressed as:

[0023]

[0024] Where Ε(x,y) represents the cross-entropy loss; x F This represents the output of the prediction module; x f and x rα represents the single-modal prediction results based on in-vehicle and out-of-vehicle video sequences, respectively; α represents the multi-task learning balance coefficient.

[0025] Preferably, the channel attention module design of the video sequence feature extraction network is similar to that of the SE block, using convolutional layers instead of fully connected layers in the SE block, and inserting a 1D convolution between two fully connected layers; given the input is Where N is the batch size, C is the number of channels, and H and W are the height and width of the input image, the channel information modeled by the channel attention module is represented as follows:

[0026]

[0027] Where X I and X O W represents the input and output of this module, respectively. C This represents the channel weights calculated by the module.

[0028] Preferably, the long-term time series module is similar to the TSM module, employing a time-shifting scheme based on a multi-module behavior recognition framework, given an input X∈R. N×T×C×H×W Divide X into 8 equal parts proportionally according to the channel, that is

[0029] X = [X1, X2, ..., X8], where, Shift X1 and X3 backward in the time series dimension, and shift X2 forward in the time series dimension.

[0030] Preferably, the short-term temporal module represents local motion information by calculating feature-level frame difference information. Referring to the ME module in TEA-Net, the short-term temporal module models motion information by calculating the feature-level frame difference between the current frame and two adjacent frames, enabling the network to spontaneously capture the difference information between adjacent frames. Therefore, the channel information modeled by the short-term temporal module is represented as follows:

[0031]

[0032] Preferably, the learning penalty factor p s , represented as:

[0033]

[0034] Among them, f i and r i Let represent the vector representations of the in-vehicle and out-of-vehicle video sequences in the i-th sample, respectively; τ represents the temperature coefficient; and S(x,y) represents the distance between the two vectors.

[0035] As a preferred method, the cosine distance is used to measure the distance between two vector representations. The cosine distance between vector x and vector y is expressed as:

[0036]

[0037] The prediction module first fuses the vector representations of in-vehicle and out-of-vehicle video sequences, then obtains preliminary prediction results through a prediction module based on residual structure and GRU classifier, and finally uses GPS information to correct the preliminary prediction results to obtain the final prediction results.

[0038] Preferably, GPS information is used to perform logical correction on the preliminary prediction results. Logical correction is performed on the category labels [gostraight, left change lane, turn left, right change lane, turn right]. This process is represented as follows:

[0039]

[0040] Where λ represents the inhibition coefficient; ω d This represents the correction weight applied to the preliminary prediction results.

[0041] The advantages of this invention compared to existing technologies are as follows: This invention achieves efficient driver intent prediction. First, to extract in-vehicle and out-of-vehicle video features with low computational cost, this invention uses a multi-module based on 2D convolution to extract channel-temporal information of video sequences, solving the problem of high computational cost of traditional 3D convolution and optical flow methods. Second, this invention performs multi-modal information fusion in multiple dimensions, achieving effective cross-modal information interaction. Most existing driver intent prediction algorithms neglect the use of GPS information; this invention proposes a decision correction scheme based on GPS information to further improve algorithm performance. Finally, to enable the proposed algorithm to extract single-modal features that are conducive to the network making correct decisions, this invention uses a multi-task learning framework to train the entire network. Attached Figure Description

[0042] Figure 1 This is a system structure diagram of a driver intent prediction method based on multi-dimensional cross-modal information interaction;

[0043] Figure 2 Network structure diagram for video sequence feature extraction;

[0044] Figure 3 This is a diagram of the prediction module structure. Detailed Implementation

[0045] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0046] As attached Figure 1 This invention provides a system structure diagram for a driver intention prediction method based on multi-dimensional cross-modal information interaction. The method enables drivers to perceive potential dangers caused by inappropriate behavior in the shortest possible time. (See attached diagram.) Figure 1 As shown, the specific steps for behavior recognition of input videos include:

[0047] Step S1: Video frame sampling. Divide the input video (including in-vehicle and out-of-vehicle video) V into T video segments at equal intervals, i.e., V = (V1, V2, ... V2). T For each video segment, one frame is randomly extracted to obtain T frames. After data augmentation, the image size is adjusted to 224×224, which is then used as input to the video sequence feature extraction network (using ResNet-50 as the backbone). The specific structure of ResNet is shown in Table 1.

[0048] Table 1: ResNet Network Structure

[0049]

[0050]

[0051] Step S2: The video sequence feature extraction network extracts features from the video frame sequences inside and outside the vehicle. For example... Figure 2 The diagram shows the structure of the video sequence feature extraction network. The video sequence feature extraction network is a ResNet-50 network with a channel attention module, a long-term time series module, and a short-term time series module inserted. In each Residual-Block, the Conv1 of the forward channel is replaced by the serial sum of the channel attention module and the short-term time series module, and the long-term time series module is inserted in the residual channel.

[0052] The channel attention module design of the video sequence feature extraction network is similar to that of the SE block, using convolutional layers instead of fully connected layers in the SE block, and inserting a 1D convolution between two fully connected layers; given the input is Where N is the batch size, C is the number of channels, and H and W are the height and width of the input image, the channel information modeled by the channel attention module is represented as follows:

[0053]

[0054] Where X I and X O W represents the input and output of this module, respectively. C This represents the channel weights calculated by the module.

[0055] The long-term time series module is similar to the TSM module, employing a time-shifting scheme based on a multi-module behavior recognition framework, given an input X∈R. N×T×C×H×W Divide X into 8 equal parts proportionally according to the channel, i.e., X = [X1, X2, ..., X8], where, Shift X1 and X3 backward in the time series dimension, and shift X2 forward in the time series dimension.

[0056] The short-term temporal module represents local motion information by calculating feature-level frame difference information. Referring to the ME module in TEA-Net, the short-term temporal module models motion information by calculating the feature-level frame difference between the current frame and two adjacent frames, enabling the network to spontaneously capture the difference information between adjacent frames. Therefore, the channel information modeled by the short-term temporal module is represented as follows:

[0057]

[0058] Step S3: The cross-modal channel-space weighting mechanism achieves effective multimodal feature information complementarity through cross-modal cross-weighting. The CBAM module is used to extract the channel and spatial dependencies of the in-vehicle and external video sequence features respectively. Then, the channel-space weights of the in-vehicle video sequence features are applied to the external video sequence features, and vice versa. The cross-modal channel-space weighting mechanism can be expressed as:

[0059]

[0060] The cross-modal channel-spatial weighting mechanism is applied to the output features of each ResNet-Block in the in-vehicle and out-of-vehicle video sequence feature extraction network.

[0061] Step S4: The contrastive learning module learns the spatial structure of vector representations by narrowing the distance between vector representations of different modalities within the same sample and widening the distance between vector representations of different samples. Vector representations of in-vehicle and out-of-vehicle video sequences within the same sample are considered a pair of positive samples, while vector representations between different samples are considered negative samples. First, cosine distance is used as a standard to measure the distance between two vector representations. The cosine distance between vector x and its corresponding vector y can be expressed as:

[0062]

[0063] Next, by contrastively learning the penalty factor to force the network to spontaneously shorten the spatial distance between positive samples and widen the spatial distance between negative samples, this can be expressed as:

[0064]

[0065] Step S5: The prediction module fuses the vector representations of the in-vehicle and out-of-vehicle video sequences to obtain preliminary prediction results, and finally uses GPS information to correct the preliminary prediction results to obtain the final prediction results. Figure 3 The diagram shows the prediction module structure. The prediction module first merges the video sequence vectors inside and outside the vehicle using a "stitching" operation, then feeds them into a one-dimensional deep neural network based on a residual structure to extract fused features. Finally, a GRU classifier is used to model the temporal dependency of multi-frame features to obtain preliminary prediction results. GPS information is used to logically correct the preliminary prediction results. For example, in situations where turning is impossible (no intersections nearby), the predicted scores for "turn left" and "turn right" are suppressed. For the category labels [go straight, left change lane, turn left, right change lane, turn right], this process can be represented as:

[0066]

[0067] Step S6: Use the fused prediction result obtained from the prediction module as the main task and the single-modal prediction result of the single-modal network as the auxiliary task. This allows the algorithm to extract effective single-modal features while achieving high-precision prediction, thus improving the overall network framework. The loss function of the multi-task learning method can be expressed as:

[0068]

[0069] The above embodiments are not intended to limit the present invention, and the present invention is not limited to the above embodiments. Any embodiment that meets the requirements of the present invention is within the protection scope of the present invention.

[0070] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A method for predicting driver intent based on multi-dimensional cross-modal information interaction, characterized in that... The method specifically includes the following steps: Step S1: Video frame sampling; Divide the input video V into T video segments at equal intervals, i.e., V = (V1, V2, ... V2). T One frame is randomly extracted from each video segment to obtain T frames. After data augmentation, the image size is adjusted and used as the input to the video sequence feature extraction network. The input video includes in-vehicle and out-of-vehicle videos. Step S2: The video sequence feature extraction network extracts features from the video frame sequences inside and outside the vehicle; In the video sequence feature extraction network of step S1, a channel attention module, a long-term temporal module, and a short-term temporal module are inserted. In each Residual-Block, the Conv1 of the forward channel is replaced by the serial sum of the channel attention module and the short-term temporal module, and the long-term temporal module is inserted in the residual channel. Feature extraction is performed on the video frame sequence processed in step S1 to obtain the features of the in-vehicle and out-of-vehicle video sequences. Step S3: The cross-modal channel-spatial weighting mechanism achieves effective multimodal feature information complementarity through cross-modal cross-weighting; The CBAM module is used to extract the channel and spatial weights of the in-vehicle and out-of-vehicle video sequence features respectively. Then, the channel-spatial weights of the in-vehicle video sequence features are applied to the out-of-vehicle video sequence features, and the channel-spatial weights of the out-of-vehicle video sequence features are applied to the in-vehicle video sequence features. The cross-modal channel-spatial weighting mechanism is applied to the output features of each ResNet-Block of the in-vehicle and out-of-vehicle video sequence feature extraction network. The cross-modal channel-spatial weighting mechanism is expressed as follows: in, and These represent the features of the in-vehicle and external video sequences to be weighted, respectively. and These represent the weighted features of the in-vehicle and out-of-vehicle video sequences, respectively. and These represent the channel and spatial weights, respectively, weighted onto the features of the in-vehicle video sequence; and These represent the channel and spatial weights, respectively, weighted onto the features of the external video sequence. Step S4: The contrastive learning module narrows the distance between vector representations of different modalities of the same sample and widens the distance between vector representations of different samples; The vector representations of in-vehicle and out-of-vehicle video sequences of the same sample are regarded as a pair of positive samples, while the vector representations between different samples are regarded as negative samples. First, the cosine distance is used as a standard to measure the distance between two vector representations. Then, the network is forced to spontaneously shorten the distance between positive samples and widen the distance between negative samples in space by learning the penalty factor through comparison. Step S5: The prediction module fuses the vector representations of the video sequences inside and outside the vehicle to obtain preliminary prediction results, and finally uses GPS information to correct the preliminary prediction results to obtain the final prediction results; The prediction module first uses a "stitching" operation to merge the in-vehicle and out-of-vehicle video sequence vectors processed in step S4, and then feeds them into a one-dimensional deep neural network based on residual structure to extract fused features. Finally, it uses a GRU classifier to model the temporal dependency of multi-frame features to obtain preliminary prediction results; and uses GPS information to perform logical correction on the preliminary prediction results. Step S6: Use the fusion prediction result obtained from the prediction module as the main task and the single-modal prediction result of the single-modal network as the auxiliary task to predict the driver's intention based on the multi-task learning framework. The loss function of the multi-task learning method is expressed as: Where Ε(x,y) represents the cross-entropy loss; x F This represents the output of the prediction module; x f and x r These represent the single-modal prediction results based on in-vehicle and out-of-vehicle video sequences, respectively; α represents the multi-task learning balance coefficient; ω d p represents the correction weights applied to the preliminary prediction results. s This represents a learning penalty factor.

2. The driver intent prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The channel attention module design of the video sequence feature extraction network is similar to that of the SE block, using convolutional layers instead of fully connected layers in the SE block, and inserting a 1D convolution between two fully connected layers; given the input is Where N is the batch size, C is the number of channels, and H and W are the height and width of the input image, the channel information modeled by the channel attention module is represented as follows: Where X I and X O W represents the input and output of this module, respectively. C This represents the channel weights calculated by the module.

3. The driver intention prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The long-term time series module is similar to the TSM module, employing a time-shifting scheme based on a multi-module behavior recognition framework, given an input X∈R. N×T×C×H×W Divide X into 8 equal parts proportionally according to the channel, i.e., X = [X1, X2, ..., X8], where, Shift X1 and X3 backward in the time series dimension, and shift X2 forward in the time series dimension.

4. The driver intent prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The short-term temporal module represents local motion information by calculating feature-level frame difference information. Referring to the ME module in TEA-Net, the short-term temporal module models motion information by calculating the feature-level frame difference between the current frame and two adjacent frames, enabling the network to spontaneously capture the difference information between adjacent frames. Therefore, the channel information modeled by the short-term temporal module is represented as follows:

5. The driver intent prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The learning penalty factor p s , is represented as: Among them, f i and r i Let represent the vector representations of the in-vehicle and out-of-vehicle video sequences in the i-th sample, respectively; τ represents the temperature coefficient; and S(x,y) represents the distance between the two vectors.

6. The driver intention prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The cosine distance is used to measure the distance between two vector representations. The cosine distance between vectors x and y is expressed as: The prediction module first fuses the vector representations of in-vehicle and out-of-vehicle video sequences, then obtains preliminary prediction results through a prediction module based on residual structure and GRU classifier, and finally uses GPS information to correct the preliminary prediction results to obtain the final prediction results.

7. The driver intent prediction method based on multi-dimensional cross-modal information interaction according to claim 1, characterized in that: The preliminary prediction results are logically corrected using GPS information. This process, which involves logically correcting the category labels [go straight, left change lane, turn left, right change lane, turn right], is represented as follows: Where λ represents the inhibition coefficient.

Images