A Multimodal Data Scene Recognition Method Based on Multi-level Interactive Fusion

Abstract

This invention discloses a multimodal data scene recognition method based on multi-level interactive fusion. Using video data and vehicle data collected by onboard sensors in autonomous driving scenarios, three single-modal features are extracted: 2D-level features from the video are obtained through multi-instance learning based on a two-stage attention mechanism; 3D spatiotemporal features from the scene video are extracted through a multi-layer spatiotemporal attention network, and onboard information feature vectors are added for training and interaction; and the onboard information feature vectors are trained. After feature extraction of the three modalities, similarity loss is calculated. During training, the similarity of the three modalities is maximized, and the features of the three modalities are interacted based on a multi-layer self-attention network. Finally, classification is performed. This method can utilize existing video and onboard information interaction to supplement information and improve the speed and accuracy of scene recognition.

Description

Technical Field

[0001] This invention belongs to the field of data recognition technology and relates to a method for the fusion and recognition of multimodal data, specifically a method for multimodal data scene recognition based on multi-level interactive fusion. Background Technology

[0002] In autonomous driving tasks such as classification, detection, and prediction, it is often necessary to extract relevant information from data acquired from multiple sensors and then perform multimodal data fusion. The fusion result can leverage the differences between different modalities and the complementarity of information, enabling the fused data to provide a more comprehensive and clear description of the target, thereby completing the task quickly.

[0003] Data fusion methods for autonomous driving can be categorized into three levels: pixel-level, feature-level, and decision-level. Pixel-level fusion yields data with more detailed information, such as edge and texture extraction, preserving as much of the original data's information as possible, which is beneficial for further analysis, processing, and understanding. Feature-level data fusion extracts and analyzes features from the original data; the fused features are significantly more accurate than the original data, consuming less memory and boasting faster computation speed. Decision-level data fusion is based on cognition and has the highest level of abstraction, minimizing the impact of noise during data processing. In deep learning-based autonomous driving technology, visual data is typically used for scene recognition tasks, but the resulting features lack richness and diversity. Multi-level interactive fusion of visual and non-visual data can improve scene recognition capabilities. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multimodal data scene recognition method based on multi-level interactive fusion. This method integrates non-visual data collected by sensors with visual data in a multimodal manner, using the information contained in the non-visual data to assist decision-making and improve the accuracy of scene recognition.

[0005] A multimodal data scene recognition method based on multi-level interactive fusion specifically includes the following steps:

[0006] Step 1: Acquire vehicle information through sensors, and convert the vehicle information into a feature vector X using a one-hot encoder. info , where X info ∈R 5×768 A three-layer fully connected neural network is established, and the SeLu activation function is used instead of the ReLu activation function to improve the network's generalization performance. The vehicle-mounted vector X is then used. info As input, the network is enhanced and trained to extract vehicle information features, thereby obtaining the vehicle information vector h. a :

[0007] h a =FN(X info ), h a ∈R 1×512 (1)

[0008] FN represents a feedforward neural network.

[0009] Step 2: Acquire scene video data X using sensors vedio ∈R T×C×H×W Where T represents the number of video frames, C represents the number of channels in the video frame, and H and W represent the length and width of the video frame, respectively. Features are extracted from each frame of the video using the ResNet50 pre-trained network model on ImageNet to obtain the feature vector h of the current frame. k ∈R 1×1024 Where k∈{1,...,T}. Each feature vector is merged in terms of the number of frames to obtain the final feature c∈R of the scene video. T×1024 .

[0010] Step 3: Establish a multi-instance learning model based on a two-stage attention mechanism. Extract 2D-level frame features by judging the richness of information content in each frame. Input the frame features c of the scene video obtained in Step 2 into the network. In the first stage, scores are assigned based on gating attention, and the two video frames with the highest scores are selected. In the second stage, the attention scores are corrected based on the distance between each video frame and the two keyframes, resulting in a 2D-level feature representation of the scene video. The specific steps are as follows:

[0011] s3.1. Establish the attention model for the first stage. The input is the feature vector c of all video frames in step two. Each feature is scored according to gating attention to obtain different attention scores α. k :

[0012]

[0013] Where V and U represent weight matrices, w represents the attention weight matrix, w, V, and U are all learnable parameter matrices, the superscript T denotes the transpose of the matrix, and h j This represents the feature vector for each frame.

[0014] s3.2, based on attention score α k For each video frame, the features are arranged in ascending order, and the two feature vectors h with the highest scores are selected. top1 h top2 , which serves as the keyframe feature of the video scene.

[0015] s3.3 Establish the attention model for the second stage, with the input being the feature vector h of the current frame from step two. k This includes the feature vectors of two keyframes, based on the feature vector h of the current frame. k Calculate query vector q k and information vector v k :

[0016]

[0017] W q and W v All are learnable weight matrices;

[0018] s3.4 Calculate the feature vector h of the current frame. k With two keyframe features h top1 h top2 Calculate the similarity between them, and then calculate the current query vector q. k The distance between keyframes is used as the attention score for the second stage, and as the final attention score. k :

[0019]

[0020] Where <,> represents the inner product, and q mi The query vector q represents the feature vectors of two keyframes. t This represents the query vector for the features of each frame.

[0021] s3.5, based on attention score k For information vector v k By performing weighted summation and dimensionality reduction operations, we finally obtain the 2D-level feature vector h of the scene video. b :

[0022]

[0023] Where L1 represents the fully connected layer operation, h b The network is used to extract 2D-level feature vectors from scene videos.

[0024] Step 4: Stack the scene video data from Step 2 along the frame dimension to divide it into 3D image blocks of equal size. Extract 3D spatiotemporal features using a multi-layer spatiotemporal attention network. First, flatten the data into vectors using convolutional layers, combine this vector with category labeling information, and then combine it with the vehicle information vector X from Step 1. info The two vectors are added together to output an embedding vector containing vehicle information and image information. The specific steps are as follows:

[0025] s4.1 Select X obtained in step two Vedio ∈R T×C×H×WAs input, each video frame is stacked in the frame dimension to obtain a 3D image patch. This image patch can be divided into four equal-sized 3D image patches X along the H and W directions. block , where X block ∈R T×C× (1 / 2)H×(1 / 2) W ;

[0026] s4.2, Move image block X block The input to the convolutional layer is mapped to an image patch vector X. p Based on the known information, a category label vector X of the same dimension is established. class Merge with it and add the vehicle information vector X info The fusion process yields the embedding layer vector X0:

[0027]

[0028]

[0029] This represents the linearized image patch vector. After the image patch is mapped to a vector, it is merged with the category label vector, which aggregates the representational information of the entire label vector, and then combined with the vehicle information vector X. info Add them together to fuse the dual-modal data and complete the embedding layer operation;

[0030] Step 5: Establish a multi-layer spatiotemporal attention network based on self-attention. This network can extract 3D spatiotemporal features h from the scene video by comparing the relationships between different vectors through a multi-head attention mechanism. spatial The specific steps are as follows:

[0031] s5.1 Build the self-attention module SA. Input the embedding layer vector X0 from step four. Transform X0 into three different matrices: query matrix Q, key matrix K, and value matrix V. Calculate the attention output matrix among the three matrices:

[0032]

[0033] Where, d key This represents the dimension of each key vector in the key matrix K.

[0034] s5.2 Build a multi-head attention module (MSA). Divide the embedding layer vector X0 into P parts along the second dimension of the input, calculate the attention function for each part, and finally merge them to obtain the output:

[0035]

[0036] MSA(Q,K,V)=Concat(head d1,...,head dp W O (9)

[0037] in W O It is a trainable parameter matrix, and Concat() represents a stacking operation.

[0038] s5.3. Based on the Multi-Head Attention Module (MSA), a Multi-Layer Perceptron (MLP) is used to output a spatiotemporal feature h that fuses vehicle information vectors. spatial :

[0039] x' l =MSA(LN(x l ))+x l (10)

[0040] x l+1 =MLP(LN(x' l ))+x' l (11)

[0041]

[0042] Where l = 1...L, LN represents the normalization method, L represents the number of layers in the spatiotemporal attention network, and x l Let x' represent the input vector of the l-th layer. l This represents the multi-head attention network of layer l and x. l The summed output vector has the initial input vector being the embedding layer vector X0 from step five. This represents the first vector output by the last layer of the network.

[0043] Step 6: Input the spatiotemporal features from Step 5 into the fully connected layer h. spatial Dimensionality reduction is performed to obtain the spatiotemporal information h of the scene video that incorporates vehicle information. c :

[0044] h c =L2(h spatial ), h c ∈R 1×512 (13)

[0045] L2 represents a fully connected layer for dimensionality reduction operations.

[0046] Step 7: Take the vehicle information vector h obtained in Step 1 a Step 3: Obtain the 2D-level feature vector h b The feature vector h obtained in step six, which contains 3D spatiotemporal information and vehicle information. cThe corresponding query vector and key vector are obtained through the weight matrix, and the similarity loss Loss1 among the three features is calculated. The similarity part is maximized and the difference part is minimized as an interaction method of multimodal features. The specific steps are as follows:

[0047] s7.1, eigenvector h a h b and h c The corresponding query vector q is obtained by calculating the weight matrix respectively. new and key vector k new :

[0048]

[0049]

[0050] s7.2 Calculate the similarity loss (Loss1) between the three feature vectors:

[0051]

[0052]

[0053] Step 8: Following the steps in Step 5, construct a multi-layer feature interaction network based on self-attention, with the feature vector h as the input. a h b and h c Feature interaction is performed, and finally the multimodal fusion feature vector h is extracted. final Input the classification layer to obtain the prediction results. The specific steps are as follows:

[0054] s8.1 Construct a 6-layer, 8-head self-attention feature interaction network to obtain the multimodal fused feature vector h. final ;

[0055] s8.2, transform the feature vector h final Input classification layer W∈R M×512 The classification prediction is obtained through the Softmax function:

[0056]

[0057] Where M represents the classification layer weight, prob = R 1×class This represents the probability of the classification result, and class represents the total number of categories.

[0058] Step 9: Calculate Loss2 by combining the predictions and labels from Step 8 using the cross-entropy loss function. Add this Loss2 to the loss from Step 8 to obtain the overall loss Loss. The specific steps are as follows:

[0059] s9.1 Calculate the loss Loss2 by combining the prediction results with the labels:

[0060] Loss2=-ylog(prob)-(1-y)log(1-prob) (19)

[0061] Where y∈{0,1,...,N} represents the label of the data sample.

[0062] s9.2. Add Loss2 and Loss1 obtained in step eight to get the total loss Loss, and then perform backpropagation for training:

[0063] Loss = Loss1 + Loss2 (20)

[0064] Step 10: Based on the training in Step 9, the optimal classification model is obtained. Input the test set data to obtain the corresponding category labels and complete the classification task.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] Current autonomous driving scene recognition methods mostly rely on visual appearance, rarely considering the use of in-vehicle information as a supplement to scene recognition tasks, and the fusion methods are relatively simple. This method, in terms of multimodal data interaction fusion, extracts features from video information at both 2D and 3D levels, and incorporates in-vehicle information for training during the 3D feature extraction process. It uses a self-attention-based network for interaction and employs contrastive learning loss to identify similarities between each modality, reducing information redundancy. This improves the accuracy and speed of scene recognition in autonomous driving. Attached Figure Description

[0067] Figure 1 This is a flowchart of a scene recognition method based on multimodal data;

[0068] Figure 2 This is a schematic diagram of the feedforward neural network in the embodiment;

[0069] Figure 3 This is a schematic diagram of a network based on a two-stage attention mechanism in the embodiment;

[0070] Figure 4 This is a schematic diagram of the multi-layer attention network in the embodiment; Detailed Implementation

[0071] The present invention will be further explained below with reference to the accompanying drawings;

[0072] like Figure 1 As shown, a multimodal data scene recognition method based on multi-level interactive fusion specifically includes the following steps:

[0073] Step 1: Acquire vehicle information through sensors, including weather, air visibility, vehicle speed, and direction of travel. Convert the vehicle information into a feature vector X using a one-hot encoder. info ∈R 5×768 A three-layer fully connected neural network is established, and the SeLu activation function is used instead of the ReLu activation function to improve the network's generalization performance. The vehicle-mounted vector X is then used. info As input, the network is enhanced and trained to extract vehicle information features, thereby obtaining the vehicle information vector h. a :

[0074] h a =FN(X info ), h a ∈R 1×512 (1)

[0075] Where FN represents a feedforward neural network, such as Figure 2 As shown.

[0076] Step 2: Acquire scene video data X using sensors vedio ∈R 64×3×224×224 The video data has 64 frames, 3 channels, and dimensions of 224 pixels in both width and height. Features are extracted from each frame of the video using a ResNet50 pre-trained network model on ImageNet to obtain the feature vector h of the current frame. k ∈R 1×1024 Where k∈{1,...,64}. Each feature vector is stacked to obtain the final feature c∈R of the scene video. 64×1024 .

[0077] Step 3, as follows Figure 3 As shown, a multi-instance learning model based on a two-stage attention mechanism is established. By judging the information richness of each frame, 2D-level frame features are extracted. The frame features c of the scene video obtained in step two are input into the network. In the first stage, gating attention is used to score the frames, and the two video frames with the highest scores are selected. In the second stage, the attention scores are corrected based on the distance between each video frame and the two keyframes, resulting in a 2D-level feature representation of the scene video. The specific steps are as follows:

[0078] s3.1. Establish the attention model for the first stage. The input is the feature vector c of all video frames in step two. Each feature is scored according to gating attention to obtain different attention scores α. k :

[0079]

[0080] Where V and U represent weight matrices, w represents the attention weight matrix, w, V, and U are all learnable parameter matrices, the superscript T denotes the transpose of the matrix, and h j This represents the feature vector for each frame.

[0081] s3.2, based on attention score α k For each video frame, the features are arranged in ascending order, and the two feature vectors h with the highest scores are selected. top1 h top2 , which serves as the keyframe feature of the video scene.

[0082] s3.3 Establish the attention model for the second stage, with the input being the feature vector h of the current frame from step two. k This includes the feature vectors of two keyframes, based on the feature vector h of the current frame. k Calculate query vector q k and information vector v k :

[0083] q k =W q h k ,v k =W v h k ,k=1,...,64 (3)

[0084] W q and W v All are learnable weight matrices;

[0085] s3.4 Calculate the feature vector h of the current frame. k With two keyframe features h top1 h top2 Calculate the similarity between them, and then calculate the current query vector q. k The distance between keyframes is used as the attention score for the second stage, and as the final attention score. k :

[0086]

[0087] Where <,> represents the inner product, and q mi The query vector q represents the feature vectors of two keyframes. t This represents the query vector for the features of each frame.

[0088] s3.5, based on attention score k For information vector v k By performing weighted summation and dimensionality reduction operations, we finally obtain the 2D-level feature vector h of the scene video. b :

[0089]

[0090] Where L1 represents the fully connected layer operation, h b The network is used to extract 2D-level feature vectors from scene videos.

[0091] Step 4: Stack the scene video data from Step 2 along the frame dimension, dividing it into 3D image blocks of equal size. Figure 4 The multi-layered spatiotemporal attention network shown extracts 3D spatiotemporal features. First, it flattens the data into a vector using convolutional layers, then combines this vector with category labeling information, and finally integrates it with the vehicle information vector X from step one. info The two vectors are added together to output an embedding vector containing vehicle information and image information. The specific steps are as follows:

[0092] s4.1 Select X obtained in step two Vedio ∈R 64×3×224×224 As input, each video frame is stacked in the frame dimension to obtain a 3D image patch. This image patch can be divided into four equal-sized 3D image patches X along the H and W directions. block , where X block ∈R 64×3×112×112 ;

[0093] s4.2, Move image block X block The input to the convolutional layer is mapped to an image patch vector X. p And establish category label vectors X of the same dimension. class Merge with it and add the vehicle information vector X info The fusion process yields the embedding layer vector X0:

[0094]

[0095] Where X0∈R (4+1)×768 X info X represents the vehicle information vector. class X represents the category label vector. p ∈R 1×768 , representing the linearized image patch vector. After the image patch is mapped to a vector, it is merged with the category label vector, which can aggregate the representational information of the entire label vector, and combined with the vehicle information vector X. info Add them together to fuse the dual-modal data and complete the embedding layer operation;

[0096] Step 5: Establish a multi-layer spatiotemporal attention network based on self-attention. This network can extract 3D spatiotemporal features h from the scene video by comparing the relationships between different vectors through a multi-head attention mechanism. spatial The specific steps are as follows:

[0097] s5.1 Build the self-attention module SA. Input the embedding layer vector X0 from step five. Transform X0 into three different matrices: query matrix Q, keyword matrix K, and value matrix V. Calculate the attention output matrix among the three matrices:

[0098]

[0099] Where d key This represents the dimension of each key vector in the key matrix K.

[0100] s5.2 Construct the Multi-Head Attention Module (MSA). The multi-head attention network divides the input X0 along the second dimension into four parts, calculates the attention function for each part separately, and finally merges them to obtain the output:

[0101]

[0102] MSA(Q,K,V)=Concat(head d1 ,...,head d4 W O (9)

[0103] in W O It is a trainable parameter matrix, and Concat() represents a stacking operation.

[0104] s5.3. Based on the Multi-Head Attention Module (MSA), a Multi-Layer Perceptron (MLP) is used to output a spatiotemporal feature h that fuses vehicle information vectors. spatial :

[0105] x' l =MSA(LN(x l ))+x l (10)

[0106] x l+1 =MLP(LN(x' l ))+x' l (11)

[0107]

[0108] Where l = 1...4, LN represents the normalization method, the spatiotemporal attention network has 4 layers, and x l Let x' represent the input vector of the l-th layer. l This represents the multi-head attention network of layer l and x. l The summed output vector has the initial input vector being the embedding layer vector X0 from step five. This represents the first vector output by the last layer of the network.

[0109] Step 6: Input the spatiotemporal features from Step 5 into the fully connected layer h. spatial Dimensionality reduction is performed to obtain the spatiotemporal information h of the scene video that incorporates vehicle information. c :

[0110] h c =L2(h spatial ), h c ∈R 1×512 (13)

[0111] L2 represents a fully connected layer for dimensionality reduction operations.

[0112] Step 7: Take the vehicle information vector h obtained in Step 1 a Step 3: Obtain the 2D-level feature vector h b The feature vector h obtained in step six, which contains 3D spatiotemporal information and vehicle information. c The corresponding query vector and key vector are obtained through the weight matrix, and the similarity loss Loss1 among the three features is calculated. The similarity part is maximized and the difference part is minimized as an interaction method of multimodal features. The specific steps are as follows:

[0113] s7.1, eigenvector h a h b and h c The corresponding query vector q is obtained by calculating the weight matrix respectively. new and key vector k new :

[0114]

[0115]

[0116] s7.2 Calculate the similarity loss (Loss1) between the three feature vectors:

[0117]

[0118]

[0119] Step 8: Following the steps in Step 5, construct a multi-layer feature interaction network based on self-attention, with the feature vector h as the input. a h b and h c Feature interaction is performed, and finally the multimodal fusion feature vector h is extracted. final Input the classification layer to obtain the prediction results. The specific steps are as follows:

[0120] s8.1 Construct a 6-layer, 8-head self-attention feature interaction network to obtain the multimodal fused feature vector h. final ;

[0121] s8.2, transform the feature vector h final Input classification layer W∈R M×512 The classification prediction is obtained through the Softmax function:

[0122]

[0123] Where M represents the classification layer weight, prob = R 1×class This represents the probability of the classification result, and class represents the total number of categories.

[0124] Step 9: Calculate Loss2 by combining the predictions and labels from Step 8 using the cross-entropy loss function. Add this Loss2 to the loss from Step 8 to obtain the overall loss Loss. The specific steps are as follows:

[0125] s9.1 Calculate the loss Loss2 by combining the prediction results with the labels:

[0126] Loss2=-ylog(prob)-(1-y)log(1-prob) (19)

[0127] Where y∈{0,1,...,N} represents the label of the data sample.

[0128] s9.2. Add Loss2 and Loss1 obtained in step eight to get the total loss Loss, and then perform backpropagation for training:

[0129] Loss = Loss1 + Loss2 (20)

[0130] Step 10: Based on the training in Step 9, the optimal classification model is obtained. Input the test set data to obtain the corresponding category labels and complete the classification task.

[0131] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims. Obvious variations or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A multimodal data scene recognition method based on multi-level interactive fusion, characterized in that: Specifically, the following steps are included: Step 1: Encode the vehicle information acquired by the sensors into a feature vector X using one-hot encoding. info ∈R 5×768 Then, three fully connected layers are built as a feedforward neural network FN, using the SeLu activation function, with the vehicle-mounted vector X as the input. info Obtain the vehicle information vector h a ∈R 1×512 : h a =FN(X info ) (0) Step 2: Transfer the scene video data X acquired by the sensor vedio ∈R T×C×H×W Feature extraction is performed frame-by-frame using the ResNet50 network model to obtain the feature vector h of the k-th frame. k ∈R 1×1024 Where T represents the number of video frames, C represents the number of channels, H and W represent the length and width of the video frame, respectively, and k∈{1,...,T}; each feature vector is merged in terms of the number of frames to finally obtain the feature c∈R of the scene video. T×1024 ; Step 3: Establish a multi-instance learning model based on a two-stage attention mechanism. The first-stage attention model, according to the gating attention mechanism, provides the feature vector h for each frame. k Assign attention score α k The second-stage attention model is based on the feature vector h k Calculate query vector q k and information vector v k Then select the attention score α k The two highest eigenvectors h top1 h top2 As a keyframe feature, the query vector q is calculated. k With two keyframe features h top1 h top2 The distance between them is used as the attention score for the second stage. k : Where <,> represents the inner product, and q mi q represents the query vector corresponding to two keyframes. t The query vector represents the features of each frame; based on the attention score. k For information vector v k By performing weighted summation and dimensionality reduction operations, we obtain the 2D-level feature vector h of the scene video. b ∈R 1×512 : Where L1 represents the fully connected layer operation; Step 4: Transfer the scene video data X vedio The frames are stacked into 3D image blocks, and then divided into four equal-sized 3D image blocks X. block ∈R T×C×(1 / 2)H×(1 / 2)W It is mapped to an image patch vector X through a convolutional layer. p ∈R 1×768 Based on the known information, a category label vector X of the same dimension is established. class By merging with it and fusing it with the vehicle information vector, an embedding layer vector X0∈R is obtained. (4+1)×768 ; Step 5: Use the self-attention module SA to transform the embedding layer vector X0 into a query matrix Q, a keyword matrix K, and a value matrix V, and calculate the attention output matrix among the three matrices: Where, d key This represents the dimension of each key vector in the key matrix K; The multi-head attention module (MSA) is used to divide the embedding layer vector X0 into P parts along the second dimension of the input. The attention function of each part is calculated separately, and then the outputs are combined. MSA(Q,K,V)=Concat(head d1 ,...,head dp )W O (6) in W O It is a trainable parameter matrix, and Concat() represents the stacking operation; based on the multi-head attention module MSA, a multilayer perceptron (MLP) is used to output a spatiotemporal feature h that fuses vehicle information vectors. spatial ; Step 6: Combine the spatiotemporal features h that incorporate vehicle information vectors spatial The input is fed into a fully connected layer, and after dimensionality reduction, a feature vector h of 3D-level spatiotemporal information and vehicle information is obtained. c ∈R 1×512 ; Step 7: Take the vehicle information vector h obtained in Step 1 a Step 3: Obtain the 2D-level feature vector h of the scene video. b The feature vector h obtained in step six, which contains 3D-level spatiotemporal information and vehicle information. c The corresponding query vector q is obtained by calculating the weight matrix. new and key vector k new : Calculate the similarity loss Loss1 among the three feature vectors: Step 8: Construct a 6-layer, 8-head self-attention feature interaction network, and input the feature vector h. a h b and h c The feature vector h after multimodal fusion is obtained. final The classification prediction result prob = R is obtained through the Softmax function. 1×class : Among them, W * ∈R M×512 The classification layer weights are represented by `class`, which represents the total number of categories. Step 9: Calculate the loss Loss2 by combining the prediction result prob obtained in Step 8 with the labels y∈{0,1,...,N} of the data samples. Loss2=-ylog(prob)-(1-y)log(1-prob) (12) Then, the network is trained by backpropagation using the overall loss Loss = Loss1 + Loss2; Step 10: Input the vehicle information and video data of the autonomous driving scene to be classified into the classification model trained in Step 9, and obtain the corresponding category label at the output end to complete the classification task.

2. The multimodal data scene recognition method based on multi-level interactive fusion as described in claim 1, characterized in that: The vehicle information acquired by the sensors includes weather, air visibility, vehicle speed, and driving direction.

3. The multimodal data scene recognition method based on multi-level interactive fusion as described in claim 1, characterized in that: The ResNet50 network model is derived from a model pre-trained on ImageNet.

4. The multimodal data scene recognition method based on multi-level interactive fusion as described in claim 1, characterized in that: Attention score α k The calculation method is as follows: Where V and U represent weight matrices, w represents the attention weight matrix, and w, V, and U are all learnable parameter matrices, with superscripts indicating their relative importance. Let h represent the transpose of the matrix. j Represents the feature vector of each frame; Query vector q k and information vector v k The calculation method is as follows: W q and W v All are learnable weight matrices.

5. The multimodal data scene recognition method based on multi-level interactive fusion as described in claim 1, characterized in that: x′ l =MSA(LN(x l ))+x l (16) x l+1 =MLP(LN(x l '))+x l ' (17) Where l = 1...L, LN represents the normalization method, L represents the number of layers in the spatiotemporal attention network, and x l Let x represent the input vector of the l-th layer. l This represents the output vector of the l-th layer, with the initial input vector x0 = X0. This represents the first vector output by the last layer of the network.

Images