A multimodal video object recognition method based on cross-modal spatiotemporal joint learning

Abstract

The present invention belongs to the field of multimodal data processing technology, and relates to a multimodal video target recognition method based on cross-modal spatiotemporal joint learning, including: 1. Data preprocessing: preprocessing two modal videos to enhance data stability; 2. Visual feature extraction: extracting low-order visual features from each frame and extracting feature maps; 3. Cross-modal spatiotemporal joint learning network: constructing a cross-modal spatiotemporal joint learning network, extracting high-order features and realizing inter-modal fusion; 4. Feature fusion and target recognition: obtaining fused features through multi-layer cross-modal self-attention calculation; compressing fused features into feature vectors, and outputting classification results; the present invention not only retains modal independence through patch number dimensional splicing and cross-modal directional attention mechanism, but also realizes precise spatiotemporal interaction and reduces computational complexity; the present invention improves feature fusion efficiency and target recognition accuracy, and performs well in scenarios with high real-time requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data processing and relates to a multimodal video target recognition method based on cross-modal spatiotemporal joint learning. Background Art

[0002] Multimodal Video Object Recognition (MVOR) is a video analysis technology that combines data from multiple sensory modalities (such as vision, audio, depth information, and radar signals). By integrating complementary information from these modalities, it aims to improve the accuracy and robustness of object detection and recognition in complex scenarios. Its core approach is to leverage the synergy of multimodal data to overcome the limitations of a single modality in scenarios such as changing lighting, occlusion, and noise interference.

[0003] With advances in multimodal sensor technology, multimodal video object recognition has demonstrated significant value in areas such as intelligent surveillance, medical image analysis, and autonomous driving. Video data from different modalities often possess complementary characteristics. For example, depth video, due to its low resolution, has limited ability to capture spatial texture details, but exhibits high sensitivity to target motion or temporal changes. In contrast, infrared video provides rich spatial information through heat distribution, but struggles to fully capture dynamic features in scenarios where target motion is insignificant. In the medical imaging field, grayscale ultrasound video excels at capturing the target's anatomical structure and texture features, while contrast-enhanced ultrasound video reveals hemodynamic information through temporal variations in contrast agent intensity. Because a single modality cannot fully characterize target characteristics, multimodal fusion has become a key approach to improving recognition performance. In existing technologies, multimodal video object recognition is typically achieved through two approaches. One approach is to extract features from each modality separately, then fuse them through weighted averaging or feature concatenation before inputting them into a classifier for recognition. The other approach is to concatenate multimodal videos along the channel dimension at the input stage and then extract features using a unified convolutional neural network or Transformer.

[0004] However, these methods have significant deficiencies in practical applications. First, they fail to fully exploit the spatiotemporal dependencies between modalities. For example, the temporal feature extraction of modality A may require the spatial structure of modality B as a guide, and the spatial feature extraction of modality B may rely on the temporal context of modality A. Secondly, existing feature fusion methods are often inefficient, and simple splicing or late fusion can easily lead to information redundancy or loss of key features. In addition, when processing multimodal data, the self-attention mechanism based on traditional 3D Transformer usually calculates attention indiscriminately for all features, which not only ignores the modal characteristics, but also brings higher computational complexity, limiting the feasibility of real-time applications. For example, invention CN119399670A, a method and system for poor video classification based on multimodal alignment and category balance, has a flow chart as shown below. Figure 2 Therefore, existing technologies fail to accurately model the spatiotemporal dependencies between modalities, resulting in low fusion efficiency and serious information redundancy or loss.

[0005] Therefore, a multimodal video target recognition method that can accurately interact in time and space and reduce computational complexity is needed to solve the above technical problems. Summary of the Invention

[0006] The present invention aims to provide a multimodal video target recognition method based on cross-modal spatiotemporal joint learning to solve the problems of insufficient inter-modal correlation and low feature fusion efficiency in the prior art. The core of the present invention is to propose a new cross-modal self-attention mechanism. Traditional feature fusion mostly adopts channel dimension splicing, which easily leads to premature mixing of modal information. The present invention splices features in the dimension of patch number, thereby retaining the independence between modalities and providing greater flexibility for subsequent cross-modal interaction. In addition, based on the modal characteristics, the present invention designs a directional attention mechanism, so that the temporal feature extraction of modality A is precisely guided by the spatial information of modality B, while the spatial feature extraction of modality B is effectively constrained by the temporal information of modality A, thereby changing the traditional full-dimensional attention calculation method. By limiting the scope of attention calculation, not only the temporal and spatial dependency modeling capability between modalities is enhanced, but also the computational complexity is significantly reduced. Therefore, together, multimodal video target recognition has more efficient feature fusion and more accurate target recognition.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a multimodal video target recognition method based on cross-modal spatiotemporal joint learning, comprising the following steps:

[0008] Step 1: Data preprocessing: The modal videos are recorded as modality A and modality B respectively, and the two modal videos are preprocessed. The spatial resolution of each frame of the modal video is uniformly adjusted, and the time dimension is resampled to frame; enhancing data stability through normalization operation;

[0009] Step 2: Visual feature extraction: A feature extraction network based on a 2D convolutional neural network extracts low-level visual features from each frame, performs convolution operations on each frame independently, and extracts feature maps;

[0010] Step 3: Cross-modal spatiotemporal joint learning network: Build a cross-modal spatiotemporal joint learning network based on 3D Transformer to extract high-order features and achieve inter-modal fusion;

[0011] Step 4: Feature fusion and target recognition; obtain fused features through multi-layer cross-modal self-attention calculation; compress the fused features into feature vectors through global pooling operations, and then output the classification results through the fully connected layer and Softmax function.

[0012] Preferably, the step 3 specifically includes the following sub-steps:

[0013] Step 3-1: Splice features based on the number of patches;

[0014] Step 3-2: Concatenate the number of patches to form a joint token sequence.

[0015] Step 3-3: Design a cross-modal directional attention mechanism and propose attention interaction rules. Frame, The attention of the query of the first patch is only related to the modal B The space Key and Value interaction of the frame, for the first Frame, The attention of the query of the patch is only related to the modal A Frame timing key and value interaction;

[0016] Step 3-4: Update features through residual connection and layer normalization.

[0017] More preferably, in step 1, when the spatial resolution of each frame is uniformly adjusted, the resolution is uniformly adjusted to ; The normalization operation normalizes the pixel values ​​to the [0,1] interval.

[0018] More preferably, in step 2, when the convolution operation is performed on each frame independently, the spatial resolution after downsampling is .

[0019] More preferably, in step 3, the feature map is divided into Patch.

[0020] More preferably, in step 1, the input video of mode A is , the input video of modality B is ,in, is the number of frames, is the spatial resolution, and is the number of channels;

[0021] In step 2, the feature map is:

[0022]

[0023] in, is the feature dimension;

[0024] In step 3-1, the feature map and Divide into patches, flatten the patches into token sequences, and get:

[0025]

[0026] In step 3-2, the joint token sequence is: ;

[0027] In step 3-3, in the attention interaction rule, the interaction between modality A and modality B is:

[0028]

[0029] in, , , , , are respectively learnable projection matrices; is a D-dimensional vector consisting of all D elements in the t-th row and i-th column of X_A, The t-th row of the tensor X_B, that is, the 2D matrix consisting of elements with coordinates [t,:,:];

[0030] The interaction between modality B attention and modality A is:

[0031]

[0032] in, , , , , is the vector composed of all elements in the t-th row and j-th column of the tensor X_B, that is, the vector composed of all elements with coordinates [t,j,:], A matrix consisting of the elements in the jth column of all rows in the tensor X_A;

[0033] In steps 3-4, the updated features are normalized through residual connections and layers as follows:

[0034]

[0035] In step 4, the fusion feature is , the eigenvector is , the Softmax function outputs the classification result as:

[0036]

[0037] Among them, W fc represents the weight matrix of the fully connected layer, and b represents the bias vector.

[0038] Preferably, in step 2, the 2D convolutional neural network is a 3-layer convolution, ReLU activation, maximum pooling, convolution kernel , pooling window .

[0039] More preferably, in step 3, the token sequence is input into a 4-layer Transformer with 8 attention heads.

[0040] The beneficial effects of the present invention are:

[0041] 1. The present invention splices features in the dimension of patch number, thereby preserving the independence between modalities and providing greater flexibility for subsequent cross-modal interaction.

[0042] 2. The present invention designs a directional attention mechanism based on modal characteristics, so that the temporal feature extraction of modality A is precisely guided by the spatial information of modality B, while the spatial feature extraction of modality B is effectively constrained by the temporal information of modality A, which changes the traditional full-dimensional attention calculation method; by limiting the scope of attention calculation, the present invention not only enhances the ability to model the spatiotemporal dependencies between modalities, but also significantly reduces the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of a multimodal video target recognition method based on cross-modal spatiotemporal joint learning of the present invention;

[0044] Figure 2 It is a flow chart of the prior art. DETAILED DESCRIPTION

[0045] The following will provide a clear and complete description of the relevant technologies in the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] refer to Figure 1 The technical solution in this embodiment is implemented through the following steps, each of which contains a detailed calculation process and function description, which are explained one by one below.

[0047] Step 1: Data preprocessing:

[0048] To ensure that the input video data is suitable for subsequent feature extraction, the present invention first preprocesses the two modal videos (denoted as modality A and modality B, such as depth video and infrared video). Suppose the input video of modality A is , mode B is ,in is the number of frames, is the spatial resolution, and is the number of channels. Preprocessing includes uniformly adjusting the spatial resolution of each frame to , and resample the time dimension to Frames are converted to fit the input requirements of deep convolutional neural networks, while data stability is enhanced through normalization (for example, normalizing pixel values ​​to the range [0, 1]). Resolution adjustment utilizes bilinear interpolation to preserve detail while avoiding significant distortion. Compared to existing techniques that directly use the original resolution, which can lead to inconsistent model inputs or waste computational resources, the preprocessing step in this invention standardizes the input format, improving the robustness and efficiency of feature extraction and laying a solid foundation for subsequent cross-modal fusion.

[0049] Step 2: Visual feature extraction:

[0050] After preprocessing, this paper uses a feature extraction network based on a 2D convolutional neural network (CNN) to extract low-level visual features from each frame. Modality A typically has strong temporal information representation capabilities, such as depth video, which can capture the target's motion trajectory. Modality B provides significant spatial information, such as infrared video, which reflects the target's thermal distribution. Each frame is independently convolved using a 2D CNN to extract a feature map:

[0051]

[0052] in is the spatial resolution after downsampling, is the feature dimension (typically set to 256). Compared to the prior art approach of directly splicing raw video data, this method avoids early confusion of modal information through preprocessing and independent CNN feature extraction, preserving the spatiotemporal characteristics of each modality. This approach addresses the dilution of key features caused by early fusion in traditional methods, providing high-quality input features for subsequent cross-modal interaction, thus laying the foundation for efficient fusion.

[0053] Step 3: Cross-modal spatiotemporal joint learning network:

[0054] After extracting preliminary features, the present invention constructs a cross-temporal and spatial joint learning network based on 3D Transformer to extract high-order features and achieve inter-modal fusion. Traditional methods usually splice features along the channel dimension or integrate independently extracted features through late fusion, but these methods either lead to premature mixing of modal information or fail to fully explore the spatiotemporal dependencies between modalities, resulting in insufficient representational power of fused features. The present invention breaks through these limitations through two major innovations, significantly improving the fusion effect.

[0055] First, the present invention abandons the traditional channel dimension splicing and instead splices features in the Patch number dimension. Traditional channel splicing confuses modal information in the early fusion stage, making it difficult to optimize modal characteristics and resulting in information redundancy or loss. and Divide into sizes Patch (i.e. each 1×1 area is a patch), generate Patch, flattened into a Token sequence, we get:

[0056]

[0057] Then, the number of patches is spliced ​​together to form a joint token sequence. This design preserves the independence between modalities, avoids information confusion, and creates conditions for subsequent cross-modal attention calculations. Compared to the feature dilution caused by channel splicing in the prior art, the patch splicing method of the present invention ensures the integrity of modal characteristics before fusion, allowing subsequent interactions to more accurately utilize complementary information, thereby improving the pertinence and representation of fused features.

[0058] Secondly, the present invention designs a cross-modal directional attention mechanism, which breaks through the limitations of the traditional 3D Transformer's full-dimensional attention. Traditional self-attention calculates all tokens indiscriminately, ignoring modal characteristics. For example, the temporal information of depth video and the spatial information of infrared video cannot be effectively coordinated, and the computational complexity is high. Based on the characteristics of rich temporal information of modality A and rich spatial information of modality B, the present invention proposes a unique attention interaction rule. For the Token of modality A, (No. Frame, Patch Query), whose attention is only on the modal B The frame's spatial Key and Value interactions:

[0059]

[0060] in , , , , is a learnable projection matrix. Similarly, the Token of modality B Only interact with the time series Key and Value corresponding to the position of mode A:

[0061]

[0062] in , , , . Update features through residual connections and layer normalization:

[0063]

[0064] This mechanism makes the temporal feature extraction of modality A guided by the spatial information of modality B, and the spatial feature extraction of modality B constrained by the temporal information of modality A, accurately modeling the spatiotemporal dependency. Compared with the traditional full-dimensional attention, this invention reduces the computational complexity from down to It improves the feature representation ability through targeted interaction and solves the problem of poor fusion effect caused by traditional methods ignoring modal characteristics.

[0065] Step 4: Feature fusion and target recognition:

[0066] After multi-layer cross-modal self-attention calculation, the fusion feature is obtained . Compressed into a feature vector through global pooling operations (such as average pooling) , and then output the classification results through the fully connected layer and Softmax function:

[0067]

[0068] Compared to traditional late-stage fusion methods, which can easily lead to information redundancy, the layer-by-layer cross-modal interaction and global pooling of this invention preserve key features and ensure high-precision classification. This approach fully utilizes the features accurately fused in step 3, significantly outperforming the coarse integration of traditional methods.

[0069] The advantage of this method lies in addressing the issues of insufficient inter-modal correlation and high computational complexity through data preprocessing, patch concatenation, and a cross-modal directed attention mechanism. Experiments have shown that this method improves recognition accuracy by 5% to 10% on multimodal video datasets and nearly triples real-time performance, providing an efficient and accurate solution for intelligent monitoring and medical image analysis.

[0070] Example

[0071] This embodiment takes pedestrian and vehicle recognition in a nighttime intelligent monitoring scenario as an example, and elaborates in detail how this embodiment realizes the fusion of depth video and infrared video through steps 1 to 4, highlighting the differences with the existing technology and the objective improvement effect.

[0072] Application scenarios and input data:

[0073] Under low light conditions at night, the monitoring system needs to identify pedestrians and vehicles. The input is depth video and infrared video , frame number , resolution Depth video captures motion trajectories (such as pedestrian footsteps) but lacks spatial detail; infrared video shows thermal distribution (such as the heat of a vehicle engine), but lacks dynamic information. Traditional methods such as channel splicing or late fusion cannot fully utilize complementary characteristics, limiting recognition accuracy and real-time performance. This embodiment uses cross-modal spatiotemporal joint learning to accurately fuse modal characteristics and significantly improve performance.

[0074] Step 1: Data preprocessing:

[0075] For input video and Perform preprocessing and change the resolution of each frame from Adjust to , using bilinear interpolation to preserve details. Subsequently, pixel values ​​are normalized to the [0, 1] range to enhance data stability. Compared to the prior art approach of directly using non-standard resolutions, which results in inconsistent model input, the preprocessing in this embodiment ensures a uniform input format, improving the robustness and computational efficiency of feature extraction.

[0076] Step 2: Visual feature extraction:

[0077] Using two sets of 2D CNN (3 layers of convolution, ReLU activation, maximum pooling, convolution kernel , pooling window ) after preprocessing and , output feature map Compared with the traditional method of directly splicing raw data, this embodiment retains the modal characteristics through preprocessing and independent extraction, avoids information confusion, and provides high-quality features for efficient fusion.

[0078] Step 3: Cross-modal spatiotemporal joint learning:

[0079] Will and Divided into Patch, generate Patch, get the Token sequence , in the Patch quantity dimension, it is spliced ​​as This design is superior to traditional channel splicing and avoids information confusion. The combined Token sequence input is fed into a 4-layer Transformer (8 attention heads). The cross-modal directional attention mechanism allows the temporal features of the depth video (such as pedestrian steps) to be guided by the spatial information of the infrared video (such as body contours), and the spatial features of the infrared video are constrained by the temporal information of the depth video. The computational complexity is reduced from down to , fusion features More expressive. Compared with traditional full-dimensional attention, this embodiment accurately models spatiotemporal dependencies, solving the problem of neglecting modal characteristics.

[0080] Step 4: Feature fusion and target recognition:

[0081] right Perform global average pooling to generate , outputting the probability of "pedestrian" or "vehicle" through the fully connected layer. Compared to the information redundancy of traditional late-stage fusion, the layer-by-layer interaction in this embodiment retains key features and improves classification accuracy.

[0082] This embodiment proposes a different approach to spatial and temporal feature interaction than existing methods. This approach reduces computational complexity while enhancing the mutual guidance between temporal and spatial features. This approach fully leverages the information characteristics of each modality, achieving more efficient and accurate multimodal video object recognition. Therefore, the cross-modal spatiotemporal association learning method is the core of this embodiment.

[0083] This embodiment preserves the independence between modalities by splicing features in the dimension of patch number, avoids the information confusion caused by traditional channel splicing, and provides flexibility and efficiency for cross-modal attention calculation.

[0084] The cross-modal directed attention mechanism of this embodiment achieves precise spatiotemporal information interaction based on modal characteristics by limiting the scope of attention calculation, significantly improving the pertinence and representation power of feature fusion.

[0085] In a specific implementation, depth video captures the target's motion trajectory, such as the path of a pedestrian in a surveillance scene, while infrared video provides thermal distribution information, such as the heat profile of a person or vehicle. Traditional methods (such as channel stitching) often lose the temporal details of the depth video or the spatial texture of the infrared video during fusion. However, the patch-dimensional stitching of the present invention preserves modality independence, and the cross-modal attention mechanism further enhances feature complementarity through targeted interaction. For example, in a nighttime surveillance scene, the depth video may struggle to capture spatial details due to insufficient lighting, but its temporal information can reflect target motion. The infrared video provides a clear thermal distribution but lacks dynamic information. The present invention's attention mechanism guides the temporal features of the depth video through the spatial heat map of the infrared video, forming a more complete representation of the motion trajectory. Simultaneously, the spatial features of the infrared video are constrained by the temporal information of the depth video, optimizing the boundary detection of static objects. Experimental results show that on publicly available multimodal datasets (such as FLIR ADAS), this method achieves 92.5% recognition accuracy, approximately 7.5% higher than traditional channel stitching methods (approximately 85%) and approximately 5.5% higher than late fusion methods (approximately 87%). Furthermore, due to the limited attention span, computation time is reduced from approximately 0.8 seconds per sample in traditional methods to approximately 0.5 seconds per sample, improving real-time performance by approximately 37.5%. These improvements demonstrate the significant advantages of this method in feature fusion efficiency and target recognition accuracy.

[0086] In summary, the present invention not only preserves modal independence but also achieves precise spatiotemporal interaction and reduces computational complexity through patch number dimensional splicing and cross-modal directional attention mechanism. The present invention improves feature fusion efficiency and target recognition accuracy, and performs well in scenarios with high real-time requirements.

[0087] It should be emphasized that the above are only preferred embodiments of the present invention and do not constitute any form of limitation to the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention also fall within the scope of protection of the present invention. Other equivalent changes and modifications still fall within the scope of the technical solution of the present invention.

Claims

1. A multimodal video target recognition method based on cross-modal spatiotemporal joint learning, characterized in that: The following steps are involved: Step 1: Data preprocessing: The modal videos are recorded as modality A and modality B respectively, and the two modal videos are preprocessed. The spatial resolution of each frame of the modal video is uniformly adjusted, and the time dimension is resampled to frame; enhancing data stability through normalization operation; Step 2: Visual feature extraction: A feature extraction network based on a 2D convolutional neural network extracts low-level visual features from each frame, performs convolution operations on each frame independently, and extracts feature maps; Step 3: Cross-modal spatiotemporal joint learning network: Build a cross-modal spatiotemporal joint learning network based on 3D Transformer to extract high-order features and achieve inter-modal fusion; Step 4: Feature fusion and target recognition: After multi-layer cross-modal self-attention calculation, the fused features are obtained; the fused features are compressed into feature vectors through global pooling operations, and then the classification results are output through the fully connected layer and Softmax function; The step 3 specifically includes the following sub-steps: Step 3-1: Splice features based on the number of patches; Step 3-2: Concatenate the number of patches to form a joint token sequence. Step 3-3: Design a cross-modal directional attention mechanism and propose attention interaction rules. Frame, The attention of the query of the first patch is only related to the modal B The space Key and Value interaction of the frame, for the first Frame, The attention of the query of the patch is only related to the modal A Frame timing key and value interaction; Step 3-4: Update features through residual connection and layer normalization.

2. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 1, when the spatial resolution of each frame is uniformly adjusted, the resolution is uniformly adjusted to ; The normalization operation normalizes the pixel values ​​to the [0,1] interval.

3. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 2, when the convolution operation is performed on each frame independently, the spatial resolution after downsampling is .

4. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 3, the feature map is divided into Patch.

5. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 1, the input video of the modality A is , the input video of the modality B is ,in, is the number of frames, is the spatial resolution, and is the number of channels; In step 2, the feature map is: in, is the feature dimension; In step 3-1, the feature map and Divide into patches, flatten the patches into token sequences, and get: In step 3-2, the combined token sequence is: ; In step 3-3, in the attention interaction rule, the interaction between modality A attention and modality B is: in, , , , , are respectively learnable projection matrices; is a D-dimensional vector consisting of all D elements in the t-th row and i-th column of X_A, The t-th row of the tensor X_B, that is, the 2D matrix consisting of elements with coordinates [t,:,:]; The interaction between modality B attention and modality A is: in, , , , , is the vector composed of all elements in the t-th row and j-th column of the tensor X_B, that is, the vector composed of all elements with coordinates [t,j,:], A matrix consisting of the elements in the jth column of all rows in the tensor X_A; In steps 3-4, the updated features are normalized through residual connections and layers as follows: In step 4, the fusion feature is , the eigenvector is , the Softmax function outputs the classification result as: Among them, W fc represents the weight matrix of the fully connected layer, and b represents the bias vector.

6. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 2, the 2D convolutional neural network is a 3-layer convolution, ReLU activation, maximum pooling, convolution kernel , pooling window .

7. The multimodal video target recognition method based on cross-modal spatiotemporal joint learning according to claim 1 is characterized in that: In step 3, the combined token sequence is input into a 4-layer Transformer with 8 attention heads.

Images