Dynamic target recognition system and method of heterogeneous sensors under multi-modal visual large model

Abstract

The present application relates to a kind of multi-modal visual large model under the dynamic target identification system of heterogeneous sensor anti-interference.The system includes: anti-background interference module, by modality convolution to multi-modal data of heterogeneous sensor, LSTM time series modeling and weighted fusion generation space-time feature, combine time / space attention mechanism and Transformer reinforced feature, use Gaussian probability density model to filter foreground pixel and extract foreground feature;Multi-modal processing module, separate each modality foreground feature and splice after refining, by Transformer extract consistent feature and complementary feature, after fusion and pooling enhancement generation reinforced fusion feature;Target identification module, fusion foreground feature and reinforced fusion feature, after convolution compression and Timesformer long time series modeling, input target detection head, output target classification and boundary box.The present application solves the problem of heterogeneous sensor data interference suppression and dynamic target accurate identification, improves robustness in complex scene.

Description

Technical Field

[0001] This invention belongs to the field of multimodal data target recognition technology, specifically relating to a dynamic target recognition system and method for heterogeneous sensors with anti-interference under a large multimodal visual model. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, dynamic target recognition has become a core technology in many application fields, especially in autonomous driving, intelligent monitoring, robot navigation, and drone tracking. In these applications, complex background changes, dynamic changes in targets, and data interference and fusion problems from multiple heterogeneous sensors pose significant challenges to the accuracy of target recognition. Especially in variable, complex, and long-term environments, traditional static target recognition and single-modal recognition often result in false positives and false negatives. To improve the accuracy of dynamic target recognition, there is an urgent need for a dynamic target recognition technology that can adaptively suppress complex dynamic backgrounds, fully integrate multimodal information from multiple heterogeneous sensors, and maintain target tracking stability over long time.

[0003] First, existing target recognition technologies struggle to overcome interference from dynamic and complex backgrounds. Traditional dynamic target detection algorithms primarily rely on convolutional neural networks, requiring stable backgrounds, stable light sources, and relatively uniform target appearance. Even the YOLO algorithm framework, with its end-to-end spatial regression, struggles to effectively handle this. For example, patent CN119832219A discloses a lightweight aerial image target detection method that improves YOLOv11. This method replaces the original sampling convolution with a lightweight DBRDown to reduce computational cost, inserts MSFLB blocks, and uses FPIoU loss to simultaneously constrain IoU and corner distances, enabling more accurate identification of distant targets from aerial images. However, this patent lacks detailed research on the interference issues in dynamic and complex environments such as strong light and background motion, failing to consider external factors. Nevertheless, handling complex background interference is crucial for dynamic target recognition, thus requiring specialized processing to improve model accuracy.

[0004] Secondly, traditional target recognition technologies are not strong in processing multimodal data. Traditional YOLO primarily targets single-frame RGB images and has poor capabilities for multimodal data recognition. Existing improved multimodal target recognition systems still face the dilemma of balancing consistency and complementarity in data fusion. On the one hand, RGB, depth images, and sensor data from IMUs have different spatiotemporal resolutions and coordinate systems, often lacking precise alignment. On the other hand, traditional fusion strategies often employ static stitching or simple weighting, failing to dynamically allocate source weights based on the scene, resulting in the dilution of the advantageous information of each modality and hindering the full utilization of complementary characteristics. Therefore, it is necessary not only to classify and extract consistency information from multimodal data but also to explore the complementary relationships between different modalities. Only in this way can the accuracy of multimodal data fusion be significantly enhanced, thereby improving recognition accuracy.

[0005] Finally, existing target recognition technologies focus on modeling short-term data and cannot effectively handle long-term dependencies. As targets continue to move, short-term methods often ignore long-term changes in the target, leading to tracking instability or decreased accuracy. For example, patent CN113191339A discloses a method and system for monitoring foreign object intrusion into a track based on video analysis. It acquires track video in real time through a camera, removes rain interference from the video, and models multi-mode backgrounds. Then, it performs dynamic target detection and deep learning-based foreign object classification, achieving all-weather, real-time, and accurate foreign object alarms. However, it is still based on short-term methods, emphasizing the replacement strategy of "old frames and new frames," and does not utilize the temporal dependencies of targets or noise over longer time spans. In long-term tracking, it is still prone to drift, false detections, and missed detections. Therefore, it is necessary to introduce deep temporal networks capable of long-term feature extraction and modeling to comprehensively improve the stability and accuracy of dynamic target recognition. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a multimodal vision large-scale model heterogeneous sensor anti-interference dynamic target recognition system, comprising:

[0007] The anti-background interference module is used to preprocess multimodal data from heterogeneous sensors to obtain time-series samples of multimodal data. The data in the time-series samples are convolved according to modality category to obtain initial features for each modality. These initial features are then input into an LSTM to obtain spatiotemporal features for each modality. The spatiotemporal features of each modality are weighted to obtain the spatiotemporal features of the multimodal data. Temporal and spatial attention are extracted from the spatiotemporal features of each modality, and these attention mechanisms are used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features are then input into the multi-head attention mechanism in the Transformer for feature enhancement, resulting in enhanced spatiotemporal features of the multimodal data. A Gaussian distribution is used to establish a pixel probability density model, which is then used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability density of all pixels is compared with a probability density threshold, and foreground pixels are selected based on the comparison results. Feature extraction is then performed on the foreground pixels to obtain foreground features.

[0008] The multimodal processing module is used to separate the foreground features of various modalities from the foreground features, refine the foreground features of various modalities to obtain refined foreground features of various modalities, concatenate the refined foreground features of various modalities to obtain refined foreground features of multimodal data, input the refined foreground features of multimodal data into Transformer to extract the consistency features of multimodal data and the complementary features of various modalities, fuse the consistency features of multimodal data and the complementary features of various modalities to obtain fused features, pool the fused features, and use the pooled fused features to enhance the fused features themselves to obtain enhanced fused features;

[0009] The target recognition module is used to concatenate foreground features and enhanced fusion features to obtain concatenated features, perform convolutional compression on the concatenated features, input the convolutionally compressed concatenated features into Timesformer to obtain concatenated features with long temporal dependencies, input the concatenated features with long temporal dependencies into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

[0010] Furthermore, in the anti-background interference module, the specific method for convolving the data in the time-series samples of multimodal data according to modality category to obtain the initial features of each type of modality data is as follows:

[0011] The modal categories covered by the multimodal data include RGB video frames, depth information, and time-series IMU.

[0012] Channel-wise convolution is used on RGB video frames, depth information, and time-series IMU data, with different channels used according to modality. Specifically, 3D convolution is used to extract spatiotemporal features from RGB video frames and depth information, while 1D convolution is applied to extract temporal features from the IMU data. Batch normalization and ReLU activation are used for optimization during convolution, and max pooling is used to maintain consistent dimensionality in the generated data. The initial feature Y of the generated RGB video frames is then used. RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU .

[0013] Furthermore, in the anti-background interference module, the initial features of various modal data are input into an LSTM to obtain the spatiotemporal features of various modal data. The spatiotemporal features of various modal data are weighted to obtain the spatiotemporal features of multimodal data. Temporal attention and spatial attention are extracted from the spatiotemporal features of various modal data. These temporal and spatial attention are then used to optimize the spatiotemporal features of the multimodal data. Finally, the optimized spatiotemporal features of the multimodal data are input into the multi-head attention mechanism in the Transformer for feature enhancement. The specific method for obtaining the enhanced spatiotemporal features of the multimodal data is as follows:

[0014] Using a single-layer LSTM network to generate the initial features Y of RGB video frames RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU Temporal feature mapping is performed, uniformly mapping to a subspace to obtain the spatiotemporal features F of various modal data. m The formula is as follows:

[0015]

[0016] Where m is the modal category, θ m λ represents the LSTM weight parameters. m The learning rate;

[0017] The spatiotemporal features O of the multimodal data are generated by weighting using the following formula:

[0018]

[0019] Where ε is a parameter used to prevent zero-painting, w m The weights w for different class modes m are assigned by the Adam optimizer. m Update the weights w of category m modality. m The optimization is dynamically adjusted based on environmental interference;

[0020] Feature enhancement is performed using the multi-head attention mechanism in the transformer, as detailed below:

[0021] Calculate spatial attention for the spatiotemporal features F of RGB video frames. RGB Spatiotemporal features F of depth information Depth The sum of the two is then subjected to spatial average pooling to generate the spatial average pooling result S. special ;

[0022] Calculate temporal attention for the spatiotemporal features F of RGB video frames. RGB The spatiotemporal characteristics F of depth information Depth Spatiotemporal characteristics F of time series IMU IMUh The sum of the three is then subjected to time-averaged pooling to generate the time-averaged pooling result S. temproal ;

[0023] The spatiotemporal features O of the multimodal data are weighted again to obtain the optimized spatiotemporal features Y of the multimodal data. STA The formula is as follows:

[0024]

[0025] The spatiotemporal features Y of the optimized multimodal data STA The input is fed into the multi-head attention mechanism in the transformer to obtain the spatiotemporal features Y of the optimized multimodal data. STA Enhanced spatiotemporal features Y of consistent multimodal data CM-STA .

[0026] Furthermore, in the anti-background interference module, a Gaussian distribution is used to establish a pixel probability density model, and this model is used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability density of all pixels is compared with a probability density threshold, and foreground pixels are selected from all pixels based on the comparison results. The specific method for extracting features from the foreground pixels to obtain foreground features is as follows:

[0027] Initialize 3 Gaussian distributions π k, μ k σ k 2 For each phase pixel, model the pixel probability density model P(r):

[0028]

[0029] Where N is the Gaussian probability density function, π k μ is the weight of the k-th Gaussian component. kLet σ be the mean of the k-th Gaussian component. k 2 Let r be the variance of the k-th Gaussian component, and r be the pixel feature value. The pixel feature value r is derived from the enhanced spatiotemporal feature Y of the multimodal data. CM-STA ;

[0030] The weight π of the k-th Gaussian component k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 All updates are performed using the EM algorithm, and the specific steps are as follows:

[0031] For the E-step, the posterior probability P(k|r) is defined as the probability that the pixel feature value r belongs to the k-th Gaussian component:

[0032]

[0033] For M steps, update the weight π of the k-th Gaussian component. k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 :

[0034]

[0035] Where M represents the total number of pixels, r i π represents the feature value of the i-th pixel. k t+1 μ k t+1 σ k 2,t+1 These represent the weight, mean, and variance of the k-th Gaussian component after iteration, respectively.

[0036] To generate a background mask, select the Gaussian component with the highest weight and lowest variance to define the background. Calculate the current P(k|r). If it is greater than the threshold γ, the pixel feature value r is identified as a background pixel; otherwise, it is identified as a foreground pixel.

[0037]

[0038]

[0039] Among them, Y bg Y represents background features. fg Indicates foreground features.

[0040] Furthermore, in the multimodal processing module, foreground features of various modalities are separated from the foreground features. After feature refinement of the foreground features of each modality, refined foreground features of each modality are obtained. The specific method for concatenating the refined foreground features of each modality into refined foreground features of the multimodal data is as follows:

[0041] According to the modal category, the corresponding channel is used to select the foreground feature Y. fg The foreground features of RGB video frames, the foreground features of depth information, and the foreground features of the time-series IMU are separated from the data.

[0042] The foreground features of RGB video frames are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of depth information (Depth) are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of the time-series IMU are subjected to 1D convolution, batch normalization, and ReLU activation to generate refined foreground features Y of RGB video frames. RGB Refined foreground features Y of depth information Depth Refined foreground features Y of depth information IMU ;

[0043] Refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMU Concatenating along the channel dimension, as shown below:

[0044]

[0045] Where Y0 represents the refined foreground features of the multimodal data.

[0046] Furthermore, in the multimodal processing module, the refined foreground features of the multimodal data are input into the Transformer to extract the consistency features and complementary features of various modalities. The consistency features and complementary features of various modalities are then fused to obtain fused features. The fused features are then pooled, and the pooled fused features are used to enhance the fused features themselves to obtain enhanced fused features. The specific method is as follows:

[0047] The refined foreground features Y0 of the multimodal data are input into the Transformer to extract the consistency features h of the multimodal data. con The refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMUInput each element into a Transformer and extract the complementary features h of the RGB video frames. RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU com ;

[0048] The first weight α and the second weight β are learned through the Adam optimizer, and the consistency feature h is fused. con Complementarity features of RGB video frames h RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU com Where α and β are both softmax(MLP(h)), and the multilayer perceptron (MLP) is a two-layer fully connected network, represented as follows:

[0049]

[0050] Where H represents the fusion feature;

[0051] The fusion feature H is subjected to global average pooling to generate the fusion weight S. FA Using fusion weight S FA Enhance the fusion feature H to generate a stronger fusion feature Y. FA The calculation formula is as follows:

[0052] .

[0053] Furthermore, in the target recognition module, the method for concatenating foreground features and enhancement fusion features to obtain concatenated features, performing convolutional compression on the concatenated features, and inputting the convolutionally compressed concatenated features into Timesformer to obtain concatenated features with long-term dependencies is as follows:

[0054] Foreground feature Y fg and fusion feature Y FA By splicing the data, we obtain the splicing feature Y. input For splicing feature Y input Perform convolution compression to obtain Y dim The formula is as follows:

[0055]

[0056] Among them, Con 1×1 This indicates a one-dimensional convolution operation with a 1*1 kernel, where K represents the kernel and b represents the bias term.

[0057] Y dim Inputting Timesformer, the independent temporal and spatial attention mechanisms within Timesformer are used to capture long-term dependencies, generating a concatenated feature Y with long-term dependencies. transformer The formula is shown below:

[0058]

[0059] Where, θ TS This represents the complete set of trainable parameters for Timesformer.

[0060] Furthermore, in the target recognition module, the specific method for inputting concatenated features with long-term time dependence into the target detection head to obtain the classification result of the target in the foreground and the bounding box of the target is as follows:

[0061] The splicing feature Y with long-term dependence transformer Enter the YOLO object detection head to generate the bounding box y of the target. t The classification probability p of the target t ;

[0062] The splicing feature Y with long-term dependence transformer Input the Gaussian uncertainty detection head to obtain the mean μ and variance α. 2 The concatenated feature Y with long-term continuation dependency is output through 1×1 convolution. transformer The mean (μ) of the center coordinates (x, y) x ,μ y The method calculates the squared deviation between the local coordinates and the center coordinates (x, y) using a 1×1 convolution, and then smooths the result using the Softplus function, outputting the mean and variance (σ) of the center coordinates. x ,σ y );

[0063] The specific steps for optimizing the loss function of the TimeSformer model and the YOLO object detection model are as follows:

[0064] For Timesformer, the loss function consists of spatiotemporal consistency loss and Gaussian uncertainty loss. For spatiotemporal consistency loss, within a given sliding window time T, the bounding box y at time t+1 should be used. t+1 The bounding box y at time t t The formula for smoothing loss is as follows:

[0065]

[0066] Among them, y t+1 and yt L represents the bounding boxes at time t+1 and time t, respectively. smooth This represents the spatiotemporal consistency smoothing loss;

[0067] For Gaussian uncertainty loss L Gaussian The formula is as follows:

[0068] ;

[0069] For the object detection model YOLO, the bounding boxes use the CIoU loss L. CIoU The classification probability uses cross-entropy loss L. cls The formula is as follows:

[0070]

[0071]

[0072] Where IoU is the sum of the predicted bounding box d and the ground truth bounding box d. gt The intersection-union ratio (IoU), ρ is the Euclidean distance between the predicted center and the ground truth center, c is the diagonal of the smallest rectangle containing both the predicted and ground truth boxes, v is the aspect ratio consistency, α is the aspect ratio consistency weighting factor, and y i ' is the true label of the i-th type of target, p i ' is the probability of predicting the target as the i-th type.

[0073] Until CIoU loses L CIoU and cross-entropy loss L cls Once convergence is achieved, the optimized target classification result and target bounding box are output.

[0074] A method for anti-interference dynamic target recognition using heterogeneous sensors under a multimodal vision large model includes:

[0075] Multimodal data from heterogeneous sensors is preprocessed to obtain time-series samples of multimodal data. Data from these time-series samples are convolved according to modality category to obtain initial features for each modality. These initial features are then input into an LSTM to obtain spatiotemporal features for each modality. The spatiotemporal features of each modality are weighted to obtain the spatiotemporal features of the multimodal data. Temporal and spatial attention are extracted from these spatiotemporal features. These attention mechanisms are then used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features are then input into a multi-head attention mechanism in a Transformer for feature enhancement, resulting in enhanced spatiotemporal features. A Gaussian distribution is used to establish a pixel probability density model, which is then used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability densities of all pixels are compared with a probability density threshold. Based on the comparison results, foreground pixels are selected from all pixels, and feature extraction is performed on these foreground pixels to obtain foreground features.

[0076] The process involves separating foreground features from various modalities, refining these features, concatenating them to form a multimodal foreground feature, inputting the multimodal foreground feature into a Transformer to extract consistency features and complementary features from various modalities, fusing these features to obtain a fused feature, pooling the fused feature, and using the pooled fused feature to enhance the fused feature itself to obtain an enhanced fused feature.

[0077] The foreground features and enhancement fusion features are concatenated to obtain concatenated features. The concatenated features are then compressed by convolution. The convolutionally compressed concatenated features are then input into Timesformer to obtain concatenated features with long temporal dependencies. The concatenated features with long temporal dependencies are then input into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

[0078] A computer program product includes a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the above-described method for anti-interference dynamic target recognition of heterogeneous sensors under a large multimodal vision model.

[0079] The beneficial effects of this invention are as follows:

[0080] 1. This invention, based on an anti-background interference module, uses a long short-term recurrent neural network feature mapping, adaptive weight optimization, and a multi-Gaussian mixture model for background modeling. This effectively suppresses interference from strong light and complex dynamic environments, significantly improving the extraction capability of foreground temporal features. This method can accurately identify multimodal features of targets in complex dynamic backgrounds, avoiding the false detection and false negative problems caused by large background changes or unstable lighting in traditional methods.

[0081] 2. This invention utilizes a multimodal processing module, employing multi-channel convolution and a multimodal visual large-model encoder to fuse consistency and complementary features, enhancing the spatiotemporal consistency between different modalities, optimizing the accuracy of multimodal data fusion, and effectively solving the challenge of heterogeneous sensor multimodal data fusion. This technology can effectively improve target recognition accuracy in dynamic and complex environments, especially in the processing and fusion of multimodal data from heterogeneous sensors such as RGB, depth images, and temporal data.

[0082] 3. This invention employs a temporal feature enhancement module, which effectively captures long-term temporal dependencies spanning seconds by introducing a long-term spatiotemporal attention mechanism model, thus optimizing the stability and accuracy of dynamic target tracking. Furthermore, by combining it with a Gaussian uncertainty loss function, it further enhances spatiotemporal consistency, effectively solving the problem that traditional short-term methods cannot capture long-term dependencies, and significantly improving the detection accuracy of dynamic targets. Attached Figure Description

[0083] Figure 1 This is a system block diagram of the present invention.

[0084] Figure 2 This is a schematic diagram of the process for generating the foreground using a Gaussian mixture model in the background interference suppression module.

[0085] Figure 3 A flowchart illustrating the modeling of the long-term spatiotemporal attention mechanism and the Gaussian uncertainty algorithm in the target recognition module.

[0086] Figure 4 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0087] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0088] In industrial manufacturing scenarios, the working environment is often accompanied by uncertainties such as strong light reflection, dust interference, and equipment movement, which can lead to overexposure, blurring, or distortion of the target area captured by the front-end image sensor, severely affecting the system's recognition accuracy. Therefore, in multi-station robotic collaborative operations, complex background changes are the main bottleneck for system recognition stability. Although current mainstream target recognition methods have achieved good performance in static image target detection, they generally rely on the premise of stable lighting, minimal background changes, and consistent target shape, making it difficult to cope with complex dynamic background interference such as strong light, shaking, and the coexistence of multiple targets, easily resulting in missed detections and false detections. Research shows that target recognition technology with anti-dynamic interference, multimodal collaborative perception, and long-term modeling capabilities has broad market demand and application prospects. To address the above challenges, the dynamic target recognition anti-interference system proposed in this patent does not simply integrate multiple deep modules, but is based on scene perception design, systematically introducing an attention mechanism to improve environmental adaptability and recognition stability. To address the dynamic background interference problem, the system integrates a modal adaptive mechanism based on a long short-term recurrent neural network and a Gaussian mixture modeling method to achieve effective differentiation between foreground and background.

[0089] In robotic arm grasping operations, the system needs to continuously track and identify dynamically moving targets over long periods. During this process, the target may experience occlusion, rotation, displacement, or even be misidentified as background. Traditional recognition algorithms often make judgments based on short time frame windows, ignoring the long-term changing trends of the target throughout the sequence, resulting in a lack of temporal continuity in the detection results, specifically manifested as the target skipping frames or suddenly disappearing. Furthermore, existing methods typically do not incorporate quantitative modeling of detection uncertainty, making it difficult to evaluate the reliability of the system output in complex dynamic environments. To address the temporal consistency problem in target tracking, this system introduces a long-term spatiotemporal attention mechanism model based on self-attention, employing a spatiotemporal decoupling structure to capture dynamic dependencies across seconds, thereby ensuring the continuity and stability of the recognition results in the temporal dimension. Simultaneously, a Gaussian uncertainty loss function is designed, explicitly modeling the mean and variance of the target bounding box, enabling the system to handle uncertainty regions with fault tolerance and risk assessment, further improving the robustness and reliability of target detection.

[0090] In intelligent security inspection of rail transit, single-modal data often struggles to reliably extract effective features due to reflective target materials, complex structures, or environmental occlusion. Therefore, engineering practice commonly employs multi-sensor fusion schemes, including RGB, depth information, and IMU, to improve the system's robustness under different operating conditions. However, the data acquired by these heterogeneous sensors exhibit significant differences in sampling frequency, spatial dimension, and data structure, making conventional feature stitching or weighted fusion methods inaccurately represent the advantages of each modality. This can even lead to information redundancy and modal conflicts, limiting recognition performance. Current multimodal perception technologies lack a universal and efficient fusion architecture, making it difficult to simultaneously extract shared information and differential features between modalities, thus limiting the system's anti-interference capability in complex environments. Addressing this research gap, this system designs a structured multimodal processing module. First, channel convolution is performed on RGB, depth, and IMU data to extract local features within each modality. Subsequently, a multimodal visual large-model encoder models the consistency and complementarity information between modalities, performing deep fusion at the feature layer. Ultimately, by integrating attention mechanisms and global pooling operations to enhance information representation capabilities, high-precision and dynamic adaptive fusion of multimodal data is achieved, effectively improving the system's recognition performance in complex scenarios.

[0091] Example 1

[0092] like Figure 1 As shown, a multimodal vision large-scale model heterogeneous sensor anti-interference dynamic target recognition system includes:

[0093] The background interference resistance module is used to preprocess multimodal data from multiple heterogeneous sensors (in scenarios such as intelligent industrial manufacturing and rail transit monitoring, heterogeneous sensors such as image vision, LiDAR, and infrared acquire dynamic time-series data of heterogeneous sensing multimodal data such as video frames (RGB), depth, and time (IMU) respectively) to obtain time-series samples of multimodal data. The data in the time-series samples of multimodal data are convolved according to modality category to obtain initial features of each modality. These initial features are then input into an LSTM to obtain the spatiotemporal features of each modality. Finally, the spatiotemporal features of each modality are weighted to obtain the spatiotemporal characteristics of the multimodal data. Features are extracted from the spatiotemporal features of various modal data, including temporal and spatial attention. Temporal and spatial attention are then used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features of the multimodal data are then input into the multi-head attention mechanism in the Transformer for feature enhancement, resulting in enhanced spatiotemporal features of the multimodal data. A Gaussian distribution is used to establish a pixel probability density model, and this model is used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability density of all pixels is compared with a probability density threshold, and foreground pixels are selected based on the comparison results. Feature extraction is then performed on the foreground pixels to obtain foreground features.

[0094] The multimodal processing module is used to separate the foreground features of various modalities from the foreground features, refine the foreground features of various modalities to obtain refined foreground features of various modalities, concatenate the refined foreground features of various modalities to obtain refined foreground features of multimodal data, input the refined foreground features of multimodal data into Transformer to extract the consistency features of multimodal data and the complementary features of various modalities, fuse the consistency features of multimodal data and the complementary features of various modalities to obtain fused features, pool the fused features, and use the pooled fused features to enhance the fused features themselves to obtain enhanced fused features;

[0095] The target recognition module is used to concatenate foreground features and enhanced fusion features to obtain concatenated features, perform convolutional compression on the concatenated features, input the convolutionally compressed concatenated features into Timesformer to obtain concatenated features with long temporal dependencies, input the concatenated features with long temporal dependencies into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

[0096] (1) The specific method for preprocessing multimodal data to obtain time series samples of multimodal data is as follows: data alignment is completed by adjusting internal and external parameters, and linear interpolation is performed to fill the time gap. Then, the multimodal data is normalized to the range of 0-1 using Min-MAX, and divided into groups of 10 frames by a sliding window to form a time-continuous input block, providing a unified format for subsequent processing.

[0097] In the anti-background interference module, the specific method for convolving the data in the time-series samples of multimodal data according to modality category to obtain the initial features of each type of modality data is as follows:

[0098] The modal categories covered by the multimodal data include RGB video frames, depth information, and time-series IMU.

[0099] Channel-wise convolution is used on RGB video frames, depth information, and time-series IMU data, with different channels used according to modality. Specifically, 3D convolution is used to extract spatiotemporal features from RGB video frames and depth information, while 1D convolution is applied to extract temporal features from the IMU data. Batch normalization and ReLU activation are used for optimization during convolution, and max pooling is used to maintain consistent dimensionality in the generated data. The initial feature Y of the generated RGB video frames is then used. RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU .

[0100] It clearly distinguishes the feature extraction methods of RGB video frames (3D convolution), Depth (3D convolution), and IMU (1D convolution), and handles the physical characteristics of different modalities in a targeted manner. The 3D convolution of RGB and Depth simultaneously captures spatiotemporal features (appearance and motion), while the 1D convolution of MU focuses on time series features (motion acceleration, etc.), improving the accuracy of feature representation of each modality and avoiding feature loss caused by forcibly unifying heterogeneous data.

[0101] By employing batch normalization (BN), ReLU activation, and max pooling, the problem of inconsistent multimodal data distribution (such as RGB pixel values ​​vs. IMU numerical range) is addressed, reducing feature dimensionality, improving subsequent computational efficiency, enhancing model convergence stability, and reducing computational redundancy.

[0102] (2) In the anti-background interference module, the initial features of various modal data are input into LSTM to obtain the spatiotemporal features of various modal data. The spatiotemporal features of various modal data are weighted to obtain the spatiotemporal features of multimodal data. Temporal attention and spatial attention are extracted from the spatiotemporal features of various modal data. Temporal attention and spatial attention are used to optimize the spatiotemporal features of multimodal data. The optimized spatiotemporal features of multimodal data are input into the multi-head attention mechanism in Transformer for feature enhancement. The specific method for obtaining the enhanced spatiotemporal features of multimodal data is as follows:

[0103] Using a single-layer LSTM network to generate the initial features Y of RGB video frames RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU Temporal feature mapping is performed, uniformly mapping to a subspace to obtain the spatiotemporal features F of various modal data. m The formula is as follows:

[0104]

[0105] Where m is the modal category, θ m λ represents the LSTM weight parameters. m The learning rate;

[0106] The spatiotemporal features O of the multimodal data are generated by weighting using the following formula:

[0107]

[0108] Where ε is a parameter used to prevent zero-painting, ε = 0.00001, w m The weights w for different class modes m are assigned by the Adam optimizer. m Update the weights w of category m modality. m The optimization is dynamically adjusted according to environmental interference, such as reducing the RGB weight under strong light and increasing the IMU weight under vibration;

[0109] Feature enhancement is performed using the multi-head attention mechanism in the transformer, as detailed below:

[0110] Calculate spatial attention for the spatiotemporal features F of RGB video frames. RGB Spatiotemporal features F of depth information Depth The sum of the two is then subjected to spatial average pooling to generate the spatial average pooling result S. special ;

[0111] Calculate temporal attention for the spatiotemporal features F of RGB video frames. RGBThe spatiotemporal characteristics F of depth information Depth Spatiotemporal characteristics F of time series IMU IMUh The sum of the three is then subjected to time-averaged pooling to generate the time-averaged pooling result S. temproal ;

[0112] The spatiotemporal features O of the multimodal data are weighted again to obtain the optimized spatiotemporal features Y of the multimodal data. STA The formula is as follows:

[0113]

[0114] The spatiotemporal features Y of the optimized multimodal data STA The input is fed into the multi-head attention mechanism in the transformer to obtain the spatiotemporal features Y of the optimized multimodal data. STA Enhanced spatiotemporal features Y of consistent multimodal data CM-STA .

[0115] By using a long short-term recurrent neural network (LSTM) with dynamic weight fusion and a multi-head self-attention mechanism, the model's ability to understand the temporal dynamic features of heterogeneous sensing multimodal relationships in complex background environments is enhanced, effectively resisting interference from complex background environments and ensuring the stability and reliability of the feature recognition system in complex backgrounds.

[0116] The Adam optimizer learns the modal weights and automatically adjusts the modal importance based on environmental disturbances (such as RGB failure caused by changes in lighting), maintaining system robustness even when some sensors fail.

[0117] Separate spatial attention S from multimodal features spatial and time attention S temporal Spatial attention focuses on key object regions, while temporal attention enhances continuous motion features, suppresses background noise, and highlights dynamic target features.

[0118] Multi-head attention mechanism for optimized feature Y STA Further refinement is performed to capture long-distance dependencies (such as the association features of occluded targets), and the output Y is obtained. CM−STA Preserving the original dimensions avoids information loss, enhances feature discrimination, and provides high-quality input for foreground segmentation.

[0119] (3) In the anti-background interference module, a Gaussian distribution is used to establish a pixel probability density model and this model is used to calculate the probability density of all pixels in the enhanced spatiotemporal features of multimodal data. The probability density of all pixels is compared with the probability density threshold. Based on the comparison result, foreground pixels are selected from all pixels. The specific method for extracting features from the foreground pixels to obtain foreground features is as follows:

[0120] Initialize 3 Gaussian distributions π k, μ k σ k 2 That is, a multi-Gaussian mixture model, which models the pixel probability density model P(r) for each phase pixel:

[0121]

[0122] Where N is the Gaussian probability density function, π k μ is the weight of the k-th Gaussian component. k Let σ be the mean of the k-th Gaussian component. k 2 Let r be the variance of the k-th Gaussian component, and r be the pixel feature value. The pixel feature value r is derived from the enhanced spatiotemporal feature Y of the multimodal data. CM-STA ;

[0123] The weight π of the k-th Gaussian component k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 All updates are performed using the EM algorithm, and the specific steps are as follows:

[0124] For the E-step, the posterior probability P(k|r) is defined as the probability that the pixel feature value r belongs to the k-th Gaussian component:

[0125]

[0126] For M steps, update the weight π of the k-th Gaussian component. k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 :

[0127]

[0128] Where M represents the total number of pixels, r i π represents the feature value of the i-th pixel. k t+1 μ k t+1 σ k 2,t+1 These represent the weight, mean, and variance of the k-th Gaussian component after iteration, respectively.

[0129] To generate a background mask, select the Gaussian component with the highest weight and lowest variance to define the background. Calculate the current P(k|r). If it is greater than the threshold γ, γ=0.5, then the pixel feature value r is identified as a background pixel; otherwise, it is identified as a foreground pixel.

[0130]

[0131]

[0132] Among them, Y bg Y represents background features. fg Indicates foreground features.

[0133] The multi-Gaussian mixture model updates parameters online using the EM algorithm, dynamically adapting to disturbances such as changes in illumination and background movement, accurately distinguishing the foreground from complex dynamic backgrounds, based on weight π. k and variance σ k 2 Background components are selected with high weights and low variance to stabilize the background, while foreground features Y are used. fg Directly extracting target features significantly reduces the false detection rate.

[0134] The core advantage of the multi-Gaussian mixture model in this step is that it can approximate any complex probability distribution by using a weighted combination of multiple Gaussian distributions (components). Each component can capture a different state in the background environment. The K components together construct a more comprehensive and flexible background probability model P(x) to more accurately describe the normal range of background pixel changes. When the value of a pixel matches the background distribution modeled by the multi-Gaussian mixture model, it is identified as background and effectively suppressed, reducing false alarms caused by dynamic changes in the background.

[0135] (4) In the multimodal processing module, the foreground features of various modal data are separated from the foreground features. After feature refinement of the foreground features of various modal data, refined foreground features of various modal data are obtained. The specific method for concatenating the refined foreground features of various modal data into refined foreground features of multimodal data is as follows:

[0136] According to the modal category, the corresponding channel is used to select the foreground feature Y. fg The foreground features of RGB video frames, the foreground features of depth information, and the foreground features of the time-series IMU are separated from the data.

[0137] The foreground features of RGB video frames are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of depth information (Depth) are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of the time-series IMU are subjected to 1D convolution, batch normalization, and ReLU activation to generate refined foreground features Y of RGB video frames.RGB Refined foreground features Y of depth information Depth Refined foreground features Y of depth information IMU By using batch normalization (to accelerate convergence and stabilize training), ReLU (to introduce nonlinearity), and max pooling (to reduce dimensionality and enhance translation invariance), we further optimize multimodal features such as YRGB, YDepth, and YIMU, ensuring that the foreground features of each modality are independently and optimally purified for subsequent multimodal fusion.

[0138] Refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMU Concatenating along the channel dimension, as shown below:

[0139]

[0140] Where Y0 represents the refined foreground features of the multimodal data.

[0141] Convolutional operations are performed independently on RGB, Depth, and IMU foreground features. RGB / Depth features are preserved using 2D convolution to maintain spatial structure, while IMU features are refined using 1D convolution to extract temporal patterns and retain modality-specific information. Concatenating and refining features along the channel dimension establishes multimodal associations, laying the foundation for consistent / complementary feature extraction.

[0142] (5) In the multimodal processing module, the refined foreground features of the multimodal data are input into the Transformer to extract the consistency features of the multimodal data and the complementary features of various types of modal data. The consistency features of the multimodal data and the complementary features of various types of modal data are fused to obtain fused features. The fused features are pooled, and the pooled fused features are used to enhance the fused features themselves to obtain enhanced fused features. The specific method is as follows:

[0143] The refined foreground features Y0 of the multimodal data are input into the Transformer to extract the consistency features h of the multimodal data. con The refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMU Input each element into a Transformer and extract the complementary features h of the RGB video frames. RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs hIMU com By processing consistency and complementarity, common clues and contributions among multiple modalities are captured simultaneously, thereby enhancing the representational ability of multimodal features and improving the accuracy and robustness of multimodal feature recognition.

[0144] The first weight α and the second weight β are learned through the Adam optimizer, and the consistency feature h is fused. con Complementarity features of RGB video frames h RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU com Where α and β are both softmax(MLP(h)), and the multilayer perceptron (MLP) is a two-layer fully connected network, represented as follows:

[0145]

[0146] Here, H represents the fusion feature; it is learned through gradient descent using the Adam optimizer, and dynamically fuses consistency and complementarity information based on the input heterogeneous sensing multimodal data to generate a discriminative fusion feature H.

[0147] The fusion feature H is subjected to global average pooling to generate the fusion weight S. FA Using fusion weight S FA Enhance the fusion feature H to generate a stronger fusion feature Y. FA The calculation formula is as follows:

[0148] .

[0149] This step involves foreground feature Y. fg Building upon this foundation, we further explore and fuse the consistency and complementarity information among different modalities to enhance the discriminativeness of the fused feature H. This adaptively strengthens important feature channels while suppressing relatively unimportant ones, generating a highly discriminative, information-rich, and robust global fused feature Y. FA This provides the optimal input for the final recognition task (such as classification or detection).

[0150] The Transformer (Multimodal Visual Large Model Encoder) outputs the consistency feature h of the multimodal data. con Complementarity features of RGB video frames h RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU comIt can take into account both overall target attributes (such as category) and modality-specific cues (such as motion jitter features of IMU). Learnable weights α and β generated by Softmax (MLP) dynamically balance the contributions of common and complementary features, avoiding suboptimal fusion caused by manually setting weights. The fusion weight S is used. FA Enhance the fusion feature H to generate a stronger fusion feature Y. FA Strengthen target-related features to improve detection confidence.

[0151] (6) In the target recognition module, the foreground features and enhancement fusion features are concatenated to obtain concatenated features, the concatenated features are convolved and compressed, and the convolutionally compressed concatenated features are input into Timesformer to obtain concatenated features with long-term dependencies. The specific method is as follows:

[0152] Foreground feature Y fg and fusion feature Y FA By splicing the data, we obtain the splicing feature Y. input For splicing feature Y input Perform convolution compression to obtain Y dim The formula is as follows:

[0153]

[0154] Among them, Con 1×1 This indicates a one-dimensional convolution operation with a 1*1 kernel, where K represents the kernel and b represents the bias term.

[0155] Y dim Inputting Timesformer, the independent temporal and spatial attention mechanisms within Timesformer are used to capture long-term dependencies, generating a concatenated feature Y with long-term dependencies. transformer The formula is shown below:

[0156]

[0157] Where, θ TS This represents the complete set of trainable parameters for Timesformer.

[0158] Reduce the splicing feature Y by using 1×1 convolution. input This reduces the computational burden on Timesformer, balancing model performance and computational efficiency. Timesformer employs a separate spatiotemporal attention mechanism to capture long-term motion trajectories and output Y... transformer It includes a global spatiotemporal context to address the forgetting problem in short-sequence models.

[0159] By integrating and enhancing long-term spatiotemporal dependent information, we can achieve information complementarity and integrity, capture complex behavioral or state evolutions that span long time periods and spaces, and provide comprehensive input for long-term spatiotemporal modeling.

[0160] (7) In the target recognition module, the specific method for inputting the spliced ​​features with long-term time dependence into the target detection head to obtain the classification result of the target in the foreground and the bounding box of the target is as follows:

[0161] The splicing feature Y with long-term dependence transformer Enter the YOLO object detection head to generate the bounding box y of the target. t The classification probability p of the target t ;

[0162] The stitched features Ytransformer with long-term dependencies are input into the Gaussian uncertainty detection head to obtain the mean μ and variance α2. A 1×1 convolution is then performed to output the mean (μ) of the center coordinates (x, y) of the stitched features Ytransformer with long-term dependencies (referring to the actual physical coordinates of each spatial location on the Ytransformer mapped back to the input image). x ,μ y The algorithm calculates the squared deviation between the local coordinates (the center point coordinates of the target bounding box) and the center coordinates (x, y) using a 1×1 convolution, and then smooths the result using the Softplus function, outputting the mean and variance (σ) of the center coordinates. x ,σ y );

[0163] The specific steps for optimizing the loss function of the TimeSformer model and the YOLO object detection model are as follows:

[0164] For Timesformer, its loss function consists of spatiotemporal consistency loss and Gaussian uncertainty loss. Spatiotemporal consistency loss is the overall loss calculated for the entire sliding window time series T, while Gaussian uncertainty loss is the loss calculated for each time unit t. These two loss functions together constitute the loss function optimization for Timesformer.

[0165] For spatiotemporal consistency loss, within a given sliding window time T, the bounding box y at time t+1 should be used. t+1 The bounding box y at time t t The formula for smoothing loss is as follows:

[0166]

[0167] Among them, y t+1 and y t L represents the bounding boxes at time t+1 and time t, respectively.smooth This represents the spatiotemporal consistency smoothing loss;

[0168] For Gaussian uncertainty loss L Gaussian The formula is as follows:

[0169] ;

[0170] For the object detection model YOLO, the bounding boxes use the CIoU loss L. CIoU The classification probability uses cross-entropy loss L. cls The formula is as follows:

[0171]

[0172]

[0173] Where IoU is the sum of the predicted bounding box d and the ground truth bounding box d. gt The intersection-union ratio (IoU), ρ is the Euclidean distance between the predicted center and the ground truth center, c is the diagonal of the smallest rectangle containing both the predicted and ground truth boxes, v is the aspect ratio consistency, α is the aspect ratio consistency weighting factor, and y i ' is the true label of the i-th type of target, p i ' is the probability of predicting the target as the i-th type.

[0174] Until CIoU loses L CIoU and cross-entropy loss L cls Once convergence is achieved, the optimized target classification result and target bounding box are output.

[0175] The bounding box is directly output using the object detection model YOLO. i And classification probability p i Predicting the mean coordinates (μ) using a Gaussian uncertainty head x ,μ y ) and variance (σ) x ,σ y This technology can quantify and detect uncertainties (such as fuzzy targets), thereby improving security decision-making capabilities. It utilizes spatiotemporal consistency loss L... smooth Constraining the bounding boxes of adjacent frames to change smoothly, using Gaussian loss L Gaussian Minimize coordinate prediction error by using YOLO's CIoU loss L CloU and cross-entropy loss L cls Optimize localization and classification to comprehensively improve detection stability and accuracy. This is achieved by optimizing the model's performance, robustness, and reliability through innovative loss function design and uncertainty modeling.

[0176] Example 2

[0177] refer to Figure 4A method for anti-interference dynamic target recognition using heterogeneous sensors under a multimodal vision large model, comprising:

[0178] Multimodal data is preprocessed to obtain time-series samples. The data in the time-series samples are convolved according to modality category to obtain initial features of each modality. The initial features of each modality are input into an LSTM to obtain spatiotemporal features of each modality. The spatiotemporal features of each modality are weighted to obtain the spatiotemporal features of the multimodal data. Temporal attention and spatial attention are extracted from the spatiotemporal features of each modality. Temporal attention and spatial attention are used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features of the multimodal data are input into the multi-head attention mechanism in the Transformer for feature enhancement to obtain enhanced spatiotemporal features of the multimodal data. A pixel probability density model is established using a Gaussian distribution, and the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data is calculated using this model. The probability density of all pixels is compared with the probability density threshold. Based on the comparison results, foreground pixels are selected from all pixels, and features are extracted from the foreground pixels to obtain foreground features.

[0179] The process involves separating foreground features from various modalities, refining these features, concatenating them to form a multimodal foreground feature, inputting the multimodal foreground feature into a Transformer to extract consistency features and complementary features from various modalities, fusing these features to obtain a fused feature, pooling the fused feature, and using the pooled fused feature to enhance the fused feature itself to obtain an enhanced fused feature.

[0180] The foreground features and enhancement fusion features are concatenated to obtain concatenated features. The concatenated features are then compressed by convolution. The convolutionally compressed concatenated features are then input into Timesformer to obtain concatenated features with long temporal dependencies. The concatenated features with long temporal dependencies are then input into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

[0181] Example 3

[0182] A computer program product includes a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition method of Embodiment 2.

[0183] The contents not described in detail in this specification are prior art known to those skilled in the art. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

Claims

1. A multimodal visual large-scale model heterogeneous sensor anti-interference dynamic target recognition system, characterized in that, include: The anti-background interference module is used to preprocess multimodal data from heterogeneous sensors to obtain time-series samples of multimodal data. The data in the time-series samples are convolved according to modality category to obtain initial features for each modality. These initial features are then input into an LSTM to obtain spatiotemporal features for each modality. The spatiotemporal features of each modality are weighted to obtain the spatiotemporal features of the multimodal data. Temporal and spatial attention are extracted from the spatiotemporal features of each modality, and these attention mechanisms are used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features are then input into the multi-head attention mechanism in the Transformer for feature enhancement, resulting in enhanced spatiotemporal features of the multimodal data. A Gaussian distribution is used to establish a pixel probability density model, which is then used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability density of all pixels is compared with a probability density threshold, and foreground pixels are selected based on the comparison results. Feature extraction is then performed on the foreground pixels to obtain foreground features. The multimodal processing module is used to separate the foreground features of various modalities from the foreground features, refine the foreground features of various modalities to obtain refined foreground features of various modalities, concatenate the refined foreground features of various modalities to obtain refined foreground features of multimodal data, input the refined foreground features of multimodal data into Transformer to extract the consistency features of multimodal data and the complementary features of various modalities, fuse the consistency features of multimodal data and the complementary features of various modalities to obtain fused features, pool the fused features, and use the pooled fused features to enhance the fused features themselves to obtain enhanced fused features; The target recognition module is used to concatenate foreground features and enhanced fusion features to obtain concatenated features, perform convolutional compression on the concatenated features, input the convolutionally compressed concatenated features into Timesformer to obtain concatenated features with long temporal dependencies, input the concatenated features with long temporal dependencies into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

2. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 1, characterized in that: In the anti-background interference module, the specific method for convolving the data in the time-series samples of multimodal data according to modality category to obtain the initial features of each type of modality data is as follows: The modal categories covered by the multimodal data include RGB video frames, depth information, and time-series IMU. Channel-wise convolution is used on RGB video frames, depth information, and time-series IMU data, with different channels used according to modality. Specifically, 3D convolution is used to extract spatiotemporal features from RGB video frames and depth information, while 1D convolution is applied to extract temporal features from the IMU data. Batch normalization and ReLU activation are used for optimization during convolution, and max pooling is used to maintain consistent dimensionality in the generated data. The initial feature Y of the generated RGB video frames is then used. RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU .

3. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 2, characterized in that: In the background interference resistance module, the initial features of various modal data are input into an LSTM to obtain the spatiotemporal features of each modal data. These spatiotemporal features are then weighted to obtain the spatiotemporal features of the multimodal data. Temporal and spatial attention are extracted from the spatiotemporal features of each modal data. These temporal and spatial attention are then used to optimize the spatiotemporal features of the multimodal data. Finally, the optimized spatiotemporal features of the multimodal data are input into the multi-head attention mechanism in a Transformer for feature enhancement. The specific method for obtaining the enhanced spatiotemporal features of the multimodal data is as follows: Using a single-layer LSTM network to generate the initial features Y of RGB video frames RGB Initial features Y of depth information (Depth) Depth Initial features Y of time series IMU IMU Temporal feature mapping is performed, uniformly mapping to a subspace to obtain the spatiotemporal features F of various modal data. m The formula is as follows: Where m is the modal category, θ m λ represents the LSTM weight parameters. m The learning rate; The spatiotemporal features O of the multimodal data are generated by weighting using the following formula: Where ε is a parameter used to prevent zero-painting, w m The weights w for different class modes m are assigned by the Adam optimizer. m Update the weights w of category m modality. m The optimization is dynamically adjusted based on environmental interference; Feature enhancement is performed using the multi-head attention mechanism in the transformer, as detailed below: Calculate spatial attention for the spatiotemporal features F of RGB video frames. RGB Spatiotemporal features F of depth information Depth The sum of the two is then subjected to spatial average pooling to generate the spatial average pooling result S. special ; Calculate temporal attention for the spatiotemporal features F of RGB video frames. RGB The spatiotemporal characteristics F of depth information Depth Spatiotemporal characteristics F of time series IMU IMUh The sum of the three is then subjected to time-averaged pooling to generate the time-averaged pooling result S. temproal ; The spatiotemporal features O of the multimodal data are weighted again to obtain the optimized spatiotemporal features Y of the multimodal data. STA The formula is as follows: The spatiotemporal features Y of the optimized multimodal data STA The input is fed into the multi-head attention mechanism in the transformer to obtain the spatiotemporal features Y of the optimized multimodal data. STA Enhanced spatiotemporal features Y of consistent multimodal data CM-STA .

4. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 3, characterized in that: In the anti-background interference module, a Gaussian distribution is used to establish a pixel probability density model, and this model is used to calculate the probability density of all pixels in the enhanced spatiotemporal features of multimodal data. The probability density of all pixels is compared with a probability density threshold, and foreground pixels are selected from all pixels based on the comparison results. The specific method for extracting features from the foreground pixels to obtain foreground features is as follows: Initialize 3 Gaussian distributions π k, μ k σ k 2 For each phase pixel, model the pixel probability density model P(r): Where N is the Gaussian probability density function, π k μ is the weight of the k-th Gaussian component. k Let σ be the mean of the k-th Gaussian component. k 2 Let r be the variance of the k-th Gaussian component, and r be the pixel feature value. The pixel feature value r is derived from the enhanced spatiotemporal feature Y of the multimodal data. CM-STA ; The weight π of the k-th Gaussian component k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 All updates are performed using the EM algorithm, and the specific steps are as follows: For the E-step, the posterior probability P(k|r) is defined as the probability that the pixel feature value r belongs to the k-th Gaussian component: For M steps, update the weight π of the k-th Gaussian component. k The mean μ of the k-th Gaussian component k The variance σ of the k-th Gaussian component k 2 : Where M represents the total number of pixels, r i π represents the feature value of the i-th pixel. k t+1 μ k t+1 σ k 2,t+1 These represent the weight, mean, and variance of the k-th Gaussian component after iteration, respectively. To generate a background mask, select the Gaussian component with the highest weight and lowest variance to define the background. Calculate the current P(k|r). If it is greater than the threshold γ, the pixel feature value r is identified as a background pixel; otherwise, it is identified as a foreground pixel. Among them, Y bg Y represents background features. fg Indicates foreground features.

5. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 4, characterized in that: In the multimodal processing module, foreground features of various modalities are separated from foreground features. After feature refinement of the foreground features of each modality, refined foreground features of each modality are obtained. The specific method for concatenating the refined foreground features of each modality into refined foreground features of the multimodal data is as follows: According to the modal category, the corresponding channel is used to select the foreground feature Y. fg The foreground features of RGB video frames, the foreground features of depth information, and the foreground features of the time-series IMU are separated from the data. The foreground features of RGB video frames are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of depth information (Depth) are subjected to 2D convolution, batch normalization, ReLU activation, and max pooling. The foreground features of the time-series IMU are subjected to 1D convolution, batch normalization, and ReLU activation to generate refined foreground features Y of RGB video frames. RGB Refined foreground features Y of depth information Depth Refined foreground features Y of depth information IMU ; Refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMU Concatenating along the channel dimension, as shown below: Where Y0 represents the refined foreground features of the multimodal data.

6. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 5, characterized in that: In the multimodal processing module, the refined foreground features of the multimodal data are input into the Transformer to extract the consistency features and complementary features of various modalities. The consistency features and complementary features of the multimodal data are then fused to obtain a fused feature. The fused feature is then pooled, and the pooled fused feature is used to enhance the fused feature itself to obtain an enhanced fused feature. The specific method is as follows: The refined foreground features Y0 of the multimodal data are input into the Transformer to extract the consistency features h of the multimodal data. con The refined foreground features Y of RGB video frames RGB Refined foreground features Y of depth information Depth Refined foreground features Y of time series IMU IMU Input each element into a Transformer and extract the complementary features h of the RGB video frames. RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU com ; The first weight α and the second weight β are learned through the Adam optimizer, and the consistency feature h is fused. con Complementarity features of RGB video frames h RGB com Complementary features of depth information h Depth com Complementary characteristics of time series IMUs h IMU com Where α and β are both softmax(MLP(h)), and the multilayer perceptron (MLP) is a two-layer fully connected network, represented as follows: Where H represents the fusion feature; The fusion feature H is subjected to global average pooling to generate the fusion weight S. FA Using fusion weight S FA Enhance the fusion feature H to generate a stronger fusion feature Y. FA The calculation formula is as follows: 。 7. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 6, characterized in that: In the target recognition module, the foreground features and enhanced fusion features are concatenated to obtain concatenated features. These concatenated features are then compressed via convolution. The specific method for inputting the convolutionally compressed concatenated features into Timesformer to obtain concatenated features with long-term temporal dependencies is as follows: Foreground feature Y fg and fusion feature Y FA By splicing the data, we obtain the splicing feature Y. input For splicing feature Y input Perform convolution compression to obtain Y dim The formula is as follows: Among them, Con 1×1 This indicates a one-dimensional convolution operation with a 1*1 kernel, where K represents the kernel and b represents the bias term. Y dim Inputting Timesformer, the independent temporal and spatial attention mechanisms within Timesformer are used to capture long-term dependencies, generating a concatenated feature Y with long-term dependencies. transformer The formula is shown below: Where, θ TS This represents the complete set of trainable parameters for Timesformer.

8. The multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition system according to claim 7, characterized in that: In the target recognition module, the specific method for inputting spliced ​​features with long-term time dependence into the target detection head to obtain the classification result of the target in the foreground and the bounding box of the target is as follows: The splicing feature Y with long-term dependence transformer Enter the YOLO object detection head to generate the bounding box y of the target. t The classification probability p of the target t ; The splicing feature Y with long-term dependence transformer Input the Gaussian uncertainty detection head to obtain the mean μ and variance α. 2 The concatenated feature Y with long-term continuation dependency is output through 1×1 convolution. transformer The mean (μ) of the center coordinates (x, y) x ,μ y The method calculates the squared deviation between the local coordinates and the center coordinates (x, y) using a 1×1 convolution, and then smooths the result using the Softplus function, outputting the mean and variance (σ) of the center coordinates. x ,σ y ); The specific steps for optimizing the loss function of the TimeSformer model and the YOLO object detection model are as follows: For Timesformer, the loss function consists of spatiotemporal consistency loss and Gaussian uncertainty loss. For spatiotemporal consistency loss, within a given sliding window time T, the bounding box y at time t+1 should be used. t+1 The bounding box y at time t t The formula for smoothing loss is as follows: Among them, y t+1 and y t L represents the bounding boxes at time t+1 and time t, respectively. smooth This represents the spatiotemporal consistency smoothing loss; For Gaussian uncertainty loss L Gaussian The formula is as follows: ； For the object detection model YOLO, the bounding boxes use the CIoU loss L. CIoU The classification probability uses cross-entropy loss L. cls The formula is as follows: Where IoU is the sum of the predicted bounding box d and the ground truth bounding box d. gt The intersection-union ratio (IoU), ρ is the Euclidean distance between the predicted center and the ground truth center, c is the diagonal of the smallest rectangle containing both the predicted and ground truth boxes, v is the aspect ratio consistency, α is the aspect ratio consistency weighting factor, and y i ' is the true label of the i-th type of target, p i 'This is the probability of predicting the target as the i-th type; Until CIoU loses L CIoU and cross-entropy loss L cls Once convergence is achieved, the optimized target classification result and target bounding box are output.

9. A method for anti-interference dynamic target recognition using heterogeneous sensors under a multimodal vision large model, characterized in that, include: Multimodal data from heterogeneous sensors is preprocessed to obtain time-series samples of multimodal data. Data from these time-series samples are convolved according to modality category to obtain initial features for each modality. These initial features are then input into an LSTM to obtain spatiotemporal features for each modality. The spatiotemporal features of each modality are weighted to obtain the spatiotemporal features of the multimodal data. Temporal and spatial attention are extracted from these spatiotemporal features. These attention mechanisms are then used to optimize the spatiotemporal features of the multimodal data. The optimized spatiotemporal features are then input into a multi-head attention mechanism in a Transformer for feature enhancement, resulting in enhanced spatiotemporal features. A Gaussian distribution is used to establish a pixel probability density model, and this model is used to calculate the probability density of all pixels in the enhanced spatiotemporal features of the multimodal data. The probability density of all pixels is compared with a probability density threshold. Based on the comparison results, foreground pixels are selected from all pixels, and feature extraction is performed on these foreground pixels to obtain foreground features. The process involves separating foreground features from various modalities, refining these features, concatenating them to form a multimodal foreground feature, inputting the multimodal foreground feature into a Transformer to extract consistency features and complementary features from various modalities, fusing these features to obtain a fused feature, pooling the fused feature, and using the pooled fused feature to enhance the fused feature itself to obtain an enhanced fused feature. The foreground features and enhancement fusion features are concatenated to obtain concatenated features. The concatenated features are then compressed by convolution. The convolutionally compressed concatenated features are then input into Timesformer to obtain concatenated features with long temporal dependencies. The concatenated features with long temporal dependencies are then input into the target detection head to obtain the classification results of the target in the foreground and the bounding box of the target.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the multimodal vision large model heterogeneous sensor anti-interference dynamic target recognition method as described in claim 9.

Images