A multi-modal target intelligent identification method based on Mamba architecture for missing modalities

Abstract

The application discloses a kind of modal missing-oriented multi-modal target intelligent identification method based on Mamba architecture, belong to multi-modal target identification technical field.The application constructs the end-to-end identification framework including heterogeneous feature coding, diffusion feature completion, dynamic fusion and mixed expert classification.Visual large model and Mamba encoder are used to extract the semantic features of visible light, infrared, signal and AIS data respectively.A diffusion completion module is designed to generate missing modal inversely in the feature space using diffusion probability model, ensuring the continuity and integrity of feature distribution.Then, a dynamic fusion module based on Mamba architecture is constructed to realize adaptive gating and deep interaction of heterogeneous features using selective state space mechanism.Finally, a multi-task mixed expert module is introduced to decouple the large class, type and action recognition tasks through dynamic routing mechanism.The application significantly improves the recognition robustness and fine-grained classification accuracy in the modal missing scenario while reducing the computational complexity.

Description

Technical Field

[0001] This invention relates to the fields of multimodal deep learning, computer vision and target recognition technology, and in particular to a method and system for intelligent identification of sea and air targets in a cross-domain environment of sea and air, targeting multi-source heterogeneous data such as visible light, infrared, signal features and AIS, and in cases where sensor data may be randomly missing. Background Technology

[0002] In modern sea and air defense systems, high-precision identification of key targets such as ships and aircraft is the core task of situational awareness systems. Single sensors are prone to failure in rainy, foggy, or strong electromagnetic countermeasure environments. Multimodal deep learning technology, which utilizes multi-source heterogeneous data such as visible light, infrared, electromagnetic signals, and AIS, significantly improves identification stability through complementarity.

[0003] In the field of feature extraction and fusion, the Transformer architecture dominates due to its global context modeling capabilities. The computational complexity of its core self-attention mechanism increases quadratically with sequence length. When processing long-term, high-resolution, heterogeneous data from the sea and air, traditional computing methods face severe computational bottlenecks and memory overhead, making them unsuitable for the real-time requirements of airborne or shipborne edge computing devices. The Mamba architecture, based on a selective state-space model (SSM), achieves linear computational complexity (…). However, there is currently no mature solution for applying it to the dynamic fusion and interaction of multimodal heterogeneous data.

[0004] In real-world environments, sensor malfunctions, communication interference, or silent control often lead to the complete loss of specific modal data. Existing technologies mostly employ zero-padding or statistical interpolation strategies to handle missing inputs. This approach disrupts the continuity of feature distribution, introducing distribution bias in subsequent fusion stages. Diffusion models perform well in image generation and temporal completion tasks, but a unified framework for semantic reconstruction of missing modalities within the low-dimensional feature space of multimodal recognition remains lacking.

[0005] Maritime and air target identification involves multiple levels of tasks, including broad category identification, fine-grained model determination, and motion analysis. Existing methods often employ simple shared fully connected layers, neglecting the differences in feature focus across different tasks (e.g., model identification focuses on texture, while motion identification focuses on trajectory). This leads to feature conflicts and negative transfer during multi-task joint learning. While general-purpose multimodal large models have strong generalization capabilities, they lack adaptability to specialized maritime and air data such as signal features and AIS, and their large size makes them unsuitable for efficient deployment on edge devices. Summary of the Invention

[0006] This invention aims to address the problems of strong heterogeneity of multi-source data, easy loss of sensor modalities in actual combat environments, and high computational complexity of traditional Transformer architecture in existing air and sea target identification. It proposes a multimodal target intelligent identification method and system based on Mamba architecture to address the problem of modal loss.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multimodal target intelligent recognition method based on Mamba architecture for modality missing, the method comprising:

[0008] Step 1: Acquire raw multimodal data of sea and air targets, including visible light images, infrared images, signal feature spectrum diagrams, and AIS time series data;

[0009] Step 2: Construct a multimodal heterogeneous feature encoding network;

[0010] To address the highly heterogeneous nature of air and sea target data, differentiated feature extraction strategies are designed. For visible light images, infrared images, and signal feature spectrograms, a pre-trained large visual model is used as the backbone network to extract features under frozen parameters. Different modalities are then mapped to a unified semantic space through a trainable projection layer. For AIS time-series data, a Mamba time-series encoder is constructed. The linear computational complexity advantage of the state-space model in processing long-series data is utilized to capture the navigation trajectory and state changes of ships or aircraft, and extract global temporal semantic representations.

[0011] Step 3: Feature space completion based on diffusion model;

[0012] To address modal loss caused by sensor malfunctions, a feature completion module is constructed; the specific method of the feature completion module includes:

[0013] Step 3.1: Initialize the locations of the modal features marked as missing by the mask to standard Gaussian noise;

[0014] Step 3.2: Generate sinusoidal time-step embedding vectors;

[0015] Step 3.3: Concatenate the standard Gaussian noise obtained in Step 3.1 with the sinusoidal time step embedding vector generated in Step 3.2;

[0016] Step 3.4: Input the concatenation result obtained in Step 3.3 into the trained multilayer perceptron to directly reconstruct a semantic feature vector that approximates the true distribution; the multilayer perceptron includes, in sequence: a first fully connected layer, a layer normalization layer, a GELU activation function, a second fully connected layer, a layer normalization layer, a GELU activation function, and a third fully connected layer; wherein, the input dimension of the first fully connected layer is "the sum of the feature dimension D and the time step embedding dimension H", and the output dimension is H, and the output dimension of the third fully connected layer is D;

[0017] Step 3.5: Replace the masked missing modal feature positions in the global temporal semantic representation obtained in Step 2 with the semantic feature vector obtained in Step 3.4 to obtain a new global temporal semantic representation;

[0018] Step 4: Cross-modal feature semantic alignment;

[0019] Step 4.1: Construct triples of anchor points, homologous positive samples, and heterologous hard negative samples;

[0020] Step 4.2: Use the triplet loss function to constrain the distribution of each modality in the common metric space to complete the semantic alignment of each modality; where the common metric space is an embedding space of uniform dimension. The distance between each modal feature is calculated after L2 normalization within this space; multimodal alignment loss. The loss for mining triples from hard samples is defined as: and order For pairwise combinations of different modes Average value; where, The distance between different modal features of the same sample. This is the minimum distance among the samples in the batch that are different from the anchor label. It is the interval constant;

[0021] Step 5: Dynamic feature fusion based on Mamba architecture;

[0022] The completed visible light image, infrared image, signal feature spectrum, and AIS time series data four-modal features are mapped as follows: First, the alignment module linearly projects them onto the common dimension. get A global alignment semantic vector is generated by the alignment network. ; then The channel dimension is concatenated and used as input for Mamba-Fusion; within the Mamba-Fusion module:

[0023] 1) The input splicing features are first mapped to the fusion latent space through the input projection layer;

[0024] 2) After expanding the vector into a sequence of length 1, selective state-space modeling is performed by sequentially inputting two stacked Mamba blocks.

[0025] 3) Mamba stacked output is compressed back via output projection Dimension, to obtain the final fusion feature ;

[0026] Each Mamba block includes the following along the data flow direction: input normalization, channel expansion, depthwise convolution, non-linear activation, channel compression, and residual connection; the depthwise convolution is a selective state space, and the selective scanning mechanism adaptively generates state parameters according to the input to realize dynamic gating and interaction of different modal information;

[0027] Step 6: Multi-task hybrid expert classification prediction;

[0028] The multi-task hybrid expert classification module configures a gating network and several expert networks for each recognition task; the multi-task hybrid expert classification module configures an independent gating network and several expert networks for each recognition task; for any task The gating network adopts the same structure as the multilayer perceptron in step 3.4, and integrates the features. Mapped to expert weight vector ;

[0029] No. A network of experts For a multilayer perceptron, output the logits for this task: The final task output is a weighted sum of the outputs from all experts:

[0030] ;

[0031] By using independent gating for each task and dynamic expert routing, different expert combination paths are formed for the three tasks of category, model, and action on the same fusion feature, thereby decoupling the task concerns and suppressing negative transfer.

[0032] During training, only the modalities that "originally exist" are used with probability during the training phase. Random masks are used to simulate missing values, and a constraint is set: for any sample, if all of its "real-world modalities" are randomly masked, then this sample will not undergo supervision completion in this round (to avoid training instability caused by complete missing values); the completion loss is only applied to the supervision mask. Calculate the position of 1;

[0033] To address the differences in feature focus among the three levels of tasks—class recognition, fine-grained model determination, and action analysis—a multi-task hybrid expert classification module is constructed. Each task is configured with an independent gating network and several expert networks. Through a dynamic routing mechanism, the gating network automatically allocates the weights of each expert network based on the input feature attributes, thereby achieving feature decoupling between tasks, avoiding feature conflicts in multi-task joint learning, and outputting recognition results for each category.

[0034] Furthermore, the specific structure of the large visual model in step 2 is as follows: the input image is obtained by image block embedding to obtain patch sequence features; the trainable class label [CLS] is concatenated to the beginning of the sequence and superimposed with position encoding, and then globally interacted through several Transformer encoding blocks; finally, the output is layer normalized and the feature corresponding to the [CLS] position is taken as the visual semantic representation, and then mapped to a unified 512-dimensional semantic space through a trainable projection layer; wherein, the image block embedding module divides the input image into several image blocks and maps them into sequence features; the class label and position encoding module concatenates the class label vector to the beginning of the sequence and superimposes the position encoding; each Transformer encoding block includes a multi-head self-attention submodule and a feedforward network submodule in sequence, and residual connections and layer normalization are set outside each submodule to complete information interaction and stable training; finally, the output is normalized and the feature corresponding to the class label is taken as the visual semantic representation, and then mapped to a unified dimensional semantic space through a trainable projection layer.

[0035] Furthermore, the total loss function is defined as:

[0036] ;

[0037] in, For the multi-task classification loss, the standard cross-entropy loss between the predicted probability distribution and the true label is calculated for the three sub-tasks of category recognition, model determination and behavior analysis, and the sum of the three is used as the total classification loss. For multimodal alignment loss, hard sample mining triplet loss is adopted;

[0038] The average of the pairwise combinations of different modes is obtained ; For feature reconstruction loss, mean squared error is used to measure the difference between the completed features generated by the Diff-Impute module and the original true features, and the calculation is only performed on the modalities that are randomly masked but originally exist. , This is the loss weight.

[0039] Furthermore, multi-task classification loss For three tasks—category identification, model determination, and behavior analysis—the standard cross-entropy loss between the predicted probability distribution and the true label is calculated separately, and the sum of these three losses is used as the total classification loss.

[0040] ;

[0041] in, For batch size, For the first The sample at the th Real labels on each task This represents the predicted probability for the corresponding task; this loss term is used to optimize the prediction accuracy for each task.

[0042] Furthermore, the multimodal alignment loss is defined as:

[0043] ;

[0044] in, For a set of modal pairs, Indicates anchor point and Positive samples distance, For anchor points and negative samples distance, The boundary constant is used; the goal of this loss is to optimize the consistency of cross-modal features, bringing homologous modes closer together and pushing heterologous modes further apart.

[0045] Furthermore, feature reconstruction loss This constraint is used to maintain consistency between the generated completed features and the original true features. For modalities that are randomly masked but whose original data truly exists, the mean squared error loss is used as a metric.

[0046] ;

[0047] in, For the generated completion features, Original features Represents element-wise product. It is a mask that ensures that loss calculation is performed only on modes that exist and are masked.

[0048] The beneficial effects of this invention are as follows: It introduces a Diff-Impute mechanism to effectively repair the semantic distribution of missing modalities within the feature space. Even when sensors partially fail or are interfered with in real-world scenarios, it maintains high recognition accuracy, overcoming the feature distribution shift problem caused by traditional zero-filling strategies. Utilizing the Mamba architecture to process long-term AIS data and multimodal fusion reduces computational complexity from quadratic to linear compared to the traditional Transformer architecture. While ensuring long-sequence modeling capabilities, it significantly reduces computational resource consumption, making it more suitable for edge computing environments. By extracting strongly generalized visual features through a visual model, combined with Mamba's dynamic gating fusion and MoE's multi-task decoupling, it effectively solves the confusion problem between similar models and complex actions, significantly improving the recognition accuracy of fine-grained tasks. Attached Figure Description

[0049] Figure 1This invention provides an overall flowchart of a multimodal target intelligent recognition method based on the Mamba architecture for addressing modal missing features, as provided in an embodiment of the present invention. The diagram illustrates the end-to-end processing from multimodal data input, feature extraction, Diff-Impute feature completion, cross-modal alignment, Mamba deep fusion, to MoE multi-task classification.

[0050] Figure 2 This is a schematic diagram of the Diff-Impute feature completion module in an embodiment of the present invention. The diagram illustrates the specific process of introducing Gaussian noise and temporal embedding into the feature space, and reconstructing the semantic features of the missing modality by predicting the noise residual through a reverse generation process.

[0051] Figure 3 This is a schematic diagram illustrating the triplet loss principle of the cross-modal alignment module in this embodiment of the invention. The diagram shows how, in the feature metric space, triplet constraints can be used to shorten the distance between homologous positive samples and widen the distance between heterologous hard negative samples, thereby eliminating the modal gap.

[0052] Figure 4 This is a schematic diagram of the internal structure of the Mamba deep fusion module (Mamba-Fusion) in an embodiment of the present invention. The diagram shows that after feature stitching, the data enters the SSM module, where a selective scanning mechanism dynamically calculates state parameters based on the input. This involves the process of implementing dynamic gating and interaction of features.

[0053] Figure 5 This is a t-SNE visualization comparison of the feature space distribution between the method of this invention and the single-modal method. (a) shows the feature distribution of the visible light mode only, and (b) shows the feature distribution after fusion according to this invention.

[0054] Figure 6 This is a schematic diagram of the confusion matrix of the method of the present invention in the model determination task.

[0055] Figure 7 This is a verification diagram of the single-sample dual-modal recognition result of the method in the embodiment of the present invention. Among them, (a) shows the recognition result of the visible light modality, and (b) shows the recognition result of the infrared modality. Both diagrams present a comparison between the predicted results of the target category, model and action and the real label, which intuitively reflects the accurate recognition performance and high confidence output characteristics of the model under different visual modalities. Detailed Implementation

[0056] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0057] Example 1:

[0058] This embodiment provides the data preparation and computing environment configuration required for implementing the method of the present invention.

[0059] The dataset required for this invention is collected through real-world data collection, web crawling, and simulation. It constructs a multimodal benchmark classification dataset covering two major target categories: ships and aircraft. This dataset includes ten types of targets (aircraft carriers, cargo ships, container ships, tugboats, oil tankers, yachts, civil airliners, fighter jets, helicopters, small rotary-wing UAVs, six types of ships, and four types of aircraft) and nine types of actions (ships docking at shore, ships sailing on water, civil airliners landing on the ground, civil airliners flying in the air, UAVs flying in the air, fighter jets flying in the air, fighter jets dropping decoys, helicopters flying in the air, and helicopters landing on the ground). This dataset achieves temporal synchronization and spatial correlation of visible light images, infrared images, feature signals, and AIS data, providing high-quality training, validation, and testing samples for Transformer-based multimodal recognition networks.

[0060] The overall dataset coverage is shown in Table 1.

[0061] Table 1. Task-modal distribution of the overall dataset

[0062]

[0063] The dataset employs a multi-level directory mapping annotation strategy, where a single original image is simultaneously associated with multi-dimensional labels through its file path. For example, the path "Visible Light Image / Category / Ships / Docked Tugboat 1.jpg" is automatically mapped to the model label "Tugboat" and the action label "Ships docked at shore". "×" indicates that no corresponding data was collected. The dataset was randomly divided into training and test sets at a 9:1 ratio, resulting in the numbers shown in Table 2. The training set contains 31,308 data points, while the test set contains 3,488 data points.

[0064] Table 2 Modal distribution of training / test sets

[0065]

[0066] Additional time-series datasets are shown in Table 3, where each row represents a visible light image, an infrared image, and AIS status data.

[0067] Table 3 Distribution of Time Series Datasets

[0068]

[0069] The simulation experiment configuration environment of this invention is shown in Table 4.

[0070] Table 4 Simulation Experiment Environment Configuration

[0071]

[0072] This invention performs a unified preprocessing procedure on the acquired multimodal raw data to adapt to network input requirements. For visible light and infrared images, the sizes are uniformly adjusted to... The pixels are normalized to meet the input standards of the backbone network; for one-dimensional signal feature data, a short-time Fourier transform (STFT) is used to convert it into a two-dimensional time-frequency spectrum, and the size is adjusted accordingly. Pixels; for AIS time-series data, it is constructed into "samples" Time step The system uses a three-dimensional tensor format with a fixed time step of 10 and a feature dimension of 3 (including latitude, longitude, speed, and heading information) to preserve the dynamic temporal features of the trajectory. Furthermore, to train the Diff-Impute diffusion completion module, a random masking strategy is applied to the input samples with a 30% probability during the training phase to simulate modal missingness. During the testing phase, data inherently missing from the sensor is processed directly without additional masking.

[0073] Example 2:

[0074] This embodiment combines Figures 1 to 4 This paper details the specific construction process and internal working mechanism of a Mamba-Multimodal Network based on the Mamba architecture for addressing modality loss, as proposed in this invention.

[0075] like Figure 1 As shown, the overall recognition framework of this invention adopts an end-to-end design, mainly composed of four core components cascaded together: a multimodal heterogeneous feature encoding network, a Diff-Impute feature completion module, a cross-modal alignment and Mamba dynamic fusion module, and a multi-task hybrid expert (MoE) classification module. The following sections, in conjunction with the accompanying drawings, provide a detailed description of each module.

[0076] Construction of a multimodal heterogeneous feature encoding network: To address the strong heterogeneity of the input data, this embodiment designs the following... Figure 1 The dual-path differential encoding architecture shown on the left, labeled "Encoder," maps raw data of different dimensions to a unified semantic feature space (dimension). ).

[0077] For visible light images (Vis), infrared images (IR), and signal characteristic spectrograms (Sig), the input size is... Image data is processed through a Transformer layer to extract deep visual features. To preserve the general representational capabilities learned by the pre-trained model on large-scale data, the backbone parameters are frozen during training. The backbone network output is then connected to a trainable linear projection layer, which consists of "LayerNorm + GELU activation function + linear layer," compressing and mapping the feature dimensions into a 512-dimensional feature vector. .

[0078] A dedicated Mamba timing encoder is built for AIS data. The input shape is... The AIS sequence data (containing latitude, longitude, and airspeed information at 10 time steps) is first mapped to a high-dimensional space through a linear layer, and then input into a two-layer stacked Mamba block. The Mamba block utilizes its unique state-space model (SSM) to efficiently capture the long-range temporal dependencies of the track data. The final output is processed by global average pooling (GAP) to obtain a 512-dimensional global semantic representation. .

[0079] Construction of the Diff-Impute Feature Completion Module: To address the challenge of modal loss caused by sensor malfunctions in real-world applications, this embodiment deploys a Diff-Impute module after the feature encoding stage (structure as follows). Figure 2 (As shown).

[0080] This module's design philosophy differs from traditional pixel-level reconstruction, operating directly in a low-dimensional feature space. For modal feature locations marked as missing by the mask, the system first initializes them with standard Gaussian noise, and then concatenates the generated sinusoidal time-step embedding vector with this noise vector as input. The core completion network employs a lightweight multilayer perceptron (MLP) structure, including... "At the hierarchical level, a single-step inverse generation strategy is adopted to directly predict the noise residual based on the input, and reconstruct the initial noise into a semantic feature vector that approximates the true distribution in one step. During the training phase, the mean squared error (MSE) loss function is used to constrain the generated complete features to be consistent with the original features that actually exist in the modality, thereby forcing the network to learn the manifold distribution of each modality feature."

[0081] Construction of Cross-Modal Alignment and Mamba Dynamic Fusion Modules: Addressing the "modal gap" caused by differences in imaging mechanisms among different sensors and the need for heterogeneous feature interaction, this embodiment adopts a cascaded architecture, sequentially constructing cross-modal alignment and Mamba deep fusion modules. Their working principles correspond to... Figure 3 and Figure 4 .

[0082] The preprocessing stage before feature fusion mainly relies on the triplet structure. The system calculates the Euclidean distance between the anchor sample and the homologous positive sample and the heterologous hard negative sample, and uses the triplet loss function to force the homologous sample distance to be closer and the hard negative sample distance to be farther to achieve feature semantic alignment; then, the aligned visible light, infrared, signal, AIS four modal features and global features are concatenated in the channel dimension and projected into the latent space, input into the stacked Mamba block, and the state parameters are dynamically calculated according to the current input sample content using a selective scanning mechanism. This mechanism enables adaptive gating of heterogeneous features (such as suppressing blurred features and amplifying clear features) through this input-dependent characteristic, and finally outputs highly discriminative multimodal fusion features through linear projection and residual connection. This mechanism endows the model with adaptive gating capability, which can automatically suppress the weights of low-quality or noisy modes and select discriminative information of high-value modes for deep interaction and fusion;

[0083] Multi-task Hybrid Expert (MoE) Classification Module Construction: Given the significant differences in feature focus among category recognition, model determination, and motion analysis tasks (such as the difference between texture details and motion trajectories), this embodiment abandons the traditional shared fully connected layer architecture and instead adopts an independent multi-task MoE classification head (e.g., ...). Figure 1 (As shown on the right), to avoid feature conflicts in multi-task joint learning.

[0084] Each classification head contains four expert MLPs focusing on different feature subdomains and a trainable gating network. During inference, the gating network receives fused features and dynamically calculates the confidence weights of each expert network for the current sample using the Softmax function. The final classification result is the weighted sum of the outputs of all experts. This mechanism achieves feature decoupling between tasks. For example, in action recognition tasks, the gating network automatically activates the experts that focus on temporal features, thereby effectively avoiding negative transfer in multi-task joint learning and significantly improving the overall recognition accuracy.

[0085] Example 3:

[0086] This embodiment, based on the dataset constructed in Embodiment 1 and the network model constructed in Embodiment 2, details the training strategy, hyperparameter configuration, and construction and optimization process of the composite loss function of the present invention.

[0087] Training hyperparameters and optimization strategy configuration: Ensuring the convergence stability of the model during training on heterogeneous multimodal data is crucial for optimization; therefore, this embodiment employs an end-to-end joint optimization strategy. Specifically, the feature embedding dimensions for all modalities are first uniformly set. The hidden layer expansion coefficient of the Mamba fusion module is set to 2, and the state dimension (State Dim) is set to 64; the optimizer is AdamW, and the initial learning rate is set to... The weight decay coefficient is To suppress overfitting, a ReduceLROnPlateau learning rate scheduler is introduced during training. When the validation set loss does not decrease within 3 consecutive epochs, the learning rate automatically decays to 0.5 times the current value. The entire training process is set to 50 epochs, with a batch size of 8, and an early stopping mechanism is configured. If the average accuracy of the validation set does not improve within 8 consecutive epochs, training is terminated early.

[0088] This invention designs a composite objective function that includes multi-task classification loss, multi-modal alignment loss, and feature reconstruction loss. Parameter updates are achieved through joint optimization, and the total loss function is defined as:

[0089]

[0090] in, For the multi-task classification loss, the standard cross-entropy loss between the predicted probability distribution and the true label is calculated for the three sub-tasks: category identification, model determination and behavior analysis. The sum of the three is then used as the total classification loss. For multimodal alignment loss, a hard-negative triplet loss is employed, aiming to minimize the distance between homologous positive sample pairs while maximizing the distance to the hardest heterogeneous negative sample within the batch. The distance margin is used to define the boundary values. Set it to 0.2; For feature reconstruction loss, mean squared error (MSE) is used to measure the difference between the completed features generated by the Diff-Impute module and the original true features. This is calculated only for modalities that are randomly masked but still exist. To balance the gradient contributions of each task, the loss weights are aligned. Weighting of Completion Loss All are set to 0.1.

[0091] Training Process Execution Logic: The training process follows a cyclical logic of "random masking - forward propagation - backpropagation update". In each training iteration step, a batch of multimodal data is first randomly masked with a 30% probability to artificially simulate a modality missing scenario. Then, the processed data is input into the network constructed in Example 2. The Diff-Impute module completes the missing features in the feature space, the Mamba module completes the fusion, and the MoE module outputs the prediction results of the three tasks. Next, the total loss is calculated according to the above composite loss function, and the gradient is calculated through the backpropagation algorithm. Finally, the network parameters are updated using the optimizer until the preset number of rounds is reached or the early stopping condition is triggered. The model weights with the highest accuracy on the validation set are saved as the final deployment model.

[0092] Example 4:

[0093] This embodiment, based on the experimental environment and dataset described in Embodiment 1, performs performance testing and comparative verification on the network model constructed in Embodiment 2. Through modality loss comparison experiments and core component ablation experiments, combined with performance benchmarking against mainstream multimodal models, the effectiveness and superiority of this invention in sea and air target recognition tasks are comprehensively verified.

[0094] This experiment focuses on scenarios where sensor modalities are randomly missing in real-world applications. By setting up three test groups—"complete modality," "single modality," and "partial modality missing"—it verifies the complementary value of multimodal fusion and the robustness of this invention in dealing with modality loss. No additional fusion baselines were set; the experiment only visually demonstrates the effectiveness of the model's core mechanism through the performance differences of different modal combinations. The specific experimental design is as follows:

[0095] Complete modal group (Ours): Input four types of complete modal data: visible light, infrared, signal, and AIS, and enable all the core mechanisms of this invention (feature completion, cross-modal alignment, Mamba dynamic fusion, and MoE classification).

[0096] Single-modal group (Only-XX): Only input data of a certain type of modality (e.g., Only-Vis only inputs visible light) to verify the limitations of single-modal recognition;

[0097] Partially missing modal groups (No-XX / No-XX-XX): Intentionally removing one or two types of modal data (e.g., No-Vis removes visible light, No-Signal-Ais removes signal and AIS), simulating modal missing scenarios caused by sensor failure or interference:

[0098] Table 5. Performance comparison of different methods on multimodal sea and air target recognition tasks.

[0099]

[0100] Table 5 clearly shows that single-modal recognition performance has significant shortcomings. The average accuracy of visible light modality alone is less than 50%, and the model recognition accuracy is only 27.46%, far from meeting practical needs. However, multimodal fusion significantly improves performance, with the average accuracy of the complete modality group (Ours) reaching 99.50% and the model recognition accuracy approaching 100%, fully demonstrating the complementary advantages of multi-source heterogeneous data. In modality-deficient scenarios, when only a single modality (such as No-Vis or No-Ais) is missing, the model's average accuracy remains above 97%, indicating that the feature completion and dynamic fusion mechanism of this invention can effectively compensate for the impact of single-modality loss. However, when infrared modalities (such as No-Ir or No-Ir-Ais) are missing, the model recognition accuracy drops sharply (to a minimum of only 25.29%), indicating that infrared modalities play an irreplaceable role in fine-grained model differentiation. When two modalities (such as No-Ir-Ais) are simultaneously missing, the performance degradation is most significant, further confirming the necessity of multimodal fusion.

[0101] To clarify the role of each core component in this invention, an ablation experiment was conducted using a leave-one-out strategy. The experiment used the complete model (Ours) as a baseline, and sequentially removed four core components: “input modality completion (F1)”, “multimodal feature alignment (F2)”, “Mamba fusion (F3)”, and “MoE classification architecture (F4)”. The contribution value of each component was quantified by comparing performance changes. Detailed results are shown in Table 6.

[0102] Table 6 Ablation Experiment Results of the Impact of Key Components on Model Performance

[0103]

[0104] As shown in Table 6, F1 represents input modality completion. When this factor exists, it indicates that missing modalities are filled with features. When this factor is ablated, a zero-filling strategy is adopted. F2 represents cross-modal feature alignment. When this factor exists, it indicates that multimodal feature alignment is first performed on the encoded multimodal features. When this factor is ablated, the encoded features directly enter the subsequent multimodal feature fusion step without any task operation. F3 represents a multimodal feature fusion strategy based on the Mamba concept. When this factor exists, it indicates that Mamba-based fusion operation is performed on the multimodal features. When this factor is ablated, a static linear concatenation strategy is adopted. F4 represents the MoE classification architecture. When this factor exists, it indicates that the three-class recognition tasks at the end adopt a classification strategy based on a hybrid expert model. When this factor is ablated, a classification strategy in which the three-class end tasks share a fully connected layer is adopted. The baseline performance refers to the original model performance without any module ablation, which is as follows: category recognition accuracy is 100%, model recognition accuracy is 99.40%, and action recognition accuracy is 99.50%.

[0105] The ablation experiments show that all four core components contribute significantly to model performance. The MoE classification architecture has the greatest impact; removing it directly reduces the accuracy of type recognition to 94.39% and action recognition to 96.68%, indicating that the feature requirements for categories, types, and actions differ, making the shared fully connected layer inadequate. Cross-modal alignment and Mamba fusion modules are also crucial; their removal significantly lowers performance, especially Mamba fusion, which outperforms static linear concatenation, demonstrating that dynamic gating mechanisms can better filter useful features. While the input modality completion module provides a smaller performance boost, it still shows that feature space completion, compared to zero-padding, better preserves feature integrity, allowing the model to maintain feature distribution continuity even with modality loss. This is more robust than the zero-padding strategy, echoing the high performance in modality loss scenarios shown in Table 5. Removing all four components results in a significant drop in model performance, with accuracy for all three categories falling to around 60%, further illustrating that these four components are key to the model's optimal performance.

[0106] This embodiment selects nine mainstream multimodal models, including CLIP, Flamingo, and Qwen3, and compares them on the same dataset. The focus is on examining the performance of three tasks: class recognition, model recognition, and action recognition. The results are shown in Table 7.

[0107] Table 7 Performance comparison of different domain models on the three sub-tasks

[0108] As shown in Table 7 and the accompanying figures, the method of this invention demonstrates superior comprehensive performance in cross-domain maritime and airspace identification tasks. In the basic category identification task, all models achieved 100% accuracy, indicating a relatively low difficulty in macroscopic semantic differentiation. In the fine-grained model identification task, this invention (0.99) maintains a first-tier level comparable to large-scale pre-trained models such as CLIP and Flamingo, significantly outperforming Uni-Perceiver (0.89). In the most challenging action recognition task, the general multimodal large model Qwen3 achieved an accuracy of only 0.86, while this invention reached 0.99. This is mainly attributed to the general large model's lack of deep understanding of AIS trajectory and electromagnetic signal data, while the Mamba encoder and MoE architecture of this invention are specifically optimized for temporal trajectory features, thus achieving performance exceeding that of the general large model in maritime and airspace specialized tasks. Furthermore, t-SNE feature visualization and confusion matrix analysis further corroborate the above conclusions. This advantage benefits from the Mamba dynamic fusion (screening high-value features) and MoE classification (decoupling task requirements) verified in Table 6, and also confirms the core value of multimodal fusion compared to single-modal fusion in Table 5. Ultimately, it surpasses general large models in professional domain tasks. The MoE classification head, by capturing trajectory features, achieves accurate differentiation between "small rotary-wing drones" and "helicopters" with extremely similar appearances, with a misclassification rate close to zero.

[0109] Example 5:

[0110] This embodiment utilizes t-SNE feature dimensionality reduction visualization technology and confusion matrix analysis method to further verify the effectiveness of the multimodal recognition model constructed in Embodiment 2 in feature space reconstruction and fine-grained classification.

[0111] This embodiment extracts the penultimate layer features of the model in the "model determination" task and projects them onto a two-dimensional plane using the t-SNE algorithm, visually demonstrating the Mamba-Fusion module's ability to reshape semantic features. Comparison results are shown in the appendix to the manual. Figure 5 As shown.

[0112] like Figure 5 As shown in (a), when using only single-mode (e.g., visible-only) data, due to insufficient nighttime lighting and sea surface clutter interference, feature points of different ship types exhibit severe aliasing in space, with blurred class boundaries. In particular, it is difficult to effectively distinguish between visually similar "cargo ships" and "container ships." In contrast, as shown in the attached figure… Figure 5As shown in (b), after the cross-modal alignment and Mamba dynamic fusion processing proposed in this invention, the feature distribution is significantly optimized: feature points of the same class exhibit extremely high intra-class compactness, while clear inter-class separation is formed between different classes. Even fine-grained classes that were originally easily confused in a single modality form independent clusters in the fused feature space. This result intuitively demonstrates that the Mamba module successfully utilizes signal features and the strong discriminative information of AIS to correct the distribution deviation of visual features, constructing a more consistent and separable multimodal semantic manifold.

[0113] To address potential classification blind spots and error correction mechanisms in the model, this embodiment statistically analyzes the prediction results for the most challenging "model determination" task and plots a confusion matrix (see attached). Figure 6 We will conduct an in-depth analysis.

[0114] Appendix Figure 6 The statistical results presented show that the model achieves an accuracy rate exceeding 98% in the vast majority of categories, exhibiting extremely high diagonal dominance. Specifically, the model not only achieves 100% accurate identification of "civilian passenger aircraft" and "cargo ships" with sufficient samples, but also performs exceptionally well in distinguishing between easily confused "small rotary-wing drones" and "helicopters." Traditional single-modal models often produce a large number of misclassifications due to the similar appearance (both have rotor structures) and similar radar cross sections (RCS) of the two. However, the MoE classification head introduced in this invention, through a dynamic routing mechanism, can automatically pay attention to subtle differences in the behavioral trajectories (actions) of the two (such as the hovering frequency of drones and the flight altitude of helicopters), thereby achieving accurate error correction with a misclassification rate close to zero. The very few remaining misclassifications in the figure are mainly concentrated between "container ships" and "oil tankers." The reason for this is that the infrared thermal features of some long-distance samples are extremely similar under adverse sea conditions, but the overall error rate is still far lower than that of existing baseline models.

[0115] In addition to statistical validation using batch samples, this embodiment also selected different modal data of typical samples in the test set for single-sample validation, and the results are shown in the appendix. Figure 7 As shown. Figure 7(a) and (b) represent the visible light and infrared modal recognition results for the same target, respectively. The results demonstrate that the model accurately outputs recognition results regardless of the visual modality input: in the category recognition task, the target is identified as a ship with a confidence level of 1.000; in the model recognition task, it is accurately identified as a yacht with a high confidence level of over 0.955; and in the action recognition task, it correctly identifies the action state of "ship docked at the shore" with a confidence level of 1.000. This result echoes the high accuracy reflected in the confusion matrix mentioned earlier, indicating that the model not only performs stably in batch testing but also maintains high accuracy and high confidence in single-modal inference scenarios, further confirming the reliability and adaptability of the method in practical sea and air target recognition.

[0116] Matters not covered in this invention are common knowledge.

[0117] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A multimodal target intelligent recognition method based on Mamba architecture for modality missing features, the method comprising: Step 1: Acquire raw multimodal data of sea and air targets, including visible light images, infrared images, signal feature spectrum diagrams, and AIS time series data; Step 2: Construct a multimodal heterogeneous feature encoding network; To address the highly heterogeneous nature of air and sea target data, a differentiated feature extraction strategy is designed. For visible light images, infrared images, and signal feature spectrograms, a pre-trained large visual model is used as the backbone network to extract features under frozen parameters. Different modalities are then mapped to a unified semantic space through a trainable projection layer. For AIS time-series data, a Mamba time-series encoder is constructed. Leveraging the linear computational complexity advantage of state-space models in processing long-sequence data, we can capture the navigation trajectory and state changes of ships or aircraft and extract global temporal semantic representations. Step 3: Feature space completion based on diffusion model; To address modal loss caused by sensor malfunctions, a feature completion module is constructed. The specific method of the feature completion module includes: Step 3.1: Initialize the locations of the modal features marked as missing by the mask to standard Gaussian noise; Step 3.2: Generate sinusoidal time-step embedding vectors; Step 3.3: Concatenate the standard Gaussian noise obtained in Step 3.1 with the sinusoidal time step embedding vector generated in Step 3.2; Step 3.4: Input the concatenation result obtained in Step 3.3 into the trained multilayer perceptron to directly reconstruct a semantic feature vector that approximates the true distribution; the multilayer perceptron includes, in sequence: a first fully connected layer, a layer normalization layer, a GELU activation function, a second fully connected layer, a layer normalization layer, a GELU activation function, and a third fully connected layer; wherein, the input dimension of the first fully connected layer is "the sum of the feature dimension D and the time step embedding dimension H", and the output dimension is H, and the output dimension of the third fully connected layer is D; Step 3.5: Replace the masked missing modal feature positions in the global temporal semantic representation obtained in Step 2 with the semantic feature vector obtained in Step 3.4 to obtain a new global temporal semantic representation; Step 4: Cross-modal feature semantic alignment; Step 4.1: Construct triples of anchor points, homologous positive samples, and heterologous hard negative samples; Step 4.2: Use the triplet loss function to constrain the distribution of each modality in the common metric space to complete the semantic alignment of each modality; where the common metric space is an embedding space of uniform dimension. The distance between each modal feature is calculated after L2 normalization within this space; multimodal alignment loss. The loss for mining triples from hard samples is defined as: and order For pairwise combinations of different modes Average value; where, The distance between different modal features of the same sample. This is the minimum distance among the samples in the batch that are different from the anchor label. It is the interval constant; Step 5: Dynamic feature fusion based on Mamba architecture; The completed visible light image, infrared image, signal feature spectrum, and AIS time series data four-modal features are mapped as follows: First, the alignment module linearly projects them onto the common dimension. get A global alignment semantic vector is generated by the alignment network. ; then The channel dimension is concatenated and used as input for Mamba-Fusion; within the Mamba-Fusion module: 1) The input splicing features are first mapped to the fusion latent space through the input projection layer; 2) After expanding the vector into a sequence of length 1, selective state-space modeling is performed by sequentially inputting two stacked Mamba blocks. 3) Mamba stacked output is compressed back via output projection Dimension, to obtain the final fusion feature ; Each Mamba block includes the following along the data flow direction: input normalization, channel expansion, depthwise convolution, non-linear activation, channel compression, and residual connection; the depthwise convolution is a selective state space, and the selective scanning mechanism adaptively generates state parameters according to the input to realize dynamic gating and interaction of different modal information; Step 6: Multi-task hybrid expert classification prediction; The multi-task hybrid expert classification module configures a gating network and several expert networks for each recognition task; the multi-task hybrid expert classification module configures an independent gating network and several expert networks for each recognition task; for any task The gating network adopts the same structure as the multilayer perceptron in step 3.4, and integrates the features. Mapped to expert weight vector ; No. A network of experts For a multilayer perceptron, output the logits for this task: The final task output is a weighted sum of the outputs from all experts: ； By using independent gating for each task and dynamic expert routing, different expert combination paths are formed for the three tasks of category, model, and action on the same fusion feature, thereby decoupling the task concerns and suppressing negative transfer. During training, only the modalities that "originally exist" are tested with probability during the training phase. Random masking is used to simulate missing features, and a constraint is set: for any sample, if all of its "real-world modalities" are randomly masked, then this sample will not undergo completion supervision in this round; the completion loss is only applied to the supervision mask. Calculate the position of 1; To address the differences in feature focus among the three levels of tasks—class recognition, fine-grained model determination, and action analysis—a multi-task hybrid expert classification module is constructed. Each task is configured with an independent gating network and several expert networks. Through a dynamic routing mechanism, the gating network automatically allocates the weights of each expert network based on the input feature attributes, thereby achieving feature decoupling between tasks, avoiding feature conflicts in multi-task joint learning, and outputting recognition results for each category.

2. The multimodal target intelligent recognition method based on Mamba architecture for modality missing as described in claim 1, characterized in that the specific structure of the large visual model in step 2 is as follows: the input image is obtained by image block embedding to obtain patch sequence features; the trainable class label [CLS] is concatenated to the beginning of the sequence and superimposed with position encoding, and then globally interacted through several Transformer encoding blocks; finally, the output is layer normalized and the features corresponding to the [CLS] position are taken as visual semantic representations, and then mapped to a unified 512-dimensional semantic space through a trainable projection layer; wherein, The image segmentation and embedding module divides the input image into several image blocks and maps them to sequence features; the category labeling and position encoding module concatenates the category label vector to the beginning of the sequence and superimposes the position encoding; each Transformer encoding block includes a multi-head self-attention submodule and a feedforward network submodule in sequence, and residual connections and layer normalization are set outside each submodule to complete information interaction and stable training; the final output is normalized and the features corresponding to the category label are taken as visual semantic representations, and mapped to a semantic space of a unified dimension through a trainable projection layer.

3. The multimodal target intelligent recognition method based on Mamba architecture for modality missing as described in claim 1, characterized in that the total loss function of the multimodal target intelligent recognition method is defined as: ； in, For the multi-task classification loss, the standard cross-entropy loss between the predicted probability distribution and the true label is calculated for the three sub-tasks of category recognition, model determination and behavior analysis, and the sum of the three is used as the total classification loss. For multimodal alignment loss, hard sample mining triplet loss is adopted; The average of the pairwise combinations of different modes is obtained ; For feature reconstruction loss, mean squared error is used to measure the difference between the completed features generated by the Diff-Impute module and the original true features, and the calculation is only performed on the modalities that are randomly masked but originally exist. , This is the loss weight.

4. The multimodal target intelligent recognition method based on Mamba architecture for modality missingness as described in claim 3, characterized in that the multi-task classification loss... For three tasks—category identification, model determination, and behavior analysis—the standard cross-entropy loss between the predicted probability distribution and the true label is calculated separately, and the sum of these three losses is used as the total classification loss. ； in, For batch size, For the first The sample at the th Real labels on each task This represents the predicted probability for the corresponding task; this loss term is used to optimize the prediction accuracy for each task.

5. A multimodal target intelligent recognition method based on Mamba architecture for modality missingness as described in claim 3, characterized in that the multimodal alignment loss is defined as: ； in, For a set of modal pairs, Indicates anchor point and Positive samples distance, For anchor points and negative samples distance, The boundary constant is used; the goal of this loss is to optimize the consistency of cross-modal features, bringing homologous modes closer together and pushing heterologous modes further apart.

6. The multimodal target intelligent recognition method based on Mamba architecture for modality missingness as described in claim 3, characterized in that, Feature Reconstruction Loss This constraint is used to maintain consistency between the generated completed features and the original true features. For modalities that are randomly masked but whose original data truly exists, the mean squared error loss is used as a metric. ； in, For the generated completion features, Original features Represents element-wise product. It is a mask that ensures that loss calculation is performed only on modes that exist and are masked.

Images