Ship target recognition method based on multi-modal attention mechanism

Abstract

The application provides a ship target recognition method based on a multi-modal attention mechanism, multi-source heterogeneous sensors are used to synchronously collect multiple characteristics of a ship target to form multi-modal original feature data, a multi-modal attention fusion network model is used to map the original features of each mode to a unified hidden space, calculate the attention weight between modes to generate enhanced features, and thus the complementary relationship between modes is mined through cross-modal interaction, the enhanced features are re-scaled through a channel attention mechanism, the feature channels with rich information are strengthened, and the channels disturbed or with little information are inhibited, and finally a deep decision network is used to output a recognition result. The method can effectively improve the accuracy and reliability of ship target recognition, and enhance the feature extraction and discrimination ability of the model for ship targets in complex environments.

Description

Technical Field

[0001] This application relates to the field of target recognition technology. Specifically, this application relates to a method for ship target recognition based on a multimodal attention mechanism. Background Technology

[0002] Traditional ship target identification methods relying on single sensors often fall short in handling complex scenarios. Radar features are susceptible to interference, and infrared signals are easily blocked; data from any single modality is insufficient to independently support highly reliable identification tasks. To address this issue, multi-sensor information fusion technology has been introduced into the field of target identification. By comprehensively utilizing sensor data with different physical characteristics, such as radar and infrared sensors, it aims to improve overall identification performance. Early fusion algorithms often employed simple methods based on feature-level concatenation or decision-level voting. While these methods achieved some effectiveness, they failed to fully consider the differences in the contributions of different modalities, resulting in low accuracy in ship target identification. Summary of the Invention

[0003] Therefore, it is necessary to provide a ship target recognition method based on a multimodal attention mechanism to address the aforementioned technical problems.

[0004] Firstly, this application provides a ship target recognition method based on a multimodal attention mechanism, including: Electromagnetic scattering characteristics, thermal radiation characteristics, and electromagnetic emission characteristics of ship targets are collected synchronously by multi-source heterogeneous sensors to form multimodal raw characteristic data; The multimodal raw feature data is input into a trained multimodal attention fusion network model, which outputs a recognition result corresponding to the ship target category, specifically including: The original multimodal feature data are input into the corresponding feature extraction branches, and after nonlinear transformation, they are mapped to a latent space of a unified dimension to obtain the latent features of each modality. The latent features of each modality are concatenated to form a joint feature, and the inter-modal attention weights are calculated based on the joint feature. The inter-modal attention weights are then used to generate enhanced features after cross-modal interaction. The enhanced features are subjected to global average pooling along the channel dimension to obtain channel descriptors. The channel descriptors are then passed through two fully connected layers to generate channel attention weights. The channel attention weights are used to rescale the enhanced features channel by channel to obtain the reconstructed features. The reconstructed features are input into a deep decision network consisting of multiple residual blocks, and the network outputs the identification result corresponding to the ship target category.

[0005] In one embodiment, the original multimodal feature data are input into the corresponding feature extraction branches, and after nonlinear transformation, mapped to a latent space of a unified dimension to obtain the latent features of each modality, specifically including: The following transformations are performed on the original feature data of each modality: ; ; in, Represents the raw feature data of each modality. This represents the activation function. This indicates a batch normalization operation. , Represents the learnable weight matrix. , Indicates the bias term. Represents the hidden features of each modality.

[0006] In one embodiment, the latent features of each modality are concatenated to form a joint feature, and inter-modal attention weights are calculated based on the joint feature. The enhanced features resulting from cross-modal interaction are then generated using these inter-modal attention weights, specifically including: The latent features of each modality are concatenated to form a joint feature; A linear transformation is performed on the joint features to generate a query vector, a key vector, and a value vector; The intermodal attention weights are obtained by performing a scaled dot product calculation based on the query vector and the key vector. The enhanced features after cross-modal interaction are calculated based on the intermodal attention weights and the value vector.

[0007] In one embodiment, the enhanced features are subjected to global average pooling along the channel dimension to obtain channel descriptors. The channel descriptors are then passed sequentially through two fully connected layers to generate channel attention weights. The enhanced features are then rescaled channel-by-channel using these channel attention weights to obtain the reconstructed features. Specifically, this includes: The enhanced features are subjected to global average pooling along the channel dimension to compress global spatial information into channel descriptors; The channel descriptors are sequentially passed through two fully connected layers to generate channel attention weights; the first fully connected layer is used to reduce the dimensionality, and the second fully connected layer is used to restore the dimensionality. The channel attention weights and the enhanced features are multiplied channel by channel to obtain the reconstructed features.

[0008] In one embodiment, the computation process of the residual block in the deep decision network is as follows: ; ; in, Indicates block input, , Represents the weight matrix. , This represents the bias term, and the output result is... .

[0009] In one embodiment, the optimization objective of the multimodal attention fusion network model is to minimize the cross-entropy loss between the predicted class and the true label, and its mathematical expression is: ; in, N Indicates batch size, Indicates sample Category The true label, This represents the probability predicted by the model.

[0010] In one embodiment, it further includes: Calculate the recognition performance index; Calculate the anti-interference performance indicators; Calculate timeliness performance indicators; The comprehensive countermeasure effectiveness index is calculated by weighting and fusing the identification effectiveness index, the anti-interference performance index, and the timeliness performance index. The performance of the multimodal attention fusion network model is evaluated based on the comprehensive adversarial effectiveness index.

[0011] In one embodiment, the recognition performance metrics include one or more of accuracy, precision, recall, and F1 score.

[0012] In one embodiment, the electromagnetic scattering characteristics include radar cross section, Doppler frequency shift, range profile characteristics, micro-Doppler characteristics, and polarization characteristics; The thermal radiation features include thermal contrast, temperature distribution statistics, contour shape factor, heat source distribution, and image texture. The electromagnetic emission characteristics include radio frequency, modulation parameters, direction of arrival, signal strength, and fingerprint characteristics.

[0013] The aforementioned ship target recognition method based on a multimodal attention mechanism simultaneously collects multiple features of ship targets from multiple heterogeneous sensors to form multimodal raw feature data. A multimodal attention fusion network model first maps the raw features of each modality to a unified latent space and calculates intermodal attention weights to generate enhanced features, thereby mining complementary relationships between modalities through cross-modal interaction. Then, a channel attention mechanism is used to rescale the enhanced features channel by channel, strengthening information-rich feature channels and suppressing channels that are interfered with or have scarce information. Finally, a deep decision network is used to output the recognition result. This method can effectively improve the accuracy and reliability of ship target recognition and enhance the model's ability to extract and discriminate features of ship targets in complex environments. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a ship target recognition method based on a multimodal attention mechanism in one embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0016] Traditional ship target recognition methods suffer from low recognition rates and poor robustness in complex electromagnetic environments. This application proposes an intelligent recognition algorithm based on a Transformer-Enhanced Fusion Network (TEFNet). It achieves cross-modal adaptive fusion through a channel attention weighting mechanism, dynamic interaction and fine alignment of features between modalities, and introduces a residual enhancement structure to strengthen feature reuse, thereby improving the model's deep representation capabilities. Simulations are conducted to demonstrate ship target recognition scenarios. The model constructs a three-modal feature space encompassing radar, infrared, and electronic support measures (ESM). An adversarial training strategy is employed to improve the model's robustness in electronic interference environments. A comprehensive Combat Effectiveness Index (CEI) evaluation system is established, assessing macro-F1 recognition performance, robustness, and latency. Experiments show that, under strong interference conditions, the proposed model algorithm outperforms the baseline methods of single Transformer Fusion Network (TFN) and Graph Fusion Network (GFN) in terms of the Comprehensive Effectiveness Index (CEI) for intelligent identification of five types of ship targets, providing efficient and reliable algorithmic support for intelligent anti-interference identification in complex electromagnetic environments.

[0017] The multimodal attention fusion network involved in this application has the following core innovations: ① Designing a multi-branch feature extraction structure to maintain modality specificity and lay the foundation for subsequent fusion; ② Introducing a lightweight channel attention mechanism to achieve adaptive fusion of cross-modal features, the core of which is to simulate the dynamic evaluation and decision-making process of sensor reliability by experts in a jammed environment; ③ Utilizing residual connections to enhance gradient propagation and feature reuse capabilities, thereby improving the model's stability in adversarial environments. To further quantify and evaluate the comprehensive performance of the algorithm in near-realistic adversarial environments, this application constructs an evaluation standard based on "accuracy," "stability," and "speed" adversarial rules. It measures intelligent recognition performance using the Macro-F1 score, employs the Projected Gradient Descent (PGD) adversarial attack algorithm for robustness quantification, measures temporal performance using model inference latency, and aggregates these multi-dimensional parameters into a Comprehensive Adversarial Performance Index (CEI). This aims to provide an effective design, verification, and evaluation method for the engineering application of intelligent ship recognition algorithms.

[0018] In one embodiment, such as Figure 1 As shown, a ship target recognition method based on a multimodal attention mechanism is provided. The method specifically includes the following steps: Step S10: Electromagnetic scattering characteristics, thermal radiation characteristics and electromagnetic emission characteristics of ship targets are collected synchronously through multi-source heterogeneous sensors to form multimodal raw feature data; Step S20 involves inputting the original multimodal feature data into a trained multimodal attention fusion network model, outputting the recognition result corresponding to the ship target category, specifically including: Step S202: Input the original feature data of the multimodal modes into the corresponding feature extraction branches respectively, and after nonlinear transformation, map them to the latent space of a unified dimension to obtain the latent features of each modality; Step S204: Concatenate the latent features of each modality to form a joint feature, calculate the intermodal attention weights based on the joint features, and use the intermodal attention weights to generate enhanced features after cross-modal interaction; Step S206: Perform global average pooling on the enhanced features along the channel dimension to obtain channel descriptors, and pass the channel descriptors through two fully connected layers in sequence to generate channel attention weights. Use the channel attention weights to rescale the enhanced features channel by channel to obtain the reconstructed features. Step S208: Input the reconstructed features into a deep decision network consisting of multiple residual blocks, and output the recognition result corresponding to the ship target category.

[0019] Specifically, TEFNet consists of four core parts: a multimodal feature extraction module, a cross-modal interaction module, an attention-based feature reconstruction module, and a residual enhancement decision module. It is innovatively designed from three levels: feature representation learning, intermodal interaction, and deep network optimization. It aims to achieve adaptive, robust, and efficient fusion of multi-source heterogeneous features to form an end-to-end trainable architecture.

[0020] (1) Multimodal feature analysis The core challenge of multimodal fusion stems from the heterogeneity between different modal data. In the radar, infrared, and ESM modalities involved in this application, this heterogeneity is mainly reflected in feature distribution and physical meaning. They originate from different physical processes (electromagnetic scattering, thermal radiation, and electromagnetic emission), resulting in significant differences in their value distribution, dimensions, and noise patterns. These differences can be quantified through statistical measures such as the KL divergence between the feature dimensions of each modality. Direct fusion would make the model difficult to learn. However, this heterogeneity also presents a strong complementarity. While radar is susceptible to deception and jamming, it provides accurate range and velocity information; while infrared is easily obscured by clouds and fog, its imaging characteristics are difficult to completely counterfeit; ESM can provide "fingerprint" information of the target radiation source, but its positioning accuracy is relatively poor. In adversarial environments, it is difficult for the enemy to simultaneously and effectively jam all modalities. Therefore, an ideal fusion model must be able to dynamically evaluate the reliability of each modality at the current moment and perform weighted fusion accordingly. This constitutes the core motivation for designing the attention fusion mechanism in this application.

[0021] (2) Multimodal feature representation learning Multimodal information fusion refers to the process of combining observational information from different sensors (information sources) according to time or space using computer technology to obtain a more consistent and accurate description of the target or environment. Based on different levels of information abstraction, fusion is typically divided into three levels: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion directly synthesizes the raw data, preserving the most complete information, but it requires high sensor homogeneity and has a heavy computational burden. Decision-level fusion integrates the decision results after each sensor has independently completed its identification process, offering good fault tolerance but resulting in the greatest information loss. The feature-level fusion adopted in this application is a compromise and efficient strategy. It first uses deep neural networks to extract high-dimensional features from heterogeneous sensor data such as radar, infrared, and ESM, and then performs fusion and reprocessing in the feature space. This approach preserves the key information of the original data while reducing the difficulty of direct fusion of heterogeneous data through feature transformation, laying a framework foundation for subsequent introduction of attention mechanisms to achieve adaptive weighted fusion.

[0022] The primary challenge of multimodal learning lies in effectively handling the distributional differences and feature scale mismatches between data from different sensors. In maritime combat scenarios, multiple heterogeneous sensors capture the electromagnetic scattering, thermal radiation, and electromagnetic emission characteristics of targets, which differ significantly in numerical distribution, physical meaning, and noise patterns. Directly using a network with shared weights ignores modal specificity, leading to a decrease in feature representation capabilities.

[0023] To address this issue, TEFNet employs a multimodal feature learning strategy. Assume the input sample consists of feature vectors from three modalities: suppose a multimodal system contains... M Several different sensor modes, each mode providing The eigenvectors of dimension . For the eigenvector of dimension . m Each mode, whose primitive feature representation can be denoted as: To handle the differences in feature distributions between different modalities, TEFNet designed an independent feature extraction branch for each modality. Each branch consists of a nonlinear transformation function. Implementation, in which Let be the dimension of the hidden layer feature space. Specifically, the feature extraction process for each modality can be implemented using a multilayer perceptron: (1) in, and For learnable weight matrix, For bias vectors, This represents the LeakyReLU activation function. This modality-specific approach maintains modality specificity while mapping features from each modality to a unified feature space.

[0024] This application focuses on intelligent target recognition for maritime aircraft. In this scenario, the aircraft typically integrates information from three modes: radar (R), infrared imaging (IR), and electronic support measures (ESM). After front-end signal processing, feature vectors containing the target's physical attributes can be extracted for each mode: ① Radar mode ( m = r ) provides the electromagnetic scattering characteristics of the target, whose feature vector includes five dimensions: radar cross section (RCS), Doppler frequency shift, range profile characteristics, micro-Doppler characteristics, and polarization characteristics; ② Infrared modes ( m = ir ) provides the thermal radiation distribution characteristics of the target, and its feature vector includes five dimensions: thermal contrast, temperature distribution statistics, contour shape factor, heat source point distribution, and image texture; ③ ESM mode ( m = esmThe algorithm provides the electromagnetic radiation signal characteristics of the target, and its feature vector includes five dimensions: radio frequency frequency, modulation parameters, azimuth of arrival, signal strength, and fingerprint features. This application's algorithm receives the aforementioned 15-dimensional raw features and aims to achieve high-precision target classification and recognition through deep learning. Through the above processing, the 5-dimensional raw features of each modality are uniformly mapped to a 32-dimensional latent space representation.

[0025] (2) superscript , , These represent radar, infrared, and ESM modes, respectively. Each mode has an independent feature extraction branch, and each branch consists of two fully connected layers. Batch normalization and LeakyReLU activation functions are introduced to enhance training stability and nonlinear expressive power. Taking the radar branch as an example, its feature transformation process can be described as follows: (3) in, , For learnable weight matrix, , This is the bias term. Similarly, the infrared characteristics can be obtained. and ESM features In this way, the features of each modality are mapped to a unified 32-dimensional feature space, while preserving the modality-specific discriminative information.

[0026] (3) Cross-modal interaction and feature enhancement Traditional multimodal fusion methods often employ simple operations such as concatenation or weighted averaging, neglecting the complex interactions between modalities. Inspired by the self-attention mechanism in the Transformer model, this application designs a lightweight cross-modal interaction module aimed at uncovering the intrinsic relationships between features of different modalities.

[0027] The core idea of ​​the attention mechanism is to mimic the human cognitive process's ability to selectively focus on important information while ignoring secondary information. In deep learning models, it dynamically allocates the contribution of different input components to the output by calculating a set of weight coefficients. The Channel Attention mechanism, which is a key component of the algorithm used in this application, is an important form of spatial attention, typically represented by the Squeeze-and-Excitation Network (SENet) module. This mechanism obtains the global descriptor of each feature channel through a "squeeze" operation (such as global average pooling), then learns the interdependencies between channels and generates the weights of each channel through an "excitation" operation (such as fully connected layers and the sigmoid function), and finally reconstructs the original features through weighted multiplication. This process enables the model to automatically strengthen feature channels that contribute significantly to the recognition task and suppress useless or interfering channels, thereby significantly improving feature representation capabilities. Applying it to multimodal fusion scenarios can be seen as recalibrating the importance of feature channels from different sensors, which is the theoretical core for achieving adaptive fusion.

[0028] The algorithm in this application concatenates the feature vectors of the three modes to form a joint representation. Then, a vector of query, key, and value is generated through linear transformation: (4) in, For learnable parameter matrix, For the projection dimension. Intermodal attention weights are calculated using scaled dot products: (5) The attention matrix The correlation strength between each pair of the three modalities was clearly quantified. Finally, the enhanced features after cross-modal interaction are represented as follows: (6) Global average pooling is performed along the channel dimension to compress global spatial information into a single channel descriptor. (7) Next, the non-linear relationship between channels is learned through two fully connected layers. The first fully connected layer reduces the dimension to introduce a bottleneck structure, and the second fully connected layer restores the original dimension. (8) in, , For learnable weights, It is the ReLU activation function. This is the Sigmoid function. The final reconstructed features are rescaled using attention weights on the original features: (9) in, This indicates channel-wise multiplication. This module enables the model to emphasize information-rich feature channels and suppress irrelevant or noisy channels, achieving adaptive feature reconstruction. This mechanism allows the model to adaptively learn complementary information between modes; for example, when radar signals are interfered with, the model can automatically enhance its dependence on infrared and ESM modes.

[0029] This application models the multimodal fusion process as an input-based dynamic optimization problem. The goal is to learn an optimal fusion weight for each sample. This makes the fused features For target category The most discriminative: (10) in, Indicates mutual information, This is a fusion function. Directly optimizing this problem is extremely difficult. The channel attention mechanism employed in this application is an efficient approximation of this problem. The calculation process of the attention weights can be viewed as a miniature reliability assessment network. It learns the "confidence score" of each modality feature by compressing global information and performing a nonlinear transformation. This score is related to the amount of discriminative information carried by the modality under the current input. Finally, the fused features... It achieves the amplification of information-rich modes and the suppression of information-sparse or interfered modes.

[0030] Classical channel attention mechanisms (such as SENet and ECA) and their variants (such as CBAM with spatial attention) have achieved great success in fields such as image recognition. However, their direct application to multimodal recognition scenarios has limitations. SENet and ECA focus on modeling the relationships between channels within a single modality (such as an image), while this application faces the challenge of fusing three heterogeneous modalities: radar, infrared, and ESM. TEFNet's attention mechanism works on the joint feature channels after cross-modal stitching, with its weights reflecting the relative importance of different modal channels, thus achieving cross-modal adaptive calibration. Its design intent is fundamentally different from SENet / ECA. Although CBAM includes both channel and spatial attention, its spatial attention module is suitable for inputs with two-dimensional spatial structures (such as images). The target features processed in this application are preprocessed feature vectors, where spatial information is highly abstracted, making the spatial attention module in CBAM ineffective. TEFNet abandons spatial attention, focusing instead on channel-dimensional fusion, resulting in a more lightweight and efficient model that better suits the computational constraints of missile platforms.

[0031] In summary, TEFNet's fusion strategy is not a simple aggregation of existing modules, but a targeted design for the specific characteristics of multimodal fusion scenarios—namely, high modal heterogeneity and dynamic changes in modal reliability under adversarial environments. Its core theoretical advantage lies in its use of lightweight channel attention to explicitly model the reliability of different sensor modalities under the current input, and then dynamically weighted fusion accordingly. This mechanism endows the model with inherent anti-interference capabilities: when a modality (such as radar) is subjected to strong interference, its overall reliability score decreases, and the model automatically shifts its decision dependence to the undisturbed infrared or ESM modal, thus achieving an intelligent decision-making process at the algorithm level similar to an expert's "selective trust." This is something that simply applying mechanisms like SENet to individual modalities does not provide.

[0032] (4) The anti-interference advantage of attention mechanism The purpose of counter-interference is to add perturbations to the input. To maximize the model loss: In multimodal scenarios, jamming typically targets a single mode (e.g., jamming only radar). Assuming the radar mode is jammed, then... For static fusion models, all modal features are treated equally, and contaminated features are excluded. This directly destroys the overall feature representation after fusion. However, under the dynamic attention mechanism of this application, the model re-evaluates the input. Ideally, the reliability score of the disturbed mode... It will decrease, while the score for clean modes (infrared, ESM) will decrease. , The value will increase accordingly. The fusion process becomes: (11) This indicates that the attention mechanism, by automatically reducing the weight of the disturbed modes, isolates the influence of the disturbance to a certain extent, making the model output more dependent on the clean modes, thereby maintaining the stability of the system.

[0033] (5) Deep decision network with residual enhancement As neural network depth increases, models encounter vanishing / exploding gradients and network degradation, which ironically leads to increased training error and decreased performance. Residual learning cleverly addresses this challenge by introducing shortcut connections. This application introduces residual blocks into the decision network portion of TEFNet to enhance feature reuse capabilities, ensuring that deep networks maintain stable training characteristics and superior performance while effectively learning complex features after multimodal fusion.

[0034] The network consists of an initial linear transformation layer and multiple residual blocks stacked together. The computation process of each residual block can be represented as follows: (12) in, For block input, , The weight matrix is ​​used to output the following results: The introduction of shortcut connections ensures that gradients can be directly backpropagated, enabling the network to be effectively trained with architectures dozens of layers deep.

[0035] Ultimately, the deep decision network will reconstruct the features Mapped to a 5-dimensional output space (corresponding to 5 target categories): (13) in , This represents the overall transformation of the residual network.

[0036] In summary, TEFNet, through four core designs—multimodal feature representation learning, cross-modal interaction, channel attention reconstruction, and residual deep decision-making—forms a hierarchical, adaptive, and robust multimodal fusion framework, providing an effective solution for intelligent target recognition in complex adversarial environments.

[0037] Experimental Design To verify the effectiveness and superiority of the TEFNet model in complex electromagnetic environments, this section designs a systematic experimental scheme and evaluation framework. The experiments revolve around a highly realistic penetration and countermeasure scenario. By constructing a multimodal adversarial scenario dataset, the recognition accuracy, anti-interference robustness, and overall adversarial effectiveness of TEFNet and baseline models are comprehensively evaluated and analyzed. All experiments are implemented using the Python 3.9 and PyTorch 1.10.1 frameworks.

[0038] The algorithm design in this application closely mirrors the realities of modern maritime electronic warfare. Through data generation and interference functions, it successfully replicates in a simulation environment the most likely and lethal soft-kill environment that maritime vehicles will encounter during terminal penetration: a large number of decoys and highly intelligent electromagnetic signal suppression and deception. The core objective of the experiment is to evaluate TEFNet's ability to accurately identify and lock onto real ship targets from a highly jammed background in such extreme electromagnetic environments.

[0039] The experimental dataset was constructed based on this scenario. We used a multidimensional Gaussian distribution model to simulate the radar, infrared, and ESM modalities of four types of real ships and decoy targets. To increase the difficulty of identification, feature overlap regions were intentionally set; for example, the radar and infrared feature parameters of ship A and ship B were set similarly to simulate the most challenging "inter-class confusion" problem in actual combat. The dataset contains 2000 samples, of which 60% are decoy targets and 40% are four types of real ships (each type accounting for 10%), a proportion consistent with the target distribution under typical saturated combat scenarios. All samples were subjected to adaptive interference. This interference module simulates the core methods of modern electronic warfare systems, effectively reconstructing the physical process of feature distortion under interference by injecting additive Gaussian noise (standard deviation 0.15) with a random range of amplitude distortion ([0.5, 1.5] times) on the original features. Finally, each sample is a 15-dimensional vector composed of 5-dimensional radar features, 5-dimensional infrared features, and 5-dimensional ESM features, and has been standardized. For model training, the dataset was randomly divided into training and test sets in a 7:3 ratio.

[0040] TEFNet is trained end-to-end, and its optimization objective is to minimize the cross-entropy loss between the predicted and true labels. (14) in, N Indicates batch size, Indicates sample Category The true label (one-hot encoding). This represents the probability predicted by the model.

[0041] To enhance the model's robustness in adversarial environments, Projected Gradient Descent (PGD) adversarial training is introduced during training. As a classic strong adversarial attack method based on the white-box assumption, it iterates multiple times, adding small perturbations to the original input along the direction that increases the model's loss function (the gradient direction), and projecting these perturbations within an allowable range (such as within an ε-sphere), thereby generating adversarial examples that can effectively deceive the model. PGD attacks are widely used to evaluate model robustness, i.e., the model's ability to maintain stable performance under adversarial inputs. This application introduces PGD attacks into the performance evaluation system to simulate a strong electronic interference environment, precisely to verify the reliability of the TEFNet model in worst-case scenarios. (15) (16) in, To disturb the boundary, Let the step size be [value]. This is achieved by minimizing the loss on adversarial examples. The model is able to learn more robust feature representations.

[0042] For intelligent identification algorithms of maritime vehicles, a single accuracy metric is insufficient to fully reflect their applicability in adversarial scenarios. Intelligent identification algorithms for maritime vehicles must be accurate, stable, and fast, achieving a highly efficient balance between identification effectiveness, anti-interference capabilities, and timeliness. Based on this concept, this application constructs the Comprehensive Adversarial Performance Index (CEI), a multi-dimensional evaluation index. This index integrates three heterogeneous indicators—the macro F1 score reflecting overall identification performance, the adversarial robustness accuracy reflecting anti-interference capabilities, and the inference latency reflecting decision-making speed—into a unified scalar value through weighted fusion. This provides a more scientific and comprehensive evaluation perspective and theoretical basis for the engineering application of the algorithm.

[0043] In machine learning, the intelligent identification of maritime aircraft can be abstracted into a classification model, and there are many indicators to measure the classification model.

[0044] ① Accuracy: Represents the proportion of samples that are correctly predicted out of the total sample. The formula is as follows: (17) in, TP (True Positive) represents the number of true cases, that is, the number of ships correctly identified. TN (TrueNegative) represents the number of true negatives, that is, the number of correctly identified as decoys; FP (False Positive) indicates a false positive, which means that the number of decoys is mistakenly identified as the number of ships; FN(False Negative) represents the number of false negatives, i.e., the number of ships that are mistakenly identified as decoys. High accuracy means a high overall accuracy rate, but this metric can be misleading when there is class imbalance (such as when the number of decoys far exceeds the number of real ships). For example, even if the model identifies all targets as decoys, it may still achieve a high accuracy rate, but this completely defeats the purpose of adversarial analysis.

[0045] ② Precision: The ratio of samples predicted to be positive to samples that are actually positive. The formula is: (18) ③ Recall: The ratio of correctly predicted positive samples to all positive samples. High recall ensures that as many true targets as possible are found. Its calculation formula is: (19) The Receiver Operating Characteristic Curve (ROC curve) depicts the true positive rate of the classifier at different decision thresholds. TPR ) and the false positive rate FPR The trade-off between the ROC curve and the top left corner indicates better classifier performance. The area under the ROC curve (Area Under the Curve) is the key factor in determining the classifier's performance. AUC This quantifies the performance of the ROC curve. AUC The closer the value is to 1, the better the overall classification and discrimination ability of the model, and the better it can separate samples of different categories in the feature space. AUC The value means that the model can maintain a strong ability to distinguish between "ships" and "decoys" under various decision-making rigors.

[0046] The precision-recall curve (PR curve) focuses on the recognition performance of positive samples, with recall on the horizontal axis and precision on the vertical axis. In scenarios with imbalanced positive and negative samples (such as when the number of decoys far exceeds the number of real ships), the PR curve reflects the true performance of the classifier better than the ROC curve. Average precision (AP) is an approximation of the area under the PR curve; a higher AP value indicates a better overall balance between precision and recall. AP The value lies in assessing the system's ability to consistently identify high-value targets. A high-value system... AP A high recall value means that the system can not only identify true targets (high recall), but also ensure that most of the identified targets are true (high precision).

[0047] ④ F1 score: The harmonic mean of precision and recall, a metric for measuring the precision of a binary classification model, taking into account both precision and recall. (20) In multi-class classification problems, macros F1 The score (Macro-F1 Score) is determined by classifying categories. F The score is calculated by taking the arithmetic mean to ensure that all categories (especially the fewer ship categories) receive equal attention. This indicator comprehensively reflects the overall performance of the identification system, and is highly effective. F A value of 1 (close to 1.0) means the system can maximize the capture of real targets while minimizing false decoys. Considering the balance of model evaluation and the comprehensiveness of intelligent recognition, this algorithm will macro... F The Macro-F1 score is used as a metric to measure the accuracy of recognition performance.

[0048] Adversarial robustness is evaluated using a Projected Gradient Descent (PGD) adversarial attack. This attack generates adversarial examples by deliberately adding minute perturbations imperceptible to the original input samples to deceive the model. The mathematical principle is shown in equations (13)-(14). The attack strength is determined by the number of iterations, the step size, and the upper limit of the perturbation (…). The parameters are jointly determined. This experiment sets the attack parameters to 18 iterations, with a step size of... upper limit of disturbance This algorithm simulates the characteristic-level deception interference generated by the enemy's advanced DRFM technology. The anti-interference robustness index reflects the stability of the model under extreme electromagnetic countermeasures environments. The higher the value (closer to 1.0), the stronger the model's ability to maintain performance under strong interference conditions, corresponding to its survivability and countermeasure reliability in complex electromagnetic environments. This algorithm uses the index as the core indicator to directly measure the "stability" of recognition performance.

[0049] The timeliness metric, measured by single-sample inference latency, is the average time (in milliseconds) required for the model to process a single input. In adversarial scenarios, the process from target identification to decision-making must be completed within an extremely short time; otherwise, the target may escape lock-on or maneuver to evade. This timeliness metric ensures that the algorithm meets the time window requirements for adversarial strategy decisions; a lower value indicates a faster model response and stronger adversarial applicability.

[0050] To quantify the adversarial effectiveness of the model, this application introduces the Comprehensive Adversarial Effectiveness Index (CEI) as a core indicator. This index combines measures of "accuracy" (Macro-F1 Score), "stability" (robustness), and "timeliness" (timeliness). The calculation formula is as follows: (twenty one) in, The weights are denoted as , representing the corresponding parameter weights. Regarding the dynamic configuration strategy for the Comprehensive Effectiveness Index (CEI) weights, the weight configuration of the three indicators "accuracy," "stability," and "speed" in the CEI is essentially based on the priority requirements of different dimensions of the identification system under different adversarial scenarios. This application aims to propose and verify the effectiveness of the CEI comprehensive evaluation framework. Due to space limitations and research focus, this paper selects the weight set (0.4, 0.4, 0.2) as a typical case for demonstration and analysis. The scenario background for this configuration is: a conventional maritime confrontation expected to take place in a medium-to-high intensity electromagnetic confrontation environment, emphasizing first-shot hit effectiveness. It reflects the equal emphasis on identification "accuracy" and anti-jamming "stability" under this scenario (each accounting for 40%), while also giving appropriate consideration to "real-time performance" (20%), provided that the processor computing power is sufficiently guaranteed. This case verifies the feasibility of the CEI framework, and the dynamic configurability of the weights ensures that this evaluation system can be flexibly adapted to other diverse adversarial scenarios.

[0051] Norm() represents the normalization function. Macro-F1 is normalized when calculating the macro-average F1 score of the model on the test set, with a value range of [0, 1]. Robustness is normalized when calculating the classification accuracy of the model on PGD adversarial attack samples, with a value range of [0, 1]. Timeliness maps decision latency through an exponential decay function, converting it into a score between [0, 1]. This function is a special normalization method for latency characteristics. (twenty two) Latency is the time (in milliseconds) that the model takes to perform forward inference on a single sample. The maximum allowable delay threshold is a key parameter set according to the decision-making time window requirements of the adversarial strategy. If the delay is lower than this threshold, no penalty will be imposed. In this application, the value is set to 20 based on a certain simulation project. This is the attenuation coefficient, which controls the severity of the score drop once the maximum delay is exceeded. This value can be adjusted according to actual needs. The smaller the value, the more severe the penalty. In this case, the value is set to 10.

[0052] This function maps the delay to the (0, 1] interval, reflecting the real-time requirement. Substituting the specific weights and constant parameters in the case study, as well as Equation 20, into Equation 19, we can obtain the formula for calculating the comprehensive adversarial effectiveness index of this experimental case: (twenty three) To comprehensively and fairly verify the effectiveness and superiority of the TEFNet model proposed in this application in complex electromagnetic warfare environments, this experiment introduces two advanced models based on cutting-edge architectures as comparative baselines, aiming to evaluate the performance boundaries of the fusion strategy from different dimensions.

[0053] The Transformer architecture, with its powerful global context modeling capabilities, has achieved revolutionary success in multiple fields such as natural language processing and multimodal learning. Its core lies in the self-attention mechanism, which can compute the dependency between any two elements in a sequence, thereby effectively capturing long-range dependencies.

[0054] This application implements a Transformer Fusion Network (TFN) baseline model, aiming to verify the potential of global attention mechanisms in multimodal feature fusion. The model first utilizes three independent fully connected layers to map the 5-dimensional original features from radar, infrared, and ESM to a 32-dimensional latent space, respectively, to obtain modality-specific feature representations. These features are then treated as a sequence. The input is fed into a standard Transformer encoder layer. This encoder globally models the interactions between all feature elements within the sequence using a multi-head self-attention mechanism and performs a non-linear transformation through a feedforward neural network. Finally, the sequence output by the encoder is average-pooled and fed into a classifier for object recognition.

[0055] The theoretical advantage of the TFN model lies in its ability to disregard feature order and distance, calculating the correlation between all modal features equally, thereby uncovering potential global complementary information. However, its limitations are also significant: First, the computational complexity of the self-attention mechanism increases quadratically with sequence length, and its inherent computational overhead remains a considerable burden in missile-borne embedded systems requiring extreme real-time performance; second, the Transformer model typically requires large-scale data for sufficient training to avoid overfitting, and its performance may not be fully realized in the limited simulation data scenario of this study; finally, its "globally equal" attention mechanism may lack the ability to dynamically perceive "modal reliability" under specific interference environments, which is crucial for intelligent recognition in adversarial scenarios.

[0056] Graph Neural Networks (GNNs) are powerful tools for processing non-Euclidean data and have been introduced into the field of multimodal learning in recent years. They explicitly model the complex topological relationships between modalities by treating modalities as nodes in a graph.

[0057] The Graph Fusion Network (GFN) baseline model implemented in this application aims to explore the applicability of topological reasoning-based fusion strategies in this task. The model defines the feature vectors of radar, infrared, and ESM modalities as three nodes in a graph structure. The initial connections between nodes are set to fully connected, forming a complete graph to allow information exchange between any modalities. The node features are the 32-dimensional latent features extracted through independent branches. A Graph Attention Network (GAT) is used as the core message passing mechanism. Learnable attention weights are used to calculate the importance of neighboring nodes (i.e., modalities) to the central node, and neighborhood information is aggregated to update the node representation. After several layers of graph convolution, the updated features of all nodes are finally concatenated and fed into the classifier.

[0058] The advantage of the GFN model lies in its strong inductive bias, explicitly emphasizing the learning of "intermodal relationships," theoretically capable of fitting complex, non-linear interaction patterns between modalities. However, its application to the recognition scenario in this study has limitations: the multimodal system in this study only contains three modalities, and the relationship structure between modalities is relatively simple and fixed, not forming a complex topological structure that requires deep graph reasoning for understanding. Therefore, introducing a complex GNN architecture may bring unnecessary model complexity, easily leading to overfitting with limited data, and its message passing computation process also introduces additional inference latency. In contrast, the channel attention-based weighted fusion method adopted by TEFNet achieves adaptive interaction between modalities in a lighter and more direct way, which may be more suitable for the scenario in this task where the number of modalities is small but efficient and reliable decision-making is required.

[0059] To comprehensively evaluate the adversarial effectiveness of the proposed intelligent recognition model in complex electromagnetic warfare environments, this section conducts systematic testing and in-depth analysis of the proposed TEFNet model and two advanced comparative models—a Transformer-based fusion network (TFN) and a graph neural network-based fusion model (GFN). Using a constructed multimodal adversarial scenario dataset, the model performance is comprehensively evaluated from multiple dimensions, including recognition accuracy, anti-interference robustness, real-time performance, and overall adversarial effectiveness. Experimental results are shown in Table 1. Table 1: Model Performance Comparison Results The Transformer Fusion Network (TFN) demonstrated superior performance in the multimodal classification task of this experiment, thanks to its powerful global context modeling capabilities. The model achieved a macro F1 score of 0.7711, ranking first among the three comparison models, indicating its strong feature fusion and classification abilities. In-depth analysis of its class recognition performance reveals that TFN excels particularly in distinguishing high-value targets with significant features, achieving near-perfect differentiation. This is attributed to the Transformer architecture's self-attention mechanism, which effectively captures global feature dependencies.

[0060] It is worth noting that the TFN model performed relatively poorly in the anti-interference robustness test, with a robustness score of only 0.4935. This result reveals the limitations of the Transformer architecture in adversarial environments: while its self-attention mechanism is powerful, it lacks the dynamic perception capability of modal reliability under specific interference conditions, making the model's decision-making process susceptible to perturbations when facing carefully designed adversarial attacks. Furthermore, the TFN model's inference latency reached 0.1884ms, the highest among the three models, mainly due to the quadratic computational complexity of the self-attention mechanism in the Transformer. Although this latency is still within the time window for adversarial strategy decision-making, it remains a significant burden for missile-borne embedded systems that require extreme real-time performance.

[0061] GFN Model Performance Analysis The graph neural network-based fusion model (GFN) achieved a macro-F1 score of 0.7703 in this experiment, which is quite close to that of TFN. This model demonstrates unique advantages in feature fusion by treating different modal features as nodes in a graph and utilizing graph attention mechanisms to learn the topological relationships between modalities. In terms of performance across categories, GFN also excels in identifying high-value targets, proving the effectiveness of graph neural networks in handling complex relationships between modalities.

[0062] In terms of real-time performance, GFN performed best, with an inference latency of only 0.0853ms, thanks to the relatively simple computational architecture of graph neural networks. However, its performance in the robustness test was worrying, with a robustness score as low as 0.4785, ranking last among the three models. This result indicates that although GNN can effectively model the relationships between modalities, its defense against adversarial perturbations is weak. We believe this may be because the graph attention mechanism struggles to accurately assess the importance weights between nodes in perturbation environments, making the model susceptible to feature-level deception.

[0063] The TEFNet model proposed in this application demonstrates the best performance in Integrated Adversarial Performance (CEI), ranking first with a score of 0.7475, reflecting its optimal balance among accuracy, robustness, and real-time performance. Although its macro F1 score (0.7478) is slightly lower than the two comparative models, this difference mainly stems from the different model design philosophy: TEFNet does not simply pursue the highest recognition accuracy, but prioritizes improving anti-interference capability and real-time performance while ensuring sufficient accuracy.

[0064] In terms of robustness against interference, TEFNet demonstrates a significant advantage, achieving a robustness score of 0.6210, far surpassing TFN (0.4935) and GFN (0.4785). This advantage is directly attributable to the multimodal attention fusion mechanism and residual enhancement structure in the model. The attention mechanism enables the model to dynamically evaluate the reliability of each modality under the current interference environment, automatically reinforcing the contribution of clean modes while suppressing disturbed modes; the residual structure enhances the model's stability, making its decision boundaries smoother and less susceptible to manipulation by minor perturbations. This design allows TEFNet to maintain a more stable and reliable output when facing adversary electronic deception and noise suppression.

[0065] In terms of real-time performance, TEFNet's inference latency is 0.1086ms, which is higher than GFN but much lower than TFN, fully meeting the time window requirements for adversarial strategy decision-making. More importantly, TEFNet's performance is more balanced across different categories.

[0066] In summary, TEFNet’s superior performance reflects the advanced nature of its design philosophy: it achieves adaptive fusion between modalities through a lightweight channel attention mechanism, which avoids the computational burden of Transformer and overcomes the shortcomings of GNN in terms of anti-interference; it improves training stability and feature reuse capability through residual enhancement structure; and finally achieves the best balance in the three key dimensions of accuracy, robustness and real-time performance.

[0067] Ablation Experimental Study To quantitatively evaluate the contribution of each core component in the TEFNet model, this section designs an ablation study. Using the complete TEFNet model as a baseline, key modules were sequentially removed or replaced to form a comparative model, which was then trained and tested under the same experimental settings. The results are shown in Table 2.

[0068] Table 2 Comparison of Ablation Test Performance w / o Attention: Removes the cross-modal channel attention module, directly concatenates the three-modal features and inputs them into the decision network. w / o Residual: Removes the residual connections in the decision network, using ordinary fully connected layers stacked. SharedBackbone: Removes the independent feature extraction branch for each modality, directly concatenates the 15-dimensional original features of the three modalities, and inputs them into a shared multilayer perceptron (MLP) for feature extraction.

[0069] Analysis of Table 2 reveals the following: The attention module is crucial: removing the attention module (without attention) has the most significant impact on robustness (a decrease of approximately 30%), and the CEI index also drops substantially. This fully demonstrates that the dynamic cross-modal fusion mechanism is the core of the model's ability to resist interference and maintain stability. The residual structure is effective: removing the residual structure (without residual) leads to a certain decrease in the Macro-F1 score, indicating that residual connections play a positive role in maintaining the feature representation ability and training stability of deep networks. Modal extraction is necessary: ​​using a shared backbone leads to a comprehensive decrease in all performance metrics, verifying the necessity of designing independent feature extraction branches for heterogeneous modalities. Directly mixing heterogeneous features severely impairs the model's representation learning ability. In terms of overall efficiency trade-offs, removing certain modules (such as attention or shared backbone) reduces the number of parameters and computation, thereby reducing latency, but this comes at the cost of accuracy and robustness. The complete model achieves the best balance between performance and efficiency.

[0070] The ablation experiments demonstrate that each component of TEFNet makes an irreplaceable contribution to the ultimate superior Integrated Countermeasures (CEI), and its design is both adequate and necessary.

[0071] This application addresses the challenge of ship target recognition in complex electromagnetic warfare environments by proposing a TEFNet intelligent recognition model based on multimodal feature fusion and an attention mechanism. By constructing highly realistic adversarial scenario scenarios and datasets, and designing a Comprehensive Performance Index (CEI) for quantitative evaluation, experimental results verify the effectiveness and application potential of the multimodal attention fusion mechanism in improving recognition accuracy and environmental adaptability. This provides a reliable technical approach to solving key problems in intelligent target recognition in modern electronic warfare.

[0072] The above description is merely a specific implementation of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included within the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention should be determined by the protection scope of the claims.

Claims

1. A ship target recognition method based on a multimodal attention mechanism, characterized in that, include: Electromagnetic scattering characteristics, thermal radiation characteristics, and electromagnetic emission characteristics of ship targets are collected synchronously by multi-source heterogeneous sensors to form multimodal raw characteristic data; The multimodal raw feature data is input into a trained multimodal attention fusion network model, which outputs a recognition result corresponding to the ship target category, specifically including: The original multimodal feature data are input into the corresponding feature extraction branches, and after nonlinear transformation, they are mapped to a latent space of a unified dimension to obtain the latent features of each modality. The latent features of each modality are concatenated to form a joint feature, and the inter-modal attention weights are calculated based on the joint feature. The inter-modal attention weights are then used to generate enhanced features after cross-modal interaction. The enhanced features are subjected to global average pooling along the channel dimension to obtain channel descriptors. The channel descriptors are then passed through two fully connected layers to generate channel attention weights. The channel attention weights are used to rescale the enhanced features channel by channel to obtain the reconstructed features. The reconstructed features are input into a deep decision network consisting of multiple residual blocks, and the network outputs the identification result corresponding to the ship target category.

2. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, The original multimodal feature data are input into the corresponding feature extraction branches, and after nonlinear transformation, mapped to a latent space of a unified dimension to obtain the latent features of each modality, specifically including: The following transformations are performed on the original feature data of each modality: ； ； in, Represents the raw feature data of each modality. This represents the activation function. This indicates a batch normalization operation. , Represents the learnable weight matrix. , Indicates the bias term. Represents the hidden features of each modality.

3. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, The latent features of each modality are concatenated to form a joint feature, and inter-modal attention weights are calculated based on the joint feature. The enhanced features resulting from cross-modal interaction are then generated using these inter-modal attention weights. Specifically, this includes: The latent features of each modality are concatenated to form a joint feature; A linear transformation is performed on the joint features to generate a query vector, a key vector, and a value vector; The intermodal attention weights are obtained by performing a scaled dot product calculation based on the query vector and the key vector. The enhanced features after cross-modal interaction are calculated based on the intermodal attention weights and the value vector.

4. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, The enhanced features are subjected to global average pooling along the channel dimension to obtain channel descriptors. These channel descriptors are then passed sequentially through two fully connected layers to generate channel attention weights. The enhanced features are then rescaled channel-by-channel using these channel attention weights to obtain the reconstructed features. Specifically, this includes: The enhanced features are subjected to global average pooling along the channel dimension to compress global spatial information into channel descriptors; The channel descriptors are sequentially passed through two fully connected layers to generate channel attention weights; the first fully connected layer is used to reduce the dimensionality, and the second fully connected layer is used to restore the dimensionality. The channel attention weights and the enhanced features are multiplied channel by channel to obtain the reconstructed features.

5. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, The calculation process of the residual block in the deep decision network is as follows: ; ； in, Indicates block input, , Represents the weight matrix. , This represents the bias term, and the output result is... .

6. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, The optimization objective of the multimodal attention fusion network model is to minimize the cross-entropy loss between the predicted class and the true label, and its mathematical expression is: ; in, N Indicates batch size, Indicates sample Category The true label, This represents the probability predicted by the model.

7. The ship target recognition method based on multimodal attention mechanism according to claim 1, characterized in that, Also includes: Calculate the recognition performance index; Calculate the anti-interference performance indicators; Calculate timeliness performance indicators; The comprehensive countermeasure effectiveness index is calculated by weighting and fusing the identification effectiveness index, the anti-interference performance index, and the timeliness performance index. The performance of the multimodal attention fusion network model is evaluated based on the comprehensive adversarial effectiveness index.

8. The ship target recognition method based on multimodal attention mechanism according to claim 7, characterized in that, The recognition performance metrics include one or more of accuracy, precision, recall, and F1 score.

9. The ship target recognition method based on multimodal attention mechanism according to claim 3, characterized in that, The electromagnetic scattering characteristics include radar cross section, Doppler frequency shift, range profile characteristics, micro-Doppler characteristics, and polarization characteristics; The thermal radiation features include thermal contrast, temperature distribution statistics, contour shape factor, heat source distribution, and image texture. The electromagnetic emission characteristics include radio frequency, modulation parameters, direction of arrival, signal strength, and fingerprint characteristics.

Images