Dynamic ship classification and identification method based on multi-modal mass change

Abstract

The invention discloses a dynamic ship classification and identification method based on multi-modal quality change. The method comprises the following steps: respectively inputting an acquired infrared light image and a visible light image into two independent branches; feature extraction is carried out on input data in the two independent branches through an embedded layer, and the two extracted features are processed in sequence through N sets of fusion processing modules to obtain an infrared light feature vector and a visible light feature vector which are finally processed; inputting the two finally processed feature vectors into a shared expert gating network, dynamically allocating weights according to the quality change of the two modals, and performing modal feature fusion; and inputting the fused features into a classification head, carrying out ship identification and classification, and outputting a final ship identification result. The method can effectively cope with the condition that the quality of infrared light and visible light images is not uniform, and improves the accuracy and robustness of ship identification in a complex scene.

Description

Technical Field

[0001] The invention relates to the technical field of image processing and pattern recognition, and in particular to a dynamic ship classification and recognition method based on multi-modal mass change. Background Art

[0002] Existing ship recognition technologies mainly rely on single-modal data, such as visible light images or infrared images for target recognition and classification. These technologies can usually achieve relatively ideal results under ideal image quality and environmental conditions. However, in actual application scenarios, due to the complexity of environmental conditions (such as lighting changes, bad weather) and fluctuations in sensor performance, the quality of single-modal data is often difficult to guarantee. In addition, the characteristic performance of ships in different environments may vary significantly. For example, visible light images under backlight conditions will lose edge information, while in high humidity or low temperature difference environments, the target contrast of infrared images may be significantly reduced. Therefore, the applicability of single-modal data in complex scenes is limited, resulting in obvious deficiencies in recognition accuracy and robustness.

[0003] Multimodal fusion technology can improve the comprehensiveness and accuracy of recognition to a certain extent by comprehensively utilizing data from different modalities. However, existing multimodal fusion methods have certain limitations. These methods usually adopt simple superposition or averaging strategies, ignoring the inherent connection and complementary characteristics between different modal data, and may even introduce information conflicts when the quality of modal data is uneven, thereby negatively affecting the overall performance after fusion. In addition, traditional multimodal methods are usually unable to flexibly optimize fusion strategies according to data quality and environmental changes, making them less than ideal in practical applications.

[0004] In recent years, deep learning technology has made significant progress in the field of ship identification. In particular, models such as convolutional neural networks (CNNs) have been widely used in image recognition tasks due to their powerful automatic feature extraction capabilities. These models can improve the recognition accuracy of single-modal data to a certain extent. However, deep learning methods still have the following problems in multimodal fusion scenarios: on the one hand, these models do not fully consider the impact of changes in modal data quality, especially when the quality of modal data is uneven or dynamically changing, the accuracy of recognition results needs to be improved; on the other hand, these models usually process multimodal data through static structures, lacking the ability to adaptively optimize data quality and environmental changes, which limits their practicality and robustness. Summary of the invention

[0005] The present invention proposes a dynamic ship classification and recognition method based on multimodal quality changes, the purpose of which is to overcome the defects of the prior art, effectively deal with the situation of uneven quality of infrared and visible light images, and improve the accuracy and robustness of ship recognition in complex scenes.

[0006] The technical solution of the present invention is as follows: A dynamic ship classification and identification method based on multi-modal mass change includes the following steps: S1: Input the collected infrared image and visible light image into two independent branches respectively; S2: The input data in the two independent branches are respectively subjected to feature extraction through the embedding layer, and the two extracted features are processed in sequence through N groups of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector; S3: Input the final infrared light feature vector and the final visible light feature vector into the shared expert gating network, dynamically assign weights according to the quality changes of the two modalities, and perform modal feature fusion; S4: Input the fused features into the classification head to perform ship identification and classification, and output the final ship identification result.

[0007] As a further improvement of the dynamic ship classification and identification method based on multimodal mass changes: the fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input into the fusion processing module to obtain the output infrared light feature vector and the output visible light feature vector; the fusion processing module also includes multiple BA modules, and the BA modules are used to interact the modal information of the infrared light feature vector and the visible light feature vector in the processing process of the two Transformer encoders, and send the feature vector obtained after the interaction back to the Transformer encoder for fusion processing.

[0008] The step S2 specifically includes: S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector respectively to obtain a normalized infrared light feature vector and a normalized visible light feature vector; S2-2: Input the normalized infrared feature vector and the normalized visible light feature vector into the multi-head self-attention mechanism layer of the corresponding Transformer encoder for processing, to obtain a preliminary processed infrared light feature vector and a preliminary processed visible light feature vector; at the same time, input the normalized infrared light feature vector and the normalized visible light feature vector into the first BA module for data interaction, to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector; S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector as the input of the next layer of the corresponding Transformer encoder; S2-4: performing layer normalization processing on the intermediate infrared light feature vector and the intermediate visible light feature vector in the next layer of their respective corresponding Transformer encoders, and passing the layer normalized vectors to the feedforward neural network of the corresponding Transformer encoder, and outputting the feedforward infrared light feature vector and the feedforward visible light feature vector respectively; at the same time, inputting the intermediate infrared light feature vector and the intermediate visible light feature vector into the second BA module for data interaction, and obtaining a second interactive infrared light feature vector and a second interactive visible light feature vector; S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector; The deep fusion infrared light feature vector and the deep fusion visible light feature vector serve as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module.

[0009] As a further improvement of the dynamic ship classification and recognition method based on multimodal mass change, the multi-head self-attention mechanism layer in step S2-2 processes the feature vector as follows:

[0010]

[0011] in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vector and the preliminary processing of visible light feature vector, respectively. and They represent the normalized infrared light feature vector and the normalized visible light feature vector respectively.

[0012] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change, the data interaction of the BA module includes token dimension reduction, linear mapping, and dimension increase processing on the received feature vector, which specifically includes the following steps: First, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer:

[0013]

[0014] in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction and D is the dimension before dimensionality reduction; and They respectively represent the infrared light feature vector and visible light feature vector input to the BA module and processed by their respective Transformer encoder processes.

[0015] Then, the BA module establishes the relationship between the two modal features through linear mapping:

[0016]

[0017] in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes.

[0018] Finally, the linearly mapped features are dimensionally upgraded to obtain the infrared light feature vector and the visible light feature vector:

[0019]

[0020] in, and are the dimension-raising projection matrices of the infrared light feature vector and the visible light feature vector, respectively. The sizes of the two dimension-raising projection matrices are D×d; and They respectively represent the infrared light feature vector and visible light feature vector after dimensionality upgrade.

[0021] As a further improvement of the dynamic ship classification and identification method based on multi-modal mass change, step S3 specifically includes: First, an expert gating network evaluates the image quality of both modalities:

[0022]

[0023] in, and are the final infrared light feature vector and the final visible light feature vector, is the quality assessment function, and Represent the quality assessment results of infrared light feature vector and visible light feature vector respectively.

[0024] Then, a corresponding weight is assigned to each modal feature according to the quality assessment results:

[0025]

[0026] in, is the weight of infrared light feature, is the weight of the visible light feature.

[0027] Finally, the feature vectors of the two modalities are weightedly fused according to the assigned weights:

[0028] in, is the fused feature vector.

[0029] As a further improvement of the dynamic ship classification and identification method based on multimodal quality changes: the quality assessment function extracts high-frequency components through discrete wavelet transform to quantify clarity, calculates global contrast using histogram statistical dynamic range, and estimates noise level using median filter residual to comprehensively evaluate image quality characteristics.

[0030] As a further improvement of the dynamic ship classification and identification method based on multi-modal mass change: and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

[0031] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change: the feature extraction network of the embedding layer in step S2 is a convolutional neural network, and the feature extraction process is as follows:

[0032]

[0033] in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared light images and visible light images; and Respectively represent the infrared light image and the visible light image input to the two independent branches in step S1; and Represent the feature vectors extracted from infrared images and visible light images respectively.

[0034] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change, in step S1, image data preprocessing is performed before the infrared light image and the visible light image are respectively input into two independent branches. The image data preprocessing process is as follows:

[0035]

[0036] in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and They represent the preprocessed infrared image and visible light image respectively, and the preprocessed infrared image and visible light image are input into two independent branches respectively.

[0037] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change: N≥3.

[0038] Compared with the prior art, the present invention has the following beneficial effects: (1) The fusion weights of different modalities are dynamically adjusted through the expert gating network, which effectively copes with the uneven quality of infrared and visible light images and improves the accuracy of recognition results; (2) The deep feature interaction of the multi-level Transformer encoder and the BA module is applied to improve the fusion effect of modal information and further improve the accuracy of ship recognition; (3) The image quality assessment method adopts multi-factor fusion evaluation and adaptive weight mechanism, combined with the expert gating network, which improves the effectiveness of information fusion between modalities, enhances the ability to cope with complex environments and quality changes, and improves the practicality and robustness of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the data processing flow of the fusion processing module. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is described in detail below: A dynamic ship classification and identification method based on multi-modal mass change includes the following steps: S1: The collected infrared images and visible light images with uneven modality quality are preprocessed respectively, and the preprocessed images are input into two independent branches respectively. Each branch processes data of one modality independently, one branch processes infrared images, and the other branch processes visible light images.

[0041] The infrared image and visible light image are represented as and , these two images have different resolutions and features and can usually be represented as a multi-channel matrix: (1) (2) in, and Respectively represent the height and width of the infrared image, is the number of channels of the infrared image (usually 1, indicating a grayscale image); and Represent the height and width of the visible light image, respectively. is the number of channels of the visible light image (usually 3, indicating an RGB image).

[0042] The image data preprocessing includes operations such as resizing and normalization to make the size and value range of the image data uniform. The image data preprocessing process is as follows: (3) (4) in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and They represent the preprocessed infrared image and visible light image respectively.

[0043] The height and width of the preprocessed image are unified to H×W, that is: (5) (6) S2: The input data in the two independent branches are respectively extracted through the embedding layer to map the high-dimensional image data into a low-dimensional feature vector for subsequent processing and fusion. The two extracted features are then processed in sequence through N groups (N ≥ 3) of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector.

[0044] Preferably, the feature extraction network of the embedding layer is a convolutional neural network (CNN), and the feature extraction process is as follows: (7) (8) in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared light images and visible light images; and They represent the feature vectors extracted from the infrared image and the visible light image, namely, the initial infrared feature vector and the initial visible light feature vector, respectively.

[0045] Assuming that the dimension of each feature vector is D, their shape can be expressed as: (9) (10) The extracted initial infrared light feature vector and initial visible light feature vector are taken as input and sent to the fusion processing module for further processing.

[0046] The fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input to the fusion processing module to obtain the output infrared light feature vector and the output visible light feature vector. The initial infrared light feature vector and the initial visible light feature vector are the input infrared light feature vector and the input visible light feature vector of the first fusion processing module, respectively, the output infrared light feature vector and the output visible light feature vector of the previous fusion processing module are the input infrared light feature vector and the input visible light feature vector of the next fusion processing module, respectively, and the output infrared light feature vector and the output visible light feature vector of the last fusion processing module are the final infrared light feature vector and the final visible light feature vector, respectively.

[0047] The fusion processing module also includes multiple BA modules, which are used to interact the modal information of the infrared light feature vectors and the visible light feature vectors in the processing of the two Transformer encoders, and send the feature vectors obtained after the interaction back to the Transformer encoder for fusion processing.

[0048] Preferably, if Figure 1 The process of processing the two input feature vectors by the fusion processing module includes the following steps: S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector respectively to obtain a normalized infrared light feature vector and a normalized visible light feature vector.

[0049] S2-2: Input the normalized infrared feature vector and the normalized visible light feature vector into the multi-head self-attention mechanism layer of their respective corresponding Transformer encoders for processing to obtain a preliminary processed infrared light feature vector and a preliminary processed visible light feature vector; at the same time, input the normalized infrared light feature vector and the normalized visible light feature vector into the first BA module for data interaction to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector.

[0050] The Transformer encoder can capture long-distance dependencies between features and global features within the modality through the self-attention mechanism. The self-attention mechanism processing process is as follows: (11) (12) in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vector and the preliminary processing of visible light feature vector, respectively. and They represent the normalized infrared light feature vector and the normalized visible light feature vector respectively, and their dimensions are both D.

[0051] Further preferably, the BA module performs data interaction including token dimension reduction, linear mapping, and dimension increase processing on the received feature vector, completes the deep interaction of modal information, strengthens the correlation between multimodal features, and obtains two features after interaction. The dimension reduction process helps to reduce the redundant information of the features, so as to focus on more representative features. The linear mapping is to construct the relationship between the two modes, and the dimension increase restores the expressive power of the features, providing more complementary information for subsequent processing.

[0052] Specifically, first, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer: (13) (14) in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction, usually d<D. and They represent the infrared light feature vector and visible light feature vector respectively, which are processed by the respective Transformer encoder processes and input to the BA module. Corresponding to the first BA module, the input here is the normalized infrared light feature vector and the normalized visible light feature vector .

[0053] Then, the BA module establishes the relationship between the two modal features through linear mapping. The linear mapping can be achieved through a bilinear operation, that is, bilinear mapping of the two reduced feature vectors to capture the interdependence between the two modalities.

[0054] The linear mapping process is as follows: (15) (16) in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes.

[0055] Finally, the BA module performs a dimensionality increase operation on the linearly mapped features to obtain the infrared light feature vector and the visible light feature vector after dimensionality increase. The dimensionality increase operation is used to restore the original dimension of the feature and enhance the expressiveness of the feature. The dimensionality increase operation is also implemented through the linear projection layer, and the process is as follows: (17) (18) in, and They are the dimension-raising projection matrices of the infrared light feature vector and the visible light feature vector, respectively. The sizes of the two dimension-raising projection matrices are D×d. The dimension-raising operation restores the dimension of the feature from d to D. and They respectively represent the infrared light feature vector and the visible light feature vector after dimensionality upgrade, corresponding to the first BA module, namely the first interactive infrared light feature vector and the first interactive visible light feature vector.

[0056] At this point, the two modal eigenvectors have undergone deep modal information interaction and can better reflect the complementary characteristics of the two modes.

[0057] S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector as the input of the next layer of the corresponding Transformer encoder.

[0058] S2-4: Perform layer normalization on the intermediate infrared light feature vector and the intermediate visible light feature vector in the next layer of their respective corresponding Transformer encoders, and pass the layer normalized vectors to the feedforward neural network of the corresponding Transformer encoder, and output the feedforward infrared light feature vector and the feedforward visible light feature vector, respectively; at the same time, input the intermediate infrared light feature vector and the intermediate visible light feature vector into the second BA module for data interaction, so as to obtain a second interactive infrared light feature vector and a second interactive visible light feature vector.

[0059] S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector.

[0060] The deep fusion infrared light feature vector and the deep fusion visible light feature vector are used as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module, until the last fusion processing module completes the processing of the modal feature data and outputs the final infrared light feature vector and the final visible light feature vector.

[0061] As mentioned above, after passing through N fusion processing modules in sequence, each fusion processing module performs data interactive fusion processing through the Transformer encoder and the BA module, so that the modal features are further optimized, and the representation ability of the multimodal features is enhanced through feature interaction.

[0062] Through the above process, the features in each branch can fully integrate the information from the other modality, thereby improving the overall feature expression ability. This multi-level processing helps to gradually optimize the expression and fusion effect of features, ensuring that good classification results can be achieved even when the modal quality is uneven.

[0063] S3: Input the final infrared light feature vector and the final visible light feature vector into the shared expert gating network, dynamically assign weights according to the quality changes of the two modalities, and perform modal feature fusion. When the quality of a modality is low, the system will reduce the weight of the modality feature in the fusion process, so as to better rely on the features of another high-quality modality.

[0064] Specifically, first, the expert gating network evaluates the image quality of the two modalities through the quality assessment function: (19) (20) in, and are the final infrared light feature vector and the final visible light feature vector, both of which have a dimension of D. is the quality assessment function, and Represent the quality assessment results of infrared light feature vector and visible light feature vector respectively. and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

[0065] The quality assessment function is based on a comprehensive assessment of multiple factors such as image contrast, clarity, noise level, etc., to provide a basis for the dynamic adjustment of the weights of the expert gating network. Preferably, the quality assessment function integrates image characteristics such as contrast, clarity and noise level, extracts high-frequency components to quantify clarity through discrete wavelet transform, calculates global contrast using histogram statistics dynamic range, and estimates noise level using median filter residual. By combining frequency domain and spatial domain, the image quality characteristics are comprehensively evaluated.

[0066] This evaluation method further adopts an adaptive weight mechanism to normalize multi-factor features and generate a comprehensive quality score. It can dynamically adjust the weight distribution according to the scene complexity and modality priority to ensure the reliability and applicability of the evaluation results in multimodal fusion tasks. This evaluation mechanism combined with the expert gating network not only significantly improves the effectiveness of inter-modal information fusion, but also enhances the model's robustness to complex environments and quality changes.

[0067] Then, a corresponding weight is assigned to each modal feature according to the quality assessment results: (twenty one) (twenty two) in, is the weight of infrared light feature, is the weight of the visible light feature.

[0068] This weight calculation method ensures that when the quality of a certain modality is high, its feature weight is large, and vice versa. For example, when the quality of the infrared image is high, will approach 1, and It approaches to 0.

[0069] Finally, the feature vectors of the two modalities are weightedly fused according to the assigned weights: (twenty three) in, is the fused feature vector. The fused feature vector combines the information of the two modalities and can give priority to the information of the high-quality modality when the quality is uneven.

[0070] S4: Input the fused features into the classification head for ship identification and classification. The classification head processes the fused features and outputs the final ship identification result.

[0071] This method can maintain stable recognition performance in complex scenarios and is suitable for a variety of ship recognition application scenarios, including bad weather and night operations.

Claims

1. A dynamic ship classification and identification method based on multimodal mass change, characterized in that: The steps include: S1: Input the collected infrared image and visible light image into two independent branches respectively; S2: The input data in the two independent branches are respectively subjected to feature extraction through the embedding layer, and the two extracted features are processed in sequence through N groups of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector; S3: Input the final infrared light feature vector and the final visible light feature vector into the shared expert gating network, dynamically assign weights according to the quality changes of the two modalities, and perform modal feature fusion; S4: Input the fused features into the classification head to perform ship identification and classification, and output the final ship identification result.

2. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 1, characterized in that: The fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input to the fusion processing module to obtain the output infrared light feature vector and the output visible light feature vector; the fusion processing module also includes a plurality of BA modules, which are used to perform modal information interaction between the infrared light feature vector and the visible light feature vector in the process of the two Transformer encoders, and send the feature vector obtained after the interaction back to the Transformer encoder for fusion processing; The step S2 specifically includes: S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector respectively to obtain a normalized infrared light feature vector and a normalized visible light feature vector; S2-2: Input the normalized infrared feature vector and the normalized visible light feature vector into the multi-head self-attention mechanism layer of the corresponding Transformer encoder for processing, to obtain a preliminary processed infrared light feature vector and a preliminary processed visible light feature vector; at the same time, input the normalized infrared light feature vector and the normalized visible light feature vector into the first BA module for data interaction, to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector; S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector as the input of the next layer of the corresponding Transformer encoder; S2-4: performing layer normalization processing on the intermediate infrared light feature vector and the intermediate visible light feature vector in the next layer of their respective corresponding Transformer encoders, and passing the layer normalized vectors to the feedforward neural network of the corresponding Transformer encoder, and outputting the feedforward infrared light feature vector and the feedforward visible light feature vector respectively; at the same time, inputting the intermediate infrared light feature vector and the intermediate visible light feature vector into the second BA module for data interaction, and obtaining a second interactive infrared light feature vector and a second interactive visible light feature vector; S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector; The deep fusion infrared light feature vector and the deep fusion visible light feature vector serve as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module.

3. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 2, characterized in that: The processing process of the multi-head self-attention mechanism layer on the feature vector in step S2-2 is as follows: ； ； in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vector and the preliminary processing of visible light feature vector, respectively. and They represent the normalized infrared light feature vector and the normalized visible light feature vector respectively.

4. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 2, characterized in that: The data interaction of the BA module includes token dimension reduction, linear mapping, and dimension increase processing on the received feature vector, which specifically includes the following steps: First, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer: ； ； in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction and D is the dimension before dimensionality reduction; and They represent the infrared light feature vector and visible light feature vector input to the BA module and processed by their respective Transformer encoder processes; Then, the BA module establishes the relationship between the two modal features through linear mapping: ； ； in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes; Finally, the linearly mapped features are dimensionally upgraded to obtain the infrared light feature vector and the visible light feature vector: ； ； in, and are the dimension-raising projection matrices of the infrared light feature vector and the visible light feature vector, respectively. The sizes of the two dimension-raising projection matrices are D×d; and They respectively represent the infrared light feature vector and visible light feature vector after dimensionality upgrade.

5. The dynamic ship classification and identification method based on multi-modal mass change according to claim 1, characterized in that: The step S3 specifically includes: First, an expert gating network evaluates the image quality of both modalities: ； ； in, and are the final infrared light feature vector and the final visible light feature vector, is the quality assessment function, and Respectively represent the quality assessment results of infrared light feature vector and visible light feature vector; Then, a corresponding weight is assigned to each modal feature according to the quality assessment results: ； ； in, is the weight of infrared light feature, is the weight of the visible light feature; Finally, the feature vectors of the two modalities are weighted fused according to the assigned weights: ； in, is the fused feature vector.

6. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 5, characterized in that: The quality assessment function extracts high-frequency components to quantify clarity through discrete wavelet transform, calculates global contrast using histogram statistics of dynamic range, and estimates noise level using median filter residuals to comprehensively assess image quality characteristics.

7. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 5, characterized in that: and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

8. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 1, characterized in that: The feature extraction network of the embedding layer in step S2 is a convolutional neural network, and the feature extraction process is as follows: ； ； in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared light images and visible light images; and Respectively represent the infrared light image and the visible light image input to the two independent branches in step S1; and Represent the feature vectors extracted from infrared images and visible light images respectively.

9. The method for dynamic ship classification and identification based on multi-modal mass change according to claim 1, characterized in that: In step S1, the infrared light image and the visible light image are respectively input into two independent branches for image data preprocessing, and the image data preprocessing process is as follows: ； ； in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and They represent the preprocessed infrared image and visible light image respectively, and the preprocessed infrared image and visible light image are input into two independent branches respectively.

10. The method for dynamic ship classification and identification based on multi-modal mass change according to any one of claims 1 to 9, characterized in that: N≥3。

Images