Dynamic ship classification and recognition method based on multimodal mass changes

Abstract

This invention discloses a dynamic ship classification and identification method based on multimodal quality changes. The method comprises: inputting collected infrared and visible light images into two independent branches; extracting features from the input data in each branch through an embedding layer, and sequentially processing the two extracted features through N sets of fusion processing modules to obtain final infrared and visible light feature vectors; inputting the two final processed feature vectors into a shared expert gating network, dynamically assigning weights based on the quality changes of the two modalities, and performing modal feature fusion; inputting the fused features into a classification head, performing ship identification and classification, and outputting the final ship identification result. This method can effectively address the situation where the quality of infrared and visible light images is uneven, improving the accuracy and robustness of ship identification in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and pattern recognition, and in particular to a dynamic ship classification and recognition method based on multimodal mass changes. Background Art

[0002] Existing ship recognition technologies primarily rely on single-modal data, such as visible light or infrared images, for target identification and classification. These technologies typically achieve relatively good results under ideal image quality and environmental conditions. However, in practical applications, due to the complexity of environmental conditions (such as varying lighting and inclement weather) and fluctuations in sensor performance, the quality of single-modal data is often difficult to guarantee. Furthermore, a ship's characteristic appearance can vary significantly in different environments. For example, visible light images in backlit conditions can lose edge information, while infrared images can significantly reduce target contrast in environments with high humidity or low temperature differences. Therefore, the applicability of single-modal data in complex scenarios is limited, resulting in significant deficiencies in recognition accuracy and robustness.

[0003] Multimodal fusion technology can improve the comprehensiveness and accuracy of recognition to a certain extent by comprehensively utilizing data from different modalities. However, existing multimodal fusion methods have certain limitations. These methods often adopt simple superposition or averaging strategies, ignoring the inherent connections and complementary characteristics between different modal data. They may even introduce information conflicts when the modal data quality is uneven, thus negatively affecting the overall performance of the fusion. In addition, traditional multimodal methods are generally unable to flexibly optimize the fusion strategy based on data quality and environmental changes, making their performance less than ideal in practical applications.

[0004] In recent years, deep learning technology has made significant progress in the field of ship identification. In particular, models such as convolutional neural networks (CNNs) have been widely used in image recognition tasks due to their powerful automatic feature extraction capabilities. These models can improve the recognition accuracy of single-modal data to a certain extent. However, deep learning methods still face the following challenges in multimodal fusion scenarios: First, these models do not fully consider the impact of variations in modal data quality. In particular, when modal data quality varies or changes dynamically, the accuracy of recognition results needs to be improved. Second, these models typically process multimodal data through static structures and lack the ability to adaptively optimize for changes in data quality and the environment, limiting their practicality and robustness. Summary of the Invention

[0005] The present invention proposes a dynamic ship classification and identification method based on multimodal quality changes, the purpose of which is to overcome the shortcomings of the existing technology, effectively deal with the situation where the quality of infrared and visible light images is uneven, and improve the accuracy and robustness of ship identification in complex scenarios.

[0006] The technical solutions of the present invention are as follows:

[0007] A dynamic ship classification and identification method based on multimodal mass change includes the following steps:

[0008] S1: Input the collected infrared image and visible light image into two independent branches respectively;

[0009] S2: The input data in the two independent branches are respectively subjected to feature extraction through the embedding layer, and the two extracted features are sequentially processed through N groups of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector;

[0010] S3: The final infrared light feature vector and the final visible light feature vector are input into a shared expert gating network, and weights are dynamically assigned according to the quality changes of the two modalities to perform modal feature fusion;

[0011] S4: Input the fused features into the classification head to perform ship identification and classification, and output the final ship identification results.

[0012] As a further improvement of the dynamic ship classification and identification method based on multimodal mass changes: the fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input into the fusion processing module to obtain the output infrared light feature vector and the output visible light feature vector; the fusion processing module also includes multiple BA modules, which are used to interact the modal information of the infrared light feature vector and the visible light feature vector in the processing process of the two Transformer encoders, and send the feature vector obtained after the interaction back to the Transformer encoder for fusion processing.

[0013] The step S2 specifically includes:

[0014] S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector, respectively, to obtain the normalized infrared light feature vector and the normalized visible light feature vector;

[0015] S2-2: Inputting the normalized infrared feature vector and the normalized visible light feature vector into the multi-head self-attention mechanism layer of their respective corresponding Transformer encoders for processing to obtain a preliminary processed infrared feature vector and a preliminary processed visible light feature vector; at the same time, inputting the normalized infrared light feature vector and the normalized visible light feature vector into the first BA module for data interaction to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector;

[0016] S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector, which serves as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector, which serves as the input of the next layer of the corresponding Transformer encoder;

[0017] S2-4: performing layer normalization processing on the intermediate infrared light feature vector and the intermediate visible light feature vector in the next layer of their respective corresponding Transformer encoders, and passing the layer-normalized vectors to the feedforward neural network of the corresponding Transformer encoder, outputting a feedforward infrared light feature vector and a feedforward visible light feature vector, respectively; at the same time, inputting the intermediate infrared light feature vector and the intermediate visible light feature vector into a second BA module for data interaction, thereby obtaining a second interactive infrared light feature vector and a second interactive visible light feature vector;

[0018] S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector;

[0019] The deep fusion infrared light feature vector and the deep fusion visible light feature vector serve as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module.

[0020] As a further improvement to the dynamic ship classification and recognition method based on multimodal mass change, the multi-head self-attention mechanism layer in step S2-2 processes the feature vector as follows:

[0021]

[0022]

[0023] in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vectors and the preliminary processing of visible light feature vectors, and represent the normalized infrared light feature vector and the normalized visible light feature vector, respectively.

[0024] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change, the data interaction of the BA module includes performing token dimension reduction, linear mapping, and dimension increase processing on the received feature vector, which specifically includes the following steps:

[0025] First, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer:

[0026]

[0027]

[0028] in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction and D is the dimension before dimensionality reduction; and They represent the infrared light feature vector and visible light feature vector input to the BA module and processed by their respective Transformer encoder processes.

[0029] Then, the BA module establishes the relationship between the two modal features through linear mapping:

[0030]

[0031]

[0032] in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes.

[0033] Finally, the linearly mapped features are subjected to a dimensionality-increasing operation to obtain the infrared light feature vector and the visible light feature vector after dimensionality increase:

[0034]

[0035]

[0036] in, and are the dimension-raising projection matrices of the infrared light eigenvector and the visible light eigenvector respectively, and the sizes of the two dimension-raising projection matrices are D×d; and They represent the infrared light feature vector and visible light feature vector after dimensionality upgrade respectively.

[0037] As a further improvement of the dynamic ship classification and identification method based on multimodal mass change, step S3 specifically includes:

[0038] First, an expert gating network evaluates the image quality of both modalities:

[0039]

[0040]

[0041] in, and are the final infrared light feature vector and the final visible light feature vector, is the quality assessment function, and Represent the quality assessment results of infrared light feature vector and visible light feature vector respectively.

[0042] Then, a corresponding weight is assigned to each modal feature based on the quality assessment results:

[0043]

[0044]

[0045] in, is the weight of infrared light features, is the weight of the visible light feature.

[0046] Finally, the feature vectors of the two modalities are weightedly fused according to the assigned weights:

[0047]

[0048] in, is the fused feature vector.

[0049] As a further improvement of the dynamic ship classification and identification method based on multimodal quality changes: the quality assessment function extracts high-frequency components through discrete wavelet transform to quantify clarity, calculates global contrast using histogram statistics dynamic range, and estimates noise level using median filter residual to comprehensively evaluate image quality characteristics.

[0050] As a further improvement to the dynamic ship classification and identification method based on multimodal mass change: and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

[0051] As a further improvement to the dynamic ship classification and recognition method based on multimodal mass change, the feature extraction network of the embedding layer in step S2 is a convolutional neural network, and the feature extraction process is as follows:

[0052]

[0053]

[0054] in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared images and visible light images; and Respectively represent the infrared image and visible light image input to the two independent branches in step S1; and represent the feature vectors extracted from infrared images and visible light images respectively.

[0055] As a further improvement to the dynamic ship classification and recognition method based on multimodal mass change, in step S1, image data preprocessing is performed before the infrared image and the visible light image are input into two independent branches respectively. The image data preprocessing process is as follows:

[0056]

[0057]

[0058] in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and They represent the preprocessed infrared image and visible light image respectively, and the preprocessed infrared image and visible light image are input into two independent branches respectively.

[0059] As a further improvement to the dynamic ship classification and identification method based on multimodal mass change: N≥3.

[0060] Compared with the existing technology, the present invention has the following beneficial effects: (1) The fusion weights of different modalities are dynamically adjusted through the expert gating network, which effectively copes with the uneven quality of infrared and visible light images and improves the accuracy of recognition results; (2) The deep feature interaction of the multi-level Transformer encoder and the BA module is applied to improve the fusion effect of modal information and further improve the accuracy of ship recognition; (3) The image quality assessment method adopts multi-factor fusion evaluation and adaptive weight mechanism, combined with the expert gating network, which improves the effectiveness of information fusion between modalities, enhances the ability to cope with complex environments and quality changes, and improves the practicality and robustness of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the data processing flow of the fusion processing module. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is described in detail below:

[0063] A dynamic ship classification and identification method based on multimodal mass change includes the following steps:

[0064] S1: Preprocess the acquired infrared and visible light images, both of varying modality quality, and feed the preprocessed images into two independent branches. Each branch independently processes data from a different modality: one branch processes infrared images, and the other processes visible light images.

[0065] The infrared image and visible light image are represented as and , these two images have different resolutions and features and can usually be represented as a multi-channel matrix:

[0066] (1)

[0067] (2)

[0068] in, and Represent the height and width of the infrared light image respectively, is the number of channels of the infrared image (usually 1, indicating a grayscale image); and Represent the height and width of the visible light image respectively, is the number of channels of the visible light image (usually 3, indicating an RGB image).

[0069] The image data preprocessing includes operations such as size adjustment and normalization to make the size and value range of the image data uniform. The image data preprocessing process is as follows:

[0070] (3)

[0071] (4)

[0072] in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and Represent the preprocessed infrared image and visible light image respectively.

[0073] The height and width of the preprocessed image are unified to H×W, that is:

[0074] (5)

[0075] (6)

[0076] S2: The input data from the two independent branches is extracted through the embedding layer, mapping the high-dimensional image data into low-dimensional feature vectors for subsequent processing and fusion. The two extracted features are then processed sequentially through N groups (N ≥ 3) of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector.

[0077] Preferably, the feature extraction network of the embedding layer is a convolutional neural network (CNN), and the feature extraction process is as follows:

[0078] (7)

[0079] (8)

[0080] in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared images and visible light images; and They represent the feature vectors extracted from the infrared image and the visible light image, namely the initial infrared feature vector and the initial visible light feature vector, respectively.

[0081] Assuming that the dimension of each eigenvector is D, their shape can be expressed as:

[0082] (9)

[0083] (10)

[0084] The extracted initial infrared light feature vector and initial visible light feature vector are taken as input and sent to the fusion processing module for further processing.

[0085] The fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input to the fusion processing module to obtain the output infrared light feature vector and the output visible light feature vector. The initial infrared light feature vector and the initial visible light feature vector are the input infrared light feature vector and the input visible light feature vector of the first fusion processing module, respectively. The output infrared light feature vector and the output visible light feature vector of the previous fusion processing module are the input infrared light feature vector and the input visible light feature vector of the next fusion processing module, respectively. The output infrared light feature vector and the output visible light feature vector of the last fusion processing module are the final infrared light feature vector and the final visible light feature vector, respectively.

[0086] The fusion processing module also includes multiple BA modules, which are used to interact the modal information of the infrared light feature vectors and the visible light feature vectors in the processing of the two Transformer encoders, and send the feature vectors obtained after the interaction back to the Transformer encoder for fusion processing.

[0087] Preferably, if Figure 1 The process of processing the two input feature vectors by the fusion processing module includes the following steps:

[0088] S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector, respectively, to obtain the normalized infrared light feature vector and the normalized visible light feature vector.

[0089] S2-2: The normalized infrared feature vector and the normalized visible light feature vector are respectively input into the multi-head self-attention mechanism layer of their respective corresponding Transformer encoders for processing to obtain a preliminary processed infrared light feature vector and a preliminary processed visible light feature vector; at the same time, the normalized infrared light feature vector and the normalized visible light feature vector are input into the first BA module for data interaction to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector.

[0090] The Transformer encoder can capture long-range dependencies between features and capture global features within the modality through the self-attention mechanism. The self-attention mechanism process is as follows:

[0091] (11)

[0092] (12)

[0093] in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vector and the preliminary processing of visible light feature vector, respectively. and They represent the normalized infrared light feature vector and the normalized visible light feature vector respectively, and their dimensions are both D.

[0094] Further preferably, the data interaction of the BA module includes token dimensionality reduction, linear mapping, and dimensionality increase processing on the received feature vectors, completing the deep interaction of modal information, strengthening the correlation between multimodal features, and obtaining two features after interaction. The dimensionality reduction process helps to reduce redundant information of features, thereby focusing on more representative features. Linear mapping is to build the relationship between the two modalities, and dimensionality increase is to restore the expressive power of features, providing more complementary information for subsequent processing.

[0095] Specifically, first, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer:

[0096] (13)

[0097] (14)

[0098] in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction, usually d<D. and They represent the infrared light feature vector and visible light feature vector input to the BA module after being processed by their respective Transformer encoder processes. Corresponding to the first BA module, the input here is the normalized infrared light feature vector and the normalized visible light eigenvector .

[0099] The bundle adjustment module then establishes the relationship between the two modal features through linear mapping. This linear mapping can be achieved through a bilinear operation, that is, performing a bilinear mapping on the two reduced feature vectors to capture the interdependence between the two modalities.

[0100] The linear mapping process is as follows:

[0101] (15)

[0102] (16)

[0103] in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes.

[0104] Finally, the BA module performs a dimensionality increase operation on the linearly mapped features to obtain the increased-dimensional infrared and visible light feature vectors. This dimensionality increase operation is used to restore the original dimension of the features and enhance their expressiveness. The dimensionality increase operation is also implemented through the linear projection layer, and the process is as follows:

[0105] (17)

[0106] (18)

[0107] in, and are the up-dimensional projection matrices of the infrared light feature vector and the visible light feature vector, respectively. The size of the two up-dimensional projection matrices is D × d. The up-dimensional operation restores the dimension of the feature from d to D. and They respectively represent the infrared light feature vector and the visible light feature vector after dimensionality upgrade, corresponding to the first BA module, that is, the first interactive infrared light feature vector and the first interactive visible light feature vector.

[0108] At this point, the two modal eigenvectors have undergone deep modal information interaction and can better reflect the complementary characteristics of the two modes.

[0109] S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector as the input of the next layer of the corresponding Transformer encoder.

[0110] S2-4: The intermediate infrared light feature vector and the intermediate visible light feature vector are respectively subjected to layer normalization processing in the next layer of their respective corresponding Transformer encoders, and the vectors after layer normalization processing are passed to the feedforward neural network of the corresponding Transformer encoder, and the feedforward infrared light feature vector and the feedforward visible light feature vector are respectively output; at the same time, the intermediate infrared light feature vector and the intermediate visible light feature vector are input into the second BA module for data interaction to obtain a second interactive infrared light feature vector and a second interactive visible light feature vector.

[0111] S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector.

[0112] The deep fusion infrared light feature vector and the deep fusion visible light feature vector serve as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module until the last fusion processing module completes processing the modal feature data and outputs the final infrared light feature vector and the final visible light feature vector.

[0113] As mentioned above, after passing through N fusion processing modules in sequence, each fusion processing module performs data interactive fusion processing through the Transformer encoder and BA module, so that the modal features are further optimized and the representation ability of multimodal features is enhanced through feature interaction.

[0114] Through this process, the features in each branch can fully integrate information from the other modality, thereby improving the overall feature expression capability. This multi-level processing helps to gradually optimize the expression and fusion of features, ensuring good classification results even when the modal quality is uneven.

[0115] S3: The final infrared and visible light feature vectors are fed into a shared expert gating network. Modal feature fusion is performed by dynamically assigning weights based on the quality changes of the two modalities. If the quality of one modality is low, the system reduces the weight of that modality in the fusion process, thereby increasing reliance on the features of the other high-quality modality.

[0116] Specifically, first, the expert gating network evaluates the image quality of the two modalities through the quality assessment function:

[0117] (19)

[0118] (20)

[0119] in, and are the final infrared light feature vector and the final visible light feature vector, both of which have a dimension of D. is the quality assessment function, and Represent the quality assessment results of infrared light feature vector and visible light feature vector respectively. and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

[0120] The quality assessment function performs a comprehensive evaluation based on multiple factors, including image contrast, clarity, and noise level, providing a basis for dynamic weight adjustments in the expert gating network. Preferably, the quality assessment function integrates image characteristics such as contrast, clarity, and noise level, extracting high-frequency components through discrete wavelet transform to quantify clarity, calculating global contrast using histogram statistics of the dynamic range, and estimating noise levels using median filter residuals. This comprehensive assessment of image quality characteristics is achieved through a combination of frequency and spatial domains.

[0121] This evaluation method further employs an adaptive weighting mechanism to normalize multi-factor features and generate a comprehensive quality score. This dynamically adjusts weighting based on scene complexity and modality priority, ensuring the reliability and applicability of the evaluation results in multimodal fusion tasks. This evaluation mechanism, combined with an expert gating network, not only significantly improves the effectiveness of intermodal information fusion but also enhances the model's robustness to complex environments and quality variations.

[0122] Then, a corresponding weight is assigned to each modal feature based on the quality assessment results:

[0123] (twenty one)

[0124] (twenty two)

[0125] in, is the weight of infrared light features, is the weight of the visible light feature.

[0126] This weight calculation method ensures that when the quality of a modality is high, its feature weight is large, and vice versa. For example, when the quality of the infrared image is high, will approach 1, and It approaches 0.

[0127] Finally, the feature vectors of the two modalities are weightedly fused according to the assigned weights:

[0128] (twenty three)

[0129] in, is the fused feature vector. The fused feature vector combines the information of the two modalities and can give priority to the information of the high-quality modality when the quality is uneven.

[0130] S4: The fused features are input to the classification head for ship identification and classification. The classification head processes the fused features and outputs the final ship identification result.

[0131] This method can maintain stable recognition performance in complex scenarios and is suitable for a variety of ship identification application scenarios, including bad weather and night operations.

Claims

1. A dynamic ship classification and identification method based on multimodal mass change, characterized in that: The steps include: S1: Input the collected infrared image and visible light image into two independent branches respectively; S2: The input data in the two independent branches are respectively subjected to feature extraction through the embedding layer, and the two extracted features are sequentially processed through N groups of fusion processing modules to obtain the final infrared light feature vector and the final visible light feature vector; The fusion processing module includes two independent Transformer encoders, which are respectively used to process the input infrared light feature vector and the input visible light feature vector input to the fusion processing module to obtain an output infrared light feature vector and an output visible light feature vector; the fusion processing module also includes multiple BA modules, which are used to perform modal information interaction on the infrared light feature vectors and the visible light feature vectors processed by the two Transformer encoders, and send the feature vectors obtained after the interaction back to the Transformer encoder for fusion processing; The step S2 specifically includes: S2-1: Two independent Transformer encoders perform layer normalization on the input infrared light feature vector and the input visible light feature vector, respectively, to obtain the normalized infrared light feature vector and the normalized visible light feature vector; S2-2: Inputting the normalized infrared feature vector and the normalized visible light feature vector into the multi-head self-attention mechanism layer of their respective corresponding Transformer encoders for processing to obtain a preliminary processed infrared feature vector and a preliminary processed visible light feature vector; at the same time, inputting the normalized infrared light feature vector and the normalized visible light feature vector into the first BA module for data interaction to obtain a first interactive infrared light feature vector and a first interactive visible light feature vector; S2-3: Add and fuse the initial infrared light feature vector, the preliminary processed infrared light feature vector, and the first interactive infrared light feature vector to obtain an intermediate infrared light feature vector, which serves as the input of the next layer of the corresponding Transformer encoder; fuse the initial visible light feature vector, the preliminary processed visible light feature vector, and the first interactive visible light feature vector to obtain an intermediate visible light feature vector, which serves as the input of the next layer of the corresponding Transformer encoder; S2-4: performing layer normalization processing on the intermediate infrared light feature vector and the intermediate visible light feature vector in the next layer of their respective corresponding Transformer encoders, and passing the layer-normalized vectors to the feedforward neural network of the corresponding Transformer encoder, outputting a feedforward infrared light feature vector and a feedforward visible light feature vector, respectively; at the same time, inputting the intermediate infrared light feature vector and the intermediate visible light feature vector into a second BA module for data interaction, thereby obtaining a second interactive infrared light feature vector and a second interactive visible light feature vector; S2-5: Add and fuse the intermediate infrared light feature vector, the feedforward infrared light feature vector, and the second interactive infrared light feature vector to obtain a deeply fused infrared light feature vector; add and fuse the intermediate visible light feature vector, the feedforward visible light feature vector, and the second interactive visible light feature vector to obtain a deeply fused visible light feature vector; The deep fusion infrared light feature vector and the deep fusion visible light feature vector serve as the output infrared light feature vector and the output visible light feature vector of the corresponding fusion processing module; S3: The final infrared light feature vector and the final visible light feature vector are input into a shared expert gating network, and weights are dynamically assigned according to the quality changes of the two modalities to perform modal feature fusion; S4: Input the fused features into the classification head to perform ship identification and classification, and output the final ship identification results.

2. The method for dynamic ship classification and identification based on multimodal mass change according to claim 1, characterized in that: The multi-head self-attention mechanism layer in step S2-2 processes the feature vector as follows: ； ； in, and Represents the self-attention processing function of the Transformer encoder of the two branches, and They represent the preliminary processing of infrared light feature vector and the preliminary processing of visible light feature vector, respectively. and represent the normalized infrared light feature vector and the normalized visible light feature vector, respectively.

3. The method for dynamic ship classification and identification based on multimodal mass change according to claim 1, characterized in that: The data interaction of the BA module includes token dimension reduction, linear mapping, and dimension increase processing on the received feature vector, which specifically includes the following steps: First, the BA module performs dimensionality reduction on the two modal feature vectors through a linear projection layer: ； ； in, and Respectively represent the infrared light feature vector and visible light feature vector after dimensionality reduction, is the dimension reduction projection matrix, The size of is d×D, where d is the dimension after dimensionality reduction and D is the dimension before dimensionality reduction; and They represent the infrared light feature vector and visible light feature vector input to the BA module after being processed by their respective Transformer encoder processes; Then, the BA module establishes the relationship between the two modal features through linear mapping: ； ； in, is the feature after linear mapping, is a bilinear mapping function, is a d×d bilinear weight matrix used to capture the interaction between the two modes; Finally, the linearly mapped features are subjected to a dimensionality-increasing operation to obtain the infrared light feature vector and the visible light feature vector after dimensionality increase: ； ； in, and are the dimension-raising projection matrices of the infrared light eigenvector and the visible light eigenvector respectively, and the sizes of the two dimension-raising projection matrices are D×d; and They represent the infrared light feature vector and visible light feature vector after dimensionality upgrade respectively.

4. The dynamic ship classification and identification method based on multimodal mass change according to claim 1, characterized in that: The step S3 specifically includes: First, an expert gating network evaluates the image quality of both modalities: ； ； in, and are the final infrared light feature vector and the final visible light feature vector, is the quality assessment function, and Represent the quality assessment results of infrared light feature vector and visible light feature vector respectively; Then, a corresponding weight is assigned to each modal feature based on the quality assessment results: ； ； in, is the weight of infrared light features, is the weight of the visible light feature; Finally, the feature vectors of the two modalities are weightedly fused according to the assigned weights: ； in, is the fused feature vector.

5. The method for dynamic ship classification and identification based on multimodal mass change according to claim 4, characterized in that: The quality assessment function extracts high-frequency components through discrete wavelet transform to quantify the clarity, calculates the global contrast using histogram statistics of the dynamic range, and estimates the noise level using median filter residuals to comprehensively evaluate the image quality characteristics.

6. The method for dynamic ship classification and identification based on multimodal mass change according to claim 5, characterized in that: and A scalar between 0 and 1 that reflects the relative quality of infrared and visible light images.

7. The method for dynamic ship classification and identification based on multimodal mass change according to claim 1, characterized in that: The feature extraction network of the embedding layer in step S2 is a convolutional neural network, and the feature extraction process is as follows: ； ； in, and The embedding layer feature extraction convolutional neural network functions of the two branches are respectively applied to infrared images and visible light images; and Respectively represent the infrared image and visible light image input to the two independent branches in step S1; and represent the feature vectors extracted from infrared images and visible light images respectively.

8. The method for dynamic ship classification and identification based on multimodal mass change according to claim 1, characterized in that: In step S1, the infrared image and the visible light image are respectively input into two independent branches for image data preprocessing. The image data preprocessing process is as follows: ； ； in, is the image data preprocessing function, and represent the original infrared image and visible light image respectively, and They represent the preprocessed infrared image and visible light image respectively, and the preprocessed infrared image and visible light image are input into two independent branches respectively.

9. The dynamic ship classification and identification method based on multimodal mass change according to any one of claims 1 to 8, characterized in that: N≥3。

Images