Missing multimodal ship classification and recognition method based on modality-invariant feature learning
By constructing a multimodal ship classification and recognition method based on modal invariant feature learning, the problem of reduced recognition performance caused by modal missing is solved, and high-performance ship classification and recognition under complex sea conditions is achieved.
Patent Information
- Application Number
- CN202510024423.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-07
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2045-01-07
AI Technical Summary
Existing multimodal learning methods cannot maintain recognition performance in the absence of modalities, resulting in a decrease in the accuracy of ship recognition in complex sea conditions such as cloud and fog obscuration and coastal background.
A multimodal ship classification and recognition method based on modal invariant feature learning is constructed. Through feature extraction fusion, invariant feature learning, relevant feature extraction and modal weight fusion, the available modalities are automatically learned, a shared representation is constructed, and high-performance ship classification and recognition is achieved.
Even when one mode is missing, the model can still maintain high performance and robustness, achieving efficient ship classification and recognition.
Smart Images
Figure CN119723217B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data recognition, and in particular relates to a ship classification and recognition method, in particular to a missing multimodal ship classification and recognition method based on modal invariant feature learning. Background Art
[0002] Ship identification plays a central role in modern intelligent maritime monitoring systems. Existing deep learning-based methods have achieved promising results. However, accurate identification in complex sea conditions, such as cloud and fog obstructions and coastal backgrounds, remains a pressing challenge.
[0003] With the continuous advancement of pattern recognition and artificial intelligence technologies, ship identification technology is gradually developing towards multi-source fusion, multi-resolution analysis, effective application of attention mechanisms, and comprehensive knowledge support. Ship image information acquired through multiple sensors is complementary, and compared to single-source sensor image recognition, ship identification using multimodal image fusion has significant advantages.
[0004] However, in practical applications, modality loss is a common problem due to factors such as weather and technology. For example, a sensor may be disconnected for some reason, temporarily unable to transmit data, or factors such as economic costs may limit the acquisition of data from certain modalities. Furthermore, changing environmental conditions may also affect the quality of data from specific modalities. For example, visible light images may be disrupted in inclement weather, making it impossible to acquire valid visible light modal images. In these situations, traditional multimodal learning methods cannot maintain their original performance and cannot perform recognition tasks properly. Summary of the Invention
[0005] The present invention proposes a missing multimodal ship classification and recognition method based on modal invariant feature learning, the purpose of which is to solve the problem that the recognition task cannot be completed when a certain mode is true under the multimodal learning framework.
[0006] The technical solutions of the present invention are as follows:
[0007] A missing multimodal ship classification and recognition method based on modal invariant feature learning, comprising the following steps:
[0008] Step 1: Construct a multimodal classification and recognition model, wherein the multimodal classification and recognition model includes a feature extraction and fusion part and a classifier;
[0009] The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain the fusion feature. The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module;
[0010] The classifier is used to Get the classification and recognition results; in the classifier, fusion features The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result;
[0011] Step 2: training the multimodal classification and recognition model;
[0012] Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and the infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one of the modalities, the image of the other modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model.
[0013] As a further improvement of the missing multimodal ship classification and recognition method based on modality-invariant feature learning: the visible light feature extraction branch is used to extract visible light high-level features from the input visible light image ;
[0014] The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image .
[0015] As a further improvement of the missing multimodal ship classification and recognition method based on modality invariant feature learning: the visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network, and the input visible light image Visible light image encoder Then obtain the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain visible light high-level features ;
[0016] The infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. The input infrared image After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .
[0017] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the invariant feature learning module includes a first convolutional neural network corresponding to the visible light feature extraction branch and a second convolutional neural network corresponding to the infrared feature extraction branch;
[0018] Convolutional neural networks extract high-level features of the input layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer.
[0019] Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .
[0020] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.
[0021] As a further improvement of the missing multimodal ship classification and recognition method based on modality-invariant feature learning, the input of the relevant feature extraction module includes input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features ;
[0022] In the relevant feature extraction module, the input feature and input features They are expanded into one-dimensional vectors respectively, and each one-dimensional vector is processed by the self-attention mechanism to obtain the same features as the input features. Corresponding related features and input features Corresponding related features .
[0023] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: in the relevant feature extraction module, the one-dimensional vector expanded by the input feature is processed in the following manner: the one-dimensional vector is divided into multiple feature tags, and each feature tag is input into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then all relevant feature blocks are reversed according to the original expansion method to restore them to relevant features with the same shape as the original input features.
[0024] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: suppose the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. For any self-attention layer, let it be the first Layer, the processing process is:
[0025] ;
[0026] ;
[0027] in, is the normalization layer, is a multi-head attention block, is a feed-forward network.
[0028] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively according to the output of the relevant feature extraction module. and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature ;
[0029] The weights are calculated by the model normalization layer: for the relevant features , set its The individual voxels are , and a total of The corresponding weight is ;
[0030] Fusion Features The calculation method is: .
[0031] As a further improvement of the missing multimodal ship classification and recognition method based on modality-invariant feature learning, step 2 specifically includes:
[0032] Step 2-1: Construct a training sample set;
[0033] The training sample set includes three types of samples: Class I samples, Class II samples, and Class III samples. The three types of samples are randomly distributed in the training sample set. Each training sample includes a visible light image and an infrared image of the same target. Class I samples include visible light images and infrared images; Class II samples only include visible light images; and Class III samples only include infrared images.
[0034] Step 2-2, training the invariant feature learning module;
[0035] The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features the differences between;
[0036] Specifically, the training samples are divided into multiple batches. In each batch, all training samples are first input into the model. During input, if the current sample lacks an image of one modality, the image of the other modality is used to replace the missing modality image and input into the multimodal classification and recognition model. Then, the corresponding high-order center distance in this batch is calculated, and the parameters of the first convolutional neural network and the second convolutional neural network are updated with the goal of minimizing the high-order center distance.
[0037] High-order central moments The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch, represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, Indicates calculation of L2 norm;
[0038] Step 2-3: Input the training samples into the multimodal classification and recognition model. During input, if the current sample lacks an image of one of the modalities, use the image of the other modality to replace the missing modality image and input it into the multimodal classification and recognition model; calculate the cross-entropy loss based on the classification results and the true category labels of the training samples, and update the network parameters of the entire multimodal classification and recognition model based on the cross-entropy loss.
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] The present invention first extracts high-level features from images of different modalities. Based on these high-level features, the invariant features of the different modalities are learned. Correlated features of the different modalities are further calculated based on the high-level and invariant features. Weights for the different modalities are then derived based on the correlated features. A fused feature is then obtained by fusing the modal weights. The model automatically learns the available modalities, constructs a shared representation, and finally obtains classification and recognition results based on the fused features. This method enables the model to maintain performance even when one modality is missing, resulting in high-performance and robust ship classification and recognition in the absence of a modality. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is an architectural diagram of the multimodal classification and recognition model of the present invention. DETAILED DESCRIPTION
[0042] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only a part of the embodiments of the present invention, rather than all the embodiments.
[0043] A missing multimodal ship classification and recognition method based on modal invariant feature learning, comprising the following steps:
[0044] Step 1: Construct a multimodal classification and recognition model, which includes a feature extraction and fusion part and a classifier. The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain the fusion feature. The classifier is used to Get the classification and recognition results.
[0045] Furthermore, if Figure 1 The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module.
[0046] (1) Visible light feature extraction branch and infrared feature extraction branch.
[0047] The visible light feature extraction branch is used to extract visible light high-level features from the input visible light image Specifically, the visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network. The input visible light image Visible light image encoder Then obtain the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain visible light high-level features .
[0048] The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image Specifically, the infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. The input infrared image After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .
[0049] The visible light image encoder and infrared image encoder both use a pre-trained ResNet network as the backbone, and extract visible light and infrared features from the original visible light image and infrared image through a series of convolution, normalization, activation, and pooling layers.
[0050] (2) Invariant feature learning module.
[0051] The invariant feature learning module includes a first convolutional neural network corresponding to the visible light feature extraction branch and a second convolutional neural network corresponding to the infrared feature extraction branch.
[0052] In convolutional neural networks, high-level input features are extracted layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer.
[0053] Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .
[0054] (3) Related feature extraction module.
[0055] The relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.
[0056] The input of the relevant feature extraction module includes the input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features .
[0057] In the relevant feature extraction module, the input feature and input features Expand them into one-dimensional vectors respectively, and process each one-dimensional vector as follows: divide the one-dimensional vector into multiple feature tags, input each feature tag into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then reverse the original expansion method to restore all relevant feature blocks to the same shape as the original input feature. The corresponding related features are recorded as , input features The corresponding related features are recorded as .
[0058] Assume the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. For any self-attention layer, let it be the first Layer, the processing process is:
[0059] ;
[0060] ;
[0061] in, is the normalization layer, is a multi-head attention block, is a feed-forward network.
[0062] In this embodiment, the feature labels need to pass through a stack consisting of eight self-attention layers to learn relevant feature blocks.
[0063] (4) Modal weight fusion module.
[0064] The modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively output by the relevant feature extraction module. and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature .
[0065] The weights are calculated by the model normalization layer: for the relevant features , set its The individual voxels are , and a total of The corresponding weight is .
[0066] Fusion Features The calculation method is: .
[0067] (5) Classifier.
[0068] In the classifier, fusion features The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result. The classification neural network uses the sigmoid function as the activation function.
[0069] Step 2: training the multimodal classification and recognition model.
[0070] Step 2-1: Construct a training sample set.
[0071] The training sample set includes three types of samples: Class I samples, Class II samples, and Class III samples. The three types of samples are randomly distributed within the training sample set. Each training sample contains a visible light image and an infrared image of the same target. Class I samples contain both visible light images and infrared images. Class II samples contain only visible light images. Class III samples contain only infrared images.
[0072] Step 2-2: Train the invariant feature learning module.
[0073] The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features The difference between.
[0074] Specifically, the training samples are divided into multiple batches. Within each batch, all training samples are first fed into the model. During input, if images of one modality are missing from the current sample, images of the other modality are substituted for the missing modality and fed into the multimodal classification and recognition model. The corresponding high-order center distances within this batch are then calculated, and the parameters of the first and second convolutional neural networks are updated with the goal of minimizing these high-order center distances.
[0075] High-order central moments The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch, represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, preferably 2 or 3, Indicates calculation of L2 norm.
[0076] In steps 2-3, the training samples are fed into the multimodal classification and recognition model. If images from one modality are missing from the current sample, images from the other modality are substituted for the missing modality. A cross-entropy loss is calculated based on the classification results and the true class labels of the training samples. The network parameters of the entire multimodal classification and recognition model are updated based on this cross-entropy loss. This step updates the parameters of all modules in the multimodal classification and recognition model to ensure that the model fully utilizes feature extraction, invariant feature learning, and feature fusion for the classification task.
[0077] Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one modality, the image of the other modality is used to replace the missing modality image and input into the multimodal classification and recognition model.
[0078] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. The scope of the present invention is defined by the claims rather than the foregoing description.
Claims
1. A missing multimodal ship classification and recognition method based on modal invariant feature learning, characterized in that the steps include: Step 1: Construct a multimodal classification and recognition model, wherein the multimodal classification and recognition model includes a feature extraction and fusion part and a classifier; The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain the fusion feature. The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module; The classifier is used to Get classification and recognition results; In the classifier, fusion features The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result; Step 2: training the multimodal classification and recognition model; Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and the infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one modality, the image of the other modality is used to replace the missing modality image and input into the multimodal classification and recognition model. The visible light feature extraction branch is used to extract visible light high-level features from the input visible light image ; The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image ; The invariant feature learning module includes a first convolutional neural network corresponding to the visible light feature extraction branch and a second convolutional neural network corresponding to the infrared feature extraction branch; Convolutional neural networks extract high-level features of the input layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer. Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .
2. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to claim 1 is characterized in that: The visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network. The input visible light image Visible light image encoder Then obtain the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain visible light high-level features ; The infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. The input infrared image After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .
3. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to claim 1, characterized in that: The relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.
4. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to claim 3 is characterized in that: The input of the relevant feature extraction module includes the input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features ; In the relevant feature extraction module, the input feature and input features They are expanded into one-dimensional vectors respectively, and each one-dimensional vector is processed by the self-attention mechanism to obtain the same features as the input features. Corresponding related features and input features Corresponding related features .
5. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to claim 4 is characterized in that: In the relevant feature extraction module, the one-dimensional vector expanded by the input feature is processed in the following way: the one-dimensional vector is divided into multiple feature tags, each feature tag is input into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then all relevant feature blocks are reversed according to the original expansion method to restore them to relevant features with the same shape as the original input features.
6. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to claim 5, characterized in that: Assume that the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. For any self-attention layer, let it be the first Layer, the processing process is: ； ； in, is the normalization layer, is a multi-head attention block, is a feed-forward network.
7. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to any one of claims 4 to 6, characterized in that: The modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively output by the relevant feature extraction module. and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature ; The weights are calculated by the model normalization layer: for the relevant features , set its The individual voxels are , and a total of The corresponding weight is ; Fusion Features The calculation method is: .
8. The missing multimodal ship classification and recognition method based on modal invariant feature learning according to any one of claims 1, 3, 4, 5 or 6, characterized in that: Step 2 specifically includes: Step 2-1, construct a training sample set; The training sample set includes three types of samples: Class I samples, Class II samples, and Class III samples. The three types of samples are randomly distributed in the training sample set. Each training sample includes a visible light image and an infrared image of the same target. Class I samples include visible light images and infrared images; Class II samples only include visible light images; and Class III samples only include infrared images. Step 2-2, training the invariant feature learning module; The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features the differences between; Specifically, the training samples are divided into multiple batches. In each batch, all training samples are first input into the model. During input, if the current sample lacks an image of one modality, the image of the other modality is used to replace the missing modality image and input into the multimodal classification and recognition model. Then, the corresponding high-order center distance in this batch is calculated, and the parameters of the first convolutional neural network and the second convolutional neural network are updated with the goal of minimizing the high-order center distance. High-order central moments The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch, represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, Indicates calculation of L2 norm; Step 2-3: Input the training samples into the multimodal classification and recognition model. During input, if the current sample lacks an image of one of the modalities, use the image of the other modality to replace the missing modality image and input it into the multimodal classification and recognition model; calculate the cross-entropy loss based on the classification results and the true category labels of the training samples, and update the network parameters of the entire multimodal classification and recognition model based on the cross-entropy loss.