Missing multi-modal ship classification and identification method based on modal invariant feature learning

Abstract

The invention discloses a missing multi-modal ship classification and identification method based on modal invariant feature learning, and belongs to the field of data identification. The method comprises the steps of firstly extracting advanced features of different modal images, then learning invariant features of different modalities based on the advanced features, further calculating to obtain related features of the different modalities according to the advanced features and the invariant features, then obtaining weights under the different modalities based on the related features, and obtaining fusion features through modal weight fusion. And automatically learning available modalities, constructing a shared representation, and finally obtaining a classification recognition result according to fusion features. According to the method, the performance of the model can be maintained even if one mode is lacked, and high-performance and high-robustness ship classification identification in the lacked mode is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of data recognition, and in particular relates to a ship classification and recognition method, in particular to a missing multimodal ship classification and recognition method based on modal invariant feature learning. Background Art

[0002] Ship identification plays a core role in modern marine intelligent monitoring systems, and existing deep learning-based methods have achieved good results. However, how to accurately identify ships in complex sea conditions such as cloud cover and coastal background is still a problem that needs to be solved urgently.

[0003] With the continuous advancement of pattern recognition and artificial intelligence technology, ship identification technology is gradually developing towards multi-source fusion, multi-resolution analysis, effective use of attention mechanisms, and comprehensive knowledge support. Ship image information obtained by multiple sensors is complementary. Compared with the recognition of single-source sensor images, ship identification by fusing multimodal images has significant advantages.

[0004] However, in actual applications, due to factors such as weather and technology, the problem of missing modalities is common. For example, the sensor may be disconnected for some reason, resulting in the temporary inability to send data, or factors such as economic costs limit the collection of certain modal data. In addition, changes in environmental conditions may also affect the quality of specific modal data. For example, visible light images may be interfered with under severe weather conditions, and effective visible light modal images cannot be obtained. In these cases, traditional multimodal learning methods cannot maintain their original performance and cannot complete the recognition work normally. Summary of the invention

[0005] The present invention proposes a missing multimodal ship classification and recognition method based on modal invariant feature learning, the purpose of which is to solve the problem that the recognition task cannot be completed when a certain mode is true under the multimodal learning framework.

[0006] The technical solution of the present invention is as follows: A missing multimodal ship classification and recognition method based on modal invariant feature learning, comprising the following steps: Step 1: construct a multimodal classification and recognition model, wherein the multimodal classification and recognition model includes a feature extraction and fusion part and a classifier; The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain the fusion feature ; The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module; The classifier is used to Get the classification and recognition results; in the classifier, the fusion features The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result; Step 2: training the multimodal classification and recognition model; Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and the infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one of the modalities, the image of the other modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model.

[0007] As a further improvement of the missing multimodal ship classification and recognition method based on modality invariant feature learning: the visible light feature extraction branch is used to extract visible light high-level features from the input visible light image ; The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image .

[0008] As a further improvement of the missing multimodal ship classification and recognition method based on modality invariant feature learning: the visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network, and the input visible light image Visible light image encoder Then we get the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain the visible light high-level features ; The infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. The input infrared image After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .

[0009] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the invariant feature learning module includes a first convolutional neural network corresponding to the visible light feature extraction branch and a second convolutional neural network corresponding to the infrared feature extraction branch; In the convolutional neural network, the high-level features of the input are extracted layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer. Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .

[0010] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.

[0011] As a further improvement of the missing multimodal ship classification and recognition method based on modality-invariant feature learning, the input of the relevant feature extraction module includes input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features ; In the relevant feature extraction module, the input feature and input features They are expanded into one-dimensional vectors respectively, and each one-dimensional vector is processed by the self-attention mechanism to obtain the same Corresponding related features and the input features Corresponding related features .

[0012] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: in the relevant feature extraction module, the one-dimensional vector expanded by the input feature is processed in the following manner: the one-dimensional vector is divided into multiple feature tags, and each feature tag is input into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then all relevant feature blocks are reversely operated according to the original expansion method to restore them to relevant features with the same shape as the original input features.

[0013] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: Assume that the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. The corresponding related feature block, for any self-attention layer, is set to Layer, the processing process is: ; ; in, is the normalization layer, is a multi-head attention block, It is a feed-forward network.

[0014] As a further improvement of the missing multimodal ship classification and recognition method based on modal invariant feature learning: the modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively output by the relevant feature extraction module and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature ; The weights are calculated by the model normalization layer: for the relevant features , set its The individual element is , and have a total of The corresponding weight is ; Fusion Features The calculation method is: .

[0015] As a further improvement of the missing multimodal ship classification and recognition method based on modality invariant feature learning, step 2 specifically includes: Step 2-1, construct a training sample set; The training sample set includes three types of samples: Class I samples, Class II samples and Class III samples, and the three types of samples are randomly distributed in the training sample set; each training sample includes a visible light image and an infrared image from the same target; Class I samples include visible light images and infrared images; Class II samples only include visible light images; Class III samples only include infrared images; Step 2-2, training the invariant feature learning module; The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features The difference between Specifically, the training samples are divided into multiple batches; in each batch, all training samples are first input into the model. During input, if the current sample lacks an image of one modality, an image of another modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model; then the corresponding high-order center distance in this batch is calculated, and the parameters of the first convolutional neural network and the second convolutional neural network are updated with the goal of minimizing the high-order center distance; High-order central moment The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch. represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, Indicates calculation of L2 norm; Step 2-3: Input the training samples into the multimodal classification and recognition model. During input, if the current sample lacks an image of one of the modalities, use the image of the other modality to replace the missing modality image and input it into the multimodal classification and recognition model; calculate the cross entropy loss based on the classification results and the true category labels of the training samples, and update the network parameters of the entire multimodal classification and recognition model based on the cross entropy loss.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention first extracts high-level features of images of different modalities, then learns invariant features of different modalities based on high-level features, and then further calculates relevant features of different modalities based on high-level features and invariant features, and then obtains weights under different modalities based on relevant features, and then obtains fused features through modal weight fusion, automatically learns available modalities, constructs shared representation, and finally obtains classification and recognition results based on fused features. The present invention enables the model to maintain performance even in the absence of one modality, and obtains high-performance and high-robustness ship classification and recognition in the missing modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is an architectural diagram of the multimodal classification and recognition model of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0019] A missing multimodal ship classification and recognition method based on modal invariant feature learning, comprising the following steps: Step 1: Construct a multimodal classification and recognition model, which includes a feature extraction and fusion part and a classifier. The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain a fusion feature. The classifier is used to Get the classification and recognition results.

[0020] Furthermore, if Figure 1 The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module.

[0021] 1.1 Visible light feature extraction branch and infrared feature extraction branch.

[0022] The visible light feature extraction branch is used to extract visible light high-level features from the input visible light image Specifically, the visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network. The input visible light image Visible light image encoder Then we get the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain the visible light high-level features .

[0023] The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image Specifically, the infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .

[0024] The visible light image encoder and the infrared image encoder both use a pre-trained ResNet network as a backbone, and extract visible light and infrared features from the original visible light image and infrared image through a series of convolution, normalization, activation, and pooling layers.

[0025] (ii) Invariant feature learning module.

[0026] The invariant feature learning module includes a first convolutional neural network corresponding to a visible light feature extraction branch and a second convolutional neural network corresponding to an infrared feature extraction branch.

[0027] In the convolutional neural network, the high-level features of the input are extracted layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer.

[0028] Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .

[0029] 3. Related feature extraction module.

[0030] The relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.

[0031] The input of the relevant feature extraction module includes the input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features .

[0032] In the relevant feature extraction module, the input feature and input features Expand them into one-dimensional vectors respectively, and process each one-dimensional vector in the following way: divide the one-dimensional vector into multiple feature tags, input each feature tag into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then reverse all relevant feature blocks according to the original expansion method to restore them to relevant features with the same shape as the original input features. The corresponding related features are recorded as , input features The corresponding related features are recorded as .

[0033] Assume the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. The corresponding related feature block, for any self-attention layer, is set to Layer, the processing process is: ; ; in, is the normalization layer, is a multi-head attention block, It is a feed-forward network.

[0034] In this embodiment, the feature labels need to pass through a stack of eight self-attention layers to learn relevant feature blocks.

[0035] (iv) Modal weight fusion module.

[0036] The modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively output by the relevant feature extraction module. and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature .

[0037] The weights are calculated by the model normalization layer: for the relevant features , set its The individual element is , and have a total of The corresponding weight is .

[0038] Fusion Features The calculation method is: .

[0039] (V) Classifier.

[0040] In the classifier, fusion features The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result. The classification neural network uses the sigmoid function as the activation function.

[0041] Step 2: training the multimodal classification and recognition model.

[0042] Step 2-1: Construct a training sample set.

[0043] The training sample set includes three types of samples: Class I samples, Class II samples and Class III samples, and the three types of samples are randomly distributed in the training sample set. Each training sample includes a visible light image and an infrared image from the same target. Class I samples include visible light images and infrared images. Class II samples only include visible light images. Class III samples only include infrared images.

[0044] Step 2-2: Train the invariant feature learning module.

[0045] The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features The difference between.

[0046] Specifically, the training samples are divided into multiple batches. In each batch, all training samples are first input into the model. When inputting, if the current sample lacks an image of one modality, the image of the other modality is used to replace the missing modality image and input into the multimodal classification and recognition model. Then the corresponding high-order center distance in this batch is calculated, and the parameters of the first convolutional neural network and the second convolutional neural network are updated with the goal of minimizing the high-order center distance.

[0047] High-order central moment The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch. represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, preferably 2 or 3, Indicates calculation of L2 norm.

[0048] Step 2-3: Input the training samples into the multimodal classification and recognition model. When inputting, if the current sample lacks an image of one of the modalities, use the image of the other modality to replace the missing modality image and input it into the multimodal classification and recognition model. Calculate the cross entropy loss based on the classification results and the true category labels of the training samples, and update the network parameters of the entire multimodal classification and recognition model based on the cross entropy loss. In this step, the parameters of all modules of the multimodal classification and recognition model are updated, so that the model can make full use of feature extraction, invariant feature learning and fusion features in the classification task.

[0049] Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and the infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one of the modalities, the image of the other modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model.

[0050] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A missing multimodal ship classification and recognition method based on modal invariant feature learning, characterized in that the steps include: Step 1: construct a multimodal classification and recognition model, wherein the multimodal classification and recognition model includes a feature extraction and fusion part and a classifier; The feature extraction and fusion part is used to extract the features of the visible light image and infrared image of the same target and perform feature fusion to obtain the fusion feature ; The feature extraction and fusion part includes a visible light feature extraction branch, an infrared feature extraction branch, an invariant feature learning module, a related feature extraction module and a modal weight fusion module; The classifier is used to Get the classification and recognition results; In the classifier, the fusion feature The probability distribution is obtained through the classification neural network, and the category corresponding to the highest probability value in the probability distribution is the classification result; Step 2: training the multimodal classification and recognition model; Step 3: Use the trained multimodal classification and recognition model for classification and recognition. During recognition, the visible light image and the infrared image are input into the multimodal classification and recognition model. If the current target lacks an image of one of the modalities, the image of the other modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model.

2. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 1, characterized in that: The visible light feature extraction branch is used to extract visible light high-level features from the input visible light image ; The infrared feature extraction branch is used to extract infrared high-level features from the input infrared image .

3. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 2 is characterized in that: The visible light feature extraction branch includes a visible light image encoder and a visible light feature extraction neural network. The input visible light image Visible light image encoder Then we get the visible light characteristics , visible light characteristics Further input into the visible light feature extraction neural network to obtain the visible light high-level features ; The infrared feature extraction branch includes an infrared image encoder and an infrared feature extraction neural network. The input infrared image After infrared image encoder Then get the infrared characteristics , infrared characteristics Further input into the infrared feature extraction neural network to obtain infrared high-level features .

4. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 2, characterized in that: The invariant feature learning module includes a first convolutional neural network corresponding to the visible light feature extraction branch and a second convolutional neural network corresponding to the infrared feature extraction branch; In the convolutional neural network, the high-level features of the input are extracted layer by layer. Each layer extracts more complex features from the output of the previous layer, and convolution and pooling operations are performed in each layer. Visible light advanced features Input into the first convolutional neural network to obtain visible light invariant features , infrared advanced features Input into the second convolutional neural network to obtain infrared invariant features .

5. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 4 is characterized in that: The relevant feature extraction module extracts relevant features from high-level features and invariant features through a self-attention mechanism.

6. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 5, characterized in that: The input of the relevant feature extraction module includes the input features corresponding to the visible light image and the input features corresponding to the infrared image , input features , input features ; In the relevant feature extraction module, the input feature and input features They are expanded into one-dimensional vectors respectively, and each one-dimensional vector is processed by the self-attention mechanism to obtain the same Corresponding related features and the input features Corresponding related features .

7. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 6 is characterized in that: In the relevant feature extraction module, the one-dimensional vector expanded by the input feature is processed in the following way: the one-dimensional vector is divided into multiple feature tags, each feature tag is input into a stack composed of several self-attention layers to learn the potential multimodal correlation to obtain relevant feature blocks, and then all relevant feature blocks are reversed according to the original expansion method to restore them to relevant features with the same shape as the original input features.

8. The missing multimodal ship classification and identification method based on modal invariant feature learning according to claim 7 is characterized in that: Assume that the stack includes self-attention layers, the input of the first self-attention layer is the feature label , No. The output of the self-attention layer is , and the 2nd to The input of each self-attention layer is the output of the previous self-attention layer, and the output of the last self-attention layer is the feature label. The corresponding related feature block, for any self-attention layer, is set to Layer, the processing process is: ； ； in, is the normalization layer, is a multi-head attention block, It is a feed-forward network.

9. The missing multimodal ship classification and identification method based on modal invariant feature learning according to any one of claims 6 to 8, characterized in that: The modal weight fusion module extracts the relevant features corresponding to the visible light image and the infrared image respectively output by the relevant feature extraction module. and related features , and obtain the corresponding visible light weights and infrared weight , and then the obtained visible light weight and infrared weight Input features of related feature extraction modules and input features Fusion is performed to obtain the fusion feature ; The weights are calculated by the model normalization layer: for the relevant features , set its The individual element is , and have a total of The corresponding weight is ; Fusion Features The calculation method is: .

10. The missing multimodal ship classification and identification method based on modal invariant feature learning according to any one of claims 4 to 8, characterized in that: Step 2 specifically includes: Step 2-1, construct a training sample set; The training sample set includes three types of samples: Class I samples, Class II samples and Class III samples, and the three types of samples are randomly distributed in the training sample set; each training sample includes a visible light image and an infrared image from the same target; Class I samples include visible light images and infrared images; Class II samples only include visible light images; Class III samples only include infrared images; Step 2-2, training the invariant feature learning module; The parameters of the first and second convolutional neural networks in the feature distance constraint invariant feature learning module based on high-order central moments are used to reduce the visible light invariant features. and infrared invariant features The difference between Specifically, the training samples are divided into multiple batches; in each batch, all training samples are first input into the model. During input, if the current sample lacks an image of one modality, an image of another modality is used to replace the image of the missing modality and input into the multimodal classification and recognition model; then the corresponding high-order center distance in this batch is calculated, and the parameters of the first convolutional neural network and the second convolutional neural network are updated with the goal of minimizing the high-order center distance; High-order central moment The calculation method is: ;in, and They are respectively the set of visible light advanced features and the set of infrared advanced features of Class C samples in this batch. represents the empirical expectation vector, Express request The vector of the order central moments, is an integer from 2 to 5, Indicates calculation of L2 norm; Step 2-3: Input the training samples into the multimodal classification and recognition model. During input, if the current sample lacks an image of one of the modalities, use the image of the other modality to replace the missing modality image and input it into the multimodal classification and recognition model; calculate the cross entropy loss based on the classification results and the true category labels of the training samples, and update the network parameters of the entire multimodal classification and recognition model based on the cross entropy loss.

Images