Multimodal ship image classification method based on Mamba Fourier calculation

Abstract

The present invention discloses a multimodal ship image classification method based on Mamba Fourier calculation, wherein the natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain a classification result. The multimodal classification model includes a sequence generation module and a classification head, and also includes N convolutional Mamba modules and N-1 Fourier attention modules. The convolutional Mamba module is used to perform convolution operations and dual-sequence Mamba calculations on the input features to obtain the basic features of the output. The Fourier attention module is used to globally interact and fuse the two sets of input features through Fourier transform and attention mechanism to obtain the output fusion features. The present invention introduces the Mamba model and Fourier calculation, realizes the efficient extraction of ship image features, and takes into account the local detail information and global semantic information of the image, thereby more comprehensively describing the characteristics of the ship, while effectively suppressing noise and background interference, thereby improving the accuracy of classification and recognition.

Description

Technical Field

[0001] The invention belongs to the field of data recognition, and in particular relates to a ship image recognition method. Background Art

[0002] In the marine field, among the many applications of remote sensing technologies, ship image classification technology is particularly critical. It significantly improves the maritime situation awareness capability by accurately identifying and classifying ship images. It can not only effectively monitor maritime traffic, but also help reduce the losses caused by marine disasters. It has extremely high value.

[0003] The ship image classification methods in the existing technology are mainly based on machine learning and deep learning. The classification accuracy is still low when dealing with complex sea conditions and multimodal data. The reasons are: on the one hand, the existing methods are easily affected by noise and background interference during the feature extraction process, resulting in low classification accuracy; on the other hand, when dealing with multimodal data fusion, the existing methods lack an effective cross-modal information interaction mechanism and cannot fully utilize the advantages of each modal data. Summary of the invention

[0004] The present invention proposes a multimodal ship image classification method based on Mamba Fourier calculation, the purpose of which is to solve the problem of low classification accuracy when processing complex sea conditions and multimodal data.

[0005] The technical solution of the present invention is as follows:

[0006] A multimodal ship image classification method based on Mamba Fourier calculation, in which the natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result;

[0007] The multimodal classification model includes a sequence generation module and a classification head, and also includes N convolutional mamba modules and N-1 Fourier attention modules, where N is greater than or equal to 3;

[0008] The sequence generation module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively;

[0009] The convolutional Mamba module is used to perform convolution operation and dual-sequence Mamba calculation on the input features to obtain the output basic features;

[0010] For the first convolutional mamba module, its input is the concatenated features of the natural light token sequence and the infrared token sequence; for the second to N convolutional mamba modules, their input is the basic features output by the previous convolutional mamba module;

[0011] The Fourier attention module is used to globally interact and fuse the two sets of input features through Fourier transform and attention mechanism to obtain output fusion features;

[0012] For the first Fourier attention module, its input is the basic features output by the first convolutional mamba module and the basic features output by the second convolutional mamba module; for any n-th Fourier attention module from the second to N-1 Fourier attention modules, its input is the fused features output by the n-1-th Fourier attention module and the basic features output by the n+1-th convolutional mamba module;

[0013] The classification head is used to obtain a classification result based on the fusion features output by the last Fourier attention module.

[0014] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation: the sequence generation module is an image block embedding module;

[0015] The calculation method of the natural light token sequence is: A = Relu (Conv (x));

[0016] The calculation method of infrared token sequence is:

[0017] Among them, Relu(·) represents the activation function in the sequence conversion module, Conv(·) represents the convolution operation in the sequence conversion module, x is the input natural light image, is the input infrared image.

[0018] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation: the convolutional Mamba module includes a convolutional module and a dual-sequence Mamba module connected in sequence.

[0019] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation: let the natural light token sequence and infrared token sequence be A, B∈R N×D , A=(a1,a2,..,a i ,…,a N ), B=(b1,b2,..,b i ,…,b N ), a i ∈R 1×D , b i ∈R 1×D , the input feature of the convolutional Mamba module is recorded as F in ∈R 2N×D , the input feature F of the first convolutional Mamba module in =(a1,a2,..,a i,…,a N ,b1,b2,..,b i ,…,b N ), then the convolution module processing process in the convolutional mamba module is:

[0020] F conv =Conv 3×3 (F in );

[0021] Among them, Conv 3×3 (·) represents the convolution operation with a kernel size of 3×3, F conv ∈R 2N×D .

[0022] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation: the processing process of the dual-sequence Mamba module in the convolutional Mamba module is:

[0023] Step A-1: ​​Based on feature F conv Calculate the attention score S∈R using the convolutional layer and activation function D×1 , to achieve preliminary feature extraction:

[0024] S = Relu(Conv 1×1 (F conv ));

[0025] Among them, Conv 1×1 (·) represents the convolution operation with a kernel size of 1×1, and Relu(·) is the activation function in the dual-sequence Mamba module;

[0026] Step A-2: Based on feature F conv Using positive sequence SSM for feature extraction:

[0027] F forward =SSM forward (Conv 1×1 (F conv ));

[0028] Among them, SSM forward (·) represents the positive sequence structured state space model;

[0029] Step A-3: Set feature F conv Arrange in reverse order to obtain the reverse feature F′ conv :

[0030] Let F conv =(c1,c2,..,c i ,…,c N ,d1,d2,..,d i ,…,d N),but:

[0031] F′ conv =(d N ,..,d i ,…,d2,d1,c N ,..,c i ,…,c2,c1);

[0032] Then, based on the reverse feature F′ conv Feature extraction using reverse SSM:

[0033] F backward =SSM backward (Conv 1×1 (F′ conv ));

[0034] Among them, SSM backward (·) represents the reverse structured state space model;

[0035] Step A-4: Use the attention score to enhance the features extracted by the forward SSM and the reverse SSM to obtain the output basic features F out ∈R 2N×D :

[0036] F out =F forward ×S+F backward ×S.

[0037] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation, the two sets of features input into the Fourier attention module are Then the processing process of the Fourier attention module is:

[0038] Step B-1: and After adding, normalization is performed to obtain feature F FT :

[0039]

[0040] Among them, Norm(·) represents normalization processing;

[0041] Step B-2: Feature F FT After the 1x1 convolution, the convolution results are subjected to three independent depth-wise separable convolutions to obtain the query matrix Q, key matrix K, and value matrix V:

[0042]

[0043] in, Represents three independent groups of depth-separable convolution operations with a convolution kernel size of 1×1, Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1;

[0044] Step B-3, perform fast Fourier calculation on the query matrix Q and the key matrix K respectively, multiply the two fast Fourier calculation results, and then perform inverse fast Fourier calculation and normalization processing in sequence, and then multiply the normalization processing result with the value matrix V, and then convolve the multiplication result, and then convolve the convolution result with the input and Perform residual connection to obtain fusion features

[0045]

[0046] Among them, FFT(·) represents fast Fourier transform calculation, and IFFT(·) represents inverse fast Fourier transform calculation.

[0047] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation: let the fusion feature output by the last Fourier attention module be The processing process of the classification head is:

[0048]

[0049] Among them, Linear(·) represents the fully connected layer of the classification head, Softmax(·) is the probability output function, and the vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

[0050] As a further improvement of the multimodal ship image classification method based on Mamba Fourier calculation, the training process of the multimodal classification model is:

[0051] Construct a training set, which is represented by in, is the i-th group of samples, X i ∈R H×W×C represents the natural light image in the i-th group of samples, represents the infrared image in the i-th group of samples, H represents the image height, W represents the image width, C represents the number of image channels, the ship in the natural light image and the infrared image in the same sample is the same ship, Y i represents the true category of the ship in the i-th group of samples, and Z represents the number of samples;

[0052] The samples in the training set are input into the multimodal classification model, the cross entropy loss function is calculated according to the classification results and the true categories, and then the cross entropy loss function is used to reversely optimize the network model.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention introduces the Mamba model and Fourier calculation to improve the accuracy of classification: the Mamba model realizes efficient extraction of ship image features through long and short sequence feature modeling, and takes into account the local detail information and global semantic information of the image, thereby more comprehensively describing the characteristics of the ship; the Fourier calculation utilizes its frequency domain transformation characteristics to effectively suppress noise and background interference, enhance the robustness and anti-interference ability of the image, and make the feature extraction process more accurate and stable.

[0055] 2. The present invention also optimizes the fusion mechanism of multimodal data. Traditional multimodal data fusion often adopts the method of feature accumulation, while the present invention uses dual-sequence Mamba calculation to first refine multimodal features into long sequence features and then uses Fourier attention calculation to perform convolution operations in an efficient manner. This not only improves the efficiency and quality of feature extraction, but also accelerates the calculation process, thereby effectively solving the problem of low classification accuracy caused by large differences in multimodal image features and difficulty in feature extraction in complex environments in existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the framework of the ship classification model of the present invention;

[0057] Figure 2 is a schematic diagram of a dual-sequence Mamba module;

[0058] Figure 3 Schematic diagram of the Fourier attention module. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0060] A multimodal ship image classification method based on Mamba Fourier calculation is proposed. The natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result.

[0061] like Figure 1 The multimodal classification model includes a sequence generation module and a classification head, and also includes N convolutional mamba modules and N-1 Fourier attention modules, where N is greater than or equal to 3. In this embodiment, there are 4 convolutional mamba modules and 3 Fourier attention modules.

[0062] The sequence generation module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively.

[0063] Specifically, the sequence generation module is an image block embedding module;

[0064] The calculation method of the natural light token sequence is: A = Relu (Conv (x));

[0065] The calculation method of infrared token sequence is:

[0066] Among them, Relu(·) represents the activation function in the sequence conversion module, Conv(·) represents the convolution operation in the sequence conversion module, x is the input natural light image, is the input infrared image. The natural light token sequence and infrared token sequence are represented as A, B∈R N×D , A=(a1,a2,..,a i ,…,a N ), B=(b1,b2,..,b i ,…,b N ), a i ∈R 1×D , b i ∈R 1×D .

[0067] The convolutional mamba module includes a convolutional module and a dual-sequence mamba module connected in sequence. The convolutional mamba module is used to perform convolution operation and dual-sequence mamba calculation on input features to obtain output basic features.

[0068] For the first convolutional Mamba module, its input is the concatenation of the natural light token sequence and the infrared token sequence: (a1, a2, .., a i ,…,a N ,b1,b2,..,b i ,…,b N ); For the 2nd to Nth convolutional Mamba modules, their input is the basic features output by the previous convolutional Mamba module.

[0069] Specifically, Figure 2 , the input feature of any convolutional Mamba module is recorded as F in ∈R 2N×D , then the convolution module processing process in the convolutional mamba module is:

[0070] F conv =Conv 3×3 (F in );

[0071] Among them, Conv 3×3 (·) represents the convolution operation with a kernel size of 3×3, F conv ∈R 2N×D .

[0072] Furthermore, the processing process of the dual-sequence Mamba module is:

[0073] Step A-1: ​​Based on feature F conv Calculate the attention score S∈R using the convolutional layer and activation function D×1 , to achieve preliminary feature extraction:

[0074] S = Relu(Conv 1×1 (F conv ));

[0075] Among them, Conv 1×1 (·) represents the convolution operation with a kernel size of 1×1, and Relu(·) is the activation function in the dual-sequence Mamba module.

[0076] Step A-2: Based on feature F conv Feature extraction using positive sequence SSM:

[0077] F forward =SSM forward (Conv 1×1 (F conv ));

[0078] Among them, SSM forward (·) represents the positive sequence structured state space model.

[0079] Step A-3: Set feature F conv Arrange in reverse order to obtain the reverse feature F′ conv :

[0080] Let F conv =(c1,c2,..,c i ,…,c N ,d1,d2,..,d i ,…,d N ),but:

[0081] F′ conv =(d N ,..,d i ,…,d2,d1,c N ,..,c i ,…,c2,c1);

[0082] Then, based on the reverse feature F′ conv Feature extraction using reverse SSM:

[0083] F backward =SSM backward (Conv 1×1 (F′ conv ));

[0084] Among them, SSM bcckward (·) represents the reverse structured state space model.

[0085] Step A-4: Use the attention score to enhance the features extracted by the forward SSM and the reverse SSM to obtain the output basic features F out ∈R 2N×D :

[0086] F out =F forward ×S+F backward ×S.

[0087] The Fourier attention module is used to globally interact and fuse the two sets of input features through Fourier transform and attention mechanism to obtain output fused features.

[0088] For the first Fourier attention module, its input is the basic features output by the first convolutional mamba module and the basic features output by the second convolutional mamba module. For any nth Fourier attention module from the second to N-1 Fourier attention modules, its input is the fused features output by the n-1th Fourier attention module and the basic features output by the n+1th convolutional mamba module.

[0089] like Figure 3 , let the two sets of features input into the Fourier attention module be Then the processing process of the Fourier attention module is:

[0090] Step B-1: and After adding, normalization is performed to obtain feature F FT :

[0091]

[0092] Among them, Norm(·) represents normalization processing.

[0093] Step B-2: Feature F FT After the 1x1 convolution, the convolution results are subjected to three independent depth-wise separable convolutions to obtain the query matrix Q, key matrix K, and value matrix V:

[0094]

[0095]

[0096] in, Represents three independent groups of depth-separable convolution operations with a convolution kernel size of 1×1, Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1.

[0097] Step B-3, perform fast Fourier calculation on the query matrix Q and the key matrix K respectively, multiply the two fast Fourier calculation results, and then perform inverse fast Fourier calculation and normalization processing in sequence, and then multiply the normalization processing result with the value matrix V, and then convolve the multiplication result, and then convolve the convolution result with the input and Perform residual connection to obtain fusion features

[0098]

[0099] Among them, FFT(·) represents fast Fourier transform calculation, and IFFT(·) represents inverse fast Fourier transform calculation.

[0100] The classification head is used to obtain the classification result according to the fusion feature output by the last Fourier attention module. Assume that the fusion feature output by the last Fourier attention module is The processing process of the classification head is:

[0101]

[0102] Among them, Linear(·) represents the fully connected layer of the classification head, Softmax(·) is the probability output function, and the vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

[0103] The training process of the multimodal classification model is:

[0104] Construct a training set, which is represented by in, is the i-th group of samples, X i ∈R H×W×C represents the natural light image in the i-th group of samples, represents the infrared image in the i-th group of samples, H represents the image height, W represents the image width, C represents the number of image channels, the ship in the natural light image and the infrared image in the same sample is the same ship, Y i represents the true category of the ship in the i-th group of samples, and Z represents the number of samples.

[0105] The samples in the training set are input into the multimodal classification model, and the cross entropy loss function is calculated according to the classification results and the true categories. The cross entropy loss function is then used to reversely optimize the network model using the Adam optimizer.

[0106] After the training is completed, the natural light image and infrared image of the same ship are input into the trained model to obtain the classification result.

[0107] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A multimodal ship image classification method based on Mamba Fourier calculation, characterized by: The natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result; The multimodal classification model includes a sequence generation module and a classification head, and also includes N convolutional Mamba modules and N-1 Fourier attention modules, where N is greater than or equal to 3; The sequence generation module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively; The convolutional Mamba module is used to perform convolution operation and dual-sequence Mamba calculation on the input features to obtain the output basic features; For the first convolutional mamba module, its input is the concatenated features of the natural light token sequence and the infrared token sequence; for the second to N convolutional mamba modules, their input is the basic features output by the previous convolutional mamba module; The Fourier attention module is used to globally interact and fuse the two sets of input features through Fourier transform and attention mechanism to obtain output fusion features; For the first Fourier attention module, its input is the basic features output by the first convolutional mamba module and the basic features output by the second convolutional mamba module; for any n-th Fourier attention module from the second to N-1 Fourier attention modules, its input is the fused features output by the n-1-th Fourier attention module and the basic features output by the n+1-th convolutional mamba module; The classification head is used to obtain a classification result based on the fusion features output by the last Fourier attention module.

2. The multimodal ship image classification method based on Mamba Fourier calculation as claimed in claim 1, characterized in that: The sequence generation module is an image block embedding module; The calculation method of the natural light token sequence is: A = Relu (Conv (x)); The calculation method of infrared token sequence is: Among them, Relu(·) represents the activation function in the sequence conversion module, Conv(·) represents the convolution operation in the sequence conversion module, x is the input natural light image, is the input infrared image.

3. The multimodal ship image classification method based on Mamba Fourier calculation as claimed in claim 1, characterized in that: The convolutional Mamba module includes a convolutional module and a dual-sequence Mamba module connected in sequence.

4. The multimodal ship image classification method based on Mamba Fourier calculation as claimed in claim 3, characterized in that: Assume that the natural light token sequence and infrared token sequence are A, B∈R N×D , A=(a1,a2,..,a i ,…,a N ), B=(b1,b2,..,b i ,…,b N ), a i ∈R 1×D , b i ∈R 1×D , the input feature of the convolutional Mamba module is recorded as F in ∈R 2N×D , the input feature F of the first convolutional Mamba module in =(a1,a2,..,a i ,…,a N ,b1,b2,..,b i ,…,b N ), then the convolution module processing process in the convolutional mamba module is: F conv =Conv 3×3 (F in ); Among them, Conv 3×3 (·) represents the convolution operation with a kernel size of 3×3, F conv ∈R 2N×D .

5. The multimodal ship image classification method based on Mamba Fourier calculation as claimed in claim 4, characterized in that: The processing process of the dual sequence Mamba module in the convolutional Mamba module is as follows: Step A-1, according to the feature F conv Calculate the attention score S∈R using the convolutional layer and activation function D×1 , to achieve preliminary feature extraction: S=Relu(Conv 1×1 (F conv )); Among them, Conv 1×1 (·) represents the convolution operation with a kernel size of 1×1, and Relu(·) is the activation function in the dual-sequence Mamba module; Step A-2: Based on feature F conv Feature extraction using positive sequence SSM: F forward =SSM forward (Conv 1×1 (F conv )); Among them, SSM forward (·) represents the positive sequence structured state space model; Step A-3: Set feature F conv Arrange in reverse order to obtain the reverse feature F′ conv : Let F conv =(c1,c2,..,c i ,…,c N ,d1,d2,..,d i ,…,d N ),but: F′ conv =(d N ,..,d i ,…,d2,d1,c N ,..,c i ,…,c2,c1); Then, based on the reverse feature F′ conv Feature extraction using reverse SSM: F backward =SSM backward (Conv 1×1 (F′ conv )); Among them, SSM backward (·) represents the reverse structured state space model; Step A-4: Use the attention score to enhance the features extracted by the forward SSM and the reverse SSM to obtain the output basic features F out ∈R 2N×D : F out =F forward ×S+F backward ×S。 6. The multimodal ship image classification method based on Mamba Fourier calculation according to claim 1, characterized in that: Assume that the two sets of features input into the Fourier attention module are Then the processing process of the Fourier attention module is: Step B-1: and After adding, normalization is performed to obtain feature F FT : Among them, Norm(·) represents normalization processing; Step B-2: Feature F FT After the 1x1 convolution, the convolution results are subjected to three independent depth-wise separable convolutions to obtain the query matrix Q, key matrix K, and value matrix V: in, Represents three independent groups of depth-separable convolution operations with a convolution kernel size of 1×1, Conv 1×1 (·) represents a convolution operation with a kernel size of 1×1; Step B-3, perform fast Fourier calculation on the query matrix Q, perform fast Fourier calculation and transpose calculation on the key matrix K, multiply the two calculation results, and then perform inverse fast Fourier calculation and normalization processing in sequence, then multiply the normalization processing result with the value matrix V, and then convolve the multiplication result, and then convolve the convolution result with the input and Perform residual connection to obtain fusion features Among them, FFT(·) represents fast Fourier transform calculation, and IFFT(·) represents inverse fast Fourier transform calculation.

7. The multimodal ship image classification method based on Mamba Fourier calculation as claimed in claim 1, characterized in that: Suppose the fusion feature output by the last Fourier attention module is The processing process of the classification head is: Among them, Linear(·) represents the fully connected layer of the classification head, Softmax(·) is the probability output function, and the vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

8. The multimodal ship image classification method based on Mamba Fourier calculation according to any one of claims 1 to 7, characterized in that: The training process of the multimodal classification model is: Construct a training set, which is represented by in, is the i-th group of samples, X i ∈R H ×W×C represents the natural light image in the i-th group of samples, represents the infrared image in the i-th group of samples, H represents the image height, W represents the image width, C represents the number of image channels, the ship in the natural light image and the infrared image in the same sample is the same ship, Y i represents the true category of the ship in the i-th group of samples, and Z represents the number of samples; The samples in the training set are input into the multimodal classification model, the cross entropy loss function is calculated according to the classification results and the true categories, and then the cross entropy loss function is used to reversely optimize the network model.

Images