Multimodal ship image classification method based on multi-sequence Mamba

Abstract

The present invention discloses a multimodal ship image classification method based on multi-sequence Mamba, wherein the natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain a classification result; the multimodal classification model comprises a sequence conversion module, a cross-attention Mamba calculation module, an alternating traversal Mamba calculation module, a spectral space state fusion module and a classification module. The present invention uses natural light images and infrared images with large differences as model inputs, first preliminarily fuses the two features through the calculation of cross-attention Mamba, then further fuses the two features through the cross-traversal Mamba calculation, and then fuses and analyzes the two features using the spectral space state fusion, thereby obtaining a rich image representation and improving the accuracy of image classification.

Description

Technical Field

[0001] The invention belongs to the field of data recognition, and in particular relates to a ship image recognition method. Background Art

[0002] Vessel image classification is one of the key technologies in the field of remote sensing image processing and analysis. It focuses on automatically classifying and identifying different scenes in vessel images. With the rapid development of remote sensing technology, people can obtain a large amount of high-resolution vessel image data, which covers a variety of scenes such as cities, forests, farmlands, and oceans. Classifying this data is of great significance for resource management, environmental monitoring, and disaster emergency response.

[0003] Traditional ship classification methods mainly rely on manual feature extraction, which has certain limitations in terms of accuracy and efficiency. In recent years, with the development of deep learning, convolutional neural networks (CNNs) have demonstrated powerful automatic feature learning capabilities in ship image classification tasks, significantly improving classification accuracy and efficiency. However, existing methods are still insufficient in extracting complex ship image features, resulting in low accuracy. Summary of the invention

[0004] The present invention proposes a multi-modal ship image classification method based on multi-sequence Mamba, the purpose of which is to solve the problems of insufficient feature extraction and low accuracy of existing classification methods.

[0005] The technical solution of the present invention is as follows:

[0006] A multimodal ship image classification method based on multi-sequence Mamba is proposed. The natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result.

[0007] The multimodal classification model includes a sequence conversion module, a cross-attention Mamba calculation module, an alternating traversal Mamba calculation module, a spectral space state fusion module and a classification module;

[0008] The sequence conversion module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively;

[0009] The cross-attention Mamba calculation module obtains two sets of first enhanced features based on the natural light token sequence and the infrared token sequence;

[0010] The alternating traversal Mamba calculation module obtains two groups of second enhanced features based on the two groups of first enhanced features;

[0011] The spectrum-space state fusion module obtains fusion features based on two sets of second enhanced features;

[0012] The classification module obtains a classification result based on the fusion features.

[0013] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba: the sequence conversion module is an image block embedding module;

[0014] The calculation method of natural light token sequence is: ;

[0015] The calculation method of infrared token sequence is: ;

[0016] in, represents the activation function in the sequence conversion module, represents the convolution operation in the sequence conversion module, is the input natural light image, is the input infrared image.

[0017] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba: Assume that the natural light token sequence , infrared token sequence , , the processing process of the cross attention Mamba calculation module is:

[0018] Step A-1: ​​Calculate the cross attention score :

[0019] ;

[0020] ;

[0021] in, represents the normalization operation, represents the activation function in the cross-attention Mamba calculation module, Represents the first linear layer in the cross-attention Mamba computation module;

[0022] Step A-2: Sequence and Perform positive sequence SSM calculation:

[0023] ;

[0024] ;

[0025] in, represents the normalization operation, represents the second linear layer in the cross-attention Mamba computation module, represents the positive sequence structured state space model;

[0026] Step A-3: Sequence and Arrange them in reverse order to get the sequence and sequence ; Then the sequence and Perform reverse SSM calculation:

[0027] ;

[0028] ;

[0029] in, represents the second linear layer in the cross-attention Mamba computation module, stands for reverse structured state space model;

[0030] Step A-4: Calculate two sets of first enhanced features based on the cross attention score, the forward SSM calculation result, and the reverse SSM calculation result and , :

[0031] ;

[0032] .

[0033] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba, the two sets of first enhanced features output by the cross-attention Mamba calculation module are assumed to be and , ,in , the processing process of alternatingly traversing the Mamba calculation module is:

[0034] Step B-1: Traverse and sort the two sets of first enhanced features to obtain features :

[0035] ;

[0036] Step B-2: Features Calculate the attention score and get the features :

[0037] ;

[0038] in, represents the normalization operation, represents the activation function in the alternate traversal Mamba calculation module, Represents alternating traversal of the first linear layer in the Mamba calculation module;

[0039] Step B-2: Features Feature extraction is performed through positive order SSM and reverse order SSM respectively to obtain the feature and :

[0040] ;

[0041] ;

[0042] in, represents the alternating traversal of the second linear layer in the Mamba computation module, represents the positive sequence structured state space model, stands for reverse structured state space model;

[0043] Step B-3: Based on the characteristics , and Compute Enhanced Features , :

[0044] ;

[0045] Step B-4: Enhance the features by traversing and sorting in the reverse way in step B-1 Split into two second enhancement features and , .

[0046] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba, the two sets of second enhanced features output by the alternate traversal Mamba calculation module are assumed to be , the processing process of the spectrum space state fusion module is:

[0047] ;

[0048] ;

[0049] ;

[0050] in, represents the normalization operation, represents the first fully connected layer in the spectral-spatial state fusion module, represents the second fully connected layer in the spectral-spatial state fusion module, represents the activation function in the spectral-spatial state fusion module, is the probability output function, fusion feature .

[0051] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba, the fusion feature obtained by the spectral spatial state fusion module is assumed to be , then the processing process of the classification module is:

[0052] ;

[0053] in, represents the fully connected layer of the classification module, is the probability output function, vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

[0054] As a further improvement of the multi-modal ship image classification method based on multi-sequence Mamba, the training process of the multi-modal classification model is:

[0055] Construct a training set, which is represented by ,in, is the i-th group of samples, represents the natural light image in the i-th group of samples, represents the infrared image in the i-th group of samples, Indicates the image height, Indicates the image width, Indicates the number of image channels. The ships in the natural light image and infrared image of the same sample are the same ship. represents the true category of the ship in the i-th group of samples, and Z represents the number of samples;

[0056] The samples in the training set are input into the multimodal classification model, the cross entropy loss function is calculated according to the classification results and the true categories, and then the cross entropy loss function is used to reversely optimize the network model.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention uses natural light images and infrared images with large differences as model inputs, and first preliminarily fuses the two features through the calculation of cross-attention Mamba, so that the two features can learn each other's features, and then further fuses the two features through cross-traversal Mamba calculation to extract richer image representation, and then uses spectral-spatial state fusion to fuse and analyze the two features, thereby obtaining rich image representation and improving the accuracy of image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the framework of the ship classification model of the present invention;

[0060] Figure 2 Schematic diagram of the cross-attention Mamba computing module;

[0061] Figure 3 A schematic diagram of alternately traversing the Mamba computing module;

[0062] Figure 4 Schematic diagram of the spectral-spatial state fusion module. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0064] A multimodal ship image classification method based on Multi-Sequence Mamba is proposed. The natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result.

[0065] like Figure 1 The multimodal classification model includes a sequence conversion module, a cross-attention Mamba calculation module, an alternating traversal Mamba calculation module, a spectral-spatial state fusion module and a classification module.

[0066] The sequence conversion module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively.

[0067] In this embodiment, the sequence conversion module is an image block embedding module;

[0068] The calculation method of natural light token sequence is: ;

[0069] The calculation method of infrared token sequence is: ;

[0070] in, represents the activation function in the sequence conversion module, represents the convolution operation in the sequence conversion module, is the input natural light image, is the input infrared image. The natural light token sequence is represented as , the infrared token sequence is expressed as , .

[0071] The cross-attention Mamba calculation module obtains two sets of first enhanced features based on the natural light token sequence and the infrared token sequence.

[0072] like Figure 2, the processing process of the cross attention Mamba calculation module is:

[0073] Step A-1: ​​Calculate the cross attention score :

[0074] ;

[0075] ;

[0076] in, represents the normalization operation, represents the activation function in the cross-attention Mamba calculation module, Represents the first linear layer in the criss-cross attention Mamba computation module.

[0077] Step A-2: Sequence and Perform positive sequence SSM calculation:

[0078] ;

[0079] ;

[0080] in, represents the normalization operation, represents the second linear layer in the cross-attention Mamba computation module, Represents the positive-sequence structured state-space model.

[0081] Step A-3: Sequence and Arrange them in reverse order to get the sequence and sequence ; Then the sequence and Perform reverse SSM calculation:

[0082] ;

[0083] ;

[0084] in, represents the second linear layer in the cross-attention Mamba computation module, Stands for reverse structured state-space model.

[0085] Step A-4: Calculate two sets of first enhanced features based on the cross attention score, the forward SSM calculation result, and the reverse SSM calculation result and , :

[0086] ;

[0087] .

[0088] The two sets of first enhanced features output can be further expressed as: , , ,in .

[0089] The alternating traversal Mamba calculation module obtains two groups of second enhanced features based on the two groups of first enhanced features.

[0090] like Figure 3 , the processing process of alternatingly traversing the Mamba calculation module is:

[0091] Step B-1: Traverse and sort the two sets of first enhanced features to obtain features :

[0092] .

[0093] Step B-2: Features Calculate the attention score and get the features :

[0094] ;

[0095] in, represents the normalization operation, represents the activation function in the alternate traversal Mamba calculation module, Represents alternating passes through the first linear layer in the Mamba computation module.

[0096] Step B-2: Features Feature extraction is performed through positive order SSM and reverse order SSM respectively to obtain the feature and :

[0097] ;

[0098] ;

[0099] in, represents the alternating traversal of the second linear layer in the Mamba computation module, represents the positive sequence structured state space model, Stands for reverse structured state-space model.

[0100] Step B-3: Based on the characteristics , and Compute Enhanced Features , :

[0101] .

[0102] Step B-4: Enhance the features by traversing and sorting in the reverse way in step B-1 Split into two second enhancement features and , .

[0103] The spectrum-space state fusion module obtains fusion features based on two groups of second enhanced features.

[0104] like Figure 4 , the processing process of the spectrum space state fusion module is:

[0105] ;

[0106] ;

[0107] ;

[0108] in, represents the normalization operation, represents the first fully connected layer in the spectral-spatial state fusion module, represents the second fully connected layer in the spectral-spatial state fusion module, represents the activation function in the spectral-spatial state fusion module, is the probability output function, fusion feature .

[0109] The classification module obtains the classification result based on the fusion feature:

[0110] ;

[0111] in, represents the fully connected layer of the classification module, is the probability output function, vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

[0112] Furthermore, the training process of the multimodal classification model is:

[0113] Construct a training set, which is represented by ,in, is the i-th group of samples, represents the natural light image in the i-th group of samples, represents the infrared image in the i-th group of samples, Indicates the image height, Indicates the image width, Indicates the number of image channels. The ships in the natural light image and infrared image of the same sample are the same ship. represents the true category of the ship in the i-th group of samples, and Z represents the number of samples.

[0114] The samples in the training set are input into the multimodal classification model, and the cross entropy loss function is calculated according to the classification results and the true categories. The cross entropy loss function is then used to reversely optimize the network model using the Adam optimizer.

[0115] After the training is completed, the natural light image and infrared image of the same ship are simultaneously input into the model to obtain the classification results.

[0116] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A multimodal ship image classification method based on multi-sequence Mamba, characterized by: The natural light image and infrared image of the same ship are simultaneously input into the multimodal classification model to obtain the classification result; The multimodal classification model includes a sequence conversion module, a cross-attention Mamba calculation module, an alternating traversal Mamba calculation module, a spectral space state fusion module and a classification module; The sequence conversion module is used to convert the input natural light image and infrared image into corresponding natural light token sequence and infrared token sequence respectively; The cross-attention Mamba calculation module obtains two sets of first enhanced features based on the natural light token sequence and the infrared token sequence; set up Natural light token sequence , infrared token sequence , , the processing process of the cross attention Mamba calculation module is: Step A-1: ​​Calculate the cross attention score : ； ； in, represents the normalization operation, represents the activation function in the cross-attention Mamba calculation module, Represents the first linear layer in the cross-attention Mamba computation module; Step A-2: Sequence and Perform positive sequence SSM calculation: ； ； in, represents the normalization operation, represents the second linear layer in the cross-attention Mamba computation module, represents the positive sequence structured state space model; Step A-3: Sequence and Arrange them in reverse order to get the sequence and sequence ; Then the sequence and Perform reverse SSM calculation: ； ； in, represents the second linear layer in the cross-attention Mamba computation module, stands for reverse structured state space model; Step A-4: Calculate two sets of first enhanced features based on the cross attention score, the forward SSM calculation result, and the reverse SSM calculation result and , : ； ； The alternating traversal Mamba calculation module obtains two groups of second enhanced features based on the two groups of first enhanced features; Assume that the two sets of first enhanced features output by the cross-attention Mamba calculation module are and , ,in , the processing process of alternatingly traversing the Mamba calculation module is: Step B-1: Traverse and sort the two sets of first enhanced features to obtain features : ； Step B-2: Features Calculate the attention score and get the features : ； in, represents the normalization operation, represents the activation function in the alternate traversal Mamba calculation module, Represents alternating traversal of the first linear layer in the Mamba calculation module; Step B-2: Features Feature extraction is performed through positive order SSM and reverse order SSM respectively to obtain the feature and : ； ； in, represents the alternating traversal of the second linear layer in the Mamba computation module, represents the positive sequence structured state space model, stands for reverse structured state space model; Step B-3: Based on characteristics , and Compute Enhanced Features , : ； Step B-4: Enhance the features by traversing and sorting in the reverse way in step B-1 Split into two second enhancement features and , ; The spectrum-space state fusion module obtains fusion features based on two sets of second enhanced features; The classification module obtains a classification result based on the fusion features.

2. The multi-modal ship image classification method based on multi-sequence Mamba as claimed in claim 1, characterized in that: The sequence conversion module is an image block embedding module; The calculation method of natural light token sequence is: ; The calculation method of infrared token sequence is: ; in, represents the activation function in the sequence conversion module, represents the convolution operation in the sequence conversion module, is the input natural light image, is the input infrared image.

3. The multi-modal ship image classification method based on multi-sequence Mamba as claimed in claim 1, characterized in that: Assume that the two sets of second enhanced features output by the alternate traversal Mamba calculation module are , the processing process of the spectrum space state fusion module is: ； ； ； in, represents the normalization operation, represents the first fully connected layer in the spectral-spatial state fusion module, represents the second fully connected layer in the spectral-spatial state fusion module, represents the activation function in the spectral-spatial state fusion module, is the probability output function, fusion feature .

4. The multi-modal ship image classification method based on multi-sequence Mamba as claimed in claim 1, characterized in that: Assume that the fusion feature obtained by the spectrum-spatial state fusion module is , then the processing process of the classification module is: ； in, represents the fully connected layer of the classification module, is the probability output function, vector Each element in represents the probability that the ship in the current input image belongs to each category, and the category corresponding to the element with the largest probability is the classification result.

5. The multimodal ship image classification method based on multi-sequence Mamba according to any one of claims 1 to 4, characterized in that: The training process of the multimodal classification model is: Construct a training set, which is represented by ,in, For the i Group samples, Indicates i Natural light images from the group sample, Indicates i Infrared images in the group sample, Indicates the image height, Indicates the image width, Indicates the number of image channels. The ships in the natural light image and infrared image of the same sample are the same ship. Indicates i The true category of the ships in the group sample, Z represents the number of samples; The samples in the training set are input into the multimodal classification model, the cross entropy loss function is calculated according to the classification results and the true categories, and then the cross entropy loss function is used to reversely optimize the network model.

Images