Ship image classification method based on local-global multi-head attention

Abstract

The present invention discloses a ship image classification method based on local global multi-head attention, which belongs to the field of data recognition. The present invention comprises the following steps: evenly dividing the input image into multiple features along the channel dimension, then extracting features by convolution, and splicing the results of feature extraction along the channel dimension to obtain feature A; fusing the width and height dimensions of feature A with the channel dimension respectively, and then obtaining feature B on the basis of the fusion; performing multi-head attention calculation on the basis of feature B to obtain local global multi-head attention features; performing linear transformation on the local global multi-head attention features to obtain the classification result of the model. The present invention reduces the complexity of the model by extracting features by channels and using local attention in multi-head attention calculation, thereby promoting model convergence and improving the accuracy of ship image classification.

Description

Technical Field

[0001] The invention belongs to the field of data recognition, and in particular relates to a ship image classification method. Background Art

[0002] As a hot research direction in the field of remote sensing, ship image classification provides a convenient and reliable solution for automatic ship identification. Ship image classification methods aim to extract features from ship images that can correctly reflect their category information.

[0003] Early optical remote sensing scene classification tasks were performed using machine learning methods. Machine learning methods such as support vector machines, decision trees, and random forests can achieve good results in classification tasks, but these machine learning methods often rely on manually extracted features, resulting in poor generalization capabilities of the models.

[0004] The deep learning-based method omits the manual feature extraction step and uses an end-to-end neural network model to automatically extract features. Visual Transformer (ViT) has achieved remarkable results as the first application of the self-attention mechanism in the field of vision. ViT can model the global features of the image due to its unique self-attention mechanism, thereby capturing long-range feature dependencies. However, ViT is a global modeling, so the computational complexity is high, and it has high requirements for hardware equipment. The model is difficult to converge, which affects the accuracy of ship image classification. Summary of the invention

[0005] The present invention proposes a ship image classification method based on local-global multi-head attention, the purpose of which is to improve the accuracy of ship image classification.

[0006] The technical solution of the present invention is as follows:

[0007] A ship image classification method based on local global multi-head attention adopts a local global multi-head attention classification model for classification; the local global multi-head attention classification model includes: an initial feature extraction module, a secondary feature extraction module, a multi-head attention calculation module and a classification output module; the method includes:

[0008] Step A: The initial feature extraction module takes the input image The feature extraction is evenly divided into multiple features along the channel dimension, and then convolution is used to extract the features. The results of the feature extraction are spliced ​​along the channel dimension to obtain the features. ;

[0009] Step B: The secondary feature extraction module extracts the features The width and height dimensions of are fused with the channel dimensions respectively, and then the features are obtained based on the fusion ;

[0010] Step C: Use the multi-head attention calculation module to calculate the feature Based on this, multi-head attention calculation is performed to obtain local and global multi-head attention features;

[0011] Step D: The classification output module performs linear transformation on the local and global multi-head attention features to obtain the classification results of the model.

[0012] As a further improvement of the ship image classification method based on local-global multi-head attention, the specific process of step A is as follows:

[0013] Step A1: Set the input image , Indicates the image height, Indicates the image width, Indicates the number of image channels; the input image Evenly divided into 4 features along the channel dimension ;

[0014] Step A2: Features According to the following formulas, convolution processing is performed respectively to obtain the features :

[0015] ;

[0016] ;

[0017] ;

[0018] in, Represents a convolution operation with a convolution kernel size of 3×3;

[0019] Step A3: Splice along the channel dimension to get the feature :

[0020] ;

[0021] in, Represents the splicing operation along the channel dimension, feature .

[0022] As a further improvement of the ship image classification method based on local global multi-head attention, the specific process of step B is:

[0023] Step B1: Perform normalization, and then fuse the width and height dimensions of the normalized image features with the channel dimension respectively:

[0024] ;

[0025] ;

[0026] in, Represents the fusion of high dimension and channel dimension, Represents the fusion of wide dimension and channel dimension, represents normalization processing, ;

[0027] Step B2: fusion of features , and the normalized features Use linear layers for feature extraction and transform the features The dimension of the extracted result is restored to the normalized feature Consistent:

[0028] ;

[0029] ;

[0030] ;

[0031] in represents a linear layer, Represents high-dimensional and channel-dimensional splitting, represents the splitting of width dimension and channel dimension, ;

[0032] Step B3: Splicing is performed along the channel dimension, and then the splicing result is input into the convolution layer to restore the channel dimension to the same as the feature Then the high dimension and wide dimension of the restored result are fused to obtain the feature :

[0033] ;

[0034] in, Represents the fusion of high dimension and wide dimension, represents a convolution operation with a convolution kernel size of 3×3, represents the splicing operation along the channel dimension, .

[0035] As a further improvement of the ship image classification method based on local-global multi-head attention, the specific process of step C is:

[0036] Step C1: After three different linear layers, we can obtain Three matrices:

[0037] ;

[0038] ;

[0039] ;

[0040] in, , , , and Represents 3 different linear layers; input image , Indicates the image height, Indicates the image width, Indicates the number of image channels;

[0041] Step C2: Perform local global multi-head attention calculation: transform the matrix The matrices are divided into three evenly along the channel dimension, resulting in nine matrices: ; Divide the above nine matrices into three groups. The first group of matrices includes , the second set of matrices includes , the third group of matrices includes , and then get the features based on the first set of matrices , based on the second set of matrices to obtain features , based on the third set of matrices, we get the features ;

[0042] Step C3: Concatenate along the channel dimension and combine with the feature Make residual connection to get features :

[0043] ;

[0044] in, represents the splicing operation along the channel dimension, ;

[0045] Step C4: After normalization, it is input into the feedforward network, and then the features are The output of the feedforward network is added and input into the multi-layer perceptron to obtain the final representation of the ship image. , that is, local-global multi-head attention features.

[0046] As a further improvement of the ship image classification method based on local global multi-head attention, in step C2, based on the first group of matrices Get Features The way is as follows:

[0047] ;

[0048] in, , represents the number of channels of the feature image, is the activation function, .

[0049] As a further improvement of the ship image classification method based on local global multi-head attention, in step C2, based on the second set of matrices Get Features The way is as follows:

[0050] Will Each of them is evenly divided into 4 parts along the width and height dimensions of the feature, resulting in twelve matrices: , ; Divide the twelve matrices into equal numbers Divided into four groups, each group is calculated independently to obtain four features , , calculated as follows:

[0051] ;

[0052] in, is the activation function, ;

[0053] Then the four features Splice along the width and height dimensions to get features :

[0054] ;

[0055] in, Represents the splicing operation along the width and height dimensions, .

[0056] As a further improvement of the ship image classification method based on local global multi-head attention, in step C2, based on the third group of matrices Get Features The way is as follows:

[0057] Will Divide the width and height dimensions evenly into 16 parts, and get 48 matrices: , ; Divide the forty-eight matrices into equal numbers Divided into sixteen groups, each group is calculated independently to obtain sixteen features , , calculated as follows:

[0058] ;

[0059] in, is the activation function, ;

[0060] Then the sixteen characteristics Splice along the width and height dimensions to get features :

[0061] ;

[0062] in, Represents the splicing operation along the width and height dimensions, .

[0063] As a further improvement of the ship image classification method based on local global multi-head attention, the specific process of step C4 is:

[0064] Specific: The characteristics After normalization, the features are obtained : , represents normalization;

[0065] The feedforward network consists of 1 DW convolution layer, 2 linear layers and 1 normalization layer. Get Features :

[0066] ;

[0067] represents a linear layer, represents the activation function, represents normalization processing, represents DW convolution processing;

[0068] The characteristics And the output of the feedforward network After addition, it is input into the multi-layer perceptron to obtain the local global multi-head attention features :

[0069] ;

[0070] in, Stands for Multilayer Perceptron.

[0071] As a further improvement of the ship image classification method based on local global multi-head attention, in step D, the classification result is calculated as follows:

[0072] ;

[0073] in, is the local global multi-head attention feature, represents a linear transformation, represents an activation function that transforms a set of arbitrary real numbers into real numbers representing a probability distribution; the vector The category represented by the item with the largest median value is the predicted category of the model.

[0074] As a further improvement of the ship image classification method based on local global multi-head attention, before performing step A, the local global multi-head attention classification model is trained in the following manner:

[0075] Step T1: Obtain ship images with image labels to form a sample data set. Each image and its corresponding category are recorded as ,in, Indicates Ship images, Represents a ship image The corresponding true label is the true category, , Indicates the number of ship images;

[0076] Step T2: Input the images in the sample data set into the local global multi-head attention classification model to obtain the corresponding classification results; input the images The corresponding classification results are recorded as ;

[0077] Step T3: Calculate the cross entropy loss between the classification result and the true label:

[0078] ;

[0079] Step T4, adjusting the parameters of the local-global multi-head attention classification model according to the cross entropy loss;

[0080] Step T5: Repeat steps T1 to T4 until the cross entropy loss is less than a preset value or the number of repetitions reaches a preset value.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] The present invention reduces the model complexity by extracting features by channels and adopting local attention in multi-head attention calculation, thereby promoting model convergence and improving the accuracy of ship image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 is a flow chart of the method of the present invention;

[0084] Figure 2Schematic diagram of the process of local-global multi-head attention calculation;

[0085] Figure 3 Schematic diagram of the calculation flow of the feedforward network. DETAILED DESCRIPTION

[0086] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0087] This embodiment provides a ship image classification method based on local-global multi-head attention, which realizes ship image classification by extracting channel features and using local attention in multi-head attention calculation to reduce model complexity.

[0088] like Figure 1 As shown, the ship image classification method based on local global multi-head attention adopts a local global multi-head attention classification model for classification. The local global multi-head attention classification model includes: an initial feature extraction module, a secondary feature extraction module, a multi-head attention calculation module and a classification output module.

[0089] Among them, the initial feature extraction module obtains the image features according to the input image The secondary feature extraction module is used in feature Dimension fusion is performed on the basis of , and then features are obtained based on the fusion The multi-head attention calculation module is based on the feature Multi-head attention calculation is performed on the basis of . When different heads perform calculations, the dimension of the matrix gradually decreases, and local-global multi-head attention features are obtained. The classification output module obtains the output of the model based on the local-global multi-head attention features, that is, the classification results of the ships in the image.

[0090] Assume the input image , Indicates the image height, Indicates the image width, Represents the number of image channels, then the local-global multi-head attention classification model processes the input image as follows:

[0091] Step A: The initial feature extraction module takes the input image The feature extraction is evenly divided into 4 features along the channel dimension, and then convolution is used to extract the features. The extracted feature results are spliced ​​along the channel dimension to obtain the feature The specific process is:

[0092] Step A1: Input image Evenly divided into 4 features along the channel dimension .

[0093] Step A2: Features According to the following formulas, convolution processing is performed respectively to obtain the features :

[0094] ;

[0095] ;

[0096] ;

[0097] in, Represents a convolution operation with a convolution kernel size of 3×3.

[0098] Step A3: Splice along the channel dimension to get the feature :

[0099] ;

[0100] in, Represents the splicing operation along the channel dimension, feature .

[0101] Step B: The secondary feature extraction module extracts the features The width and height dimensions of are fused with the channel dimensions respectively, and then the features are obtained based on the fusion The specific steps are:

[0102] Step B1: Perform normalization, and then fuse the width and height dimensions of the normalized image features with the channel dimension respectively:

[0103] ;

[0104] ;

[0105] in, Represents the fusion of high dimension and channel dimension, Represents the fusion of wide dimension and channel dimension, represents normalization processing, .

[0106] Step B2: fusion of features , and the normalized features Use linear layers for feature extraction and transform the features The dimension of the extracted result is restored to the normalized feature Consistent:

[0107] ;

[0108] ;

[0109] ;

[0110] in represents a linear layer, Represents high-dimensional and channel-dimensional splitting, represents the splitting of width dimension and channel dimension, .

[0111] Step B3: Splicing is performed along the channel dimension, and then the splicing result is input into the convolution layer to restore the channel dimension to the same as the feature Then the high dimension and wide dimension of the restored result are fused to obtain the feature :

[0112] ;

[0113] in, Represents the fusion of high dimension and wide dimension, represents a convolution operation with a convolution kernel size of 3×3, represents the splicing operation along the channel dimension, .

[0114] Step C: Use the multi-head attention calculation module to calculate the feature Based on , we can calculate the multi-head attention and obtain the local and global multi-head attention features. Figure 1 and 2 , including:

[0115] Step C1: After three different linear layers, we can obtain Three matrices:

[0116] ;

[0117] ;

[0118] ;

[0119] in, , dimensions and features same, , and Represents 3 different linear layers.

[0120] Step C2: Perform local global multi-head attention calculation: transform the matrix The matrices are divided into three evenly along the channel dimension, resulting in nine matrices: ; Divide the above nine matrices into three groups. The first group of matrices includes , the second set of matrices includes , the third group of matrices includes , and then get the features based on the first set of matrices , based on the second set of matrices to obtain features , based on the third set of matrices, we get the features .

[0121] Step C2-1: Based on the first set of matrices Get Features The way is as follows:

[0122] ;

[0123] in, , represents the number of channels of the feature image, is the activation function, .

[0124] Step C2-2: Based on the second set of matrices Get Features The way is as follows:

[0125] Will Each of them is evenly divided into 4 parts along the width and height dimensions of the feature, resulting in twelve matrices: , . Divide the twelve matrices into equal numbers Divided into four groups, each group is calculated independently to obtain four features , , calculated as follows:

[0126] ;

[0127] in, is the activation function, .

[0128] Then the four features Splice along the width and height dimensions to get features :

[0129] ;

[0130] in, Represents the splicing operation along the width and height dimensions, .

[0131] By performing attention calculations independently in groups and then concatenating the results, global calculations can be avoided, reducing the computational complexity to a certain extent.

[0132] Step C2-3, based on the third set of matrices Get Features The way is as follows:

[0133] Will Divide the width and height dimensions evenly into 16 parts, and get 48 matrices: , . Divide the forty-eight matrices into equal numbers Divided into sixteen groups, each group is calculated independently to obtain sixteen features , , calculated as follows:

[0134] ;

[0135] in, is the activation function, .

[0136] Then the sixteen characteristics Splice along the width and height dimensions to get features :

[0137] ;

[0138] in, Represents the splicing operation along the width and height dimensions, .

[0139] Step C3: Concatenate along the channel dimension and combine with the feature Make residual connection to get features :

[0140] ;

[0141] in, represents the splicing operation along the channel dimension, .

[0142] Step C4: After normalization, it is input into the feedforward network, and then the features are The output of the feedforward network is added and input into the multi-layer perceptron to obtain the final representation of the ship image. , that is, local-global multi-head attention features.

[0143] Specifically, the characteristics After normalization, the features are obtained : , Represents normalization.

[0144] like Figure 3 The feedforward network consists of 1 DW convolution layer, 2 linear layers and 1 normalization layer. Get Features :

[0145] ;

[0146] represents a linear layer, represents the activation function, represents normalization processing, Stands for DW convolution processing.

[0147] The characteristics And the output of the feedforward network After addition, it is input into the multi-layer perceptron to obtain the local global multi-head attention features :

[0148] ;

[0149] in, Stands for Multilayer Perceptron.

[0150] Step D: Classification output module for local global multi-head attention features Perform a linear transformation to obtain the classification result of the model:

[0151] ;

[0152] in, represents a linear transformation, Represents an activation function that transforms a set of arbitrary real numbers into real numbers representing a probability distribution. The vector The category represented by the item with the largest median value is the predicted category of the model.

[0153] Furthermore, the local-global multi-head attention classification model is trained in the following way:

[0154] Step T1: Obtain ship images with image labels to form a sample data set. Each image and its corresponding category are recorded as ,in, Indicates Ship images, Represents a ship image The corresponding true label is the true category, , Indicates the number of ship images.

[0155] Step T2: Input the images in the sample data set into the local global multi-head attention classification model to obtain the corresponding classification results. The corresponding classification results are recorded as .

[0156] Step T3: Calculate the cross entropy loss between the classification result and the true label:

[0157] .

[0158] Step T4: Adjust the parameters of the local-global multi-head attention classification model according to the cross entropy loss.

[0159] Step T5: Repeat steps T1 to T4 until the cross entropy loss is less than a preset value or the number of repetitions reaches a preset value.

[0160] In this embodiment, the Adam optimizer is used to optimize and solve the loss function.

[0161] It is obvious to those skilled in the art that the embodiments of the present invention are not limited to the details of the exemplary embodiments described above, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present invention is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the embodiments of the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules or devices stated in the system, device or terminal claims can also be implemented by the same unit, module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A ship image classification method based on local-global multi-head attention, characterized by: A local global multi-head attention classification model is used for classification; the local global multi-head attention classification model includes: an initial feature extraction module, a secondary feature extraction module, a multi-head attention calculation module and a classification output module; the method includes: Step A: The initial feature extraction module takes the input image The feature extraction is evenly divided into multiple features along the channel dimension, and then convolution is used to extract the features. The results of the feature extraction are spliced ​​along the channel dimension to obtain the features. ; The specific process of step A is: Step A1: Set the input image , Indicates the image height, Indicates the image width, Indicates the number of image channels; the input image Evenly divided into 4 features along the channel dimension ; Step A2: Features According to the following formulas, convolution processing is performed respectively to obtain the features : ； ； ； in, Represents a convolution operation with a convolution kernel size of 3×3; Step A3: Splice along the channel dimension to get the feature : ； in, Represents the splicing operation along the channel dimension, feature ; Step B: The secondary feature extraction module extracts the features The width and height dimensions of are fused with the channel dimensions respectively, and then the features are obtained based on the fusion ; The specific process of step B is: Step B1: Perform normalization, and then fuse the width and height dimensions of the normalized image features with the channel dimension respectively: ； ； in, Represents the fusion of high dimension and channel dimension, Represents the fusion of wide dimension and channel dimension, represents normalization processing, ; Step B2: fusion features , and the normalized features Use linear layers for feature extraction and transform the features The dimension of the extracted result is restored to the normalized feature Consistent: ； ； ； in represents a linear layer, Represents high-dimensional and channel-dimensional splitting, represents the splitting of width dimension and channel dimension, ; Step B3: Splicing is performed along the channel dimension, and then the splicing result is input into the convolution layer to restore the channel dimension to the same as the feature Then the high dimension and wide dimension of the restored result are fused to obtain the feature : ； in, Represents the fusion of high dimension and wide dimension, represents a convolution operation with a convolution kernel size of 3×3, represents the splicing operation along the channel dimension, ; Step C: Use the multi-head attention calculation module to calculate the feature Based on this, multi-head attention calculation is performed to obtain local and global multi-head attention features; Step D: The classification output module performs linear transformation on the local and global multi-head attention features to obtain the classification results of the model.

2. The ship image classification method based on local-global multi-head attention as claimed in claim 1, characterized in that: The specific process of step C is: Step C1: After three different linear layers, we can obtain Three matrices: ； ； ； in, , , , and Represents 3 different linear layers; input image , Indicates the image height, Indicates the image width, Indicates the number of image channels; Step C2: Perform local global multi-head attention calculation: transform the matrix The matrices are divided into three evenly along the channel dimension, resulting in nine matrices: ; Divide the above nine matrices into three groups. The first group of matrices includes , the second set of matrices includes , the third group of matrices includes , and then get the features based on the first set of matrices , based on the second set of matrices to obtain features , based on the third set of matrices, we get the features ; Step C3: Concatenate along the channel dimension and combine with the feature Make residual connection to get features : ； in, represents the splicing operation along the channel dimension, ; Step C4: After normalization, it is input into the feedforward network, and then the features are The output of the feedforward network is added and input into the multi-layer perceptron to obtain the final representation of the ship image. , that is, local-global multi-head attention features.

3. The ship image classification method based on local-global multi-head attention as claimed in claim 2, characterized in that: In step C2, based on the first set of matrices Get Features The way is as follows: ； in, , represents the number of channels of the feature image, is the activation function, .

4. The ship image classification method based on local-global multi-head attention as claimed in claim 2, characterized in that: In step C2, based on the second set of matrices Get Features The way is as follows: Will Each of them is evenly divided into 4 parts along the width and height dimensions of the feature, resulting in twelve matrices: , ; Divide the twelve matrices into equal numbers Divided into four groups, each group is calculated independently to obtain four features , , calculated as follows: ； in, is the activation function, ; Then the four features Splice along the width and height dimensions to get features : ； in, Represents the splicing operation along the width and height dimensions, .

5. The ship image classification method based on local-global multi-head attention as claimed in claim 2, characterized in that: In step C2, based on the third matrix Get Features The way is as follows: Will Divide the width and height dimensions evenly into 16 parts, and get 48 matrices: , ; Divide the forty-eight matrices into equal numbers Divided into sixteen groups, each group is calculated independently to obtain sixteen features , , calculated as follows: ； in, is the activation function, ; Then the sixteen characteristics Splice along the width and height dimensions to get features : ； in, Represents the splicing operation along the width and height dimensions, .

6. The ship image classification method based on local-global multi-head attention as claimed in claim 2, characterized in that: The specific process of step C4 is: Specific: The characteristics After normalization, the features are obtained : , represents normalization processing; The feedforward network consists of 1 DW convolution layer, 2 linear layers and 1 normalization layer. Get Features : ； represents a linear layer, represents the activation function, represents normalization processing, represents DW convolution processing; The characteristics And the output of the feedforward network After addition, it is input into the multi-layer perceptron to obtain the local global multi-head attention features : ； in, Stands for Multilayer Perceptron.

7. The ship image classification method based on local-global multi-head attention as claimed in claim 1, characterized in that: In step D, the classification result is calculated as follows: ； in, is the local global multi-head attention feature, represents a linear transformation, represents an activation function that transforms a set of arbitrary real numbers into real numbers representing a probability distribution; the vector The category represented by the item with the largest median value is the predicted category of the model.

8. The ship image classification method based on local-global multi-head attention according to any one of claims 1 to 7, characterized in that: Before performing step A, the local global multi-head attention classification model is trained in the following manner: Step T1: Obtain ship images with image labels to form a sample data set. Each image and its corresponding category are recorded as ,in, Indicates Ship images, Represents a ship image The corresponding true label is the true category, , Indicates the number of ship images; Step T2: Input the images in the sample data set into the local global multi-head attention classification model to obtain the corresponding classification results; The corresponding classification results are recorded as ; Step T3: Calculate the cross entropy loss between the classification result and the true label: ； Step T4: adjust the parameters of the local-global multi-head attention classification model according to the cross entropy loss; Step T5: Repeat steps T1 to T4 until the cross entropy loss is less than a preset value or the number of repetitions reaches a preset value.

Images