A remote sensing image classification method based on Transformer lightweight model

Abstract

The present invention discloses a remote sensing image classification method based on a Transformer lightweight model, which relates to the technical field of image classification. The method comprises: visualizing and preprocessing an optical remote sensing image according to actual conditions; then obtaining image coding information of the optical remote sensing image through an image coding module; then performing feature extraction on the image coding information of the optical remote sensing image through an MLP module to obtain information coding; then extracting feature information in the information coding through an Attention module; then superimposing the feature information extracted by the Attention module and the information coding obtained by the MLP module through a residual module, and then performing normalization processing through a normalization module; finally, after the normalization processing by the normalization module, the information is sent to a classification module through an FNN module, thereby obtaining a final classification result. The present invention has a higher training effect on small and medium-sized data sets than a classic Transformer model, reduces the amount of model calculation, saves computing resources, and improves the classification speed.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a remote sensing image classification method based on a Transformer lightweight model. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Image classification is a very active research direction in the fields of computer vision, pattern recognition, and machine learning. Image classification is widely used in many fields, including face recognition and intelligent video analysis in the security field, object recognition in traffic scenes, vehicle counting, and license plate recognition in the transportation field, as well as content-based image retrieval and automatic album classification in the Internet field. The purpose of image classification is to determine the type of object in the image and thus obtain the image category.

[0004] Feature extraction is an important part of image classification, which mainly includes traditional features (such as histogram of oriented gradients (HOG) combined with support vector machine (SVM)) and deep learning-based features (convolutional neural network (CNN)-based methods, Transformer-based methods, etc.).

[0005] In the Transformer-based method, the classic Transformer model encodes the original input image in blocks, and the features obtained during image encoding are relatively rough, ignoring a lot of information; at the same time, the Attention module of the classic Transformer model has problems such as high model complexity, low computational efficiency, and long time consumption, and its effect in the field of optical remote sensing image classification cannot be fully reflected. Summary of the Invention

[0006] The purpose of the present invention is to provide a remote sensing image classification method based on a lightweight Transformer model, which has a higher training effect on small and medium-sized data sets than the classic Transformer model, and reduces the amount of model calculation, saves computing resources, and improves the classification speed, thereby solving the above problems.

[0007] The technical solutions of the present invention are as follows:

[0008] A remote sensing image classification method based on a lightweight Transformer model, including:

[0009] Step S1: Select an optical remote sensing image, and visualize and preprocess the selected optical remote sensing image according to actual conditions;

[0010] Step S2: encoding the selected optical remote sensing image through an image encoding module to obtain image encoding information of the selected optical remote sensing image;

[0011] Step S3: extracting features from the image coding information of the selected optical remote sensing image through the MLP module to obtain information coding;

[0012] Step S4: extracting feature information from information encoding through the Attention module;

[0013] Step S5: The feature information extracted by the Attention module and the information encoding obtained by the MLP module are superimposed by the residual module and then normalized by the normalization module;

[0014] Step S6: After the normalization module performs normalization processing, it is sent to the classification module through the FNN module to obtain the final classification result.

[0015] Furthermore, in the step S1, visualization is to convert the selected optical remote sensing image into a picture; pre-processing includes: image rotation, clarity adjustment, and image size conversion;

[0016] The image size conversion includes: converting the image size of the selected optical remote sensing image to (224, 224).

[0017] Furthermore, the step S1 further includes:

[0018] The selected optical remote sensing images are stored in different folders according to categories, and each folder is named after the category.

[0019] Furthermore, the step S2 includes:

[0020] The image encoding module performs convolution operation on the selected optical remote sensing image through 4 convolution layers with a convolution kernel size of 3 to extract image feature information therein;

[0021] Then, a convolution layer with a convolution kernel size of 1 is used to adjust the dimension of the image feature information, and the two-dimensional image feature information is converted into one-dimensional image coding information.

[0022] Furthermore, in step S2, the feature dimension finally extracted is (batch, 28, 28, 192).

[0023] Furthermore, the step S3 includes:

[0024] Step S31: The image coding information obtained in step S2 is subjected to feature extraction by the MLP module, and features with a feature dimension of (batch, 14, 14, 192) are obtained by downsampling.

[0025] Step S32: Expand the features obtained in step S301 to obtain information encoding with a feature dimension of (batch, 196, 192);

[0026] Step S33: Split the information code obtained in step S302 according to the feature dimensions (batch, 8, 196, 24), and use the split information code as data input Q to input into the Attention module and the residual module.

[0027] Furthermore, the calculation formula of the Attention module is as follows:

[0028]

[0029] in,

[0030] d n Enter the dimension value of Q for the data;

[0031] Q T Transpose the Q matrix for the data input.

[0032] Compared with the existing technology, the beneficial effects of the present invention are:

[0033] 1. A remote sensing image classification method based on a lightweight Transformer model improves the accuracy of the Transformer model while reducing the number of model parameters, thereby obtaining better classification results; and the accuracy is greatly improved compared to the classic Transformer model and CNN model.

[0034] 2. A remote sensing image classification method based on the Transformer lightweight model can greatly reduce the huge number of model parameters and computational complexity caused by the complexity of the Transformer model.

[0035] 3. A remote sensing image classification method based on a lightweight Transformer model. By outputting features of different scales in the model, compared with the scale-invariant feature extraction of traditional Transformer models, it can better obtain features of different scales and improve the overall accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1Flowchart of a remote sensing image classification method based on the Transformer lightweight model

[0037] Figure 2 This is the overall classification flow chart in Example 1;

[0038] Figure 3 This is an example diagram of data samples;

[0039] Figure 4 This is the QAttention model structure diagram;

[0040] Figure 5 This is the MLP module structure diagram;

[0041] Figure 6 This is a comparison chart of accuracy. DETAILED DESCRIPTION

[0042] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0043] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0044] Example 1

[0045] In the Transformer-based method, the classic Transformer model encodes the original input image in blocks, and the features obtained during image encoding are relatively rough, ignoring a lot of information; at the same time, the Attention module of the classic Transformer model has problems such as high model complexity, low computational efficiency, and long time consumption, and its effect in the field of optical remote sensing image classification cannot be fully reflected.

[0046] Due to the shortcomings of the Transformer-based image classification model, such as its high complexity and difficulty in model training convergence, the training effect on small and medium-sized datasets such as optical remote sensing image classification is poor.

[0047] This embodiment proposes a remote sensing image classification method based on a lightweight Transformer model. To address the problems of the classic Transformer model, such as high model complexity, low computational efficiency, long computational time, and difficulty converging for small and medium-sized data sets, this method improves the model's fitting ability, reduces network parameters, and increases the average accuracy and training speed of optical remote sensing image classification by modifying the model structure and using an improved QAttention module, a convolutional implementation of the MLP module, and an improved image encoding module. This method can be used to obtain category recognition information for an optical remote sensing image.

[0048] Please note that Figure 2 The overall classification process includes: obtaining training samples and test samples from the optical remote sensing image classification dataset; performing data enhancement on the training samples; constructing an image encoding module, a QAttention module, an MLP module, and a classification module; inputting the training samples into the constructed classification model for training to obtain a trained model; inputting the test samples into the trained model to predict and output the optical remote sensing image classification results.

[0049] Among them, regarding the image encoding module, since the image encoding module of the classic Transformer model is relatively rough in extracting image features, it has a great impact on the convergence of the model and the final accuracy results. In order to improve the stability of model training, this embodiment extracts the feature information in the original image as perfectly as possible and adopts multi-layer convolution for downsampling, thereby greatly alleviating the problem that the Vision Transformer model is difficult to train in small-scale data.

[0050] Regarding the QAttention module, the original calculation formula of the Attention module is as follows:

[0051]

[0052] The calculation formula after removing the V parameter is as follows:

[0053]

[0054] Only the calculation formula of the Q parameter is retained, as follows:

[0055]

[0056] After experimental comparison, it was found that there was no significant difference in the test accuracy of different structures; and the number of model parameters and the amount of calculation decreased with the reduction of Q, K, and V parameters; therefore, in this embodiment, it was decided to retain only the calculation formula of the Q parameter as the calculation formula of the QAttention module, namely:

[0057]

[0058] The QAttention module structure is as follows Figure 4 shown.

[0059] Regarding the MLP module, since the implementation of the convolution operation depends on the linear operation of the elements within the receptive field, in the implementation of the MLP module, the convolution layer with a convolution kernel of 1 is considered as a fully connected model in the feature map, and two-dimensional convolution is used to replace the one-dimensional MLP module for operation, thereby further reducing the number of model parameters and improving the model's operational efficiency and stability.

[0060] The MLP module structure is as follows Figure 5 shown.

[0061] Regarding the classification module, no improvement is made to the classification module in this embodiment. Those skilled in the art should be aware of its operating principle, and no further details will be given for this purpose.

[0062] See also Figure 1-6 , a remote sensing image classification method based on the Transformer lightweight model, is essentially the process of training the model, including:

[0063] Step S1: Select an optical remote sensing image, and visualize and preprocess the selected optical remote sensing image according to actual conditions; preferably, in step S1, visualization is to convert the selected optical remote sensing image into a picture; preprocessing includes: conventional image preprocessing processes such as image rotation, clarity adjustment, and image size conversion;

[0064] The image size conversion includes: converting the image size of the selected optical remote sensing image to (224, 224); Figure 3 Schematic diagram of the data sample shown;

[0065] The step S1 further includes: storing the selected optical remote sensing images into different folders according to categories, each folder being named after the category; and then placing all optical remote sensing images belonging to the category into the folder; for example, in the NWPU-RESISC45 format, each folder contains 700 optical remote sensing images of this category.

[0066] Step S2: Encode the selected optical remote sensing image through the image encoding module to obtain image encoding information of the selected optical remote sensing image; preferably, step S2 includes:

[0067] The image encoding module performs convolution operation on the selected optical remote sensing image through 4 convolution layers with a convolution kernel size of 3 to extract image feature information therein;

[0068] Then, a convolution layer with a convolution kernel size of 1 is used to adjust the dimension of the image feature information, converting the two-dimensional image feature information into one-dimensional image coding information, so that the final extracted feature dimension is (batch, 28, 28, 192).

[0069] Step S3: extracting features from the image coding information of the selected optical remote sensing image through the MLP module to obtain information coding; preferably, step S3 includes:

[0070] Step S31: The image coding information obtained in step S2 is subjected to feature extraction by the MLP module, and features with a feature dimension of (batch, 14, 14, 192) are obtained by downsampling.

[0071] Step S32: Expand the features obtained in step S301 to obtain information encoding with a feature dimension of (batch, 196, 192);

[0072] Step S33: Split the information code obtained in step S302 according to the feature dimensions (batch, 8, 196, 24), and use the split information code as data input Q to input into the Attention module and the residual module.

[0073] Step S4: extracting feature information from the information encoding through the Attention module; preferably, the data input Q is input into the following formula for calculation, thereby extracting the feature information from the information encoding;

[0074] The calculation formula of the Attention module is as follows:

[0075]

[0076] in,

[0077] d n Enter the dimension value of Q for the data;

[0078] Q T Transpose the Q matrix for the data input.

[0079] Step S5: The feature information extracted by the Attention module and the information encoded by the MLP module are superimposed by the residual module and then normalized by the normalization module. The functions of the residual module and the normalization module are to improve the convergence ability of the model and prevent the problem of model accuracy degradation caused by gradient diffusion.

[0080] Step S6: After the normalization module performs normalization processing, it is sent to the classification module through the FNN module to obtain the final classification result; the function of the FNN module is to adjust the internal channel structure of the model so that the QAttention modules can pay attention to the feature information of different positions, thereby improving the accuracy of the model.

[0081] The QAttention module in this embodiment has 32.11M parameters and 7.07GFlops of computation, which is significantly reduced compared to the original Attention module with 48.06M parameters and 9.42GFlops of computation.

[0082] See also Figure 6 In this embodiment, a remote sensing image classification method based on a lightweight Transformer model is proposed. The classification accuracy on the NWPU-RESISC45 dataset with a 20% training set is 79.76%, which is 13.36% higher than the 66.4% of the classic Transformer model and 4.53% higher than Resnet50. The accuracy is significantly improved and the difficulty of fitting the Transformer model is greatly alleviated.

[0083] The remote sensing image classification method based on the Transformer lightweight model proposed in this embodiment has a classification speed of 824.4 images / s, which is significantly improved compared to the 759.6 images / s of the classic Transformer model.

[0084] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

[0085] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of the description in this section that did not constitute prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A remote sensing image classification method based on a Transformer lightweight model, characterized in that: include: Step S1: Select an optical remote sensing image, and visualize and preprocess the selected optical remote sensing image; Step S2: encoding the selected optical remote sensing image through an image encoding module to obtain image encoding information of the selected optical remote sensing image; Step S3: extracting features from the image coding information of the selected optical remote sensing image through the MLP module to obtain information coding; Step S4: extracting feature information from information encoding through the Attention module; Step S5: The feature information extracted by the Attention module and the information encoding obtained by the MLP module are superimposed by the residual module and then normalized by the normalization module; Step S6: After the normalization module performs normalization processing, it is sent to the classification module through the FNN module to obtain the final classification result; The calculation formula of the Attention module is as follows: in, d n Enter the dimension value of Q for the data; Q T Transpose the Q matrix for the data input.

2. A remote sensing image classification method based on a Transformer lightweight model according to claim 1, characterized in that: In the step S1, visualization is to convert the selected optical remote sensing image into a picture; Preprocessing includes: image rotation, clarity adjustment, and image size conversion.

3. The remote sensing image classification method based on the Transformer lightweight model according to claim 1 is characterized in that: The step S1 further includes: The selected optical remote sensing images are stored in different folders according to categories, and each folder is named after the category.

4. The remote sensing image classification method based on the Transformer lightweight model according to claim 1 is characterized in that: The step S2 includes: The image encoding module performs convolution operation on the selected optical remote sensing image through 4 convolution layers with a convolution kernel size of 3 to extract image feature information therein; Then, a convolution layer with a convolution kernel size of 1 is used to adjust the dimension of the image feature information, and the two-dimensional image feature information is converted into one-dimensional image coding information.

5. The remote sensing image classification method based on the Transformer lightweight model according to claim 4 is characterized in that: In step S2, the feature dimension finally extracted is (batch, 28, 28, 192).

6. The remote sensing image classification method based on the Transformer lightweight model according to claim 1, characterized in that: The step S3 includes: Step S31: The image coding information obtained in step S2 is subjected to feature extraction by the MLP module, and features with a feature dimension of (batch, 14, 14, 192) are obtained by downsampling. Step S32: Expand the features obtained in step S301 to obtain information encoding with a feature dimension of (batch, 196, 192); Step S33: Split the information code obtained in step S302 according to the feature dimensions (batch, 8, 196, 24), and use the split information code as data input Q to input into the Attention module and the residual module.

Images