Remote sensing image scene classification method fusing VanilaNet-5 network and rectangular convolution

Abstract

The invention discloses a remote sensing image scene classification method, and the method comprises the steps: obtaining an image of a remote sensing scene through Internet search, and carrying out the data enhancement preprocessing of the image; a matrix convolution maximum fusion module RCMF is introduced into the VanilaNet-5 model, and a rectangular convolution kernel convolution layer addition fusion and original input feature splicing method is adopted; a region selection module AS is added behind an RCMF, and a sequential polarization self-attention mechanism SPS is introduced; then the AS combines the SPS with a mini-region convolution module MRC to perform feature multiplication fusion; secondly, inputting the preprocessed image data into the improved model for training and testing to obtain an optimal model; and finally, inputting a to-be-classified image into the optimal model to obtain a classification result. The remote sensing image scene classification method fusing the VanilaNet-5 network and the rectangular convolution provided by the invention has a good application prospect.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image scene classification and relates to a multimodal feature fusion method for classifying remote sensing image scenes. Background Art

[0002] With the continuous advancement of remote sensing technology and the continuous emergence of large-scale remote sensing data, it has become easier to obtain high-quality remote sensing images, and the demand for remote sensing image analysis has also increased. Among them, remote sensing scene image classification is an important task in remote sensing image analysis, and is widely used in various fields such as environmental monitoring, urban planning, and agricultural management. Early remote sensing image classification research mainly used traditional methods based on artificially designed features and machine learning methods, which achieved certain success, but this method consumed a lot of manpower and material resources. However, with the rise of deep learning, methods based on deep neural networks have made major breakthroughs in this field. Deep learning models can learn feature representations from large-scale remote sensing image data, automatically discover and extract high-level semantic features in images, and thus significantly improve classification accuracy and generalization ability. Due to the complex terrain structure, rich spectral information, and massive data sets of remote sensing images, there are still many challenging problems that need to be solved.

[0003] Overall, most existing methods focus on designing complex network structures, such as end-to-end frameworks, gated bidirectional networks, etc. However, this complexity is often accompanied by the risk of overfitting, which not only limits the generalization ability of the model, but also affects its applicability on different datasets. Although the self-attention mechanism can dynamically focus on the key areas of the image and improve the accuracy of the classification task to a certain extent, the existing methods still face the challenge of not being able to fully capture all important features after integrating the attention mechanism. Therefore, designing an efficient remote sensing image scene classification algorithm that can fully utilize the advantages of the self-attention mechanism and overcome the limitations of existing methods has become an urgent need for current research. Summary of the invention

[0004] The purpose of the present invention is to provide a remote sensing image scene classification method based on deep learning technology, break through the limitations of existing research in the field of remote sensing image scene classification, and effectively improve the accuracy of remote sensing image scene classification.

[0005] A remote sensing image scene classification method integrating a VanillaNet-5 network and rectangular convolution of the present invention comprises the following steps:

[0006] (1) First, data preprocessing is performed on the remote sensing image dataset obtained through Internet search;

[0007] (2) In view of the challenges of remote sensing image scene classification and the lack of research in this task field, a remote sensing image scene classification method that integrates the VanillaNet-5 network and rectangular convolution is proposed. The method includes introducing a matrix convolution maximum fusion module (RCMF), using the method of adding and fusion of irregular convolution kernel convolution layers and splicing the original input features to obtain richer local feature information; adding a region selection module (AS) after the matrix convolution maximum fusion module (RCMF), first introducing the sequential polarization self-attention mechanism (SPS) to finely capture and extract the key features in the image, providing more information input for the deep processing stage; then the region selection module (AS) combines the sequential polarization self-attention mechanism (SPS) with the mini region convolution module (MRC) to perform feature multiplication fusion, enhance local detail features, improve spatial position relationships, and obtain more effective local feature information;

[0008] (3) The preprocessed remote sensing image data is fed into the improved VanillaNet-5 model for training and verification, and the test set is used for centralized testing. The final performance of the model is evaluated through multiple evaluation indicators;

[0009] (4) The remote sensing image to be classified is input into the improved VanillaNet-5 model trained in step c above, and the classification result is output through forward calculation.

[0010] Furthermore, the pretreatment method described in step 1 is specifically:

[0011] (1.1) Before the experiment, we first set the corresponding category for the remote sensing scene in each image; normalize the size of the image I input to the network and scale the image size to 64*64 pixels; then, we perform flip transformation, random rotation, random cropping and other operations on the image to obtain image X after amplification; finally, we randomly divide the dataset into a training set and a test set in a ratio of 5:5.

[0012] Furthermore, the remote sensing image scene classification method integrating the VanillaNet-5 network and rectangular convolution described in step 2 is specifically as follows:

[0013] (2.1) The rectangular convolution maximum fusion module (RCMF) is introduced, which uses the method of adding and fusion of irregular convolution kernel convolution layers and splicing the original input features to extract unique shallow local feature information; the sizes of its rectangular convolution kernels are 4×2 and 2×4 respectively, and then a convolution with a step size of 2 is used to supplement the coherence of spatial feature information, which is marked as Secondly, a maximum pooling layer of size 2 is used to reduce the dimension of the feature map. After the rectangular convolution kernel extracts the unique feature information, the output feature maps of the two rectangular convolution layers are added and fused, and then spliced ​​with the feature map extracted by the 4×4 convolution in VanillaNet-5 to generate more comprehensive feature information. The specific calculation formula is as follows:

[0014]

[0015] where Y R is the output feature map, Z is the feature map obtained by the Head layer in VanillaNet-5, * represents the convolution operation, Maxpool(·) represents the maximum pooling operation, and cat(·) represents the concatenation operation.

[0016] (2.2) An area selection module (AS) is added after the rectangular convolution maximum fusion module (RCMF) to enhance local detail features and improve spatial position relationships. First, a sequential polarization self-attention mechanism (SPS) is introduced to more finely capture and extract key features in the image, providing more information input for the deep processing stage. The specific calculation process of this module is as follows:

[0017]

[0018]

[0019] where Y ch ,Y sp They represent the output channel attention feature map and the output spatial attention feature map respectively, ⊙ represents the convolution multiplication operation, represents feature map multiplication, represents feature map addition, * represents convolution operation, Gap(·) represents global average pooling layer, reshape(·) represents fully connected layer, denote the convolutional layers for channel and spatial attention, respectively. Sigmoid(·) denotes the activation function, which maps the variable to between 0 and 1. SPS is the output feature map.

[0020] (2.3) Then the mini region convolution module (MRC) is introduced to help the model focus on extracting key features in a small area while suppressing noise and redundant information; the convolution kernel size used by the module is 2 and the stride is 2, which is expressed as The specific calculation process is as follows:

[0021]

[0022] where Y MRCTo output the feature map, * represents the convolution operation, Sigmiod (·) represents the activation function, and the variable is mapped to between 0 and 1; finally, the region selection module (AS) combines the sequential polarization self-attention mechanism (SPS) with the mini region convolution module (MRC) to fuse the feature multiplication, enhance the local detail features, improve the spatial position relationship, and obtain more effective local feature information. The specific calculation process of this module is as follows:

[0023] Y AS =Maxpool(Y SPS ,C,2,2)⊙Y MRC

[0024] Where Maxpool(·) represents the maximum pooling operation, C represents the channel dimension size of the pooling window, 2 represents the window size is 2 in both height and width, and Y AS is the output feature map.

[0025] Furthermore, the model training, verification and testing method of the fusion VanillaNet-5 network and rectangular convolution described in step 3 is specifically as follows:

[0026] (3.1) Input the remote sensing image dataset preprocessed in step 1 into the improved VanillaNet-5 network in step 2 for training. Set 300 iterations. After each iteration, use the validation set to verify the performance of the model generated in each iteration. Save the optimal model weight file through comparison.

[0027] (3.2) After the training iteration is completed, the test set is used to test the optimal model obtained in the final training in (3.1). The performance of the improved model is evaluated through the overall accuracy, Kappa coefficient, accuracy, precision, recall rate, F1 score, and confusion matrix of the model to verify the effectiveness and advancement of the model.

[0028] Furthermore, the method for classifying remote sensing image scenes using the improved model described in step 4 is specifically as follows: first, the remote sensing image to be classified is input into the model, and loaded into (3.1) to obtain the optimal model weight, and finally the correct remote sensing image scene category is identified through prediction.

[0029] The present invention has the following characteristics:

[0030] 1. This paper introduces a new method for remote sensing image scene classification using a remote sensing image scene classification network with multimodal feature fusion. It integrates advanced technologies such as VanillaNet-5 architecture, rectangular convolution maximum fusion module (RCMF) and area selection module (AS), effectively improving the accuracy of scene classification with rectangular or long line features.

[0031] 2. Aiming at the problem of difficulty in extracting shallow local features, the present invention introduces the rectangular convolution maximum fusion module (RCMF) module to perform parallel calculation with the first 4x4 convolution layer in VanillaNet-5 to generate more comprehensive feature information. This not only solves the problem of extracting features from slender structures, but also ensures that key spatial information is retained.

[0032] 3. Aiming at the problem of difficulty in extracting the most significant features in a local area, the present invention adds a region selection module (AS) after the rectangular convolution maximum fusion module (RCMF), introduces a sequential polarization self-attention mechanism (SPS), and combines it with the mini-region convolution module (MRC) for feature multiplication fusion, which can not only extract important feature information in the region, but also extract spatial feature information, thereby enriching the expression ability of the shallow network. This combined method helps to improve the performance of remote sensing scene classification tasks and improve the ability to accurately distinguish between key features and objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the remote sensing image scene classification algorithm proposed in the present invention.

[0034] Figure 2 This is a structural diagram of the rectangular convolution maximum fusion module (RCMF) used in the present invention.

[0035] Figure 3 Structural diagram of the sequential polarization self-attention mechanism (SPS) used in the present invention.

[0036] Figure 4 This is a structural diagram of the mini regional convolution module (MRC) proposed in the present invention.

[0037] Figure 5 The structural diagram of the area selection module (AS) proposed by the present invention

[0038] Figure 6 This is a structural diagram of the RCSFN model proposed in the present invention.

[0039] Figure 7 This is the confusion matrix diagram of the RCSFN proposed in the present invention obtained on the AID dataset.

[0040] Figure 8 This is the confusion matrix diagram of the RCSFN proposed in the present invention obtained on the NWPU-RESISC45 dataset.

[0041] Fig. 9 A detailed comparison diagram of the RCSFN proposed in the present invention and the baseline network VanillaNet-5. DETAILED DESCRIPTION

[0042] The present invention is further described below with reference to the accompanying drawings and specific implementation examples.

[0043] This paper proposes an efficient remote sensing image scene classification algorithm, combining Figures 1 to 9 The detailed instructions are as follows:

[0044] like Figure 1 The flowchart of the remote sensing image scene classification algorithm integrating VanillaNet-5 network and rectangular convolution proposed by the present invention is shown. In this process, firstly, remote sensing image data is obtained by searching the Internet, and data preprocessing is performed on the data set. Secondly, the VanillaNet-5 model is improved to improve the accuracy of the remote sensing image scene classification task. The improvement method includes introducing a rectangular convolution maximum fusion module (RCMF) to obtain richer local feature information; adding a region selection module (AS) after the rectangular convolution maximum fusion module (RCMF), introducing a sequential polarization self-attention mechanism (SPS), and providing more information input for the deep processing stage; then the sequential polarization self-attention mechanism (SPS) is combined with the mini-region convolution module (MRC) for feature multiplication fusion to enhance local detail features. Then, the image data is input into the improved VanillaNet-5 model with a fixed size of 64*64 for training and verification, and the test set is used for centralized testing. Finally, the remote sensing image to be classified is input into the model weight of the remote sensing scene automatic classification method proposed by the present invention, and the classification result is output by forward calculation.

[0045] like Figure 2 The figure shows the structure of the rectangular convolution maximum fusion module (RCMF) used in the present invention. The rectangular convolution maximum fusion module (RCMF) uses two different rectangular convolution kernels to capture different receptive fields, which are then added and fused and spliced ​​with the original input features to obtain detailed local feature information. This module not only solves the problem of extracting features from slender structures, but also ensures that key spatial information is retained.

[0046] like Figure 3This is a structural diagram of the sequential polarization self-attention mechanism (SPS) used in the present invention. The channel part of the sequential polarization self-attention mechanism (SPS) aims to deeply explore the intrinsic connection between feature channels to give priority to channels that have important contributions to the current task. The spatial self-attention part focuses on the correlation between different positions in the image, allowing the model to focus more on features that are significant at the spatial level. This method aims to improve the representation ability of the model, thereby achieving better performance in remote sensing scene classification tasks. Specifically, the sequential polarization self-attention mechanism (SPS) extracts key feature information in a targeted manner through the synergistic effect of the channel and spatial self-attention dual lines, thereby strengthening the capture of the most significant and relevant information in the image, and providing more informative input for the deep processing stage.

[0047] like Figure 4 This is a structural diagram of the mini region convolution module (MRC) proposed in the present invention. Specifically, the mini region convolution module (MRC) is designed to extract feature information of smaller objects in each region and provide position feature information. It is a component that helps the model focus on extracting key features of small regions and suppressing noise and redundant information.

[0048] like Figure 5 This is a structural diagram of the region selection module (AS) proposed in the present invention. The region selection module (AS) can enhance local detail features and improve spatial position relationships. By adaptively adjusting the weights in the feature map, the network can better capture the key features of the local region. This method of combining the attention mechanism and spatial information can improve the effectiveness of local features, allowing the module to more accurately capture important details of the target area.

[0049] The present invention proposes a RCSFN network based on VanillaNet-5. Figure 6 The structure diagram of the RCSFN model proposed in the present invention is shown in FIG. RCSFN improves classification performance through a rectangular convolution maximum fusion module (RCMF) and an area selection module (AS).

[0050] In order to analyze the contribution of each improvement strategy to improving model performance, ablation experiments are conducted to evaluate the effectiveness of the improvement strategies. For fair comparison, the same dataset and experimental environment are used in all experiments. VanillaNet-5 is used as the baseline network, and the rectangular convolution maximum fusion module (RCMF) and the area selection module (AS) are introduced in turn to optimize and improve VanillaNet-5. The ablation experiment results are shown in Table 1. The overall accuracy achieved by each improvement strategy reached 95.86% and 95.47% respectively, both exceeding VanillaNet-5. Finally, the RCSFN model proposed by combining all improvement methods achieved an overall accuracy of 96.56% and a Kappa coefficient of 96.6, which is 1.86% and 1.8% higher than VanillaNet-5.

[0051] Table 1: Ablation experiments

[0052]

[0053] In order to verify the superiority of the present invention over other advanced classification algorithms, the present invention adds comparative experiments, and the comparison results are shown in Tables 2 and 3. The RCSFN model of the present invention has the highest overall accuracy, reaching an overall accuracy of 96.56% on the AID dataset and an overall accuracy of 92.46% on the NWPU-RESISC45 dataset. Compared with other advanced classification algorithms, the classification performance is better and shows higher performance.

[0054] Table 2: Performance comparison of each model on the AID dataset

[0055]

[0056] Table 3: Performance comparison of each model on the NWPU-RESISC45 dataset

[0057]

[0058] like Figure 7 The figure shows the confusion matrix of the RCSFN proposed in the present invention on the AID dataset. It can be seen from the figure that among the 30 scene categories in the dataset, the overall accuracy (OA) of 26 categories exceeds 94%, and the OA of the remaining 4 categories is not less than 85%. These experimental results show that the RCSFN method proposed in the present invention has achieved excellent performance in scene classification.

[0059] like Figure 8The figure shows the confusion matrix diagram of the RCSFN proposed in the present invention obtained on the NWPU-RESISC45 dataset. It can be seen from the figure that in the NWPU-RESISC45 dataset, the classification accuracy of 36 scenes exceeds 90%, and the classification accuracy of 8 scenes exceeds 83%.

[0060] like Fig. 9 The figure shows the comparison details of the RCSFN proposed in the present invention and the baseline network VanillaNet-5. It can be seen from the figure that the RCSFN model proposed in the present invention effectively improves the classification accuracy, among which the overall accuracy of the bridge, farmland, industrial area, school and square categories is the most significantly improved, which is 5%, 11%, 7%, 14% and 6% respectively.

Claims

1. A remote sensing image scene classification method integrating VanillaNet-5 network and rectangular convolution, characterized in that: The VanillaNet-5 model was improved by using image preprocessing, matrix convolution maximum fusion, sequential polarization self-attention mechanism and region selection method. The preprocessed remote sensing image data was sent to the improved VanillaNet-5 model for training and verification, and the test set was used for centralized testing. The final performance of the model was evaluated by 7 evaluation indicators. The remote sensing image to be classified was input into the optimal model obtained by training, and the classification result was output by forward calculation.

2. The remote sensing image preprocessing method according to claim 1, characterized in that: Normalize the size of the input image I to the network and scale the image size to 64*64 pixels; The image augmentation method is to perform flip transformation, random rotation, random cropping and other operations on the divided image to amplify it, and obtain image X after amplification; the data set is randomly divided into a training set and a test set in a ratio of 5:

5.

3. The matrix convolution maximum fusion module according to claim 1, characterized in that: The method of adding and fusion of rectangular convolution kernel convolution layers and splicing of original input features is used to extract unique shallow local feature information; the sizes of the rectangular convolution kernels are 4×2 and 2×4, respectively, with a step size of 2, to supplement the coherence of spatial feature information, marked as Secondly, a maximum pooling layer of size 2 is used to reduce the dimension of the feature map. After the rectangular convolution kernel extracts the unique feature information, the output feature maps of the two rectangular convolution layers are added and fused, and then spliced ​​with the feature map extracted by the 4×4 convolution in VanillaNet-5 to generate more comprehensive feature information. The specific calculation formula is as follows: where Y R is the output feature map, Z is the feature map obtained by the Head layer in VanillaNet-5, * represents the convolution operation, Maxpool(·) represents the maximum pooling operation, and cat(·) represents the concatenation operation.

4. The sequential polarization self-attention mechanism of claim 1, wherein: Through the synergy of channel and spatial self-attention branches, key feature information is selectively extracted; using this dual-line self-attention mechanism, key features in the image can be finely captured and extracted, providing more information input for the deep processing stage. The specific calculation process is as follows: where Y ch ,Y sp They represent the output channel attention feature map and the output spatial attention feature map respectively, ⊙ represents the convolution multiplication operation, represents feature map multiplication, represents feature map addition, * represents convolution operation, Gap(·) represents global average pooling layer, reshape(·) represents fully connected layer, denote the convolutional layers for channel and spatial attention, respectively. Sigmoid(·) denotes the activation function, which maps the variable to between 0 and 1. SPS is the output feature map.

5. The area selection module according to claim 1, characterized in that: Help the model to suppress noise and redundant information while focusing on extracting key features in a small area; propose a mini-area convolution module with a convolution kernel size of 2 and a stride of 2, expressed as The specific calculation formula is as follows: where Y MRC is the output feature map, * represents the convolution operation, Sigmoid (·) represents the activation function, and the variable is mapped to between 0 and 1; the region selection module combines the mini region convolution module with the sequential polarization self-attention mechanism module in claim 4, which can not only extract the key feature information in the region, but also capture the spatial feature information, enriching the expression ability of the shallow network. The specific calculation formula is as follows: Y AS =Maxpool(Y SPS ,C,2,2)⊙Y MRC Where Maxpool(·) represents the maximum pooling operation, C represents the channel dimension size of the pooling window, 2 represents the window size is 2 in both height and width, and Y AS is the output feature map.

6. The training, verification and testing method of claim 1 integrating a VanillaNet-5 network and rectangular convolution, characterized in that: The preprocessed remote sensing image dataset was input into the improved network for training for 300 iterations; After each iteration, the test set is used to verify the performance of the model generated in each iteration. After 300 training and testing times, the optimal model is obtained. The test set is used to test the optimal model obtained by the final training. The performance of the improved model is evaluated by the overall accuracy, Kappa coefficient, accuracy, precision, recall rate, F1 score and confusion matrix of the model to verify the effectiveness and advancement of the model.

7. The method for model classification of remote sensing image scenes according to claim 1, characterized in that: The remote sensing image to be classified is adaptively scaled to a resolution of 64*64 pixels and loaded into the model of the present invention; the image is then input into the model with weights, and each layer module of the model uses these weights to extract features from the image and identify key features in the image; secondly, the features are finally calculated through the fully connected layer to convert the features into category probabilities; finally, the model selects the category with the highest probability as the prediction result, that is, it is considered that the input remote sensing image scene belongs to this category.

Images