Remote sensing image classification method and device based on local and global feature fusion

Abstract

The present application relates to a remote sensing image classification method and device based on the fusion of local and global features in the field of image recognition technology. The method comprises: annotating the acquired remote sensing images as training samples; constructing a remote sensing image classification model, which includes an input network, a feature extraction network, and a classification network; wherein the input network is used to perform multi-branch and multi-scale feature extraction on the training samples using dilated convolution, and the feature extraction network is used to sequentially perform global and local feature extraction and fusion using four stages consisting of one downsampling layer and several stacked RMFE modules, wherein the RMFE module simultaneously extracts local and global features using a pure convolution method; the remote sensing image classification model is trained using the training samples, and the trained remote sensing image classification model is used to classify the remote sensing images to be tested, thereby obtaining remote sensing image classification results. This method improves the accuracy of image classification while reducing the number of model parameters.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a remote sensing image classification method and device based on the fusion of local and global features. Background Art

[0002] With the advancement of science and technology, accurate and rapid acquisition of remote sensing image information is crucial for remote sensing technology to be used in fields such as geographic data acquisition, geological surveys, and environmental monitoring. Remote sensing image classification, a crucial component of remote sensing image processing, involves using computers to classify individual pixels according to specific rules or algorithms, thereby obtaining information that corresponds to the actual ground objects in the remote sensing image. This presents challenges, as detailed features of objects within the same category vary significantly, while image features of objects of different categories often exhibit similarities. Improving the effectiveness and accuracy of remote sensing image classification has long been a key research direction, aiming to better serve strategic sectors such as surveying and mapping, navigation, military affairs, and ecology.

[0003] Different remote sensing image classification methods have different conceptual characteristics, and their results can vary somewhat. In recent years, the visual Transformer model based on the self-attention module has shown excellent performance in many scenarios due to its effective global information, which makes the model more expressive. However, it also suffers from problems such as low computational efficiency and high overhead. CNNs have limited local receptive fields. Some network structures such as VAN and RepLKNet choose to use large convolution kernel designs to compensate for this problem, achieving performance similar to or even stronger than mainstream Vision Transformers. Their parameter sharing mechanism also reduces model complexity and improves model training efficiency. HorNet proposed that the key factors behind the success of the visual Transformer model are input adaptation, long-range and high-order spatial interactions, which can also be effectively achieved through convolution-based methods. Summary of the Invention

[0004] Based on this, it is necessary to provide a remote sensing image classification method and device based on the fusion of local and global features to address the above technical problems.

[0005] A remote sensing image classification method based on local and global feature fusion, the method comprising:

[0006] Remote sensing images are acquired and annotated to obtain training samples.

[0007] A remote sensing image classification model based on the fusion of local and global features is constructed, and the remote sensing image classification model includes: an input network, a feature extraction network and a classification network; the input network is used to use multiple dilated convolutions with different dilation rates to extract features of the training samples and then splice them to obtain a spliced ​​feature map; the feature extraction network is used to use 4 stages consisting of a downsampling layer and a stack of several RMFE modules to extract and fuse the local and global features of the spliced ​​feature map to obtain a fused feature map; the RMFE module is constructed based on the MetaFormer paradigm using a P2DF module and a feedforward layer; the P2DF module is used to perform channel splitting on the input feature map, and then use convolution and channel attention modules to extract the global features of the second channel splitting result, use convolution and spatial attention modules to extract the local features of the first channel splitting result, and fuse the global features and the local features; the classification network is used to perform classification according to the fused feature map to obtain a remote sensing image classification prediction result.

[0008] Training the remote sensing image classification model using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model;

[0009] The remote sensing image to be tested is input into the trained remote sensing image classification model to obtain the remote sensing image classification result.

[0010] In one embodiment, the input network includes parallel branches consisting of four dilated convolutional layers with different dilation rates.

[0011] The remote sensing image classification model is trained using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model, including:

[0012] The training samples are respectively input into the four dilated convolution modules in the input network to obtain four dilated convolution feature maps with different receptive fields, and the four dilated convolution feature maps are spliced ​​to obtain a spliced ​​feature map.

[0013] The concatenated feature map is input into the first stage of the feature extraction network to obtain a first-layer feature map.

[0014] The first feature map is input into the second stage of the feature extraction network to obtain a second-layer feature map.

[0015] The second feature map is input into the third stage of the feature extraction network to obtain a third-layer feature map.

[0016] The third feature map is input into the fourth stage of the feature extraction network to obtain a fused feature map.

[0017] The fused feature map is input into the classification network to obtain the remote sensing image classification prediction result.

[0018] The remote sensing image classification model is reversely trained using the annotations of the training samples and the remote sensing image classification prediction results to obtain a trained remote sensing image classification model.

[0019] In one embodiment, the dilation rates of the four dilated convolutional layers with different dilation rates of the input network are 1, 2, 3, and 4, respectively, and all use a 3×3 convolution kernel.

[0020] In one embodiment, the first stage of the feature extraction network includes a downsampling layer and an RMFE module; the RMFE module includes two normalization modules, a P2DF module, and a feedforward module.

[0021] The concatenated feature map is input into the first stage of the feature extraction network to obtain the first layer feature map, including:

[0022] The concatenated feature map is input into the downsampling layer of the first stage to obtain a downsampled feature map.

[0023] The downsampled feature map is input into the first normalization module of the RMFE module of the first stage to obtain a first normalized result.

[0024] The first normalization result is input into the P2DF module of the RMFE module of the first stage to obtain a fused feature map.

[0025] The fused feature map and the downsampled feature map are added to obtain a first fused feature.

[0026] The first fusion feature is input into the second normalization module of the RMFE module of the first stage to obtain a second normalized result.

[0027] The second normalized result is input into the feedforward module of the RMFE module of the first stage to obtain an enhanced feature map.

[0028] The enhanced feature map is added to the first fusion feature to obtain a first layer feature map.

[0029] In one embodiment, the P2DF module includes: a global feature extraction branch, a local feature extraction branch and a feature fusion module; wherein the global feature extraction branch includes a point convolution layer, a depth convolution layer and a channel attention module; the local feature extraction branch includes: 2 stacked depth convolution layers and 1 point convolution layer, a spatial attention module; the feature fusion module includes: a convolution module consisting of a 3×3 depth convolution layer and a skip connection, and two point convolution layers.

[0030] The first normalization result is input into the P2DF module of the RMFE module of the first stage to obtain the fused feature map, including:

[0031] The normalized result is input into the P2DF module of the RMFE module of the first stage for channel splitting to obtain a first channel splitting result and a second channel splitting result.

[0032] The first channel splitting result is input into the local feature extraction branch of the P2DF module of the RMFE module of the first stage, and processed by two stacked depth convolution layers and one point convolution layer. The obtained convolution feature is multiplied by the first channel splitting result to obtain the first channel convolution feature map.

[0033] The first channel convolution feature map is input into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a local feature map.

[0034] The second channel splitting result is input into the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and is processed by one point convolution layer and then split into the number of channels according to a preset ratio to obtain a first group of channel feature maps and a second group of channel feature maps; and the first channel feature map is processed by a 7×7 depth convolution layer and then split according to the second preset ratio to obtain a third group of channel feature maps, a fourth group of channel feature maps and a fifth group of channel feature maps. The result of multiplying the second group of channel feature maps by the third group of channel feature maps is multiplied by the fourth group of channel feature maps after dimensionality increase by point convolution, and the multiplication result is multiplied by the fifth group of channel feature maps after dimensionality increase by point convolution, and then the multiplication result is processed by point convolution to obtain a multidimensional feature map.

[0035] The multidimensional feature map is input into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map.

[0036] The local feature map and the global feature map are input into the feature fusion module of the P2DF module of the RMFE module of the first stage, and are spliced ​​in the channel dimension using a channel splicing method to obtain a global and local spliced ​​feature map; the global and local spliced ​​feature map is added to the global and local spliced ​​feature map after passing through a deep convolution layer and a GeLU activation function, and the addition result is normalized. After two point convolution layers for dimensionality increase and then dimensionality reduction, a fused feature map is obtained.

[0037] In one embodiment, the channel attention module includes a left branch, a right branch and a Sigmoid function.

[0038] The multidimensional feature map is input into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map, including:

[0039] The multidimensional feature map is input into the left branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and the left branch feature map is obtained after maximum pooling, point convolution, ReLu activation, and point convolution processing.

[0040] The multidimensional feature map is input into the right branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and the right branch feature map is obtained after average pooling, point convolution, ReLu activation, and point convolution processing.

[0041] The left branch feature map and the right branch feature map are added together and activated using a Sigmoid function, and the activation result is multiplied by the multidimensional feature map to obtain a global feature map.

[0042] In one embodiment, the spatial attention module includes: a pooling module, a convolutional layer, and a Sigmoid activation function.

[0043] The first channel convolution feature map is input into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a local feature map, including:

[0044] The first channel convolution feature map is input into the pooling module of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage. After maximum pooling and average pooling respectively, the maximum pooling results and the average pooling results are channel-spliced ​​to obtain the pooling result.

[0045] The pooling result is input into the convolution layer of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a spatial convolution feature map.

[0046] The spatial convolution feature map is activated by a Sigmoid function and then multiplied by the first channel convolution feature map to obtain a local feature map.

[0047] In one embodiment, the number of RMFE modules included in the four stages of the feature extraction network is classified as: 1, 1, 2, and 1.

[0048] In one embodiment, the convolution kernel size of the downsampling layer in the first stage of the feature extraction network is 7×7; the convolution kernel size of the downsampling layer in the second, third and fourth stages is 3×3.

[0049] A remote sensing image classification device based on local and global feature fusion, the device comprising:

[0050] The training sample acquisition module is used to acquire remote sensing images and annotate the remote sensing images to obtain training samples.

[0051] A remote sensing image classification model construction module is used to construct a remote sensing image classification model based on the fusion of local and global features. The remote sensing image classification model includes: an input network, a feature extraction network and a classification network; the input network is used to use multiple dilated convolutions with different dilation rates to extract features from the training samples and then splice them to obtain a spliced ​​feature map; the feature extraction network is used to use four stages consisting of a downsampling layer and a stack of several RMFE modules to extract and fuse the local and global features of the spliced ​​feature map to obtain a fused feature map; the RMFE module is constructed based on the MetaFormer paradigm using a P2DF module and a feedforward layer; the P2DF module is used to perform channel splitting on the input feature map, and then use convolution and channel attention modules to extract the global features of the second channel splitting result, use convolution and spatial attention modules to extract the local features of the first channel splitting result, and fuse the global features and the local features; the classification network is used to perform classification according to the fused feature map to obtain a remote sensing image classification prediction result.

[0052] A remote sensing image classification model training module is used to train the remote sensing image classification model using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model;

[0053] The remote sensing image classification module is used to input the remote sensing image to be tested into the trained remote sensing image classification model to obtain the remote sensing image classification result.

[0054] The above-mentioned remote sensing image classification method and device based on local and global feature fusion include: annotating acquired remote sensing images as training samples; constructing a remote sensing image classification model, which includes an input network, a feature extraction network, and a classification network; wherein the input network is used to perform multi-branch and multi-scale feature extraction on the training samples using dilated convolution; the feature extraction network is used to sequentially extract and fuse global and local features using four stages consisting of a downsampling layer and several stacked RMFE modules, wherein the RMFE modules simultaneously extract local and global features using a pure convolutional approach; the remote sensing image classification model is trained using the training samples, and the trained remote sensing image classification model is used to classify the remote sensing images to be tested, thereby obtaining remote sensing image classification results. This method reduces the number of model parameters while improving image classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 1 is a flow chart of a remote sensing image classification method based on local and global feature fusion in one embodiment;

[0056] Figure 2 Schematic diagram of the structure of the feedforward layer in another embodiment;

[0057] Figure 3 A schematic diagram of the structure of a remote sensing image classification model in another embodiment;

[0058] Figure 4 A schematic diagram of the structure of an input network in another embodiment;

[0059] Figure 5 This is a schematic structural diagram of an RMFE module in another embodiment;

[0060] Figure 6 A schematic diagram of the structure of a P2DF module in another embodiment;

[0061] Figure 7 Schematic diagram of the structure of the channel attention module and the spatial attention module in the P2DF module in another embodiment, where (a) is the channel attention module and (b) is the spatial attention module;

[0062] Figure 8 is a confusion matrix in another embodiment;

[0063] Figure 9 This is a comparison result of classification accuracy of several network models in another embodiment;

[0064] Figure 101 is a structural block diagram of a remote sensing image classification device based on local and global feature fusion in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0066] Residual Multi-scale Feature Extraction Block, referred to as RMFE module.

[0067] Parallel Two-Doman Fusion Block, referred to as P2DF module.

[0068] Remote sensing image classification model based on local and global feature fusion, referred to as: RMFENet model.

[0069] In one embodiment, Figure 1 As shown, a remote sensing image classification method based on the fusion of local and global features is provided, which includes the following steps:

[0070] Step 100: Acquire remote sensing images and label them to obtain training samples.

[0071] Specifically, the remote sensing images may include, but are not limited to, grassland remote sensing images, field remote sensing images, industrial area remote sensing images, river and lake remote sensing images, forest remote sensing images, residential area remote sensing images, and parking lot remote sensing images.

[0072] Step 102: Construct a remote sensing image classification model based on the fusion of local and global features. The remote sensing image classification model includes: an input network, a feature extraction network, and a classification network.

[0073] The input network is used to extract features of the training samples using multiple dilated convolutions with different dilation rates and then splice them to obtain a spliced ​​feature map.

[0074] The feature extraction network is used to extract and fuse the local and global features of the spliced ​​feature map using 4 stages consisting of a downsampling layer and several stacked RMFE modules to obtain a fused feature map.

[0075] The RMFE module is constructed based on the MetaFormer paradigm using the P2DF module and the feedforward layer. The P2DF module is used to split the input feature map into channels, and then uses the convolution and channel attention modules to extract the global features of the second channel split result. The convolution and spatial attention modules are used to extract the local features of the first channel split result, and the global features and local features are fused. The feedforward layer network includes two point convolutions, and the number of channels is first increased and then reduced. The structure of the feedforward layer is as follows: Figure 2 shown.

[0076] The classification network is used to perform classification based on the fused feature map to obtain the remote sensing image classification prediction results.

[0077] Specifically, the remote sensing image classification model first uses the input network to perform multi-branch and multi-scale feature extraction on the input training samples, and then uses 4 stages consisting of 1 downsampling layer and several stacked RMFE modules to extract and fuse global features and local features in turn. Except for the first stage, which uses a deep convolution layer with a 7×7 convolution kernel to downsample the image four times, the other three stages all use a convolution layer with a 3×3 convolution kernel to downsample by 2 times, and increase the number of channels by 64 to ensure sufficient information. In the four stages, the stacked RMFE modules are used to extract local and global information. The P2DF module in the RMFE module performs channel splitting at the input, so that the number of channels of each branch is half of the module input, reducing the number of parameters while maintaining a certain classification accuracy. The structure of the remote sensing image classification model is as follows: Figure 3 shown. Figure 3 In it, Dwconv is the depth convolution layer, Dilated conv is the hole convolution layer, and cat is the channel dimension splicing operation.

[0078] The classification network includes one fully connected layer.

[0079] The channel attention module processes the feature map input to the channel attention module through a max pooling layer and an average pooling layer, compressing the spatial dimensions and reducing the feature map from C×H×W to C×1×1. It then performs two point convolutions to reduce the number of channels and then expand them back to the original number of channels. A ReLU activation function is used to add nonlinearity in between. The two outputs are element-wise added and the output is obtained through a sigmoid activation function. This output is multiplied with the initial input feature map to learn the importance of different channels.

[0080] The spatial attention module uses average pooling and max pooling to compress the channel dimensions of the input feature map, reducing the feature map from C×H×W to 1×H×W. It then concatenates the two feature maps, fuses them through a convolutional layer, and processes them with a sigmoid activation function to produce the output. The output is multiplied by the initial input feature map to focus on the target's location information.

[0081] Step 104: using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model, the remote sensing image classification model is trained to obtain a trained remote sensing image classification model;

[0082] Step 106: Input the remote sensing image to be tested into the trained remote sensing image classification model to obtain the remote sensing image classification result.

[0083] The above-mentioned remote sensing image classification method based on the fusion of local and global features includes: annotating acquired remote sensing images as training samples; constructing a remote sensing image classification model comprising an input network, a feature extraction network, and a classification network; wherein the input network is configured to perform multi-branch and multi-scale feature extraction on the training samples using dilated convolution; and the feature extraction network is configured to sequentially extract and fuse global and local features using four stages consisting of a downsampling layer and several stacked RMFE modules, wherein the RMFE modules simultaneously extract local and global features using a pure convolutional approach; and training the remote sensing image classification model using the training samples. The trained remote sensing image classification model is then used to classify the test remote sensing images to obtain remote sensing image classification results. This method reduces the number of model parameters while improving image classification accuracy.

[0084] In one embodiment, the structure of the input network is as follows Figure 4 As shown, the input network includes parallel branches consisting of four dilated convolutional layers with different dilation rates; step 104 includes the following specific steps:

[0085] Step 200: Input the training samples into the four dilated convolution modules in the input network respectively to obtain four dilated convolution feature maps with different receptive fields, and splice the four dilated convolution feature maps to obtain a spliced ​​feature map.

[0086] Specifically, the input network is a multi-branch parallel dilated convolution module consisting of four branches, all using a 3×3 convolution kernel. The dilation rates of each branch are 1, 2, 3, and 4, respectively, and the number of channels in each branch is 16. Convolutions with different dilation rates are used, that is, at the beginning, the receptive field is enlarged and multi-scale feature information is obtained, which is then spliced ​​together as the input for the next stage.

[0087] Step 202: Input the concatenated feature map into the first stage of the feature extraction network to obtain the first layer feature map.

[0088] Step 204: Input the first feature map into the second stage of the feature extraction network to obtain a second-layer feature map.

[0089] Step 206: Input the second feature map into the third stage of the feature extraction network to obtain a third-layer feature map.

[0090] Step 208: Input the third feature map into the fourth stage of the feature extraction network to obtain a fused feature map.

[0091] Step 210: Input the fused feature map into the classification network to obtain the remote sensing image classification prediction result.

[0092] Step 212: Using the annotations of the training samples and the remote sensing image classification prediction results, reverse training is performed on the remote sensing image classification model to obtain a trained remote sensing image classification model.

[0093] In one embodiment, the dilation rates of the four dilated convolutional layers with different dilation rates of the input network are 1, 2, 3, and 4, respectively, and all use a 3×3 convolution kernel.

[0094] In one embodiment, the first stage of the feature extraction network includes a downsampling layer and an RMFE module; Figure 5 As shown, the RMFE module includes two normalization modules, one P2DF module, and one feedforward module; step 202 includes:

[0095] Step 300: Input the concatenated feature map into the downsampling layer of the first stage to obtain a downsampled feature map.

[0096] Specifically, each stage downsamples the image by dividing it into blocks. The first stage uses a 7×7 convolution kernel with a stride of 4 and a padding of 2, directly downsampling the image by a factor of four. The remaining stages use a 3×3 convolution kernel with a stride of 2 and a padding of 1.

[0097] Step 302: Input the downsampled feature map into the first normalization module of the RMFE module of the first stage to obtain a first normalized result.

[0098] Step 304: Input the first normalization result into the P2DF module of the RMFE module of the first stage to obtain a fused feature map.

[0099] Step 306: Add the fused feature map and the downsampled feature map to obtain a first fused feature.

[0100] Step 308: Input the first fusion feature into the second normalization module of the RMFE module of the first stage to obtain a second normalized result.

[0101] Step 310: Input the second normalized result into the feedforward module of the RMFE module of the first stage to obtain an enhanced feature map.

[0102] Step 312: Add the enhanced feature map to the first fusion feature to obtain a first-layer feature map.

[0103] In one embodiment, the structure of the P2DF module is as follows: Figure 6 As shown, Figure 6 The split in the above sentence is a channel splitting operation. The P2DF module includes: a global feature extraction branch, a local feature extraction branch, and a feature fusion module; wherein the global feature extraction branch includes a point convolution layer, a depth convolution layer, and a channel attention module; the local feature extraction branch includes: two stacked depth convolution layers and one point convolution layer, and a spatial attention module; the feature fusion module includes: a convolution module consisting of a 3×3 depth convolution layer and a skip connection, and two point convolution layers; step 304 includes:

[0104] Step 400: Input the normalized result into the P2DF module of the RMFE module of the first stage for channel splitting to obtain a first channel splitting result and a second channel splitting result.

[0105] Step 402: The first channel splitting result is input into the local feature extraction branch of the P2DF module of the RMFE module of the first stage, and processed through two stacked depth convolution layers and one point convolution layer, and the obtained convolution feature is multiplied by the first channel splitting result to obtain the first channel convolution feature map.

[0106] Specifically, the local feature extraction branch (left branch) uses a stacked convolution approach, consisting of a 5×5 depthwise convolutional layer, a 3×3 depthwise convolutional layer, and a 1×1 convolutional layer. Drawing on the idea of ​​large convolution kernels in VANs, it considers local contextual information and a larger receptive field, achieving adaptability in the channel dimension. Furthermore, through the spatial attention module, it enhances local details and suppresses irrelevant areas.

[0107] Step 404: Input the first channel convolution feature map into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a local feature map.

[0108] Step 406: Input the second channel splitting result into the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and after processing through one point convolution layer, split the number of channels according to a preset ratio to obtain a first group of channel feature maps and a second group of channel feature maps; and process the first channel feature map through a 7×7 depth convolution layer and split it according to a second preset ratio to obtain a third group of channel feature maps, a fourth group of channel feature maps and a fifth group of channel feature maps, multiply the result of the second group of channel feature maps by the third group of channel feature maps by performing point convolution to increase the dimension and multiply the result with the fourth group of channel feature maps, and multiply the result of the multiplication by performing point convolution to increase the dimension and multiply the result with the fifth group of channel feature maps, and then process the multiplication result by point convolution to obtain a multidimensional feature map.

[0109] Specifically, the P2DF (Parallel Two-Doman Fusion) Block consists of two branches. Visual Transformers have achieved significant success in various tasks driven by a novel spatial modeling mechanism based on dot-product self-attention for acquiring global information. Key elements behind this success are input adaptation, long-range, and high-order spatial interactions. This can be achieved within a convolutional framework.

[0110] In the P2DF Block, the global feature extraction branch (right branch) draws on the global gated convolution proposed by HorNet. First, the feature map H×W×C is transformed into twice the original number of channels 2C through a 1×1 convolution. Then, the number of channels is split according to the ratio [1:7]. The 2C-(C / 4) feature map is passed through a 7×7 dilated convolution layer and split again into three groups of channels [C / 4, C / 2, C] according to the ratio [4:2:1]. At this time, a total of four groups of feature maps [C / 4, C / 4, C / 2, C] are obtained. The first group of feature maps is multiplied with the second group of feature maps, and the dimension is increased through 1×1 convolution. It is multiplied with the third group of feature maps, and the dimension is increased again. Finally, it is multiplied with the fourth group of feature maps. In this way, the multi-dimensional spatial information interacts and captures the global dependency and adaptability well. The obtained feature maps are passed through the channel attention module to learn the importance of different channels and enhance the representation ability of global features.

[0111] Step 408: Input the multidimensional feature map into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map.

[0112] Step 410: Input the local feature map and the global feature map into the feature fusion module of the P2DF module of the RMFE module of the first stage, and obtain a global and local spliced ​​feature map after splicing in the channel dimension using a channel splicing method; add the global and local spliced ​​feature map to the global and local spliced ​​feature map after passing the global and local spliced ​​feature map through a deep convolution layer and a GeLU activation function, and normalize the addition result. After the dimensionality is increased and then reduced by two point convolution layers, a fused feature map is obtained.

[0113] In one embodiment, Figure 7 As shown in (a), the channel attention module includes a left branch, a right branch and a Sigmoid function; step 408 includes: inputting the multidimensional feature map into the left branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and obtaining the left branch feature map after sequentially performing maximum pooling, point convolution, ReLu activation, and point convolution processing; inputting the multidimensional feature map into the right branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and obtaining the right branch feature map after sequentially performing average pooling, point convolution, ReLu activation, and point convolution processing; adding the left branch feature map and the right branch feature map and then activating them with the Sigmoid function, and multiplying the obtained activation result with the multidimensional feature map to obtain the global feature map.

[0114] In one embodiment, Figure 7 As shown in (b), the spatial attention module includes: a pooling module, a convolution layer and a Sigmoid activation function; step 404 includes: inputting the first channel convolution feature map into the pooling module of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage, and after maximum pooling and average pooling respectively, performing channel splicing on the obtained maximum pooling results and average pooling results to obtain a pooling result; inputting the pooling result into the convolution layer of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage (the convolution kernel size of the preferred convolution layer is 7×7) to obtain a channel convolution feature map; activating the channel convolution feature map with the Sigmoid function and multiplying it with the first channel convolution feature map to obtain a local feature map.

[0115] In one embodiment, the four stages of the feature extraction network include RMFE modules in the following categories: 1, 1, 2, and 1.

[0116] In one embodiment, the downsampling layer in the first stage of the feature extraction network is a depthwise convolutional layer with a convolution kernel size of 7×7; the downsampling layers in the second, third, and fourth stages are convolutional layers with a convolution kernel size of 3×3.

[0117] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0118] In a verification embodiment, specific parameters of the architecture of the remote sensing image classification model based on the fusion of local and global features are set to the parameters shown in Table 1.

[0119] Table 1 Specific parameters of the network architecture

[0120]

[0121] The experimental environment configuration used in this embodiment is shown in Table 2.

[0122] Table 2 Experimental environment configuration

[0123]

[0124] In this embodiment, the dataset uses the remote sensing image dataset RSSCN7 released by Wuhan University, which contains 7 different remote sensing image scenes, namely Grass, Field, Industry, River Lake, Forest, Residential Area, and Parking, totaling 2800 RGB images. The dataset is randomly divided into two categories: training set and test set, one for training the network and the other for testing the performance of the network model, with an image ratio of 8:2. The confusion matrix is ​​as follows: Figure 8 shown.

[0125] To demonstrate the effectiveness of the RMFENet network model, experiments were conducted on the RSSCN7 dataset, a commonly used dataset for remote sensing image scene classification. Comparative experiments were conducted using ResNet, ConvNext, HorNet, VAN, and PoolFormer network models. Table 3 shows a comparison of parameter count, computational complexity, peak accuracy, and the average of the five highest accuracy rates. While the model constructed by the present invention has slightly higher parameter and computational complexity than the VAN network, its peak accuracy of 96.79% is 2 percentage points higher than that of the VAN. Compared to other classic networks, the model constructed by the present invention not only has lower computational complexity and parameter count, but also achieves higher accuracy.

[0126] Table 3 Comparison of accuracy with other models

[0127]

[0128] The RMFENet network model constructed by the present invention uses a pure convolutional approach to simultaneously obtain local and global information in the network module, effectively combining the advantages of Transformer and CNN. Experiments have shown that the model constructed by the present invention achieves better performance with less computation and parameters. The accuracy comparison results are shown in Figure 2. Figure 9 shown.

[0129] In one embodiment, Figure 10 As shown, a remote sensing image classification device based on local and global feature fusion is provided, including: a training sample acquisition module, a remote sensing image classification model construction module, a remote sensing image classification model training module and a remote sensing image classification module, wherein:

[0130] The training sample acquisition module is used to acquire remote sensing images and annotate the remote sensing images to obtain training samples.

[0131] The remote sensing image classification model construction module is used to construct a remote sensing image classification model based on the fusion of local and global features. The remote sensing image classification model includes: an input network, a feature extraction network, and a classification network; the input network is used to use multiple dilated convolutions with different dilation rates to extract features of the training samples and then splice them to obtain a spliced ​​feature map; the feature extraction network is used to extract and fuse the local and global features of the spliced ​​feature map using 4 stages consisting of a downsampling layer and a stack of several RMFE modules to obtain a fused feature map; the RMFE module is constructed based on the MetaFormer paradigm using the P2DF module and the feedforward layer; the P2DF module is used to split the input feature map into channels, and then use the convolution and channel attention modules to extract the global features of the second channel splitting result, and use the convolution and spatial attention modules to extract the local features of the first channel splitting result, and fuse the local features with the global features; the classification network is used to perform classification based on the fused feature map to obtain the remote sensing image classification prediction result.

[0132] A remote sensing image classification model training module is used to train the remote sensing image classification model using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model;

[0133] The remote sensing image classification module is used to input the remote sensing image to be tested into the trained remote sensing image classification model to obtain the remote sensing image classification result.

[0134] In one embodiment, the input network includes parallel branches consisting of four dilated convolution layers with different dilation rates; the remote sensing image classification model training module is further used to input the training samples into the four dilated convolution modules in the input network respectively to obtain four dilated convolution feature maps with different receptive fields, and splice the four dilated convolution feature maps to obtain a spliced ​​feature map; the spliced ​​feature map is input into the first stage of the feature extraction network to obtain a first-layer feature map; the first feature map is input into the second stage of the feature extraction network to obtain a second-layer feature map; the second feature map is input into the third stage of the feature extraction network to obtain a third-layer feature map; the third feature map is input into the fourth stage of the feature extraction network to obtain a fused feature map; the fused feature map is input into the classification network to obtain a remote sensing image classification prediction result; the remote sensing image classification model is reversely trained using the annotations of the training samples and the remote sensing image classification prediction results to obtain a trained remote sensing image classification model.

[0135] In one embodiment, the expansion rates of the four dilated convolutional layers with different dilation rates of the input network in the remote sensing image classification model construction module are 1, 2, 3, and 4, respectively, and all use a 3×3 convolution kernel.

[0136] In one embodiment, the first stage of the feature extraction network includes a downsampling layer and an RMFE module; the RMFE module includes two normalization modules, a P2DF module, and a feedforward module; the remote sensing image classification model training module is further used to input the spliced ​​feature map into the downsampling layer of the first stage to obtain a downsampling feature map; input the downsampling feature map into the first normalization module of the RMFE module of the first stage to obtain a first normalization result; input the first normalization result into the P2DF module of the RMFE module of the first stage to obtain a fused feature map; add the fused feature map and the downsampling feature map to obtain a first fused feature; input the first fused feature into the second normalization module of the RMFE module of the first stage to obtain a second normalization result; input the second normalization result into the feedforward module of the RMFE module of the first stage to obtain an enhanced feature map; and add the enhanced feature map and the first fused feature to obtain a first layer feature map.

[0137] In one embodiment, the P2DF module includes: a global feature extraction branch, a local feature extraction branch and a feature fusion module; wherein the global feature extraction branch includes a point convolution layer, a depth convolution layer and a channel attention module; the local feature extraction branch includes: 2 stacked depth convolution layers and 1 point convolution layer, a spatial attention module; the feature fusion module includes: a convolution module consisting of a 3×3 depth convolution layer and a jump connection and two point convolution layers; the remote sensing image classification model training module is also used to input the normalized result into the P2DF module of the RMFE module of the first stage for channel splitting to obtain the first The first channel splitting result and the second channel splitting result are input into the local feature extraction branch of the P2DF module of the RMFE module of the first stage, processed by two stacked depth convolution layers and one point convolution layer, and the convolution feature obtained is multiplied by the first channel splitting result to obtain the first channel convolution feature map; the first channel convolution feature map is input into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain the local feature map; the second channel splitting result is input into the global feature extraction branch of the P2DF module of the RMFE module of the first stage In the extraction branch, after being processed by a point convolution layer, the number of channels is split according to a preset ratio to obtain the first group of channel feature maps and the second group of channel feature maps; and the first channel feature map is processed by a 7×7 depth convolution layer and then split according to the second preset ratio to obtain the third group of channel feature maps, the fourth group of channel feature maps and the fifth group of channel feature maps. The result of multiplying the second group of channel feature maps and the third group of channel feature maps is multiplied by the fourth group of channel feature maps after dimension increase by point convolution, and the multiplication result is multiplied by the fifth group of channel feature maps after dimension increase by point convolution, and then the multiplication result is processed by point convolution to obtain a multidimensional feature map; the multidimensional feature The local feature map is input into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map; the local feature map and the global feature map are input into the feature fusion module of the P2DF module of the RMFE module of the first stage, and are spliced ​​in the channel dimension using channel splicing to obtain a global and local spliced ​​feature map; the global and local spliced ​​feature maps are added to the global and local spliced ​​feature maps after passing through a deep convolutional layer and a GeLU activation function, and the addition result is normalized. After two point convolutional layers for dimensionality increase and then dimensionality reduction, the fused feature map is obtained.

[0138] In one embodiment, the channel attention module includes a left branch, a right branch and a Sigmoid function; the remote sensing image classification model training module is also used to input the multidimensional feature map into the left branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and obtain the left branch feature map after processing through maximum pooling, point convolution, ReLu activation, and point convolution; the multidimensional feature map is input into the right branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and obtain the right branch feature map after processing through average pooling, point convolution, ReLu activation, and point convolution; the left branch feature map and the right branch feature map are added and activated with the Sigmoid function, and the activation result is multiplied by the multidimensional feature map to obtain the global feature map.

[0139] In one embodiment, the spatial attention module includes: a pooling module, a convolution layer and a Sigmoid activation function; the remote sensing image classification model training module is also used to input the first channel convolution feature map into the pooling module of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage, and after maximum pooling and average pooling respectively, the maximum pooling results and the average pooling results are channel-splicing to obtain the pooling result; the pooling result is input into the convolution layer of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a spatial convolution feature map; the spatial convolution feature map is activated by the Sigmoid function and multiplied with the first channel convolution feature map to obtain a local feature map.

[0140] In one embodiment, the number of RMFE modules included in the four stages of the feature extraction network of the remote sensing image classification model is classified as: 1, 1, 2, and 1.

[0141] In one embodiment, the downsampling layer in the first stage of the feature extraction network of the remote sensing image classification model is a depth convolution layer with a convolution kernel size of 7×7; the downsampling layers in the second, third, and fourth stages are convolution layers with a convolution kernel size of 3×3.

[0142] Regarding the specific limitations of the remote sensing image classification device based on local and global feature fusion, please refer to the limitations of the remote sensing image classification method based on local and global feature fusion above, and will not be repeated here. Each module in the above-mentioned remote sensing image classification device based on local and global feature fusion can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0143] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0144] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A remote sensing image classification method based on the fusion of local and global features, characterized in that: The method comprises: Acquiring remote sensing images and annotating the remote sensing images to obtain training samples; A remote sensing image classification model based on local and global feature fusion is constructed, and the remote sensing image classification model includes: an input network, a feature extraction network and a classification network; the input network is used to extract features of the training samples using multiple dilated convolutions with different dilation rates and then splice them to obtain a spliced ​​feature map; the feature extraction network is used to extract and fuse the local and global features of the spliced ​​feature map using four stages consisting of a downsampling layer and a stack of several RMFE modules to obtain a fused feature map; the RMFE module is constructed based on the MetaFormer paradigm using a P2DF module and a feedforward layer; the P2DF module is used to split the input feature map into channels, and then use convolution and channel attention modules to perform channel splitting on the input feature map. The block extracts the global features of the second channel splitting result, uses the convolution and spatial attention modules to extract the local features of the first channel splitting result, and fuses the global features and the local features; the classification network is used to perform classification according to the fused feature map to obtain the remote sensing image classification prediction result; the P2DF module includes: a global feature extraction branch, a local feature extraction branch and a feature fusion module; wherein the global feature extraction branch includes a point convolution layer, a depth convolution layer and a channel attention module; the local feature extraction branch includes: 2 stacked depth convolution layers and 1 point convolution layer, a spatial attention module; the feature fusion module includes: a convolution module consisting of a 3×3 depth convolution layer and a skip connection, and two point convolution layers; Training the remote sensing image classification model using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model; The remote sensing image to be tested is input into the trained remote sensing image classification model to obtain the remote sensing image classification result.

2. The method according to claim 1, characterized in that The input network comprises parallel branches consisting of four dilated convolutional layers with different dilation rates; The remote sensing image classification model is trained using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model, including: Inputting the training samples into the four dilated convolution modules in the input network respectively to obtain four dilated convolution feature maps with different receptive fields, and splicing the four dilated convolution feature maps to obtain a spliced ​​feature map; Inputting the concatenated feature map into the first stage of the feature extraction network to obtain a first-layer feature map; Inputting the first layer feature map into the second stage of the feature extraction network to obtain the second layer feature map; Inputting the second layer feature map into the third stage of the feature extraction network to obtain the third layer feature map; Inputting the third layer feature map into the fourth stage of the feature extraction network to obtain a fused feature map; Inputting the fused feature map into a classification network to obtain a remote sensing image classification prediction result; The remote sensing image classification model is reversely trained using the annotations of the training samples and the remote sensing image classification prediction results to obtain a trained remote sensing image classification model.

3. The method according to claim 2, characterized in that The expansion rates of the four dilated convolutional layers with different dilation rates of the input network are 1, 2, 3, and 4, respectively, and all use a 3×3 convolution kernel.

4. The method according to claim 2, characterized in that The first stage of the feature extraction network includes a downsampling layer and an RMFE module; the RMFE module includes two normalization modules, a P2DF module, and a feedforward module; The concatenated feature map is input into the first stage of the feature extraction network to obtain the first layer feature map, including: Input the spliced ​​feature map into the downsampling layer of the first stage to obtain a downsampled feature map; Inputting the downsampled feature map into the first normalization module of the RMFE module of the first stage to obtain a first normalized result; Input the first normalization result into the P2DF module of the RMFE module of the first stage to obtain a fused feature map; Adding the fused feature map and the downsampled feature map to obtain a first fused feature; Inputting the first fusion feature into the second normalization module of the RMFE module of the first stage to obtain a second normalized result; The second normalized result is input into the feedforward module of the RMFE module of the first stage to obtain an enhanced feature map; The enhanced feature map is added to the first fusion feature to obtain a first layer feature map.

5. The method according to claim 4, characterized in that The first normalization result is input into the P2DF module of the RMFE module of the first stage to obtain the fused feature map, including: Input the normalized result into the P2DF module of the RMFE module of the first stage for channel splitting to obtain a first channel splitting result and a second channel splitting result; The first channel splitting result is input into the local feature extraction branch of the P2DF module of the RMFE module of the first stage, and processed through two stacked depth convolution layers and one point convolution layer. The obtained convolution feature is multiplied by the first channel splitting result to obtain the first channel convolution feature map; Input the first channel convolutional feature map into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a local feature map; The second channel splitting result is input into the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and after processing through one point convolution layer, the number of channels is split according to a preset ratio to obtain a first group of channel feature maps and a second group of channel feature maps; and the first group of channel feature maps is processed through a 7×7 depth convolution layer and split according to a second preset ratio to obtain a third group of channel feature maps, a fourth group of channel feature maps and a fifth group of channel feature maps, and the result of multiplying the second group of channel feature maps and the third group of channel feature maps is multiplied with the fourth group of channel feature maps after dimensionality increase by point convolution, and the multiplication result is multiplied with the fifth group of channel feature maps after dimensionality increase by point convolution, and then the multiplication result is processed by point convolution to obtain a multidimensional feature map; Input the multidimensional feature map into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map; The local feature map and the global feature map are input into the feature fusion module of the P2DF module of the RMFE module of the first stage, and are spliced ​​in the channel dimension using a channel splicing method to obtain a global and local spliced ​​feature map; the global and local spliced ​​feature map is added to the global and local spliced ​​feature map after passing through a deep convolution layer and a GeLU activation function, and the addition result is normalized. After two point convolution layers for dimensionality increase and then dimensionality reduction, a fused feature map is obtained.

6. The method according to claim 5, characterized in that The channel attention module includes a left branch, a right branch, and a Sigmoid function; The multidimensional feature map is input into the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a global feature map, including: The multidimensional feature map is input into the left branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and the left branch feature map is obtained after maximum pooling, point convolution, ReLu activation, and point convolution processing. The multidimensional feature map is input into the right branch of the channel attention module of the global feature extraction branch of the P2DF module of the RMFE module of the first stage, and the right branch feature map is obtained after average pooling, point convolution, ReLu activation, and point convolution processing. The left branch feature map and the right branch feature map are added together and activated using a Sigmoid function, and the activation result is multiplied by the multidimensional feature map to obtain a global feature map.

7. The method according to claim 5, characterized in that The spatial attention module includes: pooling module, convolution layer and Sigmoid activation function; The first channel convolution feature map is input into the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a local feature map, including: The first channel convolution feature map is input into the pooling module of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage. After the maximum pooling and average pooling are performed respectively, the maximum pooling results and the average pooling results are channel-wise spliced ​​to obtain the pooling result; The pooling result is input into the convolution layer of the spatial attention module of the local feature extraction branch of the P2DF module of the RMFE module of the first stage to obtain a spatial convolution feature map; The spatial convolution feature map is activated by a Sigmoid function and then multiplied by the first channel convolution feature map to obtain a local feature map.

8. The method according to claim 1, characterized in that The four stages of the feature extraction network include RMFE modules in the following categories: 1, 1, 2, and 1.

9. The method according to claim 1, characterized in that The downsampling layer in the first stage of the feature extraction network is a depth-wise convolutional layer with a convolution kernel size of 7×7; the downsampling layer in the second, third, and fourth stages is a convolutional layer with a convolution kernel size of 3×3.

10. A remote sensing image classification device based on local and global feature fusion, characterized in that: The device comprises: A training sample acquisition module is used to acquire remote sensing images and annotate the remote sensing images to obtain training samples; A remote sensing image classification model construction module is used to construct a remote sensing image classification model based on local and global feature fusion. The remote sensing image classification model includes: an input network, a feature extraction network, and a classification network; the input network is used to extract features from the training samples using multiple hole convolutions with different hole rates and then splice them to obtain a spliced ​​feature map; the feature extraction network is used to extract and fuse the local and global features of the spliced ​​feature map using four stages consisting of a downsampling layer and a stack of several RMFE modules to obtain a fused feature map; the RMFE module is constructed based on the MetaFormer paradigm using a P2DF module and a feedforward layer; the P2DF module is used to split the input feature map into channels and then use convolution to perform the channel splitting. The P2DF module extracts the global features of the second channel splitting result by using the channel attention module, extracts the local features of the first channel splitting result by using the convolution and spatial attention modules, and fuses the global features and the local features; the classification network is used to perform classification according to the fused feature map to obtain the remote sensing image classification prediction result; the P2DF module includes: a global feature extraction branch, a local feature extraction branch and a feature fusion module; wherein the global feature extraction branch includes a point convolution layer, a depth convolution layer and a channel attention module; the local feature extraction branch includes: 2 stacked depth convolution layers and 1 point convolution layer, a spatial attention module; the feature fusion module includes: a convolution module consisting of a 3×3 depth convolution layer and a skip connection, and two point convolution layers; A remote sensing image classification model training module is used to train the remote sensing image classification model using the annotations of the training samples and the remote sensing image classification prediction results obtained by inputting the training samples into the remote sensing image classification model to obtain a trained remote sensing image classification model; The remote sensing image classification module is used to input the remote sensing image to be tested into the trained remote sensing image classification model to obtain the remote sensing image classification result.

Images