Remote sensing image recognition method, device, computer equipment and storage medium

Abstract

The present application relates to a remote sensing image recognition method, apparatus, computer equipment, and storage medium in the field of image recognition technology. The method includes acquiring remote sensing images, annotating them, obtaining training samples, and constructing a remote sensing image recognition model. The model includes: a convolution blocker module, a feature extraction network, and a classification network. The feature extraction network includes four stages. The first stage uses a sandglass-CA module to increase spatial connections; the second and third stages use deep convolution with a small convolution kernel and a channel multi-layer perceptron to learn the spatial and channel dimension information of the image; the fourth stage uses deep convolution with a large convolution kernel and a channel multi-layer perceptron to learn the spatial and channel dimension information of the image; the model is trained using training samples, and the trained model is used to recognize the remote sensing image to be tested. This method enables the model to have overlapping spatial information, thereby improving the accuracy of remote sensing image classification.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a remote sensing image recognition method, apparatus, computer equipment, and storage medium. Background Art

[0002] The most important aspect of remote sensing image interpretation is the recognition of image information. Remote sensing images record rich image information, complex scene compositions, and detailed features of ground features. In recent years, Transformer-based models (such as ViT (Vision Transformer)) have surpassed the performance of previous convolutional networks in many scenarios. However, these models also introduce a large number of parameters, making them unsuitable for remote sensing image classification.

[0003] MobileNeXt is a lightweight convolutional neural network model that proposes hourglass blocks, building deep convolutions and shortcuts on high dimensions. This allows it to learn more diverse high-dimensional features and effectively improves network performance, but it still lags behind Transformer-based models. The authors of Conv-Mixer proposed that the powerful performance of the ViT architecture is not only due to the Transformer's self-attention mechanism, but also to some extent due to the use of patches as input. However, the size and resolution of the image remain unchanged throughout the network, resulting in high computational complexity and memory consumption. Yu et al. proposed replacing the attention module in the Transformer with other operations such as pooling, and the effect still reached 97% of the Transformer. Therefore, using token mixer and channel MLP as a general architecture can achieve good results, but the network lacks exploration of the design of the token mixer module. ConvMLP is a lightweight, staged, joint network with convolutional layers and MLP for visual recognition, but its extraction of deep information is not sufficient. Summary of the Invention

[0004] Based on this, it is necessary to provide a remote sensing image recognition method, device, computer equipment and storage medium to address the above technical problems.

[0005] A remote sensing image recognition method, the method comprising:

[0006] Remote sensing images are acquired and annotated to obtain training samples.

[0007] A remote sensing image recognition model is constructed, which includes: a convolution blocker module, a feature extraction network and a classification network; the convolution blocker module is used to divide the training sample into multiple image blocks using a convolution module, and the feature extraction network includes: 4 stages, the first stage uses the sandglass-CA module to increase spatial connection and extract the spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to respectively learn the spatial information and channel dimension information of the spatial coordinate features; the third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage; the fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features; the classification network is used to classify remote sensing images according to the image features to obtain classification prediction results.

[0008] The remote sensing image recognition model is trained according to the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0009] The acquired remote sensing image to be tested is input into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested.

[0010] In one embodiment, the convolution blocker module includes: 3 convolution modules connected in sequence, the first and second convolution modules include 3×3 convolution layers, batch normalization layers and ReLU activation functions; the third convolution module includes 3×3 convolution layers, batch normalization layers, ReLU activation functions and maximum pooling layers.

[0011] The remote sensing image recognition model is trained according to the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model, including:

[0012] The training sample is input into the convolution blocker module, and three convolution modules are used to divide the training sample into multiple image blocks.

[0013] Inputting a plurality of the image blocks into the feature extraction network to obtain image features.

[0014] The image features are input into the classification network to obtain classification prediction results.

[0015] The remote sensing image recognition model is reversely trained according to the classification prediction results and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0016] In one embodiment, the feature extraction network further includes: 3 convolutional downsampling layers.

[0017] Inputting a plurality of the image patches into the feature extraction network to obtain image features includes:

[0018] The image block is input into the first stage, and the obtained features are input into the first convolutional downsampling layer to obtain spatial coordinate features.

[0019] The spatial coordinate features are input into the second stage, and the obtained output features are input into the second convolutional downsampling layer to obtain the second stage output features.

[0020] The output features of the second stage are input into the third stage, and the obtained output features are input into the third convolutional downsampling layer to obtain the output features of the third stage, and the obtained output features are input into the fourth stage.

[0021] In one embodiment, the first stage includes: 2 sandglass-CA modules; wherein the sandglass-CA module is composed of a sandglass module and a coordinate attention module; the sandglass module includes: a first depth convolution layer, a first point convolution layer, a second point convolution layer and a second depth convolution layer.

[0022] The image block is input into the first stage, and the obtained features are input into the first convolutional downsampling layer to obtain spatial coordinate features, including:

[0023] The image block is input into the sandglass module of the first sandglass-CA module, and activated by the ReLU6 function after passing through the first deep convolution layer to obtain spatial information features.

[0024] The spatial information features are input into the coordinate attention module of the first sandglass-CA module, and the spatial coordinate attention features are output.

[0025] The spatial information feature and the spatial coordinate attention feature are spliced ​​in the spatial dimension to obtain a first intermediate feature.

[0026] The first intermediate feature is processed by the first point convolution layer and the second point convolution layer of the sandglass module of the first sandglass-CA module, and the obtained result is processed by the second depth convolution layer of the sandglass module of the first sandglass-CA module to obtain the second intermediate feature.

[0027] The second intermediate feature is spliced ​​with the image block to obtain a first spatial coordinate feature.

[0028] The first spatial coordinate feature is input into a second sandglass-CA module to obtain a second spatial coordinate feature.

[0029] The second spatial feature is input into the first convolutional downsampling layer to obtain a spatial coordinate feature.

[0030] In one embodiment, the second stage includes: 2 DWformer modules, each of which includes: a third deep convolutional layer and a channel multi-layer perceptron, and the convolution kernel size of the third deep convolutional layer is 3.

[0031] The spatial coordinate features are input into the second stage, and the obtained output features are input into the second convolutional downsampling layer to obtain the second stage output features, including:

[0032] The spatial coordinate features are input into the third depth convolution layer of the first DWformer module of the second stage, and the obtained results are processed by layer normalization to obtain spatial dimension information.

[0033] The image spatial information is input into the channel multilayer perceptron of the first DWformer module of the second stage to obtain channel dimension information.

[0034] The spatial dimension information and the channel dimension information are fused to obtain the intermediate output features of the second stage.

[0035] The intermediate output features of the second stage are input into the second DWformer module, and the obtained results are input into the second convolutional downsampling layer to obtain the second stage output features.

[0036] In one embodiment, the third stage includes: 4 DWformer modules; wherein the DWformer module includes: a fourth depth convolution layer and a channel multi-layer perceptron, and the convolution kernel size of the fourth depth convolution layer is 3.

[0037] In one embodiment, the fourth stage includes: 2 LKDWFormer modules; wherein the LKDWFormer module includes: a fifth depth convolution layer, a GELU activation function, a batch normalization layer and a channel multi-layer perceptron, and the convolution sum size of the fifth depth convolution layer is 9.

[0038] The output features of the third stage are input into the fourth stage to obtain image features, including:

[0039] The third stage output feature is input into the fifth depth convolution layer of the first LKDWFormer module, and the obtained result is activated using the GELU activation function, and the activation result is processed using a batch normalization layer to obtain the first spatial dimension information.

[0040] The first spatial dimension information is fused with the third stage output feature to obtain a fused feature.

[0041] The fused features are input into the channel multilayer perceptron of the first LKDWFormer module to obtain the first channel dimension information.

[0042] The fusion feature is fused with the first channel dimension information to obtain a first image feature.

[0043] The first image feature is input into the second LKDWFormer module to obtain the image feature.

[0044] A remote sensing image recognition device, comprising:

[0045] The data acquisition module is used to acquire remote sensing images and annotate the remote sensing images to obtain training samples.

[0046] A remote sensing image recognition model construction module is used to construct a remote sensing image recognition model, which includes: a convolution blocker module, a feature extraction network and a classification network; the convolution blocker module is used to divide the training sample into multiple image blocks using a convolution module, and the feature extraction network includes: 4 stages, the first stage uses the sandglass-CA module to increase spatial connections and extract the spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to respectively learn the spatial information and channel dimension information of the spatial coordinate features; the third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage; the fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features; the classification network is used to classify remote sensing images according to the image features to obtain classification prediction results.

[0047] The remote sensing image recognition model training module is used to train the remote sensing image recognition model according to the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0048] The remote sensing image classification module is used to input the acquired remote sensing image to be tested into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested.

[0049] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0050] A computer-readable storage medium stores a computer program, which implements the steps of any of the above methods when executed by a processor.

[0051] The above-mentioned remote sensing image recognition method, device, computer equipment and storage medium, the method includes obtaining remote sensing images, annotating the remote sensing images, obtaining training samples, and constructing a remote sensing image recognition model, which includes: a convolution blocker module, a feature extraction network and a classification network. The convolution blocker module is used to divide the training samples into multiple image blocks. The feature extraction network includes: 4 stages, the first stage uses the sandglass-CA module to increase spatial connections and extract the spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to respectively learn the spatial information of the spatial coordinate features. The third stage uses the DWformer module to learn the spatial and channel dimensional information of the output features of the second stage. The fourth stage uses the deep convolution with a kernel size of 9 and the channel multi-layer perceptron in the LKDWFormer module to learn the spatial and channel dimensional information of the output features of the third stage to obtain image features. The classification network is used to classify remote sensing images based on image features to obtain classification prediction results. The remote sensing image recognition model is trained using the annotations of the training samples to obtain a trained remote sensing image recognition model. The acquired remote sensing image to be tested is input into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested. This method enables the model to have overlapping spatial information, improves the accuracy of remote sensing image classification, and introduces only a few parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 1 is a flow chart of a remote sensing image recognition method according to an embodiment;

[0053] Figure 2 A diagram showing the structure of a remote sensing image recognition model in one embodiment;

[0054] Figure 3 A schematic diagram of a process for training a remote sensing image recognition model in one embodiment;

[0055] Figure 4 A schematic diagram of a process for training a feature extraction network in another embodiment;

[0056] Figure 5 The structure of the SandGlass module in another embodiment;

[0057] Figure 6 Figure 1 is a structural diagram of the CA attention mechanism and the SandGlass-CA module in another embodiment, where (a) is the CA attention mechanism structural diagram and (b) is the SandGlass-CA module structural diagram;

[0058] Figure 7is a structural diagram of a DWformer module in another embodiment;

[0059] Figure 8 is a structural diagram of the LKDWFormer module in another embodiment;

[0060] Figure 9 This is a diagram of experimental results in a verification example;

[0061] Figure 10 is a structural block diagram of a remote sensing image recognition device in one embodiment;

[0062] Figure 11 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] Convolutional Tokenizer module, Convolutional Tokenizer module.

[0065] Hourglass-coordinate attention module, sandglass-CA module.

[0066] Deep convolution-channel multi-layer perceptron joint module, DWformer module.

[0067] Large convolution kernel depth convolution-channel multi-layer perceptron joint module, LKDWFormer module.

[0068] In one embodiment, Figure 1 As shown, a remote sensing image recognition method is provided, which includes the following steps:

[0069] Step 100: Acquire remote sensing images and label them to obtain training samples.

[0070] Specifically, the remote sensing images may be: grassland remote sensing images, field remote sensing images, industrial area remote sensing images, river and lake remote sensing images, forest remote sensing images, residential area remote sensing images, and parking lot remote sensing images.

[0071] Step 102: Build a remote sensing image recognition model.

[0072] The remote sensing image recognition model includes: convolution blocker module, feature extraction network and classification network.

[0073] The convolution blocker module is used to divide the training sample into multiple image blocks using the convolution module.

[0074] The feature extraction network consists of four stages. The first stage uses the sandglass-CA module to increase spatial connections and extract the spatial coordinate features of multiple image patches. The second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron (ChannelMLP) to learn the spatial information and channel dimension information of the spatial coordinate features respectively. The third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage. The fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and ChannelMLP to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features.

[0075] The classification network is used to classify remote sensing images according to image features and obtain classification prediction results.

[0076] The structure of the remote sensing image recognition model is as follows Figure 2 shown.

[0077] The first stage is a purely convolutional module, located between the Convolutional Tokenizer module and the second stage, to enhance the exchange of spatial information within the network. The first stage consists of two sandglass-CA modules. The sandglass-CA module is composed of the sandglass module and coordinate attention (CA) from MobileNeXt.

[0078] The second and third stages include the DWFormer module, which follows the general pattern of MetaFormer. This module uses deep convolution with a kernel size of 3 to learn spatial information about the image, and ChannelMLP to learn information about the channel dimension. Using 3×3 depthwise convolution in shallow stages can reduce the number of parameters while achieving good results. Figure 2 The Linear in Channel MLP is a fully connected layer, and DWconv is a depth convolution.

[0079] The fourth stage includes the LKDWFormer module. The LKDWFormer module follows the general pattern of MetaFormer. The module uses 9×9 depthwise convolution connected with GELU (Gaussian error linear units) activation function and subsequent batch normalization layer to learn the spatial information of the image, and ChannelMLP to learn the channel dimension information.

[0080] A key idea of ​​previous work is that self-attention can mix distant spatial positions, that is, they can have arbitrarily large receptive fields. Studies have shown that large convolution kernels have larger receptive fields, can bring more contextual information, help learn the relative positions between objects, and can also encode position information due to the padding effect. Therefore, this embodiment uses large convolution kernels to expand the receptive field, and uses depthwise convolution (DW convolution) with a convolution kernel size of 9 in the LKDWFormer module. In the last stage of the model, the LKDWFormer module is used to maximize the trade-off between efficiency and performance.

[0081] The classification network includes 1 fully connected layer.

[0082] Step 104: The remote sensing image recognition model is trained based on the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0083] Step 106: Input the acquired remote sensing image to be tested into the trained remote sensing image recognition model to obtain a classification result of the remote sensing image to be tested.

[0084] In the above-mentioned remote sensing image recognition method, the method includes: acquiring remote sensing images, annotating the remote sensing images, obtaining training samples, and constructing a remote sensing image recognition model, which includes: a convolution blocker module, a feature extraction network, and a classification network. The convolution blocker module is used to divide the training samples into multiple image blocks. The feature extraction network includes: 4 stages. The first stage uses the sandglass-CA module to increase spatial connections and extract spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and ChannelMLP to learn the spatial information of the spatial coordinate features and the information of the channel dimension respectively. The third stage uses the DWformer module to learn the spatial and channel dimension information of the output features of the second stage. The fourth stage uses the deep convolution with a kernel size of 9 and ChannelMLP in the LKDWFormer module to learn the spatial and channel dimension information of the output features of the third stage to obtain image features. The classification network is used to classify remote sensing images based on image features to obtain classification prediction results. The remote sensing image recognition model is trained using the annotations of the training samples to obtain a trained remote sensing image recognition model. The acquired remote sensing image to be tested is input into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested. This method enables the model to have overlapping spatial information, improves the accuracy of remote sensing image classification, and introduces only a few parameters.

[0085] In one embodiment, the convolution blocker module includes: three convolution modules connected in sequence, the first and second convolution modules include a 3×3 convolution layer, a batch normalization layer (Batch Normalization, referred to as BN) and a ReLU activation function; the third convolution module includes a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer; Figure 3 As shown, step 104 includes the following specific steps:

[0086] Step 300: Input the training sample into the convolution blocker module, and use three convolution modules to divide the training sample into multiple image blocks.

[0087] Step 302: Input multiple image blocks into the feature extraction network to obtain image features.

[0088] Step 304: Input the image features into the classification network to obtain the classification prediction result.

[0089] Step 306: reversely train the remote sensing image recognition model based on the classification prediction results and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0090] In one embodiment, the feature extraction network further includes: 3 convolutional downsampling layers; Figure 4 As shown, step 302 specifically includes the following steps:

[0091] Step 400: Input the image block into the first stage, and input the obtained features into the first convolutional downsampling layer to obtain spatial coordinate features.

[0092] Step 402: Input the spatial coordinate features into the second stage, and input the obtained output features into the second convolutional downsampling layer to obtain the second stage output features.

[0093] Step 404: Input the second stage output feature into the third stage, and input the obtained output feature into the third convolutional downsampling layer to obtain the third stage output feature.

[0094] Step 406: Input the output features of the third stage into the fourth stage to obtain image features.

[0095] In one embodiment, Figure 2As shown in Figure 2, the first stage includes: 2 sandglass-CA modules; the sandglass-CA module consists of a sandglass module and a coordinate attention module; the sandglass module includes: the first depth convolution layer, the first point convolution layer, the second point convolution layer, and the second depth convolution layer; the structure of the sandglass module is shown in Figure 2. Figure 5 As shown in the figure, the structure of the CA attention mechanism and the sandglass-CA module is as follows Figure 6 As shown, (a) is the CA attention mechanism structure diagram, and (b) is the SandGlass-CA module structure diagram. Step 400 includes: inputting the image block into the sandglass module of the first sandglass-CA module, and using the ReLU6 function to activate after the first depth convolution layer to obtain spatial information features; inputting the spatial information features into the coordinate attention module of the first sandglass-CA module to output spatial coordinate attention features; splicing the spatial information features and the spatial coordinate attention features in the spatial dimension to obtain a first intermediate feature; processing the first intermediate feature through the first point convolution layer and the second point convolution layer of the sandglass module of the first sandglass-CA module, and processing the obtained result through the second depth convolution layer of the sandglass module of the first sandglass-CA module to obtain a second intermediate feature; splicing the second intermediate feature with the image block to obtain a first spatial coordinate feature; inputting the first spatial coordinate feature into the second sandglass-CA module to obtain a second spatial coordinate feature; inputting the second spatial feature into the first convolution downsampling layer to obtain a spatial coordinate feature.

[0096] Specifically, the structure of the SandGlass module is as follows Figure 5 As shown in the figure, the SandGlass module builds both shortcut and deep convolution on high-dimensional features, which can learn more spatial information. The first dimension is the number of channels M, the second dimension is the height H, and the third dimension is the width W. After a layer of 3×3 depthwise convolution is performed to extract spatial information features, a ReLU6 activation function is used to enhance effective information and suppress irrelevant or redundant information, while keeping the feature size unchanged. To simulate the bottleneck structure, two consecutive 1×1 point convolutions are used in the middle to reduce and expand the dimensions. The features after dimensionality reduction are M / t×H×W, where t is the reduction ratio, and the features after dimensionality expansion are M×H×W, where N is the number of channels after expansion. Finally, a layer of 3×3 depthwise convolution with stride=s is performed to extract spatial information features, followed by a linear mapping (i.e., a linear activation function). This helps avoid the occurrence of zeroing and thus reduces information loss. The final feature size is M×H / s×W / s.

[0097] The coordination attention (CA) mechanism decomposes the 2D global pooling operation into two one-dimensional encoding processes, which can more effectively capture the relationship between position information and channel information, thereby achieving better performance. The coordinate attention mechanism is an improvement on the SE attention mechanism that focuses on channel information, and introduces coordinate information into SE attention. Given an input feature tensor X = [x1, x2, ..., x c ], which is processed by the coordinate attention mechanism to become Y=[y1,y2,…,y c ].

[0098] CA attention mechanism and SandGlass-CA module Figure 6 As shown, the input of the SandGlass-CA module The first dimension is the number of channels, C; the second dimension is the height, H; and the third dimension is the width, W. The input is average pooled in the X and Y directions, respectively, to produce intermediate outputs, C×H×1 and C×1×W. This captures global information in the X and Y coordinates while preserving precise information in both the Y and X coordinates. A spatial concatenation operation is then performed to produce intermediate features, C×(H+W). This is followed by a convolution operation, which yields features, C / r×(H+W). This is followed by batch normalization and a nonlinear transformation (preferably a ReLU activation function), to produce intermediate features, C / r×(H+W). Compared to the fully connected layers in the SE block, convolution is sparse, reducing complexity. The intermediate features are spatially split to produce features, C×H×1 and C×1×W. Both are then processed using the Sigmoid activation function, enhancing valid coordinate and channel information while suppressing irrelevant or redundant coordinate and channel information.

[0099] In one embodiment, the second stage includes: 2 DWformer modules; Figure 7 As shown, the DWformer module includes: a third depth convolution layer and a ChannelMLP module, and the convolution kernel size of the third depth convolution layer is 3; step 402 includes: inputting the spatial coordinate feature into the third depth convolution layer of the first DWformer module of the second stage, and obtaining the spatial dimension information after the result is layer normalized; inputting the image spatial information into the channel multi-layer perceptron of the first DWformer module of the second stage to obtain channel dimension information; fusing the spatial dimension information and the channel dimension information to obtain the intermediate output feature of the second stage; inputting the intermediate output feature of the second stage into the second DWformer module, and inputting the obtained result into the second convolution downsampling layer to obtain the second stage output feature.

[0100] In one embodiment, the third stage includes: 4 DWformer modules; wherein the DWformer module includes: a fourth depth convolution layer and a ChannelMLP module, and the convolution kernel size of the fourth depth convolution layer is 3.

[0101] In one embodiment, the fourth stage includes: 2 LKDWFormer modules; wherein, Figure 8 As shown, the LKDWFormer module includes: a fifth depth convolution layer, a GELU activation function, a batch normalization layer and a ChannelMLP module, and the convolution sum size of the fifth depth convolution layer is 9; step 406 includes: inputting the third stage output feature into the fifth depth convolution layer of the first LKDWFormer module, and activating the result using the GELU activation function, and processing the activation result using the batch normalization layer to obtain first spatial dimension information; fusing the first spatial dimension information with the third stage output feature to obtain a fused feature; inputting the fused feature into the channel multi-layer perceptron of the first LKDWFormer module to obtain first channel dimension information; fusing the fused feature with the first channel dimension information to obtain a first image feature; and inputting the first image feature into the second LKDWFormer module to obtain an image feature.

[0102] It should be understood that although Figure 1 、 Figure 3-Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 、 Figure 3-Figure 4At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0103] In a verification embodiment, the experimental environment uses Python language under the Pytorch framework, and the model is implemented on the NVIDIA GeForce RTX 3080Ti server.

[0104] The dataset used in this experiment consists of RGB remote sensing images. The images are categorized into seven categories: grassland, cultivated land, factory, woodland, water, housing, and parking. The dataset contains 2,800 images, of which 2,240 were randomly selected as the training set, and the remaining 560 images were used as the test set.

[0105] The remote sensing image recognition model first divides the input image into multiple patches by Convolutional Tokenizer. Then it passes through the Sandglass-CA Block to increase spatial connections. Then there is a joint stage of three Conv and MLP. Deep convolution with small convolution kernels is used in the shallow stage, and deep convolution with large convolution kernels is used in the deep stage to achieve the goal of balancing efficiency and performance. After each stage, a 3×3 convolution layer with a stride of 2 is used for downsampling. This allows the network to have overlapping spatial information. This improves classification accuracy while introducing only a few parameters. The structure of the remote sensing image recognition model is as follows Figure 2 shown.

[0106] The overall architecture of the remote sensing image recognition model is shown in Table 1.

[0107] Table 1 Overall architecture of remote sensing image recognition model

[0108]

[0109]

[0110] In order to demonstrate the recognition effect of the remote sensing image recognition model of the present invention on the data set, comparative experiments were conducted using the ConvMLP network model, the MobileNeXt network model, and the coatNet model. Table 2 shows the number of parameters, the amount of computation, the highest accuracy, and the average of the five highest accuracy rates. Table 2 shows that the classification accuracy of the LKDWformer network model is about 2% higher than that of the ConvMLP network model and the MobileNeXt, and about 1% higher than that of the CoatNet network model, and has very small number of parameters and computation. The experimental results are as follows Figure 9 shown.

[0111] Table 2 Comparison of various indicators

[0112]

[0113] In one embodiment, Figure 10 As shown, a remote sensing image recognition device is provided, comprising: a data acquisition module, a remote sensing image recognition model construction module, a remote sensing image recognition model training module and a remote sensing image classification module, wherein:

[0114] The data acquisition module is used to acquire remote sensing images, label the remote sensing images, and obtain training samples.

[0115] The remote sensing image recognition model construction module is used to construct the remote sensing image recognition model. The remote sensing image recognition model includes: a convolution blocker module, a feature extraction network and a classification network; the convolution blocker module is used to divide the training samples into multiple image blocks using the convolution module, and the feature extraction network includes: 4 stages. The first stage uses the sandglass-CA module to increase spatial connections and extract the spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the spatial coordinate features respectively; the third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage; the fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features; the classification network is used to classify remote sensing images according to image features to obtain classification prediction results.

[0116] The remote sensing image recognition model training module is used to train the remote sensing image recognition model based on the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0117] The remote sensing image classification module is used to input the acquired remote sensing image to be tested into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested.

[0118] In one embodiment, the convolution blocker module includes: 3 convolution modules connected in sequence, the first and second convolution modules include 3×3 convolution layers, batch normalization layers (Batch Normalization, abbreviated as BN) and ReLU activation functions; the third convolution module includes a 3×3 convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer; the remote sensing image recognition model training module is also used to input training samples into the convolution blocker module, and use three convolution modules to divide the training samples into multiple image blocks; input the multiple image blocks into a feature extraction network to obtain image features; input the image features into a classification network to obtain classification prediction results; and reverse train the remote sensing image recognition model according to the classification prediction results and the annotations of the training samples to obtain a trained remote sensing image recognition model.

[0119] In one embodiment, the feature extraction network also includes: 3 convolutional downsampling layers; a remote sensing image recognition model training module, which is also used to input a small image block into the first stage, and input the obtained features into the first convolutional downsampling layer to obtain spatial coordinate features; input the spatial coordinate features into the second stage, and input the obtained output features into the second convolutional downsampling layer to obtain second stage output features; input the second stage output features into the third stage, and input the obtained output features into the third convolutional downsampling layer to obtain third stage output features; input the third stage output features into the fourth stage to obtain image features.

[0120] In one embodiment, the first stage includes: 2 sandglass-CA modules; wherein the sandglass-CA module is composed of a sandglass module and a coordinate attention module; the sandglass module includes: a first depth convolution layer, a first point convolution layer, a second point convolution layer and a second depth convolution layer; the remote sensing image recognition model training module is also used to input the image block into the sandglass module of the first sandglass-CA module, and after passing through the first depth convolution layer, the ReLU6 function is used for activation to obtain spatial information features; the spatial information features are input into the coordinate attention module of the first sandglass-CA module, and the spatial coordinate attention is output. features; splicing the spatial information features and the spatial coordinate attention features in the spatial dimension to obtain the first intermediate features; processing the first intermediate features through the first point convolution layer and the second point convolution layer of the sandglass module of the first sandglass-CA module, and processing the obtained results through the second depth convolution layer of the sandglass module of the first sandglass-CA module to obtain the second intermediate features; splicing the second intermediate features with the image blocks to obtain the first spatial coordinate features; inputting the first spatial coordinate features into the second sandglass-CA module to obtain the second spatial coordinate features; inputting the second spatial features into the first convolution downsampling layer to obtain the spatial coordinate features.

[0121] In one embodiment, the second stage includes: 2 DWformer modules; wherein the DWformer module includes: a third deep convolution layer and a ChannelMLP module, and the convolution kernel size of the third deep convolution layer is 3; the remote sensing image recognition model training module is also used to input the spatial coordinate features into the third deep convolution layer of the first DWformer module of the second stage, and obtain the spatial dimension information after the obtained result is layer-normalized; the image spatial information is input into the channel multi-layer perceptron of the first DWformer module of the second stage to obtain channel dimension information; the spatial dimension information and the channel dimension information are fused to obtain the intermediate output features of the second stage; the intermediate output features of the second stage are input into the second DWformer module, and the obtained result is input into the second convolution downsampling layer to obtain the second stage output features.

[0122] In one embodiment, the third stage in the remote sensing image recognition model training module includes: 4 DWformer modules; wherein the DWformer module includes: a fourth deep convolutional layer and a ChannelMLP module, and the convolution kernel size of the fourth deep convolutional layer is 3.

[0123] In one embodiment, the fourth stage includes: 2 LKDWFormer modules; wherein the LKDWFormer module includes: a fifth deep convolutional layer, a GELU activation function, a batch normalization layer and a ChannelMLP module, and the convolution sum size of the fifth deep convolutional layer is 9; the remote sensing image recognition model training module is also used to input the third stage output feature into the fifth deep convolutional layer of the first LKDWFormer module, and activate the obtained result using the GELU activation function, and process the activation result using the batch normalization layer to obtain first spatial dimension information; the first spatial dimension information is fused with the third stage output feature to obtain a fused feature; the fused feature is input into the channel multi-layer perceptron of the first LKDWFormer module to obtain the first channel dimension information; the fused feature is fused with the first channel dimension information to obtain the first image feature; the first image feature is input into the second LKDWFormer module to obtain the image feature.

[0124] The specific definitions of the remote sensing image recognition device can be found in the definitions of the remote sensing image recognition method above and will not be repeated here. Each module in the remote sensing image recognition device described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules described above may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0125] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a remote sensing image recognition method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0126] Those skilled in the art will understand that Figure 11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0127] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0129] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0130] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A remote sensing image recognition method, characterized in that: The method comprises: Acquiring remote sensing images and annotating the remote sensing images to obtain training samples; A remote sensing image recognition model is constructed, which includes: a convolution blocker module, a feature extraction network, and a classification network; the convolution blocker module is used to divide the training sample into multiple image blocks using a convolution module, and the feature extraction network includes: 4 stages, the first stage uses the sandglass-CA module to increase spatial connections and extract spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to respectively learn the spatial information and channel dimension information of the spatial coordinate features; the third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage; the fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features; the classification network is used to classify remote sensing images according to the image features to obtain classification prediction results; Training the remote sensing image recognition model according to the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model; The acquired remote sensing image to be tested is input into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested.

2. The method according to claim 1, characterized in that The convolution blocker module includes: 3 convolution modules connected in sequence, the first and second convolution modules include 3×3 convolution layers, batch normalization layers and ReLU activation functions; the third convolution module includes 3×3 convolution layers, batch normalization layers, ReLU activation functions and maximum pooling layers; The remote sensing image recognition model is trained according to the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model, including: Inputting the training sample into the convolution blocker module, and using three convolution modules to divide the training sample into multiple image blocks; Inputting a plurality of the image blocks into the feature extraction network to obtain image features; Inputting the image features into the classification network to obtain a classification prediction result; The remote sensing image recognition model is reversely trained according to the classification prediction results and the annotations of the training samples to obtain a trained remote sensing image recognition model.

3. The method according to claim 2, characterized in that The feature extraction network also includes: 3 convolutional downsampling layers; Inputting the plurality of image patches into the feature extraction network to obtain image features includes: Input the image block into the first stage, and input the obtained features into the first convolutional downsampling layer to obtain spatial coordinate features; Input the spatial coordinate features into the second stage, and input the obtained output features into the second convolutional downsampling layer to obtain the second stage output features; Input the output features of the second stage into the third stage, and input the obtained output features into the third convolutional downsampling layer to obtain the third stage output features; The output features of the third stage are input into the fourth stage to obtain image features.

4. The method according to claim 3, characterized in that The first stage includes two sandglass-CA modules, each consisting of a sandglass module and a coordinate attention module. The sandglass module includes a first depthwise convolutional layer, a first pointwise convolutional layer, a second pointwise convolutional layer, and a second depthwise convolutional layer. The image block is input into the first stage, and the obtained features are input into the first convolutional downsampling layer to obtain spatial coordinate features, including: The image block is input into the sandglass module of the first sandglass-CA module, and activated with the ReLU6 function after passing through the first deep convolutional layer to obtain spatial information features; Input the spatial information features into the coordinate attention module of the first sandglass-CA module, and output the spatial coordinate attention features; Concatenate the spatial information feature and the spatial coordinate attention feature in the spatial dimension to obtain a first intermediate feature; Processing the first intermediate feature through the first point convolution layer and the second point convolution layer of the sandglass module of the first sandglass-CA module, and processing the obtained result through the second depthwise convolution layer of the sandglass module of the first sandglass-CA module to obtain a second intermediate feature; Splicing the second intermediate feature with the image block to obtain a first spatial coordinate feature; Inputting the first spatial coordinate feature into a second sandglass-CA module to obtain a second spatial coordinate feature; The second spatial coordinate feature is input into the first convolution downsampling layer to obtain the spatial coordinate feature.

5. The method according to claim 3, characterized in that The second stage includes: 2 DWformer modules, each of which includes: a third depth convolution layer and a channel multi-layer perceptron, and the convolution kernel size of the third depth convolution layer is 3; The spatial coordinate features are input into the second stage, and the obtained output features are input into the second convolutional downsampling layer to obtain the second stage output features, including: The spatial coordinate features are input into the third depth convolution layer of the first DWformer module of the second stage, and the obtained results are processed by layer normalization to obtain spatial dimension information; Input the spatial dimension information into the channel multilayer perceptron of the first DWformer module of the second stage to obtain channel dimension information; Fusing the spatial dimension information and the channel dimension information to obtain the intermediate output features of the second stage; The intermediate output features of the second stage are input into the second DWformer module, and the obtained results are input into the second convolutional downsampling layer to obtain the second stage output features.

6. The method according to claim 3, characterized in that The third stage includes: 4 DWformer modules; wherein the DWformer module includes: a fourth depth convolution layer and a channel multi-layer perceptron, and the convolution kernel size of the fourth depth convolution layer is 3.

7. The method according to claim 3, characterized in that The fourth stage includes two LKDWFormer modules, each of which includes a fifth depthwise convolutional layer, a GELU activation function, a batch normalization layer, and a channel multilayer perceptron. The convolution kernel size of the fifth depthwise convolutional layer is 9. The output features of the third stage are input into the fourth stage to obtain image features, including: Input the third stage output feature into the fifth depth convolutional layer of the first LKDWFormer module, activate the obtained result using the GELU activation function, and process the activation result using a batch normalization layer to obtain the first spatial dimension information; Fusing the first spatial dimension information with the third stage output feature to obtain a fused feature; Input the fused features into the channel multilayer perceptron of the first LKDWFormer module to obtain the first channel dimension information; Fusing the fusion feature with the first channel dimension information to obtain a first image feature; The first image feature is input into the second LKDWFormer module to obtain the image feature.

8. A remote sensing image recognition device, characterized in that: The device comprises: A data acquisition module is used to acquire remote sensing images and annotate the remote sensing images to obtain training samples; A remote sensing image recognition model construction module is used to construct a remote sensing image recognition model, which includes: a convolution blocker module, a feature extraction network, and a classification network; the convolution blocker module is used to divide the training sample into multiple image blocks using a convolution module, and the feature extraction network includes: 4 stages, the first stage uses the sandglass-CA module to increase spatial connections and extract the spatial coordinate features of multiple image blocks; the second stage uses the deep convolution with a convolution kernel size of 3 in the DWformer module and the channel multi-layer perceptron to respectively learn the spatial information and channel dimension information of the spatial coordinate features; the third stage uses the DWformer module to learn the spatial information and channel dimension information of the output features of the second stage; the fourth stage uses the deep convolution with a convolution kernel size of 9 in the LKDWFormer module and the channel multi-layer perceptron to learn the spatial information and channel dimension information of the output features of the third stage to obtain image features; the classification network is used to classify remote sensing images according to the image features to obtain classification prediction results; A remote sensing image recognition model training module is used to train the remote sensing image recognition model based on the classification prediction results obtained by inputting the training samples into the remote sensing image recognition model and the annotations of the training samples to obtain a trained remote sensing image recognition model; The remote sensing image classification module is used to input the acquired remote sensing image to be tested into the trained remote sensing image recognition model to obtain the classification result of the remote sensing image to be tested.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images