Remote sensing image classification method and device based on spatial and channel feature extraction

Abstract

The application relates to a remote sensing image classification method and device based on spatial and channel feature extraction in the technical field of image recognition. The method comprises the following steps: after a pretreated remote sensing image is subjected to multiple deep convolution processing in an input network, sequence position coding is performed, and the coding result is input into a feature extraction network; four stages are adopted to realize extraction of spatial and channel fusion features, wherein each stage comprises a plurality of stacked CW2P modules; after the spatial and channel fusion features pass through a Neck layer, classification is performed by using a classification head, and a remote sensing image classification result is obtained. The method takes global and local feature fusion as a baseline, adopts a pyramid pooling attention module to extract global information, adopts an improved local window attention module to extract local information, and adopts a linear attention module to interact and mix global and local information, so that channel dimension features are reserved. By using the method, the calculation amount and parameter amount are reduced, and the accuracy of remote sensing image classification is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a remote sensing image classification method and device based on spatial and channel feature extraction. BACKGROUND

[0002] Remote sensing image classification is to analyze the spectral information and spatial information of various ground objects in remote sensing images, select features, divide different categories according to certain rules or algorithms, and then obtain the corresponding information of the actual ground object in the remote sensing image, so as to realize the classification of the remote sensing image. As one of the main tasks of computer vision.

[0003] Image classification mainly uses convolutional neural network (CNN) to extract local information, which is essential for extracting detailed features of ground object elements; however, due to the richness of remote sensing image information and the complexity of the scene, the strictness of the classification accuracy is determined, although CNN covers more local information through local convolution within the receptive field, thereby effectively extracting high-frequency representation, but using CNN as the feature extraction method to realize high-precision remote sensing image classification is far from enough. In recent years, the application of Transformer in image classification tasks has achieved obvious results, and the key factor of its success is the self-attention mechanism, which can better capture global context information. Based on the different advantages of CNN and transformer in the field of vision, many works are now committed to combining the two to improve the accuracy of image classification. For example, NextViT proposes to use CNN to extract local features at the shallow layer and add transformer to aggregate global information at the deep layer, and reduce the quadratic computational complexity of ViT self-attention, which is the main reason that hinders the rapid application of transformer in high-resolution images. However, due to the direct combination of CNN and transformer, the model parameter amount will increase significantly, and many works are also committed to finding improvement methods for ViT self-attention to reduce the model complexity, such as GCViT, which distinguishes between local and global in the self-attention mechanism. Window self-attention is used as local feature extraction to reduce the complexity of the model, and a global generator is used to extract global features; this way of using different levels of feature fusion in the transformer self-attention part can be seen to be successful in other works, such as LightVit. However, the self-attention part mainly extracts spatial information, and the weights are shared in the channel dimension, resulting in weak modeling ability in the channel dimension. SUMMARY

[0004] Therefore, it is necessary to provide a remote sensing image classification method and device based on spatial and channel feature extraction in view of the above technical problems.

[0005] A remote sensing image classification method based on spatial and channel feature extraction, the method comprising:

[0006] Acquiring a remote sensing image, and preprocessing the remote sensing image.

[0007] Inputting the preprocessed remote sensing image into an input network to obtain sequence encoding features; the input network is used to perform sequence position encoding on the training sample after multiple deep convolutions.

[0008] Inputting the sequence encoding features into a feature extraction network to obtain spatial and channel fusion features; the feature extraction network includes four stages, wherein the first stage includes a plurality of stacked CW2P modules, and the remaining three stages include sequence embedding and position encoding layers and a plurality of stacked CW2P modules; the CW2P module is used to extract local information of the sequence encoding features using an improved local window attention module, extract global information of the sequence encoding features using a pyramid pooling attention module, and interactively mix global and local information using a linear attention module; the outputs of the improved local window attention module, the pyramid pooling attention module and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimension using a feedforward network including a gated spatial attention unit.

[0009] Inputting the spatial and channel fusion features into a Neck layer for channel dimension expansion.

[0010] Inputting the output of the Neck layer into a classification head for classification to obtain a remote sensing image classification result.

[0011] A remote sensing image classification device based on spatial and channel feature extraction, the device comprising:

[0012] A remote sensing image preprocessing module for acquiring a remote sensing image and preprocessing the remote sensing image.

[0013] A remote sensing image local feature extraction module for inputting the preprocessed remote sensing image into an input network to obtain sequence encoding features; the input network is used to perform sequence position encoding on the training sample after multiple deep convolutions.

[0014] The spatial and channel fusion feature extraction module is configured to input the sequence coding feature into a feature extraction network to obtain spatial and channel fusion features; the feature extraction network includes four stages, wherein the first stage includes a plurality of stacked CW2P modules, and the remaining three stages include a sequence embedding and position coding layer and a plurality of stacked CW2P modules; the CW2P module is configured to extract local information of the sequence coding feature by using an improved local window attention module, extract global information of the sequence coding feature by using a pyramid pooling attention module, and interactively mix the global and local information by using a linear attention module; and the outputs of the improved local window attention module, the pyramid pooling attention module, and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimensions by using a feedforward network including a gated spatial attention unit.

[0015] The classification module is configured to input the spatial and channel fusion features into a Neck layer for channel dimension expansion, input the output of the Neck layer into a classification head for classification, and obtain a remote sensing image classification result.

[0016] The remote sensing image classification method and device based on spatial and channel feature extraction include the following steps: after a preprocessed remote sensing image is processed by a plurality of deep convolution in an input network, sequence position coding is performed; then the coding result is input into a feature extraction network; spatial and channel fusion features are extracted in the feature extraction network by using four stages, wherein each stage includes a plurality of stacked CW2P modules; after the spatial and channel fusion features pass through a Neck layer, a classification head is used for classification to obtain a remote sensing image classification result. The method takes global and local feature fusion as a baseline, uses a pyramid pooling attention to extract global information, uses an improved local window attention module to extract local information, and then uses a linear attention module to interactively mix the global and local information, thereby retaining the feature map information in the channel dimension and extracting more image information in two dimensions. The method reduces the amount of calculation and the number of parameters, and improves the accuracy of remote sensing image classification. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 FIG. 1 is a flowchart of a remote sensing image classification method based on spatial and channel feature extraction according to an embodiment of the present application;

[0018] Figure 2 FIG. 4 is a structural diagram of a pyramid pooling attention module according to another embodiment of the present application;

[0019] Figure 3 FIG. 5 is a structural diagram of an improved local window attention module according to another embodiment of the present application;

[0020] Figure 4The overall structure of a remote sensing image classification network based on spatial and channel feature extraction in another embodiment;

[0021] Figure 5 The structure diagram of a channel attention module in another embodiment;

[0022] Figure 6 The structure diagram of a gated spatial attention unit in another embodiment;

[0023] Figure 7 The structure block diagram of a remote sensing image classification device based on spatial and channel feature extraction in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0025] Channel window pyramid pooling network, channel window pyramid pooling network, abbreviated as: CW2PNet.

[0026] Channel window pyramid pooling module, channel window pyramid pooling block, abbreviated as: CW2P module.

[0027] Channel window pyramid pooling multi-head self-attention module, channel window pyramid pooling multi-head self-attention, abbreviated as CW2P-MHSA module.

[0028] In one embodiment, as shown in Figure 1 A remote sensing image classification method based on spatial and channel feature extraction is provided, which comprises the following steps:

[0029] Step 100: obtaining a remote sensing image and pre-processing the remote sensing image.

[0030] Specifically, the remote sensing image can be, but is not limited to, grassland remote sensing image, field remote sensing image, industrial area remote sensing image, river and lake remote sensing image, forest remote sensing image, residential area remote sensing image, and parking lot remote sensing image.

[0031] Step 102: inputting the pre-processed remote sensing image into an input network to obtain sequence encoding features; the input network is used to perform sequence position encoding on the training sample after multiple deep convolutions.

[0032] Specifically, the input network is used to encode the training sample by a plurality of deep convolutions and then encode the sequence position by a deep convolution layer.

[0033] Step 104: input the sequence encoding feature into the feature extraction network to obtain the spatial and channel fusion feature.

[0034] The feature extraction network includes four stages, wherein the first stage includes a plurality of stacked CW2P modules, and the remaining three stages include a sequence embedding and position encoding layer and a plurality of stacked CW2P modules. As preferred, the first stage includes 2 stacked CW2P modules, the second stage includes 2 stacked CW2P modules, the third stage includes 6 stacked CW2P modules, and the fourth stage includes 2 stacked CW2P modules.

[0035] The CW2P module is used to extract local information of the sequence encoding feature by using an improved local window attention module, extract global information of the sequence encoding feature by using a pyramid pooling attention module, and interactively mix the global and local information by using a linear attention module. The outputs of the improved local window attention module, the pyramid pooling attention module and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimension by using a feedforward network including a gated spatial attention unit.

[0036] Specifically, the pyramid pooling refers to passing the feature map through different ratio average pooling layers to generate a pyramid feature map to extract global and context information for a specific task. The pyramid pooling is embedded into the pooling-based global feature extraction attention block of the remote sensing image classification network (CW2PNet) based on spatial and channel feature extraction to form a pyramid pooling attention module, which can simultaneously reduce the picture sequence length and learn powerful context feature representation. The structure of the pyramid pooling attention module is as shown in Figure 2 .

[0037] The ordinary vision Transformer (ViTs) has achieved success in image classification, but its computational cost will grow to the quadratic complexity of the image size. The local window-based method divides the image tokens into a plurality of non-overlapping windows by using local window attention, and performs self-attention within each window, and the computational complexity is linearly related to the image size. The improved local window attention module in the present application is multiplied by the channel information feature map X channel on the value tensor before the output of the local window attention module, so as to simultaneously model the intra-window and inter-window relationships to expand the receptive field and extract more features. The structure of the improved local window attention module is as shown in Figure 3 .

[0038] The limited receptive field and weak global dependency of the improved local window attention module determine the unsatisfactory classification results. Therefore, the key (K global ), value (V global ) global tokens generated by the pyramid pooling attention module are input into a linear attention block, and then the local query (Q local ) generated by the improved local window attention module is transmitted to the linear attention block, so as to realize the local-global information exchange attention through the linear attention.

[0039] The local window attention is used to extract local information. Since the local window attention mainly extracts information in the spatial dimension, a channel attention branch is added to supplement the information in the channel dimension and to aggregate it into the window attention to generate local feature representation. The pyramid pooling attention module is used to extract global information. Then, the linear attention module is used to interact and mix the global and local information, to retain the feature map information in the channel dimension, and to extract more image information in two dimensions.

[0040] Step 106: input the spatial and channel fusion features into the Neck layer for channel dimension expansion.

[0041] Specifically, the Neck layer is a fully connected layer.

[0042] Step 108: input the output of the Neck layer into the classification head for classification, to obtain the remote sensing image classification result.

[0043] Specifically, the classification head is composed of an average pooling layer and a fully connected layer.

[0044] The remote sensing image classification network based on spatial and channel feature extraction is composed of an input network, a feature extraction network, a Neck layer and a classification head. The overall structure diagram of the remote sensing image classification network based on spatial and channel feature extraction is shown in Figure 4 .

[0045] The remote sensing image classification method based on spatial and channel feature extraction includes the following steps: after the preprocessed remote sensing image is processed by multiple deep convolution in the input network, sequence position coding is performed, and then the coding result is input into the feature extraction network; in the feature extraction network, four stages are adopted to realize the extraction of spatial and channel fusion features, each stage includes a plurality of stacked CW2P modules; after the spatial and channel fusion features pass through the Neck layer, a classification head is used for classification to obtain the remote sensing image classification result. The method takes global and local feature fusion as the baseline, uses pyramid pooling attention to extract global information, uses an improved local window attention module to extract local information, and then uses a linear attention module to interact and mix global and local information, retaining the feature map information of the channel dimension, and extracting more image information in two dimensions. Using the method not only reduces the amount of calculation and the amount of parameters, but also improves the accuracy of remote sensing image classification.

[0046] In one of the embodiments, the input network includes: 3 deep convolutions, 1 sequence position coding layer; the sequence position coding layer includes 1 deep convolution layer.

[0047] In one of the embodiments, the CW2P module includes a CW2P-MHSA module and a feedforward network; the CW2P-MHSA module includes: a pyramid pooling attention module, an improved local window attention module, and a linear attention module; step 104 includes:

[0048] The sequence coding feature is input into the CW2P-MHSA module of the first CW2P module in the first stage to obtain a fused attention feature; the fused attention feature is:

[0049] X CW2P-MHSA =X+ LN(concat((X l o cal +X linear ),X gl o bal )) (1)

[0050] Wherein, X CW2P-MHSA is the fused attention feature, X is the sequence coding feature, and concat is a concatenation operation along the channel dimension of the feature map, wherein X global is the output of the pyramid pooling attention module, X local is the output of the improved local window attention module, and X linear is the output of the linear attention module.

[0051] The fused attention feature is input into a feedforward network of the first CW2P module in the first stage to obtain a first CW2P module output; the first CW2P module output is taken as input of a second CW2P module in the first stage, and the process is repeated until a last CW2P module in the first stage to obtain a first spatial and channel fused feature; the first spatial and channel fused feature is input into a second stage to obtain a second spatial and channel fused feature; the second spatial and channel fused feature is input into a third stage to obtain a third spatial and channel fused feature; and the third spatial and channel fused feature is input into a fourth stage to obtain a spatial and channel fused feature.

[0052] In one embodiment, as shown in FIG. 1, the improved local window attention module includes a deep convolutional layer, a local window attention module and a channel attention module; in the improved local window attention module of the CW2P-MHSA module of the first CW2P module in the first stage: Figure 3

[0053] The feature map input into the improved local window attention module is subjected to feature extraction by the deep convolutional layer; the extracted feature is input into the channel attention module to obtain a channel-dimension attention feature map; and the feature map input into the improved local window attention module and the channel-dimension are input into the local window attention module to obtain an output of the improved local window attention module as:

[0054]

[0055]

[0056] wherein Q local ,K local ,V local are respectively a query tensor, a key tensor and a value tensor of the improved local window attention module, X channel is the channel-dimension attention feature map, is a conversion matrix for generating the query tensor, the key tensor and the value tensor of the improved local window attention module, is a channel dimension of the key tensor of the improved local window attention module.

[0057] Specifically, a 7x7 window size is used in the local window to extract local information representation, and then multiplied by the channel information feature map X channel to expand the receptive field and extract more features. The whole process is expressed as formula (3), and then Q local ,K local ,V​local The input local window attention module calculates the output of the improved local window attention module, and the calculation expression is shown as formula (2).

[0058] In one of the embodiments, as shown in Figure 4 The channel attention module is composed of an average pooling layer, a first point convolution layer, a batch normalization layer, an activation function GELU, a second point convolution layer, and a sigmoid activation function. The extracted features are input into the channel attention module to obtain the channel-dimension attention feature map, including: the extracted features are input into the channel attention module, and are processed by the average pooling layer for pooling. Then, the first point convolution layer is used for dimension lifting, and the batch normalization layer is used for normalization. The normalized result is activated by the activation function GELU, and then is dimension reduced by the second point convolution layer. Finally, the channel-dimension attention feature map is output by the sigmoid activation function.

[0059] Specifically, the feature input into the improved local window attention module is input into the channel attention module after convolution by the depth convolution layer. The output information of the depth convolution is processed by the average pooling layer, and then is dimension lifted by the 1x1 ordinary convolution, is normalized by the batch normalization, is activated by the activation function GELU, is dimension reduced by the 1x1 ordinary convolution, and finally is output by the sigmoid activation function to obtain the channel-dimension attention feature map X channel , which is multiplied by the value (V local ) in the local window self-attention, adds the information extracted in the channel dimension to the attention extracted in the local window space dimension, and enriches the local detail feature representation of the feature map.

[0060] In one of the embodiments, in the linear attention module of the CW2P-MHSA module of the first CW2P module in the first stage:

[0061] The keys and values generated by the pyramid pooling attention module and the local query generated by the local window attention module are input into the linear attention module to obtain the output of the linear attention module as:

[0062]

[0063] , where K global is the key tensor generated by the pyramid pooling attention module, V global is the value tensor generated by the pyramid pooling attention module, Q local is the query tensor generated by the local window attention module, is the channel dimension of the key tensor generated by the pyramid pooling attention module.

[0064] Specifically, unlike the conventional visual transformer (VIT), the application does not apply dot product along the spatial dimension between the query Q and the value K T , i.e. (HWxC)·(CxHW), where C is the number of feature map channels, H is the height of the feature map, and W is the width of the feature map, but applies dot product along the channel dimension between Q T and K, i.e. (CxHW)·(HWxC), to generate a CxC softmax scaled attention score matrix, so as to retain more information of the channel dimension. In order to obtain the final attention map, the softmax scaled attention score is multiplied by V to obtain the linear attention module output, and the expression of the linear attention module output is shown as formula (4).

[0065] In one embodiment, the structure of the pyramid pooling attention module is shown as Figure 3 In the pyramid pooling attention module of the CW2P-MHSA module of the first CW2P module in the first stage:

[0066] The feature map input into the pyramid pooling attention module is processed by different scale pooling layers, normalized, and spliced to obtain a feature P as:

[0067] P = LN(concat(AvgPool1(X), AvgPool2(X), …, AvgPool n (X)) (5)

[0068] wherein AvgPool1, AvgPool2, …, AvgPool n are n different scale pooling layers; LN is a normalization operation;

[0069] The query tensor, the key tensor, and the value tensor of the multi-head attention module are calculated, and the formula expressions of the query tensor, the key tensor, and the value tensor are as follows:

[0070] (Q gl o bal ,K gl o bal ,V gl o bal )=(XW gl o bal q ,PW gl o bal k ,PW gl o bal v ) (6)

[0071] wherein Q global ,Kglobal ,V global query tensor, key tensor and value tensor generated by the pyramid pooling attention module, W global q query tensor, key tensor and value tensor generated by the pyramid pooling attention module, W global k query tensor, key tensor and value tensor generated by the pyramid pooling attention module, W global v query tensor, key tensor and value tensor generated by the pyramid pooling attention module, W

[0072] The query tensor, the key tensor and the value tensor of the multi-head attention module are input into the multi-head attention module to obtain the output of the pyramid pooling attention module, and the output expression of the pyramid pooling attention module is as follows:

[0073]

[0074] wherein, is the channel dimension of the key tensor generated by the pyramid pooling attention module.

[0075] Specifically, the specific process of the pyramid pooling attention module is as follows: first, the feature map X input into the pyramid pooling attention module is reshaped into a two-dimensional space; then, n average pooling layers with different scales are applied to the reshaped feature map X to generate a pyramid feature map.

[0076] The feature maps after the n different scale pooling layers are normalized and spliced to form P. In this way, if the pooling scale is large enough, P can be a shorter sequence than the feature map X input into the pyramid pooling attention module. In addition, P contains the context abstraction of the input X, so it can be used as a strong substitute for the input X when calculating the pyramid pooling attention module.

[0077] Suppose the query tensor, the key tensor and the value tensor in the pyramid pooling attention module are Q gl o bal ,K gl o bal and V gl o bal , and the formula is expressed as (Q, K, V) = (XW q ,PW k ,PW v ), wherein W q , W k and W v represent the weight matrix of the linear transformation for generating the query tensor, the key tensor and the value tensor. Then, Q, K and V are input into the attention module to calculate the global attention, and the output of the pyramid pooling attention module is obtained, and the output expression of the pyramid pooling attention module is as shown in formula (7).

[0078] In one of the embodiments, in the feedforward network of the CW2P-MHSA module of the first CW2P module in the first stage: the fused attention features output by the CW2P-MHSA module are input into a gated spatial attention unit after normalization, the result after point convolution and deep convolution in the first parallel branch is multiplied by the result after point convolution in the second parallel branch, and the multiplied result is point-convoluted and added to the fused attention features output by the CW2P-MHSA module to obtain the output of the CW2P module. The structure of the gated spatial attention unit (GSAU) is as shown in Figure 6

[0079] Specifically, the gated spatial attention unit (GSAU) implements an adaptive gating mechanism to reduce the number of parameters and the amount of calculation. After the X CW2P-MHSA input is processed by two parallel branches of 1x1 ordinary convolution, one branch adopts a single-layer deep convolution to weight the feature map for more effective acquisition of spatial information, and then the result is multiplied by the result of the other branch. After output, a 1x1 convolution is performed, a short connection operation is performed with the X CW2P-MHSA , and finally the output result is obtained. The gated attention unit block (GSAU) is combined with the layer normalization (LN) to form the FFN part as a multi-layer perception (MLP) block. The specific expression is as follows:

[0080] X CW2P_out = X CW2P-MHSA + GSAU (LN (X CW2P-MHSA )) (8)

[0081] X CW2P_out is the output of the CW2P module.

[0082] In one of the embodiments, the sequence embedding and position encoding layer includes one convolution layer and one deep convolution layer.

[0083] It should be understood that, although the steps in the flowchart of Figure 1 are shown in sequence according to the arrows, these steps are not necessarily executed in sequence according to the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, Figure 1 at least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with other steps or sub-steps or stages of other steps.

[0084] In one verification embodiment, the CW2PNet overall architecture is shown in Figure 4 Fig. 1, which is divided into a CW2P multi-head self-attention part (CW2P-MHSA) and a feed-forward network part (FFN) according to a transformer structure, the FFN is composed of a layer normalization (LN) and a GSAU block, the feature extraction network of the CW2PNet is composed of four stages, each stage stacks different numbers of CW2P modules, specifically, the input feature map first passes through a stem module composed of 3 deep convolutions, which down-samples the features by 4 times to extract local features; this block can be regarded as the first block sequence embedding block, then the sequence position encoding is performed through the PE layer, patch&PE is used for sequence embedding and position encoding in stages 2, 3, and 4; then the channel dimension of the feature extraction network output is expanded through the Neck layer, and finally the classification result is output through the classification head, the specific parameter settings of the network model are shown in Table 1, and the experimental environment settings are shown in Table 2.

[0085] Table 1 Network architecture parameter settings

[0086]

[0087]

[0088] Table 2 Experimental environment configuration

[0089]

[0090] In this embodiment, the data set uses the remote sensing image data set RSSCN7 released by Wuhan University, which contains 7 different remote sensing image scenes, namely grass (Grass), field (Field), industrial area (Industry), river and lake (RiverLake), forest (Forest), residential area (Resident), and parking lot (Parking), a total of 2800 RGB images. The data set is randomly divided into two types of training set and test set, one of which is used to train the network, and the other is used to test the performance of the network model, and the image ratio is 8:2. The experimental results obtained by using the data set for experiment are shown in Table 3.

[0091] Table 3 Experimental results

[0092]

[0093] In one embodiment, as shown in Figure 7 , a remote sensing image classification device based on spatial and channel feature extraction is provided, which includes a remote sensing image preprocessing module, a remote sensing image local feature extraction module, a spatial and channel fusion feature extraction module, and a classification module, wherein:

[0094] a remote sensing image preprocessing module, configured to acquire a remote sensing image and pre-process the remote sensing image;

[0095] a remote sensing image local feature extraction module, configured to input the pre-processed remote sensing image into an input network to obtain sequence encoding features; the input network is configured to perform sequence position encoding on a training sample after multiple depth convolutions;

[0096] a spatial and channel fusion feature extraction module, configured to input the sequence encoding features into a feature extraction network to obtain spatial and channel fusion features; the feature extraction network includes four stages, wherein the first stage includes a plurality of stacked CW2P modules, and the remaining three stages include a sequence embedding and position encoding layer and a plurality of stacked CW2P modules; the CW2P module is configured to extract local information of the sequence encoding features using an improved local window attention module, extract global information of the sequence encoding features using a pyramid pooling attention module, and interactively mix the global and local information using a linear attention module; the outputs of the improved local window attention module, the pyramid pooling attention module, and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimensions using a feedforward network including a gated spatial attention unit;

[0097] a classification module, configured to input the spatial and channel fusion features into a Neck layer for channel dimension expansion; input the output of the Neck layer into a classification head for classification to obtain a remote sensing image classification result.

[0098] In one of the embodiments, the input network in the remote sensing image local feature extraction module includes 3 depth convolutions and 1 sequence position encoding layer; the sequence position encoding layer includes 1 depth convolution layer.

[0099] In one of the embodiments, the CW2P module includes a CW2P-MHSA module and a feedforward network; the CW2P-MHSA module includes a pyramid pooling attention module, an improved local window attention module, and a linear attention module; the sequence encoding features are input into the CW2P-MHSA module of the first CW2P module in the first stage to obtain fused attention features; the expression of the fused attention features is shown in equation (1).

[0100] input the fused attention feature into a feedforward network of a first CW2P module in the first stage to obtain a first CW2P module output; take the first CW2P module output as input of a second CW2P module in the first stage, and take the output of the second CW2P module as input of a third CW2P module in the first stage, and so on until the last CW2P module in the first stage, to obtain a first spatial and channel fused feature; input the first spatial and channel fused feature into the second stage to obtain a second spatial and channel fused feature; input the second spatial and channel fused feature into the third stage to obtain a third spatial and channel fused feature; and input the third spatial and channel fused feature into the fourth stage to obtain a spatial and channel fused feature.

[0101] In one of the embodiments, the improved local window attention module includes a depth convolution layer, a local window attention module and a channel attention module; in the improved local window attention module of the CW2P-MHSA module of the first CW2P module in the first stage:

[0102] The feature map input to the improved local window attention module is subjected to feature extraction by the depth convolution layer; the extracted feature is input into the channel attention module to obtain a channel-dimension attention feature map; and the feature map input to the improved local window attention module and the channel dimension are input into the local window attention module to obtain an expression of the output of the improved local window attention module as shown in formula (2) and formula (3).

[0103] In one of the embodiments, the channel attention module is composed of an average pooling layer, a first point convolution layer, a batch normalization layer, an activation function GELU, a second point convolution layer and a sigmoid activation function; and the spatial and channel fused feature extraction module is further configured to input the extracted feature into the channel attention module, perform pooling processing by the average pooling layer, then perform dimension lifting by the first point convolution layer, perform normalization processing by the batch normalization layer, activate the obtained normalized result by the activation function GELU, perform dimension reduction by the second point convolution layer, and finally output a channel-dimension attention feature map by the sigmoid activation function.

[0104] In one of the embodiments, the spatial and channel fused feature extraction module is further configured to, in the linear attention module of the CW2P-MHSA module of the first CW2P module in the first stage:

[0105] The key and value generated by the pyramid pooling attention module and the local query generated by the local window attention module are input into the linear attention module to obtain an expression of the output of the linear attention module as shown in formula (4).

[0106] In one of the embodiments, the spatial and channel fusion feature extraction module is further configured to, in the pyramid pooling attention module of the CW2P-MHSA module of the first CW2P module in the first stage:

[0107] The feature maps input into the pyramid pooling attention module are processed by different ratio pooling layers, normalized, and spliced to obtain the expression of the feature P, as shown in equation (5).

[0108] The query tensor, the key tensor, and the value tensor of the multi-head attention module are calculated, and the formula expressions of the query tensor, the key tensor, and the value tensor are as shown in equation (6).

[0109] The query tensor, the key tensor, and the value tensor of the multi-head attention module are input into the multi-head attention module to obtain the output of the pyramid pooling attention module, and the expression of the output of the pyramid pooling attention module is as shown in equation (7).

[0110] In one of the embodiments, the spatial and channel fusion feature extraction module is further configured to, in the feedforward network of the CW2P-MHSA module of the first CW2P module in the first stage: input the normalized fusion attention features output by the CW2P-MHSA module into the gated spatial attention unit, multiply the results processed by the point convolution and the depth convolution in the first parallel branch with the results processed by the point convolution in the second parallel branch, and add the multiplied results to the fusion attention features output by the CW2P-MHSA module after point convolution to obtain the output of the CW2P module.

[0111] In one of the embodiments, the sequence embedding and position encoding layer in the spatial and channel fusion feature extraction module includes one convolution layer and one depth convolution layer.

[0112] For specific limitations of the remote sensing image classification device based on spatial and channel feature extraction, refer to the limitations of the remote sensing image classification method based on spatial and channel feature extraction in the above, which will not be described here. Each module in the above remote sensing image classification device based on spatial and channel feature extraction can be realized by software, hardware, and combinations thereof, in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0113] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

[0114] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a more specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A remote sensing image classification method based on spatial and channel feature extraction, characterized in that, The method comprises: acquiring a remote sensing image, and preprocessing the remote sensing image; inputting the preprocessed remote sensing image into an input network to obtain sequence encoding features; the input network is used for performing sequence position encoding on the preprocessed remote sensing image after multiple deep convolutions; inputting the sequence encoding features into a feature extraction network to obtain spatial and channel fusion features; the feature extraction network comprises four stages, wherein the first stage comprises a plurality of stacked CW2P modules, and the remaining three stages comprise a sequence embedding and position encoding layer and a plurality of stacked CW2P modules; the CW2P module is used for extracting local information of the sequence encoding features by using an improved local window attention module, extracting global information of the sequence encoding features by using a pyramid pooling attention module, and interacting and mixing the global and local information by using a linear attention module; the outputs of the improved local window attention module, the pyramid pooling attention module and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimensions by using a feedforward network comprising a gated spatial attention unit; inputting the spatial and channel fusion features into a Neck layer for channel dimension expansion; inputting the output of the Neck layer into a classification head for classification to obtain a remote sensing image classification result; wherein the CW2P module comprises a CW2P-MHSA module and a feedforward network; the CW2P-MHSA module comprises a pyramid pooling attention module, an improved local window attention module and a linear attention module; inputting the sequence encoding features into the feature extraction network to extract features by using the four stages to obtain spatial and channel fusion features, comprising: inputting the sequence encoding features into the CW2P-MHSA module of the first CW2P module in the first stage to obtain fused attention features; the fused attention features are: wherein, is a fused attention feature, X is a sequence encoding feature, is a concatenation operation along the channel dimension of the feature maps, wherein is an output of the pyramid pooling attention module, is an output of the improved local window attention module, is an output of the linear attention module; inputting the fused attention features into the feedforward network of the first CW2P module in the first stage to obtain the output of the first CW2P module; taking the output of the first CW2P module as the input of the second CW2P module in the first stage, and similarly until the last CW2P module in the first stage to obtain the first spatial and channel fusion features; inputting the first spatial and channel fusion features into the second stage to obtain the second spatial and channel fusion features; inputting the second spatial and channel fusion features into the third stage to obtain the third spatial and channel fusion features; inputting the third spatial and channel fusion features into the fourth stage to obtain the spatial and channel fusion features.

2. The method of claim 1, wherein, The input network comprises three deep convolutions and one sequence position encoding layer; the sequence position encoding layer comprises one deep convolution layer.

3. The method of claim 1, wherein, The improved local window attention module comprises a deep convolution layer, a local window attention module and a channel attention module; In the improved local window attention module of the CW2P-MHSA module of the first CW2P module in the first stage: The feature map input into the improved local window attention module is subjected to feature extraction by a deep convolution layer; The extracted features are input into a channel attention module to obtain a channel-dimension attention feature map; The feature map input into the improved local window attention module and the channel-dimension are input into a local window attention module, and the output of the improved local window attention module is: wherein, are a query tensor, a key tensor and a value tensor of the improved local window attention module, respectively, is an attention feature map of the channel dimension, are transformation matrices for generating the query tensor, the key tensor and the value tensor of the improved local window attention module, is a channel dimension of the key tensor of the improved local window attention module.

4. The method of claim 3, wherein, The channel attention module is composed of an average pooling layer, a first point convolution layer, a batch normalization layer, an activation function GELU, a second point convolution layer, and a sigmoid activation function; The extracted features are input into a channel attention module to obtain a channel-dimension attention feature map, including: The extracted features are input into a channel attention module, subjected to pooling processing by an average pooling layer, and then subjected to dimension lifting by a first point convolution layer, normalization processing by a batch normalization layer, activation processing by an activation function GELU, dimension reduction by a second point convolution layer, and finally output of a channel-dimension attention feature map by a sigmoid activation function.

5. The method of claim 3, wherein, In the linear attention module of the CW2P-MHSA module of the first CW2P module in the first stage: The keys and values generated by the pyramid pooling attention module and the local queries generated by the local window attention module are input into the linear attention module, and the output of the linear attention module is: wherein, a key tensor generated for the pyramid-pooling attention module, a value tensor generated for the pyramid-pooling attention module, a query tensor generated for the local window attention module, a channel dimension of the key tensor generated for the pyramid-pooling attention module.

6. The method of claim 1, wherein, In the pyramid pooling attention module of the CW2P-MHSA module of the first CW2P module in the first stage: The feature map input to the pyramid pooling attention module is processed by different proportion pooling layers, normalized, and spliced to obtain a feature P is: wherein, , , is n different proportion pooling layers; is a normalization operation; The query tensor, key tensor and value tensor of the multi-head attention module are calculated, and the formula expressions of the query tensor, key tensor and value tensor are: wherein, query tensors, key tensors and value tensors generated for the pyramid-pooling attention module, respectively represent weight matrices of linear transformations used to generate the query tensors, the key tensors and the value tensors. The query tensor, key tensor and value tensor of the multi-head attention module are input into the multi-head attention module to obtain the output of the pyramid pooling attention module, and the output expression of the pyramid pooling attention module is: wherein, Channel dimension of the key tensor generated for the pyramid-pooling attention module.

7. The method of claim 1, wherein, In the feedforward network of the CW2P-MHSA module of the first CW2P module in the first stage: The normalized fusion attention features output by the CW2P-MHSA module are input into the gated spatial attention unit, the results processed by the point convolution and deep convolution in the first parallel branch are multiplied by the results processed by the point convolution in the second parallel branch, the multiplied results are subjected to point convolution, and then added to the fusion attention features output by the CW2P-MHSA module to obtain the output of the CW2P module.

8. The method of claim 1, wherein, The sequence embedding and position encoding layer includes one convolution layer and one deep convolution layer.

9. A remote sensing image classification apparatus based on spatial and channel feature extraction, characterized in that, The device comprises: A remote sensing image preprocessing module is configured to acquire a remote sensing image and preprocess the remote sensing image; A remote sensing image local feature extraction module is configured to input the preprocessed remote sensing image into an input network to obtain sequence encoding features; the input network is configured to perform sequence position encoding on the preprocessed remote sensing image after multiple deep convolutions; The spatial and channel fusion feature extraction module is configured to input the sequence encoding feature into a feature extraction network to obtain spatial and channel fusion features; the feature extraction network includes four stages, wherein the first stage includes a plurality of stacked CW2P modules, and the remaining three stages include a sequence embedding and position encoding layer and a plurality of stacked CW2P modules; the CW2P module is configured to extract local information of the sequence encoding feature by using an improved local window attention module, extract global information of the sequence encoding feature by using a pyramid pooling attention module, and interactively mix the global and local information by using a linear attention module; the outputs of the improved local window attention module, the pyramid pooling attention module, and the linear attention module are fused, and the fused results are aggregated in the channel and spatial dimensions by using a feedforward network including a gated spatial attention unit; The classification module is configured to input the spatial and channel fusion features into a Neck layer for channel dimension expansion; and input the output of the Neck layer into a classification head for classification to obtain a remote sensing image classification result. The CW2P module includes a CW2P-MHSA module and a feedforward network; the CW2P-MHSA module includes a pyramid pooling attention module, an improved local window attention module, and a linear attention module; the spatial and channel fusion feature extraction module is further configured to input the sequence encoding feature into the CW2P-MHSA module of the first CW2P module in the first stage to obtain fused attention features; the fused attention features are: wherein, is a fused attention feature, X is a sequence encoding feature, is a concatenation operation along the channel dimension of the feature maps, wherein is an output of the pyramid pooling attention module, is an output of the improved local window attention module, is an output of the linear attention module; The fused attention features are input into the feedforward network of the first CW2P module in the first stage to obtain a first CW2P module output; the first CW2P module output is taken as an input of a second CW2P module in the first stage, and the process is repeated until the last CW2P module in the first stage to obtain a first spatial and channel fusion feature; the first spatial and channel fusion feature is input into a second stage to obtain a second spatial and channel fusion feature; the second spatial and channel fusion feature is input into a third stage to obtain a third spatial and channel fusion feature; and the third spatial and channel fusion feature is input into a fourth stage to obtain the spatial and channel fusion features. The classification module is configured to input the spatial and channel fusion features into a Neck layer for channel dimension expansion; and input the output of the Neck layer into a classification head for classification to obtain a remote sensing image classification result. The CW2P module includes a CW2P-MHSA module and a feedforward network; the CW2P-MHSA module includes a pyramid pooling attention module, an improved local window attention module, and a linear attention module; the spatial and channel fusion feature extraction module is further configured to input the sequence encoding feature into the CW2P-MHSA module of the first CW2P module in the first stage to obtain fused attention features; the fused attention features are: The fused attention features are input into the feedforward network of the first CW2P module in the first stage to obtain a first CW2P module output; the first CW2P module output is taken as an input of a second CW2P module in the first stage, and the process is repeated until the last CW2P module in the first stage to obtain a first spatial and channel fusion feature; the first spatial and channel fusion feature is input into a second stage to obtain a second spatial and channel fusion feature; the second spatial and channel fusion feature is input into a third stage to obtain a third spatial and channel fusion feature; and the third spatial and channel fusion feature is input into a fourth stage to obtain the spatial and channel fusion features.

Images