An attention enhancement and dense multi-scale based feature classification network model
By introducing the DenseASPP-SE module and DAM into the DeepLabv3+ model, the problems of insufficient receptive field and inadequate utilization of feature maps in remote sensing image land cover classification are solved, improving classification accuracy and refinement, and achieving more efficient semantic segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2023-05-23
- Publication Date
- 2026-05-29
Smart Images
Figure CN116630704B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer image processing and is based on deep learning methods to classify ground features in remote sensing images. Background Technology
[0002] In recent decades, remote sensing image land cover classification technology has achieved significant developments and generated widespread positive impacts in practical applications, becoming an important field of remote sensing technology application. However, with the large-scale acquisition of remote sensing data and the continuous improvement of high-resolution observation technology, the challenge of a sharp increase in the diversity of data volume and data types has emerged. This not only places higher demands on the application of remote sensing image land cover classification methods but also brings greater challenges to the development of remote sensing technology. Although remote sensing image land cover classification methods have become relatively mature thanks to the advancement of deep learning technology, the sheer number and variety of land cover types in remote sensing images, covering multiple categories such as urban areas, agriculture, forests, grasslands, and water bodies, along with the complex morphology and texture of land cover with varying scales and orientations, makes classification even more difficult. Furthermore, remote sensing images are typically large-scale datasets requiring block processing, which can lead to errors in land cover classification at block boundaries. Therefore, due to these issues, deep learning-based network models have shown poor segmentation performance for land cover identification in high-resolution remote sensing images, and the level of refinement urgently needs improvement. Currently, DeepLabv3+ is a deep semantic segmentation model based on an encoder-decoder architecture, demonstrating superior performance in image semantic segmentation. DeepLabv3+ proposes an Atrous-convolution Space Pyramid Pooling (ASPP) structure, which increases the receptive field of Convolutional Neural Networks (CNNs), enabling a better understanding of image context. To further improve model efficiency and accuracy, DeepLabv3+ employs global pooling, reducing the number of model parameters while better capturing global image information. In the decoder stage, DeepLabv3+ combines shallow and deep information through upsampling and skip connections, further enhancing semantic segmentation accuracy. High resolution is a key characteristic of remote sensing images. High-precision ground feature separation using deep learning requires CNNs with a sufficiently large receptive field. The atrous convolutions in the ASPP structure of DeepLabv3+ can expand the receptive field using a sufficiently large dilation rate. However, as the dilation rate increases to 24, it negatively impacts the atrous convolutions, even causing them to lose their modeling ability. Meanwhile, in the encoder stage, DeepLabv3+ does not fully utilize the multi-level feature maps generated by the encoder. Instead, it directly performs a fourfold bilinear upsampling of the feature maps and fuses them with a low-resolution feature map, which is detrimental to the acquisition of pixel-level information from high-resolution remote sensing images. Therefore, this invention improves the network to address these issues. Summary of the Invention
[0003] To overcome the shortcomings of existing technologies, this invention discloses a method for constructing a ground cover classification network model based on attention enhancement and dense multi-scale methods. Building upon DeepLabv3+, this invention improves the model structure to address the deficiencies in its encoder and decoder stages. In the encoder stage, this paper designs a densely-connected atrous spatial pyramid pooling (DenseASPP-SE) with Squeeze and Excitation to replace the original atrous spatial pyramid pooling structure. Simultaneously, a dual attention mechanism (DAM) is introduced in the extraction of global features to help achieve a balance between feature representation capability and spatial positioning accuracy of remote sensing images. In the encoder stage, a dense connection approach is used to upsample the image using multi-level feature maps. The technical route of this invention is as follows: Figure 1 As shown.
[0004] The technical steps employed in this invention are as follows:
[0005] Step 1: Use ResNet-101 to extract features from the RGB three-channel image to ensure the network's information capture capability;
[0006] Step 2: The DenseASPP-SE module is then used to obtain a larger receptive field, enabling more accurate differentiation of details between different land cover categories. DenseASPP-SE consists of a densely-connected atrous spatial pooling (DenseASPP) layer and a global feature enhancement layer. Its output is then subjected to feature aggregation and channel compression, such as... Figure 2 As shown;
[0007] Step 3: After obtaining the enhanced global features, DAM is introduced, which can effectively extract useful information from remote sensing images. The DAM implementation refines the output features of DenseASPP-SE through Channel Attention Mechanism (CAM) and Spatial Attention Mechanism (SAM). CAM weights the features of each channel, strengthening important channel features and suppressing unimportant ones; SAM weights the features of each pixel, strengthening features corresponding to important pixels and suppressing features corresponding to unimportant pixels. The features processed by CAM and SAM are then aggregated and compressed into channels, and further refined using CAM, such as... Figure 3 As shown;
[0008] Step 4: In the decoder, the output of the fourth stage of the backbone network is fused with the output of the third stage of the backbone network to form feature map F1. The output of the fourth stage of the backbone network is fused with the output feature map F3 of DAM to form F2. Then, the resolution of feature maps F2 and F3 is upsampled to the same size as the resolution of feature map F1 and then the features are aggregated. Finally, semantic information is discriminated to achieve land cover classification of remote sensing images.
[0009] Furthermore, the specific method for step 2 is as follows:
[0010] Step 2.1: The feature information X extracted in Step 1 is passed through the DenseASPP-SE module, specifically through the DenseASPP module and the global feature enhancement layer module.
[0011] Step 2.2: In DenseASPP, the input feature vector X is first subjected to a dilated convolution with a dilation rate of 3 to obtain feature map Y1. Then Y1 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 6 to obtain Y2. Then Y1, Y2 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 12 to obtain Y3. After that, Y1, Y2, Y3 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 18 to obtain Y4.
[0012] Step 2.3: In the global feature enhancement layer, feature refinement is performed through activation and compression operations to obtain the weighted feature map Y5;
[0013] Step 2.4: After feature aggregation of feature maps Y1, Y2, Y3, Y4, and Y5, channel compression is performed through a 1×1 convolutional layer to obtain feature map Y.
[0014] Furthermore, the specific method for step 3 is as follows:
[0015] Step 3.1: First, process the feature map Y obtained in Step 2 through SAM and CAM in the DAM module;
[0016] Step 3.2: As Figure 4 As shown, in SAM, the input feature map is Y∈R C×H×W Here, C is the number of channels, and H and W are the height and width of the feature map, respectively. Three convolutions with the same structure are applied to create three new features: a query vector, a key vector, and a value vector, represented by vectors Q, K, and V, with dimensions C×H×W. These three are the same value, key, and query vectors. To obtain the similarity matrix, the i-th key and j-th query are first calculated. Then, V and K are modified to C×N, where N has dimensions H×W. A softmax layer is then used to obtain the normalized spatial attention weights S1∈R. C´×H×W ;
[0017] Step 3.3: For the input feature map Y, the spatial attention weight is obtained by multiplying the spatial location information and the attention weight S1, and then multiplied by the scale parameter α. Then, the sum is performed element-wise with the input feature map Y to obtain the feature map after spatial feature refinement.
[0018] Step 3.4: As Figure 5 As shown, in the CAM module, for the input feature map Y∈R C×H×W CAM first reshapes the input feature Y into a size C×H×W, making it C×N, where N is H×W. Then, it transposes X to obtain a channel attention module of size N×N, and uses the Softmax function to obtain the spatial attention weights S2∈R. C´×H×W ;
[0019] Step 3.5: Input feature map X, and obtain the channel attention-weighted feature map by multiplying the elements in each channel with the attention weights of the corresponding channels;
[0020] Step 3.6: After concatenating the feature map refined from spatial features and the feature map refined from channel features along the channel, perform channel compression, and then use the channel attention mechanism to obtain the refined feature map F3.
[0021] Furthermore, the specific method for step 4 is as follows:
[0022] Step 4.1: As Figure 6 As shown, the resolution of the output feature map of Stage 4 is upsampled to the same size as the resolution of the output feature map of Stage 3, and then the features are aggregated. The aggregated feature map is then refined by two 3×3 convolutional layers with BN layers and ReLU activation functions to obtain feature map F1.
[0023] Step 4.2: Upsample the resolution of the output feature map in Step 3 to the same size as the resolution of the output feature map in Stage 4, and then perform feature aggregation. The aggregated feature map is then refined by two 3×3 convolutional layers with BN and ReLU activation functions to obtain feature map F2.
[0024] Step 4.3: After upsampling F3 by 4 times and F2 by 2 times, the two samples are concatenated with F1 along the channel and then refined through a 1×1 convolutional layer with BN and ReLU activation functions. Finally, semantic information is discriminated to achieve the classification of remote sensing ground objects.
[0025] Compared with the prior art, the beneficial effects of the present invention are:
[0026] (1) The present invention designs a DenseASPP-SE module to replace the ASPP structure of DeepLabv3+, thereby increasing the receptive field of CNN for high-resolution remote sensing images and better understanding the contextual scene information of the images;
[0027] (2) The present invention introduces DAM after obtaining enhanced global features, which can help the land cover classification model achieve a balance between feature representation capability and spatial positioning accuracy, and realize more refined classification of target areas in remote sensing images;
[0028] (3) In the decoder stage, the present invention uses a dense connection method to upsample the image by utilizing the multi-level feature map of the image, making full use of the multi-scale information of the image and improving the accuracy of the ground feature classification of the image. Attached Figure Description
[0029] Figure 1 This invention presents the network structure of a land cover classification network model based on attention enhancement and dense multi-scale.
[0030] Figure 2 DenseASPP-SE network structure
[0031] Figure 3 The dual attention mechanism module structure of the present invention
[0032] Figure 4 The spatial attention mechanism flow of the present invention
[0033] Figure 5 The channel attention mechanism flow of the present invention
[0034] Figure 6 The feature fusion module of the present invention Detailed Implementation
[0035] The invention will be further described below with reference to the accompanying drawings.
[0036] This invention designs a ground cover classification network model based on attention enhancement and dense multi-scale. Building upon the DeepLabv3+ network, an attention-enhanced dense hollow pyramid pooling model is designed to replace the original hollow spatial pyramid pooling structure. Simultaneously, a dual-attention mechanism is introduced afterward. In the encoder stage, a dense connection approach is used to upsample the image using multi-level feature maps, such as... Figure 1 The diagram shows the network structure of this invention.
[0037] First, ResNet-101 is used as the backbone network to encode semantic information and establish a feature extraction module.
[0038] Then, the input consists of a densely connected dilated spatial pooling layer and a global feature enhancement layer, and the output is subjected to feature aggregation and channel compression by the DenseASPP-SE module, such as... Figure 2 As shown, this is done to increase the receptive field of CNNs for high-resolution remote sensing images, and to better understand the contextual scene information of the images.
[0039] Then, DAM is used to further improve the representational power of the feature maps. For example... Figure 3 As shown, the input features are processed by SAM and CAM respectively, as follows: Figure 4 As shown, SAM weights the features of each pixel, thus strengthening the features of important pixels and suppressing the features of unimportant pixels. Figure 5 As shown, CAM weights the features of each channel, enhancing important channel features and suppressing unimportant ones. The refined features are then concatenated along the channels and compressed, before being further refined using CAM to enhance the channel feature information.
[0040] Finally, feature fusion is performed. For example... Figure 6 As shown, this module first uses bilinear interpolation to upsample the resolution of high-level features to the same size as the resolution of low-level feature maps. Then, it refines the upsampled high-level feature maps using a 3×3 convolutional layer with BN and ReLU activation functions. Next, it fuses the output of the fourth stage of the backbone network with the output of the third stage to form feature map F1. It then fuses the output of the fourth stage of the backbone network with the output feature map F3 of the DAM to form F2. A feature aggregation operation concatenates the refined high-level and low-level feature maps to obtain the fused feature map. Finally, a 1×1 convolutional layer with BN and ReLU activation functions is used to obtain the final output feature map of the feature fusion module.
Claims
1. A method for constructing a land cover classification network model based on attention enhancement and dense multi-scale, characterized in that, Includes the following steps: Step 1: Use ResNet-101 to extract features from the RGB three-channel image to ensure the network's information capture capability; Step 2: Then, the DenseASPP-SE module is used to obtain a larger receptive field, which can more accurately distinguish the details between different land cover categories. DenseASPP-SE consists of a densely connected hollow spatial pooling layer DenseASPP and a global feature enhancement layer. After that, its output is subjected to feature aggregation and channel compression. Step 2.1: The feature information X extracted in Step 1 is passed through the DenseASPP-SE module, specifically through the DenseASPP module and the global feature enhancement layer module. Step 2.2: In DenseASPP, the input feature vector X is first subjected to a dilated convolution with a dilation rate of 3 to obtain feature map Y1. Then Y1 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 6 to obtain Y2. Then Y1, Y2 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 12 to obtain Y3. After that, Y1, Y2, Y3 and X are concatenated and passed through a dilated convolution layer with a dilation rate of 18 to obtain Y4. Step 2.3: In the global feature enhancement layer, feature refinement is performed through activation and compression operations to obtain the weighted feature map Y5; Step 2.4: After feature aggregation of feature maps Y1, Y2, Y3, Y4, and Y5, channel compression is performed through a 1×1 convolutional layer to obtain feature map Y; Step 3: After obtaining the enhanced global features, DAM is introduced, which can effectively extract useful information from remote sensing images. The implementation of DAM refines the output features of DenseASPP-SE through the channel attention mechanism CAM and the spatial attention mechanism SAM. CAM strengthens important channel features and suppresses unimportant channel features by weighting the features of each channel. SAM strengthens the features corresponding to important pixels and suppresses the features corresponding to unimportant pixels by weighting the features of each pixel. The features processed by CAM and SAM are aggregated and then channel compressed, and the channel features are refined again by CAM. Step 3.1: First, process the feature map Y obtained in Step 2 through SAM and CAM in the DAM module; Step 3.2: In SAM, the input feature map is Y∈R C×H×W Where C is the number of channels, and H and W are the height and width of the feature map, respectively; three convolutions with the same structure are applied to create three new features, namely the query vector, the key vector, and the value vector, which are represented by vectors Q, K, and V, with dimensions C×H×W. These three are the same value, key, and query vectors; in order to obtain the similarity matrix, the i-th key and the j-th query are first calculated, and then V and K are modified to C×N, where the dimension of N is H×W. Through the Softmax layer, the normalized spatial attention weights S1 are obtained. Step 3.3: For the input feature map Y, the spatial attention weight is obtained by multiplying the spatial location information and the attention weight S1, and then multiplied by the scale parameter α. Then, the sum is performed element-wise with the input feature map Y to obtain the feature map after spatial feature refinement. Step 3.4: In the CAM module, for the input feature map, CAM first reshapes the input feature Y by size C×H×W to make it C×N, where the dimension of N is H×W; then X is transposed to obtain a channel attention module of size N×N, and the spatial attention weight S2 is obtained by using the Softmax function. Step 3.5: Input feature map X, and obtain the channel attention-weighted feature map by multiplying the elements in each channel with the attention weights of the corresponding channels; Step 3.6: After concatenating the feature map refined from spatial features and the feature map refined from channel features along the channel, perform channel compression, and then use the channel attention mechanism to obtain the refined feature map F3; Step 4: In the decoder, the output of the fourth stage of the backbone network is fused with the output of the third stage of the backbone network to form feature map F1. The output of the fourth stage of the backbone network is fused with the output feature map F3 of DAM to form F2. Then, the resolution of feature maps F2 and F3 is upsampled to the same size as the resolution of feature map F1 and then the features are aggregated. Finally, semantic information is discriminated to achieve land cover classification of remote sensing images. Step 4.1: Upsample the resolution of the output feature map of Stage 4 to the same size as the resolution of the output feature map of Stage 3, and then perform feature aggregation. The aggregated feature map is then refined by two 3×3 convolutional layers with BN and ReLU activation functions to obtain feature map F1. Step 4.2: Upsample the resolution of the output feature map in Step 3 to the same size as the resolution of the output feature map in Stage 4, and then perform feature aggregation. The aggregated feature map is then refined by two 3×3 convolutional layers with BN and ReLU activation functions to obtain feature map F2. Step 4.3: After upsampling F3 by 4 times and F2 by 2 times, the two samples are concatenated with F1 along the channel and then refined through a 1×1 convolutional layer with BN and ReLU activation functions. Finally, semantic information is discriminated to achieve the classification of remote sensing ground objects.
2. The method as described in claim 1, characterized in that, In step 2, DenseASPP-SE uses dilated convolutions with dilation rates of 3, 6, 12, and 18, along with compression and excitation mechanisms, to expand the network's receptive field.
3. The method as described in claim 1, characterized in that, Step 3 utilizes a dual attention mechanism to refine the spatial and channel features of the image.
4. The method as described in claim 1, characterized in that, Step 4 uses bilinear interpolation to upsample the resolution of the high-level feature map to the same size as the resolution of the low-level feature map. Then, the upsampled high-level feature map is refined through a 3×3 convolutional layer with BN and ReLU activation functions. Next, the module concatenates the refined high-level feature map with the low-level feature map through feature aggregation to obtain a fused feature map. Finally, the final output feature map is obtained through a 1×1 convolutional layer with BN and ReLU activation functions.