Remote sensing image semantic segmentation method based on fusion pyramid attention

Abstract

This invention relates to the field of remote sensing image processing technology, specifically to a remote sensing image semantic segmentation method based on fused pyramid attention. The method includes: acquiring the backbone of a semantic segmentation network from a remote sensing image to obtain shallow and deep feature maps; inputting the shallow feature maps into a convolutional neural network of a spatial attention module to obtain first, second, and third feature maps; obtaining a feature matrix based on the first and second feature maps; obtaining a spatial attention feature map based on the feature matrix and the third feature map; inputting the spatial attention feature map into a pyramid attention module to obtain a global feature map; reshaping the deep feature maps using a channel attention module to obtain first, second, and third channel feature maps; obtaining a channel attention matrix based on the first and second channel feature maps; obtaining an enhanced feature map based on the channel attention matrix; and fusing the global feature map and the enhanced feature map to obtain a semantically segmented image. This invention can better distinguish fine boundaries.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and specifically to a remote sensing image semantic segmentation method based on fused pyramid attention. Background Technology

[0002] Semantic segmentation has wide applications in remote sensing data extraction and classification. Existing semantic segmentation networks focus on capturing contextual information in various ways and fusing features at different levels to improve the accuracy of semantic segmentation. However, low-level semantic features lack spatial context guidance, while high-level semantic features tend to encode large objects, resulting in coarse spatial details and making it easy to cause poor recognition of small targets in remote sensing images. Summary of the Invention

[0003] To address the technical problem of low recognition accuracy of small targets in remote sensing images by existing semantic segmentation networks, the present invention aims to provide a semantic segmentation method for remote sensing images based on fused pyramid attention. The specific technical solution adopted is as follows:

[0004] The shallow and deep feature maps are obtained by inputting remote sensing images into the backbone of the semantic segmentation network.

[0005] The shallow feature maps are input into the convolutional neural network of the spatial attention module to obtain the first, second, and third feature maps. The feature matrix is ​​obtained based on the first and second feature maps. The spatial attention feature map is obtained based on the feature matrix and the third feature map.

[0006] The spatial attention feature map is input into the pyramid attention module. After concatenating dilated convolutions of different scales, it is divided into three branches. The outputs of the three branches are fused to obtain the global feature map.

[0007] The deep feature map is reshaped using the channel attention module to obtain the first, second, and third channel feature maps. The channel attention matrix is ​​obtained based on the first and second channel feature maps. The enhanced feature map is obtained based on the channel attention matrix and the third channel feature map. The global feature map and the enhanced feature map are fused to obtain the semantic segmentation image.

[0008] Preferably, obtaining the feature matrix based on the first and second feature maps specifically involves:

[0009] The feature matrix is ​​obtained by multiplying the feature map obtained by reshaping the first feature map and then transposing it with the feature map obtained by reshaping the second feature map.

[0010] Preferably, obtaining the spatial attention feature map based on the feature matrix and the third feature map specifically involves:

[0011] The product of the reshaped feature map of the third feature map and the transpose of the feature matrix is ​​calculated. The product is multiplied by a preset first scaling factor and then summed with the shallow feature map to obtain the spatial attention feature map.

[0012] Preferably, the method for obtaining the feature matrix is ​​as follows:

[0013]

[0014] Among them, S ji Let be the value of the element at the i-th row and j-th column in the feature matrix. Let Q be the value of the element at the i-th row of the matrix corresponding to the transpose of the first feature map after reshaping. j The value of the element at the j-th column in the matrix corresponding to the reshaped second feature map is represented by N = H × W, where H is the height of the first feature map and W is the width of the first feature map.

[0015] Preferably, the method for obtaining the spatial attention feature map is as follows:

[0016]

[0017] Among them, K j Let be the value of the element at the j-th column in the spatial attention feature map, and α be the first scaling factor. V represents the value of the element at the i-th row and j-th column in the transpose of the characteristic matrix. i Let I be the value of the element at the i-th row position in the matrix corresponding to the third feature map. j Let N be the value of the element at the j-th column of the matrix corresponding to the shallow feature map, where N = H × W, H is the height of the first feature map, and W is the width of the first feature map.

[0018] Preferably, obtaining the channel attention matrix based on the first and second channel feature maps specifically involves:

[0019] The first channel feature map is transposed and then multiplied with the second channel feature map. The result of the multiplication is then used to obtain the channel attention matrix using the softmax function.

[0020] Preferably, obtaining the enhanced feature map based on the channel attention matrix and the third channel feature map specifically involves:

[0021] Multiply the channel attention matrix with the matrix corresponding to the transposed third channel feature map, calculate the product of the product and the preset second scaling factor, and use the sum of the product and the deep feature map as the enhanced feature map.

[0022] Preferably, the backbone of the semantic segmentation network is a ResNet50 network.

[0023] Preferably, the semantic segmentation network employs a multivariate learning rate decay strategy, where the learning rate is multiplied by [a certain factor] in each round. Where iter represents the current iteration round number, and total_iter represents the total number of iteration rounds.

[0024] The embodiments of the present invention have at least the following beneficial effects:

[0025] The semantic segmentation network of this invention embeds the attention mechanism into high-level and low-level features through two parallel branches: a spatial attention module and a channel attention module. Convolutional layers extract spatial feature information from the feature map through the spatial attention module and channel feature information through the channel attention module, effectively aiding information propagation and improving model accuracy. Simultaneously, the pyramid attention module is improved by concatenating three dilated convolutions corresponding to different dilation coefficients, resulting in only three branches for each dilated convolution. The outputs of these three branches are then fused to obtain the global feature map, reducing computational cost while enhancing the model's ability to extract target boundaries. This semantic segmentation network expands the receptive field, better integrates low-level semantic information, and achieves more accurate recognition of fine target boundaries. Attached Figure Description

[0026] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a remote sensing image semantic segmentation method based on fused pyramid attention according to the present invention;

[0028] Figure 2 This is a schematic diagram of the network structure of the spatial attention module according to an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the network structure of the pyramid attention module according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the network structure of the channel attention module in an embodiment of the present invention. Detailed Implementation

[0031] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a remote sensing image semantic segmentation method based on fused pyramid attention proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] The following description, in conjunction with the accompanying drawings, details a specific scheme for a remote sensing image semantic segmentation method based on fused pyramid attention provided by this invention.

[0034] Example:

[0035] Please see Figure 1 The diagram illustrates a flowchart of a remote sensing image semantic segmentation method based on fused pyramid attention, according to an embodiment of the present invention, comprising the following steps:

[0036] Step 1: Obtain the shallow feature map and deep feature map by inputting the remote sensing image into the backbone of the semantic segmentation network; input the shallow feature map into the convolutional neural network of the spatial attention module to obtain the first, second and third feature maps, and obtain the feature matrix based on the first and second feature maps; obtain the spatial attention feature map based on the feature matrix and the third feature map.

[0037] It's important to note that attention mechanisms apply human perception to machine learning. Essentially, they aim to make the model focus more on important information in an image while ignoring irrelevant background information. This is achieved by assigning different weights to each input item, with the magnitude of the weight representing the degree of attention that input deserves within the overall context. Based on this, attention mechanisms are categorized into global and local attention mechanisms. Local attention mechanisms have only weights of 0 and 1, while global attention mechanisms have weights between 0 and 1.

[0038] To fully utilize the effective information from deep and shallow networks, better integrate multi-layer convolutional features, and balance the relationship between receptive field and resolution, a semantic segmentation network architecture is constructed to extract targets from remote sensing images by strengthening low-level features and contextual features.

[0039] Feature maps at different levels are obtained through the ResNet backbone and fed into two branches. The shallow features extracted from the backbone are fed into the spatial attention module to gather spatial information from the shallow features, and then into the pyramid attention module to obtain multi-scale information. The other branch feeds the feature maps output from the backbone into the channel attention module. Finally, the outputs of the two branches are fused to obtain a more accurate segmentation result.

[0040] The semantic segmentation model in this embodiment of the invention includes both a spatial attention module and a channel attention module, both of which are global attention modules. The convolutional layer extracts spatial feature information from the feature map through the spatial attention module and extracts channel feature information from the feature map through the channel attention module. This can effectively help the network to propagate information and improve the model accuracy.

[0041] Then, based on the pre-trained ResNet50 network, downsampling was removed and dilated convolutions were used in the last two blocks to reduce the number of model parameters. This network was then used as the backbone of the semantic segmentation network. Remote sensing images were acquired, and shallow and deep features were extracted using the backbone network. In other words, the remote sensing images were input into the backbone network of the semantic segmentation network, which output shallow and deep feature maps.

[0042] In this embodiment, the remote sensing images used are building images. Specifically, the datasets for the semantic segmentation network are the WHU building aerial image dataset and the ISPRS Potsdam dataset. The WHU building aerial image dataset is a binary classification dataset, namely Building and Others, and the dataset is divided into a training set of 4736 images, a test set of 2416 images, and a validation set of 1036 images.

[0043] ISPRS Potsdam is a relatively common remote sensing image dataset, containing 38 6000×6000 orthophotos and a standardized DSM. Considering that many remote sensing datasets lack a DSM, the standardized DSM dataset was not used in this embodiment. The dataset was divided into six categories: background, vehicles, buildings, trees, low vegetation, and opaque surfaces. Eighteen images were randomly selected as the training set, and six as the validation set. Because the individual images in the ISPRS Potsdam dataset are very large, which is not conducive to training the semantic segmentation network, it was divided into 256×256 pixels. The dataset was expanded by rotation and translation operations and fed into the network in batches for training.

[0044] Finally, a branch is added to the shallow backbone network to feed the shallow features extracted from the backbone into the spatial attention module. This demonstrates that feature maps at different levels in a convolutional neural network have different expressive power. While shallow networks contain significant background noise, which can negatively impact segmentation, they also contain valuable information such as edge details and target locations that can be fully utilized. Therefore, the spatial attention module is introduced to filter background information, allowing the model to focus more on target features and generate more effective feature maps.

[0045] The shallow feature maps are input into the convolutional neural network of the spatial attention module to obtain the first, second and third feature maps. The feature matrix is ​​obtained based on the first and second feature maps. The spatial attention feature map is obtained based on the feature matrix and the third feature map.

[0046] like Figure 2 In the schematic diagram of the network structure of the spatial attention module shown, the shallow feature map I output by the backbone network has a size of C×H×W, where C represents the number of channels of the shallow feature map, H represents the height of the shallow feature map, and W represents the width of the shallow feature map. The shallow feature map I is input into the convolutional layer to generate the first feature map P, the second feature map Q, and the third feature map V.

[0047] The first feature map has dimensions C×H×W. It is reshaped, resulting in a feature map of size C×N, where N = H×W, and then transposed. The second feature map Q has dimensions C×H×W. It is also reshaped, resulting in a feature map of size C×N, where N = H×W. Multiplying the reshaped second feature map with the reshaped and transposed first feature map yields the feature matrix S, which has dimensions N×N, expressed by the formula:

[0048]

[0049] Among them, S ji The value of the element at the i-th row and j-th column in the feature matrix represents the correlation between the i-th and j-th elements in the shallow feature map. The more similar the features of the two positions, the greater the correlation. Let Q be the value of the element at the i-th row of the matrix (C×N) corresponding to the transpose of the first feature map after reshaping. j This represents the value of the element at the j-th column in the matrix (C×N) corresponding to the reshaped second feature map. N = H×W, where H is the height of the first feature map and W is the width of the first feature map.

[0050] The product of the reshaped feature map V and the transpose of the feature matrix S is calculated. This product, multiplied by a scaling factor, is then summed with the shallow feature map to obtain the spatial attention feature map K. This involves fusing the captured spatial correlation information with the information in the original feature map, multiplying by a preset first scaling factor α to adjust the result to the same size as the original input, and finally summing the result with the original feature map to obtain the final feature map, expressed by the formula:

[0051]

[0052] Among them, K j Let α be the value of the element at the j-th column of the spatial attention feature map, and let α be the first proportionality coefficient, which can be set by the implementer according to the specific implementation scenario. V represents the value of the element at the i-th row and j-th column in the transpose of the characteristic matrix. i Let I be the value of the element at the i-th row position in the matrix corresponding to the third feature map. j Let N be the value of the element at the j-th column of the matrix corresponding to the shallow feature map, where N = H × W, H is the height of the first feature map, and W is the width of the first feature map.

[0053] It should be noted that, in Figure 2 In the network structure diagram of the spatial attention module, Conv represents a convolutional layer, reshape represents reshaping, Transpose represents transpose, and multiply represents multiplication.

[0054] Step 2: Input the spatial attention feature map into the pyramid attention module, concatenate dilated convolutions of different scales to form three branches, and fuse the outputs of the three branches to obtain the global feature map.

[0055] It's important to note that in semantic segmentation tasks, the most crucial factor in improving segmentation accuracy is contextual information, and increasing the receptive field can effectively increase the information available to the network. Conventional neural networks often perform downsampling to increase the receptive field, but while this increases the receptive field, it also lowers the image resolution, leading to information loss. Dilated convolutions can explicitly control the resolution of feature response computation in deep convolutional networks and effectively increase the receptive field. The pyramid attention module (AASPP) in this embodiment of the invention reduces the computational load of the pyramid pooling module and increases information transfer between dilated convolutions.

[0056] like Figure 3The diagram shows the network structure of the pyramid attention module, where K represents the spatial attention feature map and Y represents the global feature map. The spatial attention feature map K is input into dilated convolutions of different sizes. Three dilated convolutions with dilation coefficients of 6, 12, and 18 are concatenated. This increases the receptive field and enhances the correlation between convolutional branches. Simultaneously, an adaptive mean pooling module is used to obtain global features. A 1×1 convolutional layer is then used to further extract features and reduce the dimensionality of the previous layer's output. Finally, the output is upsampled back to its original size.

[0057] In this embodiment of the invention, the pyramid attention module concatenates three dilated convolutions corresponding to different dilation coefficients, so that the dilated convolution is divided into only three branches. Finally, the outputs of the three branches are fused to obtain the global feature map Y, which reduces the amount of computation and enhances the model's ability to extract target boundaries.

[0058] Step 3: Reshape the deep feature map using the channel attention module to obtain the first, second, and third channel feature maps; obtain the channel attention matrix based on the first and second channel feature maps; obtain the enhanced feature map based on the channel attention matrix and the third channel feature map; and fuse the global feature map and the enhanced feature map to obtain the semantic segmentation image.

[0059] In convolutional neural networks, different feature channels act on different semantic information. By constructing channel attention modules, the connections between channels can be used to enhance specific semantic features, so that similar semantic features can promote each other.

[0060] Deep networks have a larger receptive field, and the extracted features are abstract. Because the overlapping areas between their receptive fields increase, the image information is compressed and more global information is obtained. Therefore, the first feature map output from the backbone network is input into the channel attention module to establish the relationship between various feature channels. Based on the characteristics of the task, the effective channel information is enhanced and the useless channel information is suppressed, thereby improving the feature representation capability.

[0061] like Figure 4 In the network structure diagram of the channel attention module shown, the deep feature map U is reshaped to obtain the first channel feature map I1, the second channel feature map I2, and the third channel feature map I3, each with a size of C×N, where N=H×W. The first channel feature map I1 is transposed and multiplied with the second channel feature map I2. The result of the multiplication is then used with the softmax function to obtain the channel attention matrix X, with a size of C×C, expressed by the formula:

[0062]

[0063] Among them, X jiX represents the value of the element at the i-th row and j-th column in the channel attention matrix. The channel attention matrix contains relevant information between different channels of the channel feature map. ji The influence of the i-th channel on the j-th channel was measured. For the value of the element at the i-th row of the matrix (C×N) corresponding to the reshaped and transposed feature map of the first channel, I2 j This represents the value of the element at the j-th column in the matrix (C×N) corresponding to the reshaped feature map of the second channel, where C is the number of channels.

[0064] Similar to spatial attention, to ensure that the correlation information in the channel attention map can be transferred to the final feature map, the channel attention matrix X is multiplied by the matrix corresponding to the transpose of the third channel feature map. The product of the multiplication result and a preset second scaling factor is calculated. The result is then multiplied by the preset second scaling factor β and reshaped back to its original size, i.e., C×H×W. The sum of the product result and the deep feature map is the enhanced feature map G, expressed by the formula:

[0065]

[0066] Among them, K j Let X be the value of the element at the j-th column of the spatial attention feature map, β be the second scaling factor (which can be set by the implementer according to the specific implementation scenario), and X be the value of the element at the j-th column of the spatial attention feature map. ji Let be the value of the element at the i-th row and j-th column in the channel attention matrix. U represents the value of the element at the i-th row of the matrix corresponding to the transposed third channel feature map. j Let be the value of the element at the j-th column of the matrix corresponding to the deep feature map.

[0067] The semantic segmentation image is obtained by fusing shallow and deep features, that is, by fusing the global feature map and the enhanced feature map. Since dense connections can promote feature aggregation and help information propagation, they are used to integrate the enhanced shallow and deep features, thereby improving the semantic segmentation performance.

[0068] Furthermore, the intersection-over-union ratio (IoU) measures the degree of overlap between the predicted image and the input image, while pixel precision measures whether the label corresponding to each pixel in the predicted input image belongs to the correct category. Therefore, the average IoU and pixel precision can be used to evaluate the quality of the output results of the semantic segmentation network.

[0069] It should also be noted that in this embodiment of the invention, all datasets are trained using a multivariate learning rate decay strategy, with the learning rate multiplied by [missing information] in each round. Where `iter` represents the current iteration epoch, and `total_iter` represents the total number of iterations. The optimizer uses the SGD algorithm with a momentum of 0.9, an initial learning rate of 1e-4, and a batch size of 8. During training, images were randomly flipped to expand the dataset, and the model was trained from scratch for 100 epochs. The backbone network ResNet50 used in this embodiment was pre-trained on the ImageNet dataset, while the other convolutional kernels were randomly initialized.

[0070] In summary, this invention removes downsampling operations and uses dilated convolutions in the last two blocks on the basis of a pre-trained ResNet50 network, which helps to reduce the number of model parameters, and then uses it as the backbone of the semantic segmentation network.

[0071] By adding a branch to the shallow backbone network, the network can utilize the rich information in the shallow feature space, make use of more fine-grained features, and extract information such as texture boundaries from the feature map. In addition, the overlapping area of ​​the receptive field corresponding to each pixel is very small, and the network can capture more detailed information.

[0072] Meanwhile, to compensate for the small receptive field of shallow networks, the shallow features extracted from the backbone network are fed into the spatial attention module to gather the spatial information in the shallow features. Then, they are fed into the pyramid attention module to extract multi-scale information through dilated convolutions of different scales, thereby increasing the network's receptive field without losing resolution.

[0073] Deep networks have larger receptive fields, and the features they extract are abstract because the overlapping areas between their receptive fields increase. The image information is compressed, and what is obtained is more global information.

[0074] The output of the backbone network, i.e., the deep feature map, is input into the channel attention module to establish relationships between various feature channels. Effective channel information is enhanced while useless channel information is suppressed, improving feature representation capabilities. Finally, the shallow and deep features are fused, i.e., the global feature map and the enhanced feature map are fused to obtain the semantic segmentation image. Since dense connections can promote feature aggregation and facilitate information propagation, they are used to integrate the enhanced shallow and deep features, thus improving semantic segmentation performance.

[0075] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A remote sensing image semantic segmentation method based on fusion pyramid attention, characterized in that, The method includes: The shallow and deep feature maps are obtained by inputting remote sensing images into the backbone of the semantic segmentation network. The shallow feature maps are input into the convolutional neural network of the spatial attention module to obtain the first, second, and third feature maps. The feature matrix is ​​obtained based on the first and second feature maps. The spatial attention feature map is obtained based on the feature matrix and the third feature map. The spatial attention feature map is input into the pyramid attention module. After concatenating dilated convolutions of different scales, it is divided into three branches. The outputs of the three branches are fused to obtain the global feature map. The deep feature map is reshaped using a channel attention module to obtain the first, second, and third channel feature maps. The channel attention matrix is ​​obtained based on the first and second channel feature maps. The enhanced feature map is obtained based on the channel attention matrix and the third channel feature map. The global feature map and the enhanced feature map are fused to obtain the semantic segmentation image. The spatial attention module is used to filter background noise in the shallow feature map and focus on the target location features, while the pyramid attention module extracts multi-scale information based on the noise-filtered feature map.

2. The remote sensing image semantic segmentation method based on fusion pyramid attention according to claim 1, characterized in that, The specific steps for obtaining the feature matrix based on the first and second feature maps are as follows: The feature matrix is ​​obtained by multiplying the feature map obtained by reshaping the first feature map and then transposing it with the feature map obtained by reshaping the second feature map.

3. The remote sensing image semantic segmentation method based on fusion pyramid attention according to claim 1, characterized in that, The specific steps for obtaining the spatial attention feature map based on the feature matrix and the third feature map are as follows: The product of the reshaped feature map of the third feature map and the transpose of the feature matrix is ​​calculated. The product is multiplied by a preset first scaling factor and then summed with the shallow feature map to obtain the spatial attention feature map.

4. The remote sensing image semantic segmentation method based on fusion pyramid attention according to claim 2, characterized in that, The specific method for obtaining the feature matrix is ​​as follows: wherein, is the value of the element in the i-th row and j-th column of the feature matrix, is the value of the element in the i-th row of the corresponding matrix after the first feature map is reshaped, is the value of the element in the j-th column of the corresponding matrix after the second feature map is reshaped, N = H x W, H is the height of the first feature map, and W is the width of the first feature map.

5. The remote sensing image semantic segmentation method based on fusion pyramid attention as described in claim 3, characterized in that, The method for obtaining the spatial attention feature map is as follows: in, Let j represent the value of the element at the j-th column of the spatial attention feature map. The first proportionality coefficient, Let be the value of the element at the i-th row and j-th column in the transpose of the characteristic matrix. Let be the value of the element at the i-th row of the matrix corresponding to the third feature map. Let N be the value of the element at the j-th column of the matrix corresponding to the shallow feature map, where N = H × W, H is the height of the first feature map, and W is the width of the first feature map.

6. The remote sensing image semantic segmentation method based on fusion pyramid attention as described in claim 1, characterized in that, The specific steps for obtaining the channel attention matrix based on the first and second channel feature maps are as follows: The first channel feature map is transposed and then multiplied with the second channel feature map. The result of the multiplication is then used to obtain the channel attention matrix using the softmax function.

7. The remote sensing image semantic segmentation method based on fusion pyramid attention as described in claim 1, characterized in that, The process of obtaining the enhanced feature map based on the channel attention matrix and the third channel feature map is as follows: Multiply the channel attention matrix with the matrix corresponding to the transposed third channel feature map, calculate the product of the product and the preset second scaling factor, and use the sum of the product and the deep feature map as the enhanced feature map.

8. The remote sensing image semantic segmentation method based on fusion pyramid attention as described in claim 1, characterized in that, The backbone of the semantic segmentation network is the ResNet50 network.

9. A remote sensing image semantic segmentation method based on fusion pyramid attention as described in claim 1, characterized in that, The semantic segmentation network employs a multivariate learning rate decay strategy, where the learning rate is multiplied by [value] in each round. Where iter represents the current iteration round number, and total_iter represents the total number of iteration rounds.

Images