A remote sensing image semantic segmentation method and device
By enhancing feature mapping through sparse space and channel attention modules, and combining it with a multi-level feature fusion decoder, the problems of insufficient feature representation and sampling loss in semantic segmentation of remote sensing images are solved, achieving high-precision and efficient remote sensing image segmentation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HOHAI UNIV
- Filing Date
- 2022-05-05
- Publication Date
- 2026-06-05
AI Technical Summary
Existing deep convolutional neural networks have failed to effectively capture the spatial and channel domain dependencies in semantic segmentation of remote sensing images, resulting in insufficient feature representation and unavoidable transformation loss introduced by sampling recovery in the decoder stage, which affects segmentation accuracy.
We employ sparse spatial attention modules and sparse channel attention modules to enhance the spatial and channel dependencies of feature maps. Combined with a multi-level feature fusion decoder, we reduce feature recovery loss by using a small number of matrix calculations, thereby achieving high-precision segmentation.
In the encoder stage, spatial location and inter-channel correlation are extracted, and in the decoder stage, a multi-level feature fusion strategy is used to reduce loss, thereby achieving high-precision and low-time-consumption semantic segmentation of remote sensing images.
Smart Images
Figure CN115187775B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and specifically to a method and apparatus for semantic segmentation of remote sensing images. Background Technology
[0002] In recent years, accurate interpretation of remote sensing images has played an increasingly crucial role in fields such as natural resource management, ecological environmental protection, and smart city planning. However, in practical production applications, the deployment of deep neural network models is limited by both equipment constraints and model accuracy. To address these issues, current methods for capturing and incorporating spatial and channel domain dependencies of features in neural networks have a direct impact on the optimization of encoded features. Furthermore, in the decoder stage, the organic fusion of multi-level feature maps while ensuring low-loss sampling recovery is crucial for supporting the final probabilistic inference.
[0003] Currently, deep convolutional neural networks have further developed, and their powerful feature learning and representation capabilities, especially their significant achievements in natural image processing, have made them an important method for semantic segmentation of remote sensing images. The introduction of fully convolutional neural networks has significantly improved the performance of semantic segmentation tasks for remote sensing images, and semantic segmentation network models built with convolution as the basic unit have become mainstream. Subsequently, with the definition and development of encoder-decoder neural network structures, the stepwise sampling and recovery process can significantly reduce feature transformation loss, further enhancing the accuracy of semantic segmentation.
[0004] Existing technologies primarily use convolutional neural networks as basic units to construct convolutional encoder-decoder semantic segmentation networks for semantic segmentation. In the encoder stage, although local feature patterns are learned relatively comprehensively, the ability of long-distance dependencies to enhance feature representation is neglected, and it cannot fully capture the correlations at the spatial and channel domain levels. In the decoder stage, while progressive sampling and reconstruction decoders can achieve better loss control, the widely used bicubic upsampling introduces unavoidable transformation loss. Summary of the Invention
[0005] The purpose of this invention is to provide a method and apparatus for semantic segmentation of remote sensing images, which enhances the distinguishability of representations and improves the accuracy of semantic segmentation at the cost of a small amount of matrix computation.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0007] This invention provides a method for semantic segmentation of remote sensing images, comprising:
[0008] The remote sensing image data is input into the trained encoder skeleton network to form a feature mapping matrix;
[0009] The feature mapping matrix is processed to obtain a feature mapping F with spatial dependency enhancement. p and the feature map F with channel dependency enhancement c And concatenate them to obtain feature maps with contextual information;
[0010] The feature map with contextual information is input into a multi-level feature fusion decoder for decoding, restoring it to the same size as the original input remote sensing image, thus obtaining the semantic segmentation result.
[0011] Furthermore, it also includes training the encoder backbone network as follows:
[0012] The original remote sensing image data and manually labeled ground truth values are obtained and then divided into sub-blocks. The original remote sensing image is digitally bit-stretched and randomly divided into training set, validation set and test set according to a fixed ratio.
[0013] The training set is input into the encoder backbone network to obtain the encoded feature mapping matrix. Where C, H, and W refer to the number of channels, height, and width of the current feature map, respectively;
[0014] The feature map matrix output from the encoder skeleton network is input into the parallel sparse spatial attention module and sparse channel attention module to obtain the feature map F with spatial dependency enhancement. p and the feature map F with channel dependency enhancement c They are then cascaded and superimposed to form feature maps with contextual information;
[0015] The feature map with contextual information is input into a multi-level feature fusion decoder for decoding to obtain a semantic segmentation result, which is then restored to the same size as the original input remote sensing image.
[0016] The obtained semantic segmentation results are compared with the ground truth, the loss is calculated, and the encoder skeleton network parameters are adjusted accordingly. Through continuous training, a well-trained encoder skeleton network is obtained.
[0017] Furthermore, the feature mapping matrix is processed to obtain a feature mapping F with spatial dependency enhancement. p ,include:
[0018] After performing 1×1 convolution on the feature mapping matrix output by the encoder skeleton network, feature anchor points are resampled through pyramid pooling.
[0019] Calculate the similarity matrix between the feature mapping matrix F output by the encoder skeleton network and the resampled feature anchor matrix P2 to construct a sparse spatial attention matrix A. p Element A in the matrixp (i,j) is calculated as follows:
[0020]
[0021] in, This represents the association between the j-th position feature in the resampled feature anchor matrix and the i-th position feature in the input feature map. The feature mapping matrix The feature is formed by transposing after 1×1 convolution. F1(i) is the i-th position feature in F1, P1(j) is the j-th position feature in the resampled feature anchor matrix, C, H, and W are the number of channels, height, and width of the feature mapping matrix, respectively, and L is the total number of resampled feature anchors.
[0022] According to the spatial attention matrix A p Calculate the feature map F with spatial dependency enhancement p as follows:
[0023]
[0024] in, This represents the feature anchor matrix for resampling. For attention matrix A p The transpose of , where μ is a learnable coordination parameter.
[0025] Furthermore, the feature mapping matrix is processed to obtain a feature mapping F with channel dependency enhancement. c ,include:
[0026] Perform a 1×1 convolution operation on the feature mapping matrix output by the encoder skeleton network to obtain the channel-compressed feature mapping.
[0027] Calculate the relationship matrix between all channel features and the channel-compressed feature maps, and construct a sparse channel attention matrix A. c Element A in the matrix c (i,j) is calculated as follows:
[0028]
[0029] Among them, F s1 (i) represents the feature map F after channel compression. s1 The i-th channel feature, C1(j) represents the j-th channel feature of the input feature mapping matrix F, and S is the number of feature points after channel compression;
[0030] Based on the channel attention matrix A c Compute the channel-dependent enhanced feature map F c as follows:
[0031]
[0032] Where γ is a learnable coordination parameter, It is the feature map F after channel compression. s1 transpose, Let C be the input feature mapping matrix, and let H and W be the number of channels, height, and width of the feature mapping matrix, respectively.
[0033] Furthermore, the feature map with contextual information is input into a multi-level feature fusion decoder for decoding, including:
[0034] The context-informed feature map F formed after concatenation d (i) Perform stage-by-stage fusion as follows to obtain a feature map with the same spatial size as the original input remote sensing image:
[0035]
[0036] in, This represents the fused feature map. f represents the summation between elements. d (·) indicates data-dependent upsampling, and i represents different stages in the encoder skeleton network;
[0037] The fused and restored feature maps are used for probabilistic inference through the Softmax function, and the classification of the target pixels is determined according to the maximum class probability principle.
[0038] Furthermore, it also includes:
[0039] The ratio between the spatial dimensions of the feature mapping at different stages and the original dimensions to be recovered is set as follows:
[0040]
[0041] in, F represents d (i) Spatial dimensions, where H×W represents the original image size.
[0042] The present invention also provides a remote sensing image semantic segmentation device, comprising:
[0043] The encoder skeleton network is used to encode the feature mapping matrix formed by remote sensing image data;
[0044] A sparse module is used to process the feature mapping matrix to obtain a feature mapping F with spatial dependency enhancement. p and the feature map F with channel dependency enhancement c And concatenate them to obtain feature maps with contextual information;
[0045] The decoder is used to fuse and decode the concatenated feature maps with contextual information, restoring them to the same size as the original input remote sensing image, thus obtaining the semantic segmentation result.
[0046] Furthermore, the encoder skeleton network adopts VGG 19.
[0047] Furthermore, the sparse module includes two branches: a parallel sparse spatial attention module and a sparse channel attention module.
[0048] The sparse space attention module is used for,
[0049] After performing 1×1 convolution on the feature mapping matrix output by the encoder skeleton network, feature anchor points are resampled through pyramid pooling.
[0050] Calculate the similarity matrix between the feature mapping matrix F output by the encoder skeleton network and the resampled feature anchor matrix P2 to construct a sparse spatial attention matrix A. p Element A in the matrix p (i,j) is calculated as follows:
[0051]
[0052] in, This represents the association between the j-th position feature in the resampled feature anchor matrix and the i-th position feature in the input feature map. The feature mapping matrix The feature is formed by transposing after 1×1 convolution. F1(i) is the i-th position feature in F1, P1(j) is the j-th position feature in the resampled feature anchor matrix, C, H, and W are the number of channels, height, and width of the feature mapping matrix, respectively, and L is the total number of resampled feature anchors.
[0053] According to the spatial attention matrix A p Calculate the feature map F with spatial dependency enhancement p as follows:
[0054]
[0055] in, This represents the feature anchor matrix for resampling. For attention matrix A p The transpose of μ, where μ is a learnable coordination parameter;
[0056] The sparse channel attention module is used for,
[0057] Perform a 1×1 convolution operation on the feature mapping matrix output by the encoder skeleton network to obtain the channel-compressed feature mapping.
[0058] Calculate the relationship matrix between all channel features and the channel-compressed feature maps, and construct a sparse channel attention matrix A. c Element A in the matrix c (i,j) is calculated as follows:
[0059]
[0060] Among them, F s1 (i) represents the feature map F after channel compression. s1 The i-th channel feature, C1(j) represents the j-th channel feature of the input feature mapping matrix F, and S is the number of feature points after channel compression;
[0061] Based on the channel attention matrix A c Compute the channel-dependent enhanced feature map F c as follows:
[0062]
[0063] Where γ is a learnable coordination parameter, It is the feature map F after channel compression. s1 transpose, Let C be the input feature mapping matrix, and let H and W be the number of channels, height, and width of the feature mapping matrix, respectively.
[0064] Furthermore, the decoder includes a multi-level feature fusion module, which is specifically used for:
[0065] The context-informed feature map F formed after concatenation d (i) Perform stage-by-stage fusion as follows to obtain a feature map with the same spatial size as the original input remote sensing image:
[0066]
[0067] in, This represents the fused feature map. f represents the summation between elements. d (·) indicates data-dependent upsampling, and i represents different stages in the encoder skeleton network;
[0068] The fused and restored feature maps are used for probabilistic inference through the Softmax function, and the classification of the target pixels is determined according to the maximum class probability principle.
[0069] Also used for,
[0070] The ratio between the spatial dimensions of the feature mapping at different stages and the original dimensions to be recovered is set as follows:
[0071]
[0072] in, F represents d (i) Spatial dimensions, where H×W represents the original image size.
[0073] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:
[0074] 1. The semantic segmentation method proposed in this invention proposes a sparse spatial attention module and a sparse channel attention module in the encoder stage to extract the correlation between spatial locations and channels, thereby achieving representation enhancement with a small computational cost. In the decoder stage, a multi-level feature fusion strategy based on data-dependent upsampling is proposed to address the sampling loss and multi-level feature fusion problems. This strategy reduces the loss in the feature recovery stage in a learnable manner and ensures the fidelity of the representation transformation process.
[0075] 2. This invention utilizes the stability of the encoder-decoder architecture, combined with attention-based visual representation optimization theory and multi-source feature fusion theory, to achieve high-precision and low-time-consumption model training and prediction. The method of this invention is not only applicable to multi-resolution satellite remote sensing images and UAV remote sensing image segmentation, but also possesses high classification accuracy and operational efficiency. Attached Figure Description
[0076] Figure 1 This is a schematic diagram of the remote sensing image semantic segmentation method provided in the embodiments of the present invention;
[0077] Figure 2 This is an example of multi-level feature fusion provided in the embodiments of the present invention;
[0078] Figure 3 This is a schematic diagram of DeepGlobe experimental data and results in an embodiment of the present invention;
[0079] Figure 4 This is a schematic diagram of the experimental data and results of ISPRS Potsdam in an embodiment of the present invention. Detailed Implementation
[0080] The present invention will now be further described. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.
[0081] Example 1
[0082] This embodiment provides a method for semantic segmentation of remote sensing images. See [link to relevant documentation] Figure 1The specific implementation process is as follows:
[0083] (1) The original remote sensing image data and the manually labeled ground truth are cut into sub-blocks respectively, and the original remote sensing image is digitally bit stretched. Finally, the training set, validation set and test set are randomly divided according to a fixed ratio.
[0084] (2) Input the training set into the encoder backbone network to obtain the encoded feature mapping matrix. Where C, H, and W refer to the number of channels, height, and width of the current feature map, respectively.
[0085] (3) Input the feature mapping matrix output by the encoder skeleton network into the two parallel branches of the sparse spatial attention module and the sparse channel attention module to obtain the feature mapping F with spatial dependency enhancement. p and channel-dependent enhanced feature mapping F c .
[0086] (4) The obtained feature map F with spatial dependency enhancement p and channel-dependent enhanced feature mapping F c By cascading and stacking, a feature map containing rich contextual information is formed for decoding reasoning.
[0087] (5) Input the cascaded feature map into the multi-level feature fusion decoder, and gradually expand the feature map space size by upsampling until it is restored to the same size as the original input remote sensing image.
[0088] (6) Compare the generated prediction results with the ground truth, calculate the loss, and adjust the network parameters accordingly. Through continuous training, a well-trained backbone network is obtained.
[0089] (7) Input the remote sensing image data to be predicted into the trained skeleton network to obtain the semantic segmentation results of the pixels in the image.
[0090] In a preferred embodiment, VGG19 is used as the encoder backbone network to progressively extract higher-level feature maps, resulting in feature maps after 19 layers of convolutional neural network and pooling. Where C, H, and W refer to the number of channels, height, and width of the current feature map, respectively.
[0091] In a preferred embodiment, the feature mapping matrix output by the encoder skeleton network is input into the sparse spatial attention module. The operation process in the sparse spatial attention module is as follows: the feature mapping matrix output by the encoder skeleton network is processed by a 1×1 convolution; then, pyramid pooling is used to resample the feature anchor points, resulting in a significant reduction in the total number of resampled feature anchor points compared to the number of pixels in the original features; finally, a sparse spatial attention matrix A is constructed by calculating the similarity matrix between the feature mapping matrix output by the encoder skeleton network and the resampled feature anchor points. p Element A in the matrix p (i,j) is calculated as follows:
[0092]
[0093] in This represents the association between the j-th position feature in the resampled feature map and the i-th position feature in the input feature map. Original features The feature is formed by transposing the 1×1 convolution. F1(i) is the feature at the i-th position in F1, P1(j) is the feature at the j-th position in the resampled feature map, and L is the total number of resampled feature anchors.
[0094] Calculate the feature map F with spatial dependency reinforcement based on the spatial attention matrix. p as follows:
[0095]
[0096] in, This represents the optimized features of the sparse space attention module. This represents the feature anchor matrix for resampling. For attention matrix A p transpose, Let be the input feature mapping matrix, and μ be the learnable coordination parameter.
[0097] Specifically, assuming the feature map input H×W=256×256, the complexity of the matrix operation of the top sparse space attention module is approximately 1 / 1311 of the original computation.
[0098] In a preferred embodiment, the feature mapping matrix output by the encoder skeleton network is input into the sparse channel attention module. The operation process in the sparse channel attention module is as follows: the feature mapping matrix output by the encoder skeleton network is subjected to a 1×1 convolution operation to obtain the channel-compressed feature mapping; the relationship matrix between all channels and the compressed channels is calculated to construct a sparse channel attention matrix A. c Element A in the matrix c(i,j) is calculated as follows:
[0099]
[0100] Where F s1 (i) represents the feature map F after channel compression. s1 The i-th channel feature, C1(j) represents the j-th channel feature of the input feature mapping matrix, A c (i,j) represents the relationship between the two.
[0101] Calculate the channel-dependent reinforcement feature map F based on the channel attention matrix. c as follows:
[0102]
[0103] in, This represents the optimized features of the sparse channel attention module, where γ is a learnable coordination parameter. It is the feature map F s1 transpose, Let S be the input feature mapping matrix, and S be the number of feature points after channel compression.
[0104] Specifically, assuming the channel resampling is S, which is much smaller than C, the computation complexity of the bottom sparse channel attention module array is approximately S / C of the original computation.
[0105] In a preferred embodiment, the concatenated feature map is input into a multi-level feature fusion decoder, and the feature map space size is gradually expanded by upsampling. (See [link to relevant documentation]). Figure 2 The specific implementation process is as follows:
[0106] Let F be the feature map formed after cascading. d (i), where i represents different stages in the encoder and is related to the selection of the skeleton network. In this embodiment, the encoder goes through 5 stages, i.e., 1≤i≤5.
[0107] The decoding stage primarily considers changes in spatial size; therefore, the number of channels is not represented for now. The ratio between the spatial size of different levels of feature mapping and the original size to be recovered can be defined as:
[0108]
[0109] in F represents d (i) Spatial dimensions, where H×W represents the original image size.
[0110] Therefore, the process of fusing features at adjacent levels can be summarized as follows:
[0111]
[0112] in Indicates the characteristics after fusion. f represents the summation between elements (similar to residual calculation). d (·) indicates data-dependent upsampling.
[0113] To ensure feature fidelity and low distortion, each feature size is incorporated into the feature map of the corresponding stage of the encoder. Finally, the fused and recovered feature map is used for probabilistic inference through the Softmax function to determine the classification of the target pixel based on the maximum class probability principle.
[0114] Specifically, if the initial setting i is 5, then four fusion operations are required during the decoding stage, ultimately resulting in a feature map with the same spatial size as the original input remote sensing image.
[0115] Example 2
[0116] This embodiment provides a remote sensing image semantic segmentation device, including:
[0117] The encoder skeleton network is used to encode remote sensing images to obtain a feature mapping matrix. Where C, H, and W refer to the number of channels, height, and width of the current feature map, respectively;
[0118] A sparse module is used to process the feature mapping matrix to obtain a feature mapping F with spatial dependency enhancement. p and the feature map F with channel dependency enhancement c These features are then cascaded and stacked to output a feature map that encompasses rich contextual information.
[0119] The decoder is used to perform row fusion decoding on the concatenated feature maps to restore them to the same size as the original input remote sensing image, thus obtaining the semantic segmentation result.
[0120] As a preferred implementation, the encoder skeleton network uses VGG19 to progressively extract higher-level feature maps, resulting in feature maps after 19 layers of convolutional neural networks and pooling. Where C, H, and W refer to the number of channels, height, and width of the current feature map, respectively.
[0121] As a preferred implementation, the sparse module includes two branches: a parallel sparse spatial attention module and a sparse channel attention module.
[0122] The top branch sparse spatial attention module is used to perform 1×1 convolution on the feature map matrix output by the encoder skeleton network; resample feature anchors through pyramid pooling; and construct a sparse spatial attention matrix A by calculating the similarity matrix between all pixels of the feature map and the resampled feature anchors. p Element A in the matrix p (i,j) is calculated as follows:
[0123]
[0124] in, This represents the association between the j-th position feature in the resampled feature anchor matrix and the i-th position feature in the input feature map. The feature mapping matrix The feature is formed by transposing after 1×1 convolution. F1(i) is the i-th position feature in F1, P1(j) is the j-th position feature in the resampled feature anchor matrix, C, H, and W are the number of channels, height, and width of the feature mapping matrix, respectively, and L is the total number of resampled feature anchors.
[0125] Based on spatial attention matrix A p Calculate the feature map F with spatial dependency enhancement p as follows:
[0126]
[0127] in, This represents the feature anchor matrix for resampling. For attention matrix A p The transpose of , where μ is a learnable coordination parameter.
[0128] The bottom branch sparsity is used by the attention module to perform a 1×1 convolution operation on the feature mapping matrix output by the encoder skeleton network to obtain the channel compressed feature mapping; the relationship matrix between the original features of all channels and the compressed features of the channels is calculated to construct a sparse channel attention matrix A. c Element A in the matrix c (i,j) is calculated as follows:
[0129]
[0130] Among them, F s1 (i) represents the feature map F after channel compression. s1 The i-th channel feature, C1(j) represents the j-th channel feature of the input feature mapping matrix F, and S is the number of feature points after channel compression;
[0131] Based on the channel attention matrix A cCompute the channel-dependent enhanced feature map F c as follows:
[0132]
[0133] Where γ is a learnable coordination parameter, It is the feature map F after channel compression. s1 transpose, Let C be the input feature mapping matrix, and let C, H, and W be the number of channels, height, and width of the feature mapping matrix, respectively.
[0134] In a preferred embodiment, the decoder includes a multi-level feature fusion module, which is specifically used for:
[0135] The concatenated feature maps will be fused in stages as follows:
[0136]
[0137] in, Indicates the characteristics after fusion. f represents the summation between elements (similar to residual calculation). d (·) indicates that the data depends on upsampling, F d (i) represents the feature map formed after cascading, where i represents different stages in the encoder skeleton network.
[0138] Furthermore, the multi-level feature fusion module sets the ratio between the spatial dimensions of different levels of feature mapping and the original dimensions to be recovered as follows:
[0139]
[0140] in, F represents d (i) Spatial dimensions, where H×W represents the original image size.
[0141] Example 3
[0142] This embodiment uses two different datasets: the DeepGlobe satellite remote sensing image dataset and the ISPRS Potsdam UAV remote sensing image data. The semantic segmentation method from Embodiment 1 is used, and the final semantic segmentation result is as follows: Figure 3 and Figure 4As shown, the method of this invention can be applied to satellite remote sensing images and UAV remote sensing images. It verifies that after calculating sparse spatial attention through spatial resampling and optimizing feature mapping by compressing feature channels to calculate sparse channel attention, the image segmentation prediction results show outstanding accuracy. The annotation results generated by the model prediction have a high degree of consistency with the ground truth, high boundary matching, and good continuity within the main object region.
[0143] Example 4
[0144] This embodiment provides a computing device, including a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, it implements a remote sensing image semantic segmentation method as described in Embodiment 1 above.
[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for semantic segmentation of remote sensing images, characterized in that, include: The remote sensing image data is input into the trained encoder skeleton network to form a feature mapping matrix; The feature mapping matrix is processed to obtain a feature mapping with spatial dependency enhancement. and feature maps with channel dependency enhancement And concatenate them to obtain feature maps with contextual information; The feature mapping matrix is processed to obtain a feature mapping with spatial dependency enhancement. ,include: The feature mapping matrix output by the encoder skeleton network is then processed. After convolution, feature anchors are resampled using pyramid pooling. Calculate the feature mapping matrix output by the encoder skeleton network. With resampled feature anchor matrix Construct a sparse spatial attention matrix from the similarity matrix Elements in the matrix The calculation is as follows: ， in, Represents the i-th feature anchor point in the resampled feature anchor matrix In the mapping between the location feature and the input feature, the first position feature is... The correlation between location features The feature mapping matrix It is formed by transposing after a 1×1 convolution. for The Middle Location features, The first element in the resampled feature anchor matrix Location features, , , These represent the number of channels, height, and width of the feature mapping matrix, respectively. This represents the total number of feature anchor points resampled. Based on the spatial attention matrix Compute feature maps with spatial dependency enhancement as follows: ， in, This represents the feature anchor matrix for resampling. Attention matrix transpose, These are learnable coordination parameters; The feature mapping matrix is processed to obtain a feature mapping with channel dependency enhancement. ,include: Perform a 1×1 convolution operation on the feature mapping matrix output by the encoder skeleton network to obtain the channel-compressed feature mapping. Calculate the relationship matrix between all channel features and the channel-compressed feature maps, and construct a sparse channel attention matrix. Elements in the matrix The calculation is as follows: ， in, Represents the feature map after channel compression The Channel characteristics, The feature mapping matrix represents the input. The j-th channel feature, This represents the number of feature points after channel compression. Based on the channel attention matrix Compute channel-dependent enhanced feature maps as follows: ， in, These are learnable coordination parameters. It is the feature map after channel compression. transpose, The input feature mapping matrix, , , These represent the number of channels, height, and width of the feature mapping matrix, respectively. The feature map with contextual information is input into a multi-level feature fusion decoder for decoding, restoring it to the same size as the original input remote sensing image, thus obtaining the semantic segmentation result.
2. The remote sensing image semantic segmentation method according to claim 1, characterized in that, It also includes training the encoder backbone network as follows: The original remote sensing image data and manually labeled ground truth values are obtained and then divided into sub-blocks. The original remote sensing image is digitally bit-stretched and randomly divided into training set, validation set and test set according to a fixed ratio. The training set is input into the encoder backbone network to obtain the encoded feature mapping matrix. ,in , , These refer to the number of channels, height, and width of the current feature map, respectively. The feature map matrix output from the encoder skeleton network is input into the parallel sparse spatial attention module and sparse channel attention module to obtain feature maps with spatial dependency enhancement. and feature maps with channel dependency enhancement They are then cascaded and superimposed to form feature maps with contextual information; The feature map with contextual information is input into a multi-level feature fusion decoder for decoding to obtain a semantic segmentation result, which is then restored to the same size as the original input remote sensing image. The obtained semantic segmentation results are compared with the ground truth, the loss is calculated, and the encoder skeleton network parameters are adjusted accordingly. Through continuous training, a well-trained encoder skeleton network is obtained.
3. The remote sensing image semantic segmentation method according to claim 1, characterized in that, The feature map with contextual information is input into a multi-level feature fusion decoder for decoding, including: The context-informed feature maps formed after cascading The feature maps with the same spatial size as the original input remote sensing image are obtained by performing a step-by-step fusion process as follows: ， in, This represents the fused feature map. This indicates the summation between elements. This indicates that the data depends on upsampling. This represents different stages in the encoder skeleton network; The fused and restored feature maps are used for probabilistic inference through the Softmax function, and the classification of the target pixels is determined according to the maximum class probability principle.
4. The remote sensing image semantic segmentation method according to claim 3, characterized in that, Also includes: The ratio between the spatial dimensions of the feature mapping at different stages and the original dimensions to be recovered is set as follows: ， in, express Space dimensions, Indicates the original image size.
5. A remote sensing image semantic segmentation device, characterized in that, include: The encoder skeleton network is used to encode the feature mapping matrix formed by remote sensing image data; The sparse module is used to process the feature mapping matrix to obtain a feature mapping with spatial dependency enhancement. and feature maps with channel dependency enhancement And concatenate them to obtain feature maps with contextual information; The sparse module includes two branches: a parallel sparse spatial attention module and a sparse channel attention module. The sparse space attention module is used for, The feature mapping matrix output by the encoder skeleton network is then processed. After convolution, feature anchors are resampled using pyramid pooling. Calculate the feature mapping matrix output by the encoder skeleton network. With resampled feature anchor matrix Construct a sparse spatial attention matrix from the similarity matrix Elements in the matrix The calculation is as follows: ， in, Represents the i-th feature anchor point in the resampled feature anchor matrix In the mapping between the location feature and the input feature, the first position feature is... The correlation between location features The feature mapping matrix It is formed by transposing after a 1×1 convolution. for The Middle Location features, The first element in the resampled feature anchor matrix Location features, , , These represent the number of channels, height, and width of the feature mapping matrix, respectively. This represents the total number of feature anchor points resampled. Based on the spatial attention matrix Compute feature maps with spatial dependency enhancement as follows: ， in, This represents the feature anchor matrix for resampling. Attention matrix transpose, These are learnable coordination parameters; The sparse channel attention module is used for, Perform a 1×1 convolution operation on the feature mapping matrix output by the encoder skeleton network to obtain the channel-compressed feature mapping. Calculate the relationship matrix between all channel features and the channel-compressed feature maps, and construct a sparse channel attention matrix. Elements in the matrix The calculation is as follows: ， in, Represents the feature map after channel compression The Channel characteristics, The feature mapping matrix represents the input. The j-th channel feature, This represents the number of feature points after channel compression. Based on the channel attention matrix Compute channel-dependent enhanced feature maps as follows: ， in, These are learnable coordination parameters. It is the feature map after channel compression. transpose, The input feature mapping matrix, , , These represent the number of channels, height, and width of the feature mapping matrix, respectively. The decoder is used to fuse and decode the concatenated feature maps with contextual information, restoring them to the same size as the original input remote sensing image, thus obtaining the semantic segmentation result.
6. The remote sensing image semantic segmentation device according to claim 5, characterized in that, The encoder skeleton network uses VGG 19.
7. A remote sensing image semantic segmentation device according to claim 5, characterized in that, The decoder includes a multi-level feature fusion module, which is specifically used for: The context-informed feature maps formed after cascading The feature maps with the same spatial size as the original input remote sensing image are obtained by performing a step-by-step fusion process as follows: ， in, This represents the fused feature map. This indicates the summation between elements. This indicates that the data depends on upsampling. This represents different stages in the encoder skeleton network; The fused and restored feature maps are used for probabilistic inference through the Softmax function, and the classification of the target pixels is determined according to the maximum class probability principle. Also used for, The ratio between the spatial dimensions of the feature mapping at different stages and the original dimensions to be recovered is set as follows: ， in, express Space dimensions, Indicates the original image size.