Edge-guided multi-scale feature fusion based optical remote sensing image classification method
By processing optical remote sensing images through a backbone network and an edge feature fusion module, the problem of insufficient multi-scale feature fusion in traditional methods is solved, and high-precision ground feature classification is achieved.
Patent Information
- Application Number
- CN202211317049.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-26
- Publication Date
- 2025-12-30
- Estimated Expiration
- 2042-10-26
AI Technical Summary
Traditional optical remote sensing image land cover classification methods struggle to effectively preserve spatial detail features and perform multi-scale feature fusion under high-resolution conditions, resulting in insufficient classification accuracy.
Optical remote sensing images are processed using a backbone network and an edge feature extraction module. Multi-scale feature fusion is performed through an edge-guided feature fusion module. Combined with supervised learning and matrix transformation, edge precision and target cohesion are improved.
It improves the segmentation precision and accuracy of ground feature classification in optical remote sensing images, and effectively integrates high-level semantic information with low-level detailed information.
Smart Images
Figure CN115909081B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of optical remote sensing image processing technology, specifically to an optical remote sensing image land cover classification method based on edge-guided multi-scale feature fusion. Background Technology
[0002] With the continuous development of the aerospace industry, remote sensing images are increasingly widely used in military, exploration, environmental monitoring, and many other fields. In particular, with the advancement of imaging technology, the resolution of optical remote sensing images is becoming increasingly higher, bringing both opportunities and challenges to ground feature classification within these images. Ground feature classification in remote sensing images involves labeling and classifying targets and backgrounds within the image, ultimately obtaining a segmented image with pixel-by-pixel labeled categories. This plays a crucial role in information extraction and intelligent processing of remote sensing images, directly impacting the quality of information acquired by subsequent systems.
[0003] Traditional image segmentation algorithms often employ manually designed feature extraction and classification algorithms. However, as the resolution of remote sensing images increases and the ground features become more complex, traditional methods struggle to achieve satisfactory segmentation results.
[0004] With the development of deep learning technology and convolutional neural networks, image feature acquisition and representation techniques have made great progress. However, due to the downsampling operation in the convolutional neural network structure, image resolution and spatial detail information are inevitably lost, which seriously affects the segmentation performance of small and complex targets in remote sensing images. In addition, there is the problem of information redundancy in the multi-scale feature fusion process. Simple fusion operations such as addition and stitching are difficult to retain, fuse and represent effective information, often resulting in unsatisfactory accuracy in ground object classification.
[0005] Therefore, how to effectively preserve the spatial details of the target and effectively integrate multi-scale features to improve the accuracy and precision of land cover classification is an urgent problem to be solved. Summary of the Invention
[0006] In view of this, the present invention provides an optical remote sensing image land cover classification method based on edge-guided multi-scale feature fusion, which can preserve spatial detail feature information and effectively fuse high-level semantic information and low-level detail information to obtain fine and accurate land cover classification results.
[0007] To achieve the above objectives, the technical solution of the present invention includes the following steps:
[0008] S1: The backbone network processes the input optical remote sensing image. The backbone network contains multiple network layers, and each network layer obtains a first multi-scale feature map. The backbone network outputs multiple first multi-scale feature maps and sends them to the edge feature extraction module.
[0009] S2: The edge feature extraction module receives multiple convolutional layers from each first multi-scale feature map input, learns multi-scale feature information, and obtains a second multi-scale feature map.
[0010] The edge feature extraction module unifies the size of the second multi-scale feature map, and stitches multiple second multi-scale feature maps of the same size into a single feature map through a stitching operation. Then, the feature maps are fused through a convolutional layer to output multi-scale edge-aware features.
[0011] Multi-scale edge-aware features are fed into the edge-guided feature fusion module.
[0012] S3: The edge-guided feature fusion module performs matrix transformation on multi-scale edge-aware features, transforming them from three-dimensional to two-dimensional. The transformed matrix is the first two-dimensional matrix. The first two-dimensional matrix is multiplied by its transpose to obtain a pixel-level autocorrelation coefficient map.
[0013] The edge-guided feature fusion module inputs the first multi-scale feature map of the highest layer obtained from S1 into the convolutional layer, and obtains the second two-dimensional matrix through matrix transformation; the second two-dimensional matrix is multiplied with the pixel-level autocorrelation coefficient map to obtain the third two-dimensional matrix; the third two-dimensional matrix is transformed into a three-dimensional form to obtain the edge fusion feature map.
[0014] The edge-guided feature fusion module performs adaptive weighted summation on the edge fusion feature map through a convolutional layer and the first multi-scale feature map of the highest layer obtained by S1 to obtain the summation result. The summation result is then linearly upsampled until it is gradually restored to the same resolution as the input remote sensing image to obtain the land cover classification result.
[0015] Furthermore, in S1, the backbone network adopts ResNet101, which contains multiple network layers. Each layer consists of convolutional layers and pooling layers. The backbone network processes the input remote sensing image through multiple network layers of convolutional and pooling layers.
[0016] Furthermore, in S2, each first multi-scale feature map is input into multiple convolutional layers to learn multi-scale feature information, resulting in a second multi-scale feature map, specifically:
[0017] The first multi-scale feature maps output from different network layers are input into the edge feature perception network composed of 1×1 convolutional layers and 3×3 convolutional layers, respectively, and the output is the second multi-scale feature map.
[0018] Furthermore, in S3, the second multi-scale feature maps are resized using interpolation, and multiple resized second multi-scale feature maps are stitched together into a single feature map. Then, a convolutional layer is used to fuse the stitched feature maps, outputting multi-scale edge-aware features. Specifically: the second multi-scale feature maps are resized to 1 / 8 of the input image size using bilinear interpolation; the resized second multi-scale feature maps are then stitched along the channel dimension to obtain features with 1024 channels and a size of 1 / 8 of the input image, which are then input into a 3×3 convolutional layer for further fusion, outputting multi-scale edge-aware features. .
[0019] Furthermore, the edge feature extraction module also includes a supervision process: the edge feature extraction module inputs the second multi-scale feature map into the convolutional layer to obtain the third multi-scale feature map, and obtains the supervised edge feature map through the summation operation; the edge ground map is extracted through the ground map of the optical remote sensing image, and the edge ground map is used to supervise the learning process of the supervised edge feature map, thereby supervising the entire edge feature extraction module.
[0020] Furthermore, the edge feature extraction module inputs the second multi-scale feature map into the convolutional layer to obtain the third multi-scale feature map, and then obtains the supervised edge feature map through a summation operation, specifically:
[0021] Each second multi-scale feature map is further input into a 1×1 convolutional layer, which outputs a third multi-scale feature map; its number of channels changes from 256 to 2, and a sigmoid activation function is used.
[0022] The supervised edge feature map is obtained by summing the third multi-scale feature maps.
[0023] Furthermore, an edge ground truth map is extracted from the ground truth map of the optical remote sensing image, and this edge ground truth map is used to supervise the learning process of the supervised edge feature map, thereby supervising the entire edge feature extraction module. Specifically:
[0024] The ground truth map of the edge is extracted as follows: the ground truth map label of the optical remote sensing image is converted into a one-hot map, the distance from the non-zero point in the one-hot map to the background is calculated, and a boundary threshold is set. The value within the threshold is set to 1, and the rest is set to 0. The edge labels of all categories can be obtained by superimposing all one-hot maps.
[0025] Edge features are supervised by the ground truth map of the edges, and binary classification supervision is performed by the cross-entropy loss function to realize the learning process of the edge feature extraction module.
[0026] Furthermore, the input to the edge-guided feature fusion module is: multi-scale edge-aware features. and the highest level features The output is: the segmentation result. ;
[0027] enter Transform from 3D to 2D form using matrix transformation. The product of this product with its transpose is then performed to obtain a pixel-level autocorrelation coefficient map. :
[0028]
[0029] in Pixel-level autocorrelation coefficient plot The element in the i-th row and j-th column of M; i For matrix The i-th element in for The j-th element in M, where N is the total number of elements in M;
[0030] Input top-level features Channel dimension reduction is performed using 1×1 convolution, followed by matrix dimension transformation to obtain a 2D form. ;
[0031] right and Perform matrix multiplication and dimensional transformation to obtain edge-guided fusion features. :
[0032]
[0033] in, For dimensional transformation, This is matrix multiplication;
[0034] at last, Through 1×1 convolutional layers and high-level features Adaptive matrix element summation is performed, and the resolution is restored to that of the input image through upsampling to obtain the segmentation result.
[0035] The segmentation results are supervised using the ground truth map of the original image through the cross-entropy loss function.
[0036] Beneficial effects:
[0037] This invention provides an optical remote sensing image land cover classification method based on edge-guided multi-scale feature fusion. Addressing the shortcomings of traditional segmentation methods, such as insufficient spatial detail acquisition, low target segmentation edge precision, and difficulty in fusing high-level and low-level features, this invention offers an edge feature extraction method supervised by edge ground truth and designs a multi-scale feature fusion method guided by edge features. This improves edge precision, enhances target cohesion, and effectively interacts high-level semantic information with low-level detail information, thereby simultaneously improving segmentation precision and accuracy. Attached Figure Description
[0038] Figure 1 This is the overall network structure of the optical remote sensing image land cover classification method based on edge-guided multi-scale feature fusion provided in the embodiments of the present invention;
[0039] Figure 2 This is a schematic diagram of the edge feature extraction module according to an embodiment of the present invention;
[0040] Figure 3 This is a schematic diagram of the edge-guided feature fusion module in an embodiment of the present invention. Detailed Implementation
[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0042] A method for classifying ground features in optical remote sensing images based on edge-guided multi-scale feature fusion, the process of which is as follows: Figure 1 As shown, the method includes the following steps:
[0043] S1: The backbone network processes the input optical remote sensing image. The backbone network contains multiple network layers, and each network layer obtains a first multi-scale feature map. The backbone network outputs multiple first multi-scale feature maps and sends them to the edge feature extraction module.
[0044] In this embodiment of the invention, the backbone network adopts ResNet101, which contains multiple network layers. Each layer consists of convolutional layers and pooling layers. The backbone network processes the input remote sensing image through multiple network layers of convolutional and pooling layers, and each network layer outputs a first multi-scale feature map.
[0045] In this embodiment of the invention, the input image The processing is performed by a backbone network, which is a ResNet101 network containing multiple network layers. Each network layer includes convolutional layers and pooling layers; in this invention, it is divided into four different network layers. The backbone network outputs four different levels of feature maps, which are respectively... , , , , which is the first multi-scale feature map.
[0046] Next, the features at four different levels will be... , , , Input edge feature extraction module.
[0047] S2: The edge feature extraction module receives each first multi-scale feature map as input to multiple convolutional layers, learns multi-scale feature information, and obtains a second multi-scale feature map; in this embodiment of the invention, the first multi-scale feature maps output from different network layers are respectively input to the edge feature perception network composed of 1×1 convolutional layers and 3×3 convolutional layers, and the output is the second multi-scale feature map.
[0048] The edge feature extraction module unifies the size of the second multi-scale feature map through interpolation, and stitches multiple second multi-scale feature maps with the same size into one feature map through a stitching operation. Then, the stitched feature map is fused through a convolutional layer to output multi-scale edge-aware features.
[0049] Multi-scale edge-aware features are fed into the edge-guided feature fusion module.
[0050] The edge feature extraction module in this embodiment of the invention is described in detail below. Figure 2 The input to this module includes: the first multi-scale feature map output by the backbone network. , , , The output is: multi-scale edge-aware features. and monitoring edge features .
[0051] Feature maps at different levels are input into an edge feature perception network consisting of 1×1 convolutional layers and 3×3 convolutional layers (multiple convolutional layers), respectively, and the output is a second multi-scale feature map. , , , It contains feature information at different levels. High-level features contain abstract semantic information, while low-level features contain more detailed information.
[0052] The second multi-scale feature map is standardized to 1 / 8 of the input image size using bilinear interpolation. This standardized second multi-scale feature map is then concatenated along the channel dimension to obtain features with 1024 channels and a size of 1 / 8 of the input image. These features are then fed into a 3×3 convolutional layer for further fusion, outputting multi-scale edge-aware features. ,in, .
[0053] The edge feature extraction module also includes a supervision process: the edge feature extraction module inputs the second multi-scale feature map into the convolutional layer to obtain the third multi-scale feature map, and obtains the supervised edge feature map through the summation operation; the edge ground map is extracted through the ground map of the optical remote sensing image, and the edge ground map is used to supervise the learning process of the supervised edge feature map, thereby supervising the entire edge feature extraction module;
[0054] Each second-scale feature map is further input into a 1×1 convolutional layer, which outputs a third-scale feature map. , , , Its channel count is reduced from 256 to 2, and the sigmoid activation function is used. The above feature maps are summed, as shown in Equation 1.
[0055]
[0056] in This represents the addition of matrix elements. And on Perform bilinear interpolation to restore the input image size and obtain supervised edge features. .
[0057] The edge ground truth map is obtained as follows: convert the ground truth map labels into a one-hot map, calculate the distance from the non-zero points in the one-hot map to the background, and set a boundary threshold. Set the threshold to 1 within the threshold and 0 for the rest. Superimpose all one-hot maps to obtain the edge labels for all categories.
[0058] Edge features are supervised by the ground truth map of the edges, and binary classification supervision is performed by the cross-entropy loss function to realize the learning process of the edge feature extraction module.
[0059] S3: The edge-guided feature fusion module performs matrix transformation on multi-scale edge-aware features, transforming them from three-dimensional to two-dimensional. The transformed matrix is the first two-dimensional matrix. The first two-dimensional matrix is multiplied by its transpose to obtain a pixel-level autocorrelation coefficient map.
[0060] The edge-guided feature fusion module inputs the first multi-scale feature map of the highest layer obtained from S1 into the convolutional layer, and obtains the second two-dimensional matrix through matrix transformation; the second two-dimensional matrix is multiplied with the pixel-level autocorrelation coefficient map to obtain the third two-dimensional matrix; the third two-dimensional matrix is transformed into a three-dimensional form to obtain the edge fusion feature map;
[0061] The edge-guided feature fusion module performs adaptive weighted summation between the edge fusion feature map and the highest layer feature map through a convolutional layer to obtain the summation result. The summation result is then linearly upsampled until it is gradually restored to the same resolution as the input remote sensing image to obtain the land cover classification result.
[0062] Edge-guided feature fusion module:
[0063] See Figure 3 This module takes multi-scale edge-aware features as input. and high-level characteristics (Highest layer features). Output: Segmentation result. .
[0064] enter Transform from 3D to 2D form using matrix transformation. And multiply it by its transpose, as shown in Formula 2.
[0065]
[0066] in Pixel-level autocorrelation coefficient plot The element in the i-th row and j-th column of M; i For matrix The i-th element in for The j-th element in M, where N is the total number of elements in M.
[0067] Obtain pixel-level autocorrelation coefficient map .
[0068] enter Channel dimension reduction is performed using 1×1 convolution, followed by matrix dimension transformation to obtain a 2D form. .
[0069] right and Perform matrix multiplication and dimensional transformation to obtain edge-guided fusion features. As shown in Formula 3
[0070]
[0071] in, For dimensional transformation, This is matrix multiplication.
[0072] at last, Through 1×1 convolutional layers and high-level features Adaptive matrix element summation is performed, and the resolution is restored to that of the input image through upsampling to obtain the segmentation result.
[0073] The segmentation results are supervised using the ground truth map of the original image through the cross-entropy loss function.
[0074] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An optical remote sensing image ground object classification method of edge-guided multi-scale feature fusion, characterized in that, The method comprises the following steps: S1: the backbone network processes the input optical remote sensing image, the backbone network comprises multiple levels of network levels, each network level obtains a first multi-scale feature map; the backbone network outputs multiple first multi-scale feature maps and inputs an edge feature extraction module; S2: the edge feature extraction module receives each first multi-scale feature map input multiple convolution layers, learns multi-scale feature information, and obtains a second multi-scale feature map, specifically: the first multi-scale feature map output by different network levels is respectively input into an edge feature perception network composed of a 1*1 convolution layer and a 3*3 convolution layer, and a second multi-scale feature map is output; The edge feature extraction module unifies the size of the second multi-scale feature map, splices multiple second multi-scale feature maps with unified size into one feature map through a splicing operation, and then fuses the feature map through a convolution layer to output a multi-scale edge perception feature; The multi-scale edge perception feature is input into an edge guided feature fusion module; S3: the edge guided feature fusion module performs matrix transformation on the multi-scale edge perception feature to change from three dimensions to two dimensions; the transformed matrix is a first two-dimensional matrix, and the first two-dimensional matrix is multiplied by its transpose matrix to obtain a pixel-level self-correlation coefficient map; The edge guided feature fusion module inputs the first multi-scale feature map of the highest layer obtained from S1 into a convolution layer and obtains a second two-dimensional matrix through matrix transformation; the second two-dimensional matrix is multiplied by the pixel-level self-correlation coefficient map to obtain a third two-dimensional matrix; the third two-dimensional matrix is transformed into a three-dimensional form to obtain an edge fusion feature map; The edge guided feature fusion module performs adaptive weighted summation on the edge fusion feature map through a convolution layer and the first multi-scale feature map of the highest layer obtained from S1 to obtain a summation result; the summation result is linearly up-sampled until the resolution gradually recovers to the same size as the input remote sensing image to obtain a ground object classification result. 2.The edge-guided multi-scale feature fused optical remote sensing image ground object classification method of claim 1, wherein, In S1, the backbone network adopts ResNet101, which comprises multiple network levels, each level being composed of a convolution layer and a pooling layer, and the backbone network processes the input remote sensing image through multiple network levels of convolution layers and pooling layers. 3.The edge-guided multi-scale feature fused optical remote sensing image ground object classification method of claim 1, wherein, In S3, the second multi-scale feature map is unified in size through an interpolation method, and multiple second multi-scale feature maps with unified size are spliced into one feature map through a splicing operation, and then the spliced feature map is fused through a convolution layer to output a multi-scale edge perception feature, specifically: The second multi-scale feature map is unified in size to 1 / 8 of the input image size by bilinear interpolation; the second multi-scale feature map of the unified size is spliced in the channel dimension to obtain a feature with a channel number of 1024 and a size of 1 / 8 of the input image, and input into a 3*3 convolution layer to further fuse the spliced feature, and output a multi-scale edge perception feature . 4.The method of edge-guided multi-scale feature fused optical remote sensing image ground object classification according to claim 1, 2 or 3, characterized in that, The edge feature extraction module further comprises a supervision process: The edge feature extraction module further inputs the second multi-scale feature map into a convolution layer to obtain a third multi-scale feature map, and obtains a supervised edge feature map through summation operation; The edge true value map is extracted from the true value map of the optical remote sensing image, and the supervised edge feature map is supervised by using the edge true value map, so as to supervise the learning process of the entire edge feature extraction module.
5. The edge-guided multi-scale feature fused optical remote sensing image ground object classification method of claim 4, wherein, The edge feature extraction module further inputs the second multi-scale feature map into a convolution layer to obtain a third multi-scale feature map, and obtains a supervised edge feature map through summation operation, specifically: Each second multi-scale feature map is further input into a 1x1 convolutional layer, and a third multi-scale feature map is output; the number of channels thereof is changed from 256 to 2, and a sigmoid activation function is used; The third multi-scale feature maps are added to obtain a supervised edge feature map.
6. The edge-guided multi-scale feature fused optical remote sensing image ground object classification method of claim 5, wherein, The edge ground truth map is extracted from the ground truth map of the optical remote sensing image, and the learning process of the entire edge feature extraction module is supervised by using the edge ground truth map to supervise the supervised edge feature map, specifically as follows: The edge ground truth map is extracted by the following method: converting the ground truth map label of the optical remote sensing image into one-hot map form, calculating the distance from the non-zero point to the background in the one-hot map, setting a boundary threshold, setting 1 within the threshold, and setting 0 for the rest, and superimposing all one-hot maps to obtain the edge label of all categories; The supervised edge feature is supervised by the edge ground truth map through a cross-entropy loss function for binary classification to realize the learning process of the edge feature extraction module.
7. The edge-guided multi-scale feature fused optical remote sensing image ground object classification method of claim 6, wherein, The edge guidance feature fusion module has an input of multi-scale edge perception features and highest layer features ; Output is: segmentation result ; Input Convert from 3 dimensions to 2 dimensions by matrix transformation and multiply with its transpose to get the pixel-level autocorrelation coefficient map : wherein is a pixel-level autocorrelation coefficient map is the element in the i-th row and j-th column M i is the ith element in the matrix is the ith element in the matrix is the jth element in the matrix is the jth element in the matrix, and N is the number of elements in M. input highest layer feature channel dimension reduction is performed by 1x1 convolution, and matrix dimension transformation is performed to obtain a 2-dimensional form ; and perform matrix multiplication and dimension transformation to obtain edge-guided fusion features : wherein, is a dimension transformation, is a matrix multiplication; Finally, By 1x1 convolutional layer with high-level features Adaptive matrix element summation is performed, and the resolution of the input image is restored by upsampling to obtain the segmentation result; The segmentation result is supervised by the original image ground truth map through a cross-entropy loss function.