Remote sensing image building semantic segmentation method based on HRNet

Abstract

The invention discloses a remote sensing image building semantic segmentation method based on HRNet, and the method comprises the steps: building a multi-scale context aggregation system network architecture, enhancing a receptive field, and effectively extracting the multi-level features of an image; a dynamic region aggregation module (DRAM) is embedded in a high-resolution branch, and a global spatial relationship analysis module (GSRAM) is introduced into a low-resolution branch, so that the capturing capability of a fine structure in a high-resolution image and the global context understanding capability of a low-resolution feature are improved; a Point Rend module is adopted to process edge information during building segmentation, and details and edge performance are optimized through point sampling and point rendering. A Poly Loss framework is introduced to design and optimize a loss function, a focus loss function and a polynomial term are combined, the problem of class imbalance is effectively solved, and the segmentation effect is improved. The invention provides an efficient and accurate method for a remote sensing image building segmentation task, and is suitable for the fields of digital city construction, smart city management and the like.

Description

Technical Field

[0001] The present invention relates to a remote sensing image building semantic segmentation method based on HRNet. Background Art

[0002] Building segmentation is a classic problem in the field of automatic extraction of remote sensing image information. Its main purpose is to identify and segment building areas in remote sensing images. The results of building segmentation are of great significance for understanding the construction and update of urban spatial databases, timely grasping urban building information, urban development planning, and building digital cities.

[0003] Early methods mainly completed the building segmentation of remote sensing images by classification through artificially designed features. However, this traditional method faces the following three limitations in application: (1) The buildings in remote sensing images are complex in shape and diverse in distribution, making complete segmentation difficult and inefficient; (2) Buildings have rich edge information and complex topological relationships with surrounding objects, which increases the difficulty of segmentation; (3) Due to the differences in building materials and ages, their spectral characteristics vary greatly, making it difficult to accurately distinguish them. These limitations have greatly hindered the progress of building segmentation in remote sensing images, making it difficult for segmentation results to meet the needs of modern applications.

[0004] With the development of artificial intelligence, deep learning technology based on convolutional neural networks has been applied to the task of building segmentation in remote sensing images. The deep learning algorithm based on convolutional neural networks (CNN) has a multi-layer structure, which can automatically learn from massive remote sensing data sets and efficiently extract multi-level features of images, accurately capturing the complex structure of buildings and their fine topological relationship with the surrounding environment. Compared with traditional methods, this method shows significant advantages in the accuracy, processing speed and robustness of building segmentation, providing strong technical support for the in-depth application of remote sensing images in digital city construction and smart city management. Summary of the invention

[0005] In view of the problems encountered in traditional methods, the present invention provides a remote sensing image building semantic segmentation method based on HRNet. The method enhances the receptive field by constructing a multi-scale context aggregation architecture, adopts the Point Rend module to process the edge information during building segmentation, and introduces the Poly Loss framework to customize the optimal loss function for the remote sensing building segmentation task.

[0006] The multi-scale context aggregation system constructed by the present invention relies on four parallel branch structures, each branch processing feature maps of different spatial resolutions and numbers of channels respectively, laying a solid foundation for image feature extraction. In view of the problem that high-resolution images are rich in details but the local perception of traditional networks is limited in the task of building segmentation in remote sensing images, a dynamic region aggregation module (DRAM) is designed in the system and embedded in the first two high-resolution branches. The DRAM module effectively enhances the coherence and integrity of feature representation by integrating cross-level local context information, and improves the ability to capture subtle structures in high-resolution images. At the same time, in view of the fact that the low-resolution branches contain rich semantic information despite multiple downsampling, the system designs a global spatial relationship parsing module (GSRAM). This module aims to simulate and strengthen the long-range spatial correlation in low-resolution features, inject global context understanding into high-level feature maps, and effectively make up for the limitations of high-resolution features in semantic expression.

[0007] The central idea of ​​the Point Rend module is to regard the image segmentation task as a rendering process, and process the image through two stages: point sampling and point rendering. In the point sampling stage, Point Rend densely samples the image and randomly selects a set of points as sampling points. These sampling points can cover the details and edge parts in the image. Compared with traditional pixel-level sampling, the number of sampling points is small, which can reduce the amount of calculation. Point sampling can effectively deal with the problem of detail information loss caused by the resolution limit of remote sensing images. In the point rendering stage, for each sampling point, Point Rend uses a small convolutional neural network to process it and generate the corresponding segmentation result. This small network can be regarded as a renderer that converts the local information at the sampling point into a segmentation result. Through local rendering, Point Rend can better handle small structures and edges. In addition, Point Rend also introduces a point resolution loss function to balance global consistency and local detail performance, which is conducive to restoring the complete outline of the building. This loss function encourages the renderer to better capture details and edge information while maintaining global consistency. By minimizing this loss, Point Rend can improve the details and edge performance of the segmentation results.

[0008] The Poly Loss framework is introduced to design and optimize the loss function. The Poly Loss framework regards the loss function as a linear combination of polynomial functions, allowing the importance of different polynomial bases to be easily adjusted according to the remote sensing image building segmentation task and dataset. The loss function designed by this framework allows the model to give a higher loss weight to a minority of building categories during training, thereby effectively solving the problem of reduced classification effect caused by category imbalance. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The algorithm framework diagram proposed by the present invention;

[0010] Figure 2 This is the overall structure diagram of the multi-scale context aggregation system proposed by the present invention;

[0011] Figure 3 This is a network structure diagram of the dynamic area aggregation module (DRAM) proposed by the present invention;

[0012] Figure 4 This is a network structure diagram of the Global Spatial Relationship Resolution Module (GSRAM) proposed by the present invention;

[0013] Figure 5 Shown is a schematic diagram of the adaptive subdivision steps used in the point selection strategy reasoning stage;

[0014] Figure 6 Shown is a schematic diagram of the random sampling strategy used in the point selection strategy training phase. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings and implementation schemes in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0016] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0017] The present invention comprises the following steps:

[0018] Figure 1 The H proposed in the present invention 2 Ret algorithm architecture, which cleverly combines the multi-scale context aggregation system network architecture based on the HRNet baseline network, the boundary refinement module and the Poly loss framework to optimize the building segmentation task in remote sensing images.

[0019] Figure 2 The network architecture diagram of the multi-scale context aggregation system proposed by the present invention specifically includes:

[0020] The network architecture of the multi-scale context aggregation system is based on the HRNet baseline model. The baseline model maintains and fuses multi-scale features through four parallel branches, ensuring that the feature maps extracted from the input image contain both different spatial resolutions and different numbers of channels. In view of the fact that high-resolution features have limited receptive fields but are rich in details, the present invention embeds DRAM modules in its branches, which can effectively aggregate local context information across levels, enhance the contextual understanding ability of high-resolution features, and keep details from being lost. In the low-resolution branch, the present invention introduces a GSRAM module to capture and strengthen the long-distance dependencies between features, and compensate for the global information that may be lost due to reduced resolution. The network architecture of the multi-scale context aggregation system of the present invention includes the following steps:

[0021] Step S1.1, assuming that B1, B2, B3 and B4 are the four feature branches generated by HRNet. After being processed by the DRAM module, a high-resolution feature can be generated by cascading B1 and B2: high = {DRAM(B1),f u [DRAM(B2)]}, where f u is an upsampling operation.

[0022] Step S1.2, the low-resolution branches are cascaded and processed by the GSRAM module, expressed as: F low =GSRAM{B3,f u (B4)}.

[0024] Step S1.3, execute F high and F low The fusion between them obtains the final feature: F fuse = {R1(F high ),f u [R2(F low )]}, where R1 and R2 are both 1×1 convolutions, which can reduce the channel size. fuse Go through another convolutional block to embed into the output.

[0025] Step S1.4, for the final feature F fuse Perform a 1×1 convolution operation to adjust the number of channels to the appropriate dimension: Among them C out Can be set according to specific tasks.

[0026] The DRAM network structure of the dynamic area aggregation module is shown in the figure below: Figure 3 As shown in the figure, it contains average pooling layers with different pooling windows, which effectively integrates multi-scale information. The size of the pooling window is adaptively calculated by the downsampling rate of the feature. It specifically includes the following steps:

[0027] Step S2.1, given the input feature X∈R C×H×W , first apply 1×1 convolution to compress the feature space, reducing the number of channels of X to c=C / 4.

[0028] Step S2.2 then adaptively constructs four parallel pooling operations to generate the descriptor d∈R c×s×s , s∈S d These descriptors are further upsampled to the same spatial size and concatenated with D∈R 2C×H×W .

[0029] In step S2.3, a convolution operation is finally performed on D to reduce the channel dimension, so that its size is restored to C×H×W, ensuring that the input and output of the network match, facilitating seamless integration with subsequent modules.

[0030] The network structure of the global spatial relationship analysis module GSRAM is as follows Figure 4 As shown in Figure 2, the GSRAM module is inspired by the idea of ​​non-local reasoning, and its core goal is to model the long-range spatial correlation of different image sub-regions. First, given an input feature map X∈R C×H×W , we hope to generate a position attention map A∈R HW×HW To highlight the weights between pixel pairs. Then combine the input feature map X and the position attention map A to generate the enhanced feature E'. Then multiply the learned gating signal G by the enhanced feature E' element by element to obtain the gated enhanced feature E * Finally, the gated enhanced feature E * Adding it to the original feature X gives the output feature E. The calculation from X to E can be summarized as follows:

[0031] Step S3.1, apply 1×1 convolution to X to generate three projection features, which help to extract and represent different aspects of the input features and provide a basis for subsequent attention calculations. Denoted as X'∈R C×H×W and U,V∈R C ' ×H×W , C' is C multiplied by the reduction ratio r.

[0032] Step S3.2, reshape the feature map from three dimensions to two dimensions for matrix operations. That is, reshape X' into R C×N , reshape U,V into R C ' ×N , where N = H × W.

[0033] Step S3.3, use the transpose of U, V and the Softmax function σ to calculate the position attention map A∈R N×N :A=σ(U TV). By calculating the attention weights between pixels, the long-range spatial relationships between different sub-regions in the image can be modeled and captured. The Softmax function ensures that the attention weights are between 0 and 1, so that the relationship between each pair of pixels can be effectively and evenly considered.

[0034] Step S3.4, generate enhanced feature E' with X, X' and A: where f r Indicates converting the size of the feature to R C×H×W The reshaping function, Represents an element-wise sum operation.

[0035] Step S3.5, introduce a gating mechanism to control the flow of information, and learn a gating signal G through the convolutional layer: G = Conv 1×1 (X).

[0036] In step S3.6, the sigmoid activation function is used to limit the value of G to between [0, 1] so that the features can be smoothly controlled.

[0037] Step S3.7, multiply the generated gated signal G by the original enhanced feature E' element by element to obtain the gated enhanced feature E * :E * =G×E′, this step can ensure that the model is more sensitive to important features and can dynamically adjust the importance of each feature.

[0038] Step S3.8, finally, in order to ensure that the model can not only utilize the enhanced features but also retain the key information in the original features, the gated enhanced features E * Add element-wise to the original feature X: in Represents element-level addition. This step helps avoid the model from over-relying on enhanced features, thereby improving the stability and robustness of the model.

[0039] The Point Rend module is designed to perform efficient image segmentation through point representation. It is inspired by rendering algorithms in computer graphics, especially those that deal with oversampling and undersampling problems. By applying these methods to image segmentation tasks, Point Rend is able to output high-resolution and clear-boundary segmentation results while maintaining computational efficiency. The Point Rend module mainly includes the following steps:

[0040] Step S4.1, point selection strategy. A small number of ground truth points are selected to perform predictions to avoid over-computation of all pixels in the high-resolution output grid. In point selection, it is hoped that points that predict segmentation labels can be adaptively selected in the image plane, such as points near the edge of the image. To achieve this goal, point selection strategies for the inference stage and the training stage are proposed.

[0041] Step S4.1.1, as Figure 5 Figure 1 shows the steps of adaptive subdivision used in the inference phase of the point selection strategy. The subdivision technique is calculated only at locations where the value is very different from the neighboring values. For other locations, the value is obtained by interpolating the already calculated output values. For each region, it is iterated from low resolution to high resolution.

[0042] Step S4.1.2, as Figure 6 The figure shows a schematic diagram of the random sampling strategy used in the training phase of the point selection strategy. The random sampling strategy randomly selects points from the high-resolution feature map, focusing on areas with rich information or near the boundaries. The random sampling strategy randomly selects N points from the feature map for training. In order to make these points biased towards uncertain areas while retaining a certain degree of uniform coverage, it follows the three principles of overgeneration, importance sampling, and coverage sampling. Through the combination of these three principles, the point selection strategy in the training phase emphasizes areas with high uncertainty while maintaining a certain degree of uniformity, so that the model can effectively learn the characteristics of each area and improve the accuracy of prediction.

[0043] Step S4.2, point feature representation. For each selected point, Point Rend extracts point feature representations from the feature map by bilinear interpolation. These feature representations contain the contextual information around the point, enabling the model to predict more accurate segmentation results.

[0044] Step S4.3, point classification head. The point classification head is a small neural network that receives point feature representations as input and predicts the label of each point independently. This network can be a multi-layer perceptron (MLP) to learn the fine details of the segmentation boundary from the point features.

[0045] The Poly loss framework regards the loss function as a linear combination of polynomial functions, allowing the importance of different polynomial bases to be easily adjusted according to the remote sensing image building segmentation task and dataset. The framework provides a flexible method to adjust the weights of different polynomial bases, thereby customizing the optimal loss function for specific tasks and datasets. Specifically, PolyLoss is compatible with standard cross-entropy loss (CE) and focal loss (FL), incorporating them into its framework as special cases, and refining these loss functions by adjusting the coefficients of the polynomials, which helps to improve the performance and generalization ability of the model when dealing with class imbalanced data.

[0046] The focal loss is designed to solve the problem of class imbalance. It reduces the weight of simple samples and increases the weight of difficult samples, making the model pay more attention to difficult-to-classify samples. The focal loss function is in the form of: L FL = -α(1-P t ) γ log(P t ), where α is the category weight, γ is the focus coefficient, which is used to adjust the weight of easy-to-classify samples, and P t is the probability of predicting the target label.

[0047] The present invention integrates the idea of ​​Poly Loss into Focal Loss by adding a negative polynomial term to Focal Loss: L FL = -α(1-P t ) γ log(P t )+λ(1-P t ), where λ is a hyperparameter of Poly Loss, which is used to control the influence of the polynomial term. By adjusting α and λ, the degree to which the model pays attention to samples of different difficulty levels can be controlled. λ(1-P t ) is a polynomial term borrowed from the PolyLoss framework, which is used to further adjust the loss function to make it more robust when dealing with class imbalance. The introduction of this term can be seen as imposing additional penalties on samples with low model prediction probabilities (i.e., difficult to classify), thereby further increasing the model's attention to these samples.

[0048] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A remote sensing image building semantic segmentation method based on HRNet, characterized in that: The following steps are involved: a) Construct a multi-scale context aggregation system network architecture, which includes four parallel branch structures, each branch processes feature maps with different spatial resolutions and channel numbers, embeds a dynamic region aggregation module in the high-resolution branch, and introduces a global spatial relationship analysis module in the low-resolution branch; b) The Point Rend module is used to process the edge information of building segmentation. The image is processed through two stages: point sampling and point rendering. In the point sampling stage, the image is densely sampled and a group of points are randomly selected as sampling points. In the point rendering stage, a small convolutional neural network is used to process each sampling point to generate the corresponding segmentation result. c) The Poly Loss framework is introduced to design and optimize the loss function. The loss function is regarded as a linear combination of polynomial functions. The importance of different polynomial bases is adjusted according to the remote sensing image building segmentation task and dataset. Combined with the focal loss function, the loss function is refined by adjusting the coefficients of the polynomial.

2. The method for semantic segmentation of remote sensing images of buildings based on HRNet as claimed in claim 1, characterized in that: The dynamic area aggregation module (DRAM) includes an average pooling layer with different pooling windows to effectively integrate multi-scale information, and the size of the pooling window is adaptively calculated by the downsampling rate of the feature.

3. The method for semantic segmentation of remote sensing image buildings based on HRNet as claimed in claim 1, characterized in that: The global spatial relationship parsing module (GSRAM) includes the process of generating a position attention map, generating enhanced features by modeling the long-range spatial correlations of different image sub-regions, and introducing a gating mechanism to control the flow of information.

4. The method for semantic segmentation of remote sensing images of buildings based on HRNet according to claim 1, characterized in that: The method of constructing the multi-scale context aggregation system network architecture is as follows: Step S1.1, set B1, B2, B3 and B4 as the four feature branches generated by HRNet; After being processed by the DRAM module, a high-resolution feature is generated by cascading B1 and B2: high = {DRAM(B1),f u [DRAM(B2)]}, where f u is an upsampling operation; Step S1.2, the low-resolution branches are cascaded and processed by the GSRAM module, expressed as: F low =GSRAM{B3,f u (B4)}; Step S1.3, execute F high and F low The fusion between them obtains the final feature: F fuse = {R1(F high ),f u [R2(F low )]}, where R1 and R2 are both 1×1 convolutions, which can reduce the channel size; F fuse Go through another convolutional block to embed into the output; Step S1.4, for the final feature F fuse Perform a 1×1 convolution operation to adjust the number of channels to the appropriate dimension: Among them C out Can be set according to specific tasks.

5. The method for semantic segmentation of remote sensing image buildings based on HRNet according to claim 4, characterized in that: The dynamic area aggregation module implements the following method: Step S2.1, given the input feature X∈R C×H×W , first apply 1×1 convolution to compress the feature space, reducing the number of channels of X to c=C / 4; Step S2.2 then adaptively constructs four parallel pooling operations to generate the descriptor d∈R c×s×s , s∈S d ; These descriptors are further upsampled to the same spatial size and concatenated, denoted as D∈R 2C×H×W ; In step S2.3, a convolution operation is finally performed on D to reduce the channel dimension, so that its size is restored to C×H×W, ensuring that the input and output of the network match, facilitating seamless integration with subsequent modules.

6. The method for semantic segmentation of remote sensing images of buildings based on HRNet according to claim 5, characterized in that: The global spatial relationship parsing module adopts the following method: Step S3.1, apply 1×1 convolution to X to generate three projection features, which help to extract and represent different aspects of the input features and provide a basis for subsequent attention calculations. Denoted as X'∈R C×H×W and U,V∈R C'×H×W , C' is C multiplied by the reduction ratio r; Step S3.2, reshape the feature map from three dimensions to two dimensions for matrix operations. That is, reshape X' into R C×N , reshape U,V into R C ' ×N , where N = H × W; Step S3.3, use the transpose of U, V and the Softmax function σ to calculate the position attention map A∈R N×N :A=σ(U T V). By calculating the attention weights between pixels, the long-range spatial relationships between different sub-regions in the image can be modeled and captured; the Softmax function ensures that the attention weights are between 0 and 1, so that the relationship between each pair of pixels can be effectively and evenly considered; Step S3.4, generate enhanced feature E' with X, X' and A: where f r Indicates converting the size of the feature to R C×H×W The reshaping function, Represents element-by-element sum operation; Step S3.5, introduce a gating mechanism to control the flow of information, and learn a gating signal G through the convolutional layer: G = Conv 1×1 (X); Step S3.6, use the sigmoid activation function to limit the value of G to [0,1] so that the feature can be smoothly controlled; Step S3.7, multiply the generated gated signal G by the original enhanced feature E' element by element to obtain the gated enhanced feature E * :E * =G×E′, this step can ensure that the model is more sensitive to important features and can dynamically adjust the importance of each feature; Step S3.8, finally, in order to ensure that the model can not only utilize the enhanced features but also retain the key information in the original features, the gated enhanced features E * Add element-wise to the original feature X: in Represents element-wise addition.

Images