Remote sensing image semantic segmentation method and system based on stacked deep residual network

By stacking depth residual networks to perform semantic segmentation in high-resolution remote sensing images, the problem that traditional methods are difficult to utilize context information and spatial details is solved, and a more efficient and accurate semantic segmentation effect is achieved.

CN117611817BActive Publication Date: 2024-11-26SUN YAT SEN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311609856.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-29
Publication Date
2024-11-26
Estimated Expiration
2043-11-29

AI Technical Summary

Technical Problem

There are high inter-class similarity and high in-class differences in semantic segmentation of high-resolution remote sensing images, resulting in complex classification tasks and it is difficult for traditional methods to effectively utilize context information and spatial details, resulting in low semantic segmentation efficiency and poor accuracy.

Method used

Using stacked deep residual network (SDRNet), semantic features are extracted to improve semantic segmentation performance by building backbone networks and hierarchical connected subnets, using expanded residual blocks and attention modules. .

Benefits of technology

It improves the accuracy and efficiency of semantic segmentation, is better than the prior art, and can better utilize the context information and spatial details in high-resolution images, significantly improving the performance of semantic segmentation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117611817B_ABST
    Figure CN117611817B_ABST
Patent Text Reader

Abstract

The present invention relates to the fields of computer vision and remote sensing image processing, and specifically to a remote sensing image semantic segmentation method and system based on a stacked deep residual network, the method comprising: constructing a stacked deep residual network to extract deep features of remote sensing images; using residual learning to scale the depth of the stacked deep residual network; using an expanded residual block to aggregate multi-scale context features; using an intermediate loss to supervise the stacked deep residual network; and using the stacked deep residual network after supervised learning to semantically segment remote sensing image data. The present invention adopts a more computationally efficient stacked deep residual network to improve the network model to handle image land cover classification problems, and extracts semantic features from different levels of the network backbone to improve semantic segmentation performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and remote sensing image processing, and in particular to a remote sensing image semantic segmentation method and system based on a stacked deep residual network. Background Art

[0002] The surface land cover images generated by semantic segmentation of high-resolution remote sensing images provide key decision support and technical guarantee for the formulation of urban planning, resource management and social development policies. However, high inter-class similarity and high intra-class differences are common in land objects, which makes the classification task complicated. In addition, in high-resolution remote sensing images, large objects occlude smaller objects. This object occlusion poses a great challenge to accurate semantic segmentation, making it difficult to distinguish objects of a specific category.

[0003] Traditional methods based on prior knowledge have difficulty in distinguishing complex features in high-resolution remote sensing images. Most shallow classifiers do not fully utilize the rich contextual information in high-resolution images, and are inefficient, time-consuming, and dependent on prior knowledge when performing classification tasks. Some shallow classifiers can achieve good results in certain situations, but may fail to perform in other situations.

[0004] Convolutional neural networks (CNNs) have a complex and deep network structure that can accurately segment information-rich, multi-source heterogeneous remote sensing image data. CNNs use alternating convolutional layers, subsampling layers, and activation functions to hierarchically and automatically learn complex image features. They have a strong learning ability in identifying complex relationships in high-resolution spatial data and can efficiently identify and analyze fine-grained images. In addition, CNNs can make full use of existing computing resources and use GPUs and distributed computing to accelerate computing in parallel or distributed manner.

[0005] Deep convolutional neural networks (DCCNs) are more advanced versions of CNNs with deeper layers designed to learn high semantic representations from large amounts of data. The superior performance of DCCNs in image-related tasks such as object recognition, object detection, and semantic segmentation, as well as their ability to handle complex remote sensing big data, make them undisputedly more popular than shallow models in remote sensing image analysis tasks.

[0006] In most CNNs structures, the subsampling layer gradually reduces the spatial details of the image through pooling operations, thereby capturing a larger field of view and reducing the number of learnable parameters. However, multiple downsampling operations of CNNs will greatly reduce the details of image features, resulting in coarse feature images. Deeper networks are very effective in processing nonlinear and complex relationships between data, retaining more spatial detail information that is essential for accurate semantic segmentation, and are widely used to extract detailed structural features of images in high-resolution images. In addition, residual learning is often used to solve the gradient vanishing problem in DCNNs. Scholars have proposed fully convolutional neural networks, SegNet, etc. to improve the accuracy of semantic segmentation tasks, but how to obtain enough contextual information and make full use of spatial details is still a difficulty in semantic segmentation of high-resolution images, which requires a robust remote sensing data semantic segmentation network model. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a remote sensing image semantic segmentation method and system based on stacked deep residual network, adopting a more computationally efficient framework, namely stacked deep residual network (SDRNet), to improve the network model to deal with the image land cover classification problem, and extract semantic features from different levels of the network backbone to improve the semantic segmentation performance.

[0008] The segmentation method of the present invention specifically adopts the following technical solution: a remote sensing image semantic segmentation method based on a stacked deep residual network, comprising the following steps:

[0009] S1, construct a stacked deep residual network to extract deep features from remote sensing images;

[0010] S2. Scaling the depth of stacked deep residual networks using residual learning;

[0011] S3, using dilated residual blocks to aggregate multi-scale context features;

[0012] S4, supervised learning of stacked deep residual networks using intermediate losses;

[0013] S5. Use the stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data.

[0014] Preferably, the stacked deep residual network of step S1 includes a backbone network and two hierarchically connected first and second subnetworks, each subnetwork including an encoder, an extraction unit and a decoder, wherein the extraction unit includes an expanded residual block and an attention module; the output of the decoder of the first subnetwork is directly transmitted to the encoder of the second subnetwork.

[0015] Further preferably, the encoder of the first sub-network is constructed using five structural blocks, and the high-level image semantics is improved by the decoder of the first sub-network to achieve the upsampling task, and the image size is upsampled to the initial image size;

[0016] Using the encoder of the first sub-network and the encoder of the second sub-network to perform feature encoding on the input image;

[0017] Four structural blocks are used to construct the encoder and decoder of the second sub-network. The decoder of the second sub-network upsamples the image size to the initial image size.

[0018] Construct skip connections from the encoder of the first subnetwork to the decoder of the second subnetwork, concatenating the feature map from the encoder with the output feature map; after each skip connection, perform two additional convolution operations;

[0019] The inputs of the 3rd, 4th, and 5th layers of the encoder of the first sub-network and the inputs of the 2nd, 3rd, and 4th layers of the encoder of the second sub-network are used to construct a self-attention mechanism to form an attention module to extract multi-level features.

[0020] Preferably, step S3 comprises:

[0021] Use dilated residual blocks to integrate multi-scale contextual information;

[0022] Progressive dilation rates are used in different layers of the dilated residual block to obtain the global receptive field.

[0023] The segmentation system of the present invention specifically adopts the following technical solution, a remote sensing image semantic segmentation system based on a stacked deep residual network, including the following modules:

[0024] Feature extraction module, used to construct a stacked deep residual network to perform deep feature extraction on remote sensing images;

[0025] A depth scaling module that uses residual learning to scale the depth of stacked deep residual networks;

[0026] Feature aggregation module, which uses dilated residual blocks to aggregate multi-scale contextual features;

[0027] A supervised learning module that uses intermediate losses to perform supervised learning on stacked deep residual networks;

[0028] The semantic segmentation module uses a stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data.

[0029] After adopting the above technical solution, the present invention has the following advantages compared with the prior art:

[0030] 1. This paper proposes a novel stacked deep residual network to solve the challenging task of semantic segmentation of high-resolution images. The network architecture uses stacked encoder-decoder subnetworks to advance multi-level feature learning and attention mechanisms to refine basic learnable features.

[0031] 2. The expanded residual block in the stacked deep residual network of the present invention expands the receptive field and uses multi-scale reasoning to help further enrich contextual features and improve the accuracy of semantic segmentation.

[0032] 3. A large number of experiments have shown that the lightweight residual network framework proposed in this invention performs well in semantic segmentation tasks and is superior to the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0034] Figure 2 Schematic diagram of the SDRNet framework proposed in the present invention;

[0035] Figure 3 This is a structural diagram of the DRB proposed in the present invention;

[0036] Figure 4 This is a structural diagram of the core of the DRB proposed by the present invention;

[0037] Figure 5 Schematic diagram of the segmentation result of the present invention on the Vaihingen dataset; wherein (a) is the input image, (b) is the ground truth, and (c) is the prediction result;

[0038] Figure 6 Schematic diagram of the segmentation results of the present invention on the Potsdam dataset, where (a) is the input image, (b) is the ground truth, and (c) is the prediction result. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Example 1

[0041] This embodiment is based on the remote sensing image semantic segmentation method of stacked deep residual network. The implementation process can be found in Figure 1 , specifically including the following steps:

[0042] S1. Construct a stacked deep residual network to perform deep feature extraction on remote sensing images.

[0043] The remote sensing images are high-resolution remote sensing image data. The Vaihingen dataset and Potsdam dataset of ISPRS are used for model training, verification and testing. The two datasets are divided into six common land cover categories: impervious surfaces (white), buildings (blue), low vegetation coverage areas (cyan), trees (green), cars (yellow) and debris (red). The Potsdam dataset contains 38 images with a size of 6000×6000 pixels and a spatial resolution of 5cm. The experiment selects RGB images of 14 of them as test images. The Vaihingen dataset contains 33 images with a size of 2494×2064 pixels and a spatial resolution of 9cm. The experiment uses the near-infrared, red and green bands.

[0044] The specific steps for deep feature extraction in this step include the following:

[0045] S11. The stacked deep residual network used in this embodiment is also called a stacked encoder-decoder network, which includes a backbone network and two hierarchically connected sub-networks, and the backbone network uses the ResNet50 model. Each sub-network includes an encoder, an extraction unit including dilated residual blocks (DRB) and an attention module, and a decoder.

[0046] In the stacked deep residual network, the two encoders help generate robust features from the input image, and the two decoders enable reconstruction of spatial details. In addition, the output of the decoder 1 of the first sub-network is directly transmitted to the encoder 2 of the second sub-network, which can reduce feature loss. The SDRNet framework in this embodiment is as follows: Figure 2 shown.

[0047] S12. A pre-trained ResNet50 network is used on encoder 1 of the first sub-network to enrich the basic feature learning ability of the network and reduce the network's demand for massive training labels.

[0048] Since the scene is different from the ordinary images used in the original training dataset, this embodiment redesigns the encoder 2 of the second sub-network to activate deeper layers and effectively learn more advanced features.

[0049] In the stacked deep residual network of this embodiment, the symmetrical encoder and decoder are stacked to form a spatial reconstruction subnetwork, which encodes the rich spatial details in the image and generates a new feature map. The encoder 1 of the first subnetwork uses five structural blocks to gradually reduce the image dimension as the image channel features increase; the decoder 1 of the symmetrical first subnetwork improves the high-level image semantics to achieve the upsampling task, and the image size is upsampled to the initial image size. A 2×2 bilinear upsampling operation is performed in the decoder.

[0050] S13. Use encoder 1 of the first sub-network and encoder 2 of the second sub-network to perform feature encoding on the input image. Each convolution block in each encoder performs two convolution operations with a convolution kernel size of 3×3, and then performs batch normalization to reduce internal covariate shift, accelerate network training and enhance the stability of the model.

[0051] S14. Apply the ReLU activation function to enhance the nonlinearity of the model, and perform the maximum pooling operation to reduce the size of the feature map, reduce the number of model parameters and the amount of calculation, while retaining the most important feature information.

[0052] S15. The encoder 2 and decoder 2 of the second sub-network are constructed using four structural blocks to reduce the number of learnable features and network complexity. Since each structural block in the decoder performs a two-by-two bilinear upsampling operation on the feature input, the dimension of the input feature map is doubled. The decoder 2 of the second sub-network upsamples the image size to the initial image size.

[0053] S16. Construct skip connections from the encoder 1 of the first sub-network to the decoder 2 of the second sub-network to maintain high spatial resolution and improve the overall quality of the output feature map. After each skip connection, two additional 3×3 convolution operations are performed, each followed by a batch normalization step and a ReLU activation function; finally, mapping is performed through a softmax function.

[0054] Therefore, both decoders upsample the image size to the initial image size before connecting to the multi-class softmax function.

[0055] This embodiment connects the feature map from the encoder to the output feature map through a jump connection. There is a jump connection from the encoder 1 of the first sub-network to the decoder 1 of the first sub-network, and a jump connection from the encoder 1 of the first sub-network and the encoder 2 of the second sub-network to the decoder 2 of the second sub-network, in order to maintain a high spatial resolution and improve the overall quality of the output feature map.

[0056] S17. Use the inputs of the 3rd, 4th, and 5th layers of encoder 1 of the first sub-network and the inputs of the 2nd, 3rd, and 4th layers of encoder 2 of the second sub-network to construct a self-attention mechanism to form an attention module to extract multi-level features.

[0057] Since high-resolution remote sensing images contain redundant and unnecessary features that are irrelevant to the output class, the attention mechanism limits irrelevant areas in the input image to highlight the features of certain specific categories. Inspired by the human visual system, the self-attention mechanism improves the network to focus on specific important areas, thereby significantly reducing the cost of learning features from irrelevant areas and redundant data. In addition, since it is practically impossible to activate all learnable parameters in a deep convolutional neural network, the network is made to limit features and only utilize features related to a specific category.

[0058] This embodiment avoids using the first layer (ie, the initial layer) of the encoder because the features learned by the initial layer are simple and basic and cannot provide sufficient information to complete complex segmentation tasks, so the first layer of the encoder is not used.

[0059] In this embodiment, the self-attention mechanism is defined as:

[0060] M s (F) = σ(F a×a ([AvrPool(F); MaxPool(F)]))

[0061] Among them, σ represents the sigmoid function, F a×a represents a convolution operation with a filter size of a×a, AvrPool(F) represents an average pooling operation, and MaxPool(F) represents a maximum pooling operation.

[0062] S2. Scaling the depth of stacked deep residual networks using residual learning.

[0063] A residual learning strategy is used to solve the gradient vanishing problem, and the stacked convolution blocks are replaced with identity mapping to build a deep network that is not affected by the gradient vanishing problem. Through skip connections, the network gradient can be transferred without passing through nonlinear activation functions, thereby alleviating gradient explosion or vanishing. In addition, skip connections also improve the gradient flow during backpropagation, that is, improve the backward gradient flow, and accelerate the convergence of deep networks.

[0064] Accordingly, by decomposing the image into the encoding stage and the decoding stage, and gradually adding more detail information in the decoding stage, the original image can be better reconstructed.

[0065] The residual function is defined as:

[0066] y=F(x,W)+x

[0067] Where x represents the input, y represents the output feature map, F(x, W) represents the residual connection function, and W represents the weight parameter.

[0068] S3. Use the dilated residual block DRB to aggregate multi-scale context features. The specific steps include:

[0069] S31. Integrating multi-scale contextual information using dilated residual blocks.

[0070] In order to obtain more contextual feature information and achieve multi-scale contextual information aggregation without sacrificing image resolution, the dilated convolution uses larger sparse kernels in the pooling layer and convolution layer.

[0071] In this embodiment, the dilated residual block is proposed based on a two-dimensional dilated convolution operation, and the two-dimensional dilated convolution operation is defined as:

[0072]

[0073] Where y(m,n) is the m-dimensional and n-dimensional output of the dilated convolution, x(m,n) is the m-dimensional and n-dimensional input of the dilated convolution, w(i,j) is the filter of size i×j of the dilated convolution, and the parameter r represents the dilation rate.

[0074] When the dilation rate is equal to 1, the dilated convolution works normally; when the dilation rate is less than 1, it indicates a detailed segmentation of the feature map, which requires more training time; when the dilation rate is greater than 1, the receptive field increases without increasing the number of parameters or computational requirements. The receptive field range can be adjusted by different dilation rates.

[0075] S32. Use progressive dilation rates in different layers of the dilated residual block to obtain the global receptive field.

[0076] Based on the concept of dilated convolution, a dilated residual block is adopted as the dilated residual block, which uses progressive dilation rates in different layers to achieve the full receptive field without losing resolution or coverage.

[0077] This embodiment adopts a dilation kernel with a progressive dilation rate and a jump connection to enhance the information flow between connection layers, and reduces the grid effect and information continuity loss by ensuring that the obtained dilated convolution kernel can cover the responses of all areas in the image without increasing the number of convolution kernel parameters.

[0078] The gridding problem is the loss of information continuity caused by not all pixels getting kernel responses through the "chessboard effect". In order to solve the problem that large dilation rates will cause gridding effects and inhibit the performance of dilated kernels, the DRB block (dilated residual block) uses dilated kernels with progressive dilation rates to complete pixel-level dense prediction tasks. Connections are gradually and extensively established between layers through jump connections, which enhances the flow of information between connected layers and efficiently transmits information. By ensuring that the generated dilated kernel can get responses from all areas of the image without increasing the number of kernel parameters, the effect of image processing is significantly improved.

[0079] In this embodiment, the convolution kernel of the dilated residual block DRB is defined as:

[0080] M i′ =max[M i′+1 -2r i′ , M i′+1 -2(M i′+1 -r i′ ), r i′ ]

[0081] where r i′ is the dilation rate of the i′th layer, M i′ is the maximum generation rate of the i′th layer, and the total number of layers is k. In this embodiment, the structure of the dilated residual block is as follows: Figure 3 As shown, the kernel of the dilated residual block is Figure 4 shown.

[0082] S4. Supervised learning of stacked deep residual networks using intermediate losses.

[0083] In this embodiment, an intermediate loss is introduced at the end of the sub-network to overcome the gradient decay problem and improve the effective learning from deep to shallow layers in the deep network; and the optimization of the intermediate layer is improved and optimized by calculating the performance difference between the model and the actual situation to improve the gradient flow and enhance the learning during the back propagation process. The specific steps include:

[0084] S41. Place the loss function at the end of the two sub-networks respectively, and define the loss function as:

[0085]

[0086] Where N is the number of categories, W j represents the weight of category j, p j and p k Represent the predicted value and true value of category j respectively.

[0087] S42. Define the total loss function as:

[0088] L T =(α×MainLoss)+(β×InterL1)

[0089] Where α and β are the respective weights in the network, InterL1 is the loss value of the output layer, and MainLoss is the loss value at the end of the first sub-network (the weights are scaled 1:1).

[0090] S5. Use the stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data.

[0091] In this embodiment, the segmentation results of the Vaihingen dataset and the Potsdam dataset are as follows: Figure 5 , Figure 6 shown.

[0092] Example 2

[0093] This embodiment is based on the same inventive concept as Embodiment 1, and provides a remote sensing image semantic segmentation system based on a stacked deep residual network, including the following modules:

[0094] Feature extraction module, used to construct a stacked deep residual network to perform deep feature extraction on remote sensing images;

[0095] A depth scaling module that uses residual learning to scale the depth of stacked deep residual networks;

[0096] Feature aggregation module, which uses dilated residual blocks to aggregate multi-scale contextual features;

[0097] A supervised learning module that uses intermediate losses to perform supervised learning on stacked deep residual networks;

[0098] The semantic segmentation module uses a stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data.

[0099] The modules of this embodiment are respectively used to implement the steps of Embodiment 1. For the detailed implementation process, please refer to Embodiment 1.

[0100] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A remote sensing image semantic segmentation method based on stacked deep residual network, characterized in that: The following steps are involved: S1, construct a stacked deep residual network to extract deep features from remote sensing images; S2. Scaling the depth of stacked deep residual networks using residual learning; S3, using dilated residual blocks to aggregate multi-scale context features; S4, supervised learning of stacked deep residual networks using intermediate losses; S5. Use the stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data; The stacked deep residual network of step S1 includes a backbone network and two hierarchically connected first and second subnetworks, each of which includes an encoder, an extraction unit, and a decoder, wherein the extraction unit includes a dilated residual block and an attention module; The output of the decoder of the first sub-network is directly transmitted to the encoder of the second sub-network; The encoder of the first sub-network is constructed using five structural blocks. The decoder of the first sub-network improves high-level image semantics to achieve the upsampling task and upsamples the image size to the initial image size. Using the encoder of the first sub-network and the encoder of the second sub-network to perform feature encoding on the input image; Four structural blocks are used to construct the encoder and decoder of the second sub-network. The decoder of the second sub-network upsamples the image size to the initial image size. Build a skip connection from the encoder of the first sub-network to the decoder of the second sub-network, connecting the feature map from the encoder with the output feature map; After each skip connection, two additional convolution operations are performed; Using the inputs of the 3rd, 4th, and 5th layers of the encoder of the first sub-network and the inputs of the 2nd, 3rd, and 4th layers of the encoder of the second sub-network, a self-attention mechanism is constructed to form an attention module to extract multi-level features; Step S3 includes: Use dilated residual blocks to integrate multi-scale contextual information; Using progressive dilation rates in different layers of the dilated residual block to obtain the global receptive field; Adopt dilated residual blocks as dilated residual blocks, using progressive dilation rates in different layers to achieve a complete receptive field; The convolution kernel of the dilated residual block DRB is defined as: M i′ =max[M i′+1 -2r i′ ,M i′+1 -2(M i′+1 -r i′ ),r i′ ] where r i′ is the dilation rate of the i′th layer, M i′ is the maximum generation rate of the i′th layer, and the total number of layers is k.

2. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The self-attention mechanism is defined as: M s (F)=σ(F a×a ([AvrPool(F);MaxPool(F)])) Among them, σ represents the sigmoid function, F a×a represents a convolution operation with a filter size of a×a, AvrPool(F) represents an average pooling operation, and MaxPool(F) represents a maximum pooling operation.

3. The remote sensing image semantic segmentation method according to claim 1, characterized in that: In step S2, the stacked convolution blocks are replaced by identity mapping to build a deep network that is not affected by the gradient vanishing problem.

4. The remote sensing image semantic segmentation method according to claim 1, characterized in that: In step S2, the residual function is defined as: y=F(x,W)+x Where x represents the input, y represents the output feature map, F(x, W) represents the residual connection function, and W represents the weight parameter.

5. The remote sensing image semantic segmentation method according to claim 1, characterized in that: The dilated residual block is proposed based on the two-dimensional dilated convolution operation, which is defined as: Where y(m,n) is the m-dimensional and n-dimensional output of the dilated convolution, x(m,n) is the m-dimensional and n-dimensional input of the dilated convolution, w(i,j) is the filter of size i×j of the dilated convolution, and the parameter r represents the dilation rate.

6. A remote sensing image semantic segmentation system based on a stacked deep residual network, used to implement the remote sensing image semantic segmentation method according to claim 1, characterized in that: Includes the following modules: Feature extraction module, used to construct a stacked deep residual network to perform deep feature extraction on remote sensing images; A depth scaling module that uses residual learning to scale the depth of stacked deep residual networks; Feature aggregation module, which uses dilated residual blocks to aggregate multi-scale contextual features; A supervised learning module that uses intermediate losses to perform supervised learning on stacked deep residual networks; The semantic segmentation module uses a stacked deep residual network after supervised learning to perform semantic segmentation on remote sensing image data.