Semi-supervised remote sensing image semantic segmentation system and method

Abstract

The invention relates to the technical field of image processing, in particular to a semi-supervised remote sensing image semantic segmentation system and method.The system comprises a fully supervised network and a semi-supervised network of a Mean Tester framework, and a depolarization branch in the semi-supervised network obtains a content-free input image of a label-free remote sensing image; extracting a deviation item of the category of the remote sensing image from the content-free input image; the consistency branch inputs the remote sensing image added with random disturbance into a Mean Teamer framework, and generates adversarial disturbance with the optimal direction through a virtual adversarial training strategy; and inputting the remote sensing image added with the adversarial disturbance into a Mean Teacher framework to obtain an adversarial disturbance pseudo tag, and calculating the consistency loss between the adversarial disturbance pseudo tag and the predicted segmented image. According to the semi-supervised remote sensing segmentation method, the precision, the boundary recovery capability and the robustness of the semi-supervised remote sensing segmentation under the low-annotation condition are improved through cooperative work of the Mean Teacher framework, the depolarization branch and the consistency branch.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a semi-supervised remote sensing image semantic segmentation system and method. Background Technology

[0002] Semantic segmentation of remote sensing images requires assigning semantic labels to each pixel for applications such as urban planning and environmental monitoring. However, high-resolution remote sensing images have many pixels and complex ground features, making pixel-level annotation costly and scarce, which limits the application of supervised depth segmentation methods.

[0003] Semi-supervised learning can alleviate insufficient labeling by utilizing a large amount of unlabeled imagery, but existing semi-supervised methods suffer from problems such as poor pseudo-label quality, class imbalance, weak performance of boundary / small targets, and poor robustness to strong perturbations in remote sensing scenes. Specifically: (1) Class imbalance and confirmation bias of pseudo-labels: Pseudo-labels generated from unlabeled data often favor the majority class or network priors, leading to model learning bias and amplification of errors; (2) Weak boundary and texture modeling capabilities lead to false detection and missed detection of details (small targets / boundaries): Conventional attention / convolution methods are insufficient in responding to noisy and complex texture regions; (3) The model is weak in responding to disturbances / uncertainties in unlabeled samples, resulting in insufficient robustness: In remote sensing scenarios with noise, occlusion or spectral changes, the model is prone to producing unstable predictions. Summary of the Invention

[0004] In view of this, the present invention aims to provide a semi-supervised remote sensing image semantic segmentation system and method, which improves the accuracy, boundary recovery ability and robustness of semi-supervised remote sensing segmentation under low annotation conditions by working together with the bias removal branch and the consistency branch under the Mean Teacher framework.

[0005] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A semi-supervised remote sensing image semantic segmentation system includes a fully supervised network and a semi-supervised network. The fully supervised network segments an input first remote sensing image to obtain a predicted segmentation result and shares the network weights from the fully supervised network with the semi-supervised network. The semi-supervised network includes: a bias removal branch, which acquires a content-free input image of the unlabeled remote sensing image and extracts a bias term representing the category of the remote sensing image from the content-free input image; a student branch, which performs hard enhancement on a second remote sensing image and segments the hard-enhanced image according to the shared weights to obtain pseudo-labels; a teacher branch, which performs soft enhancement on the second remote sensing image, extracts features from the soft-enhanced image according to the shared weights, subtracts the bias term from the extracted features, decodes the features after subtracting the bias term, and obtains the predicted segmentation image corresponding to the second remote sensing image; and a consistency branch, which inputs a second remote sensing image with added random perturbations into the student branch and generates an optimal adversarial perturbation through a virtual adversarial training strategy; inputs the second remote sensing image with added adversarial perturbations into the student branch to obtain adversarial perturbation pseudo-labels, and calculates the consistency loss between the adversarial perturbation pseudo-labels and the predicted segmentation image.

[0006] Furthermore, the fully supervised network includes a first encoder, an attention enhancement module, and a first decoder, wherein: the first encoder encodes the features of the input remote sensing image, and inputs the obtained encoded features into the attention enhancement module for feature enhancement; the first decoder decodes the features from the attention enhancement module to obtain the corresponding labels.

[0007] Furthermore, both the teacher and student branches include a second encoder, an attention enhancement module, and a second decoder. The weight parameters of the second encoder and the second decoder are shared by the first encoder and the first decoder. In the student branch, the second encoder extracts features from the hard-enhanced image, inputs the extracted features into the attention enhancement module, and the enhanced features are input into the second decoder to obtain pseudo-labels. In the teacher branch, the second encoder extracts features from the soft-enhanced image, inputs the extracted features into the attention enhancement module, subtracts the bias term from the enhanced features, and then inputs them into the second decoder for decoding to obtain the predicted segmentation image.

[0008] Furthermore, the bias removal branch includes a content removal module, a second encoder, and an attention enhancement module, wherein: the content removal module performs full-image mean normalization on the unlabeled remote sensing image to obtain a content-free input image; the second encoder extracts features from the content-free input image and inputs the extracted features into the attention enhancement module to obtain the bias term.

[0009] Furthermore, the attention enhancement module includes channel attention and spatial attention. In the channel attention process: the input features are subjected to max pooling, average pooling, and median pooling; the features after each pooling are processed by a weight-sharing multilayer perceptron to obtain channel weights, which are then appended to the input features to obtain channel attention features. In the spatial attention process: the channel attention features are subjected to multi-scale dilated convolution, and the resulting multi-scale features are fused to obtain the spatial weights of the channel attention features; the spatial weights are then assigned to the corresponding values ​​in the channel attention features to obtain the output features.

[0010] Furthermore, soft enhancement includes rotating, resizing, and translating the second remote sensing image; hard enhancement includes cropping the second remote sensing image and then stitching it together.

[0011] A semi-supervised remote sensing image semantic segmentation method includes: S1: Obtain labeled and unlabeled remote sensing image datasets, preprocess the remote sensing image datasets to obtain the training set; S2: Construct a semi-supervised remote sensing image semantic segmentation system as provided in this invention, and train the semantic segmentation system using the training set obtained in step S1 to obtain a semantic segmentation model; S3: Repeat step S2 and adjust the hyperparameters during training until the optimal semantic segmentation model is obtained; S4: Input the remote sensing image to be semantically labeled into the optimal semantic segmentation model obtained in step S3 to obtain semantic segmentation labels.

[0012] Compared with the prior art, the present invention can achieve the following beneficial effects: (1) In the semi-supervised remote sensing image semantic segmentation system and method described in this invention, the bias removal branch dynamically reduces the pseudo-label bias caused by the majority class or network prior, making the pseudo-labels more balanced. The model will not be weakened by the minority class during training, reducing the negative impact of confirmation bias on model training. The consistency branch trains the model to be consistent with the perturbation of unlabeled samples, making the model less sensitive to input perturbation, spectral interference or sensor noise, improving the robustness of the model. At the same time, adversarial training helps the model output a more stable probability distribution in the boundary / mixed pixel region, reducing the propagation of false pseudo-labels and improving the reliability of pseudo-labels in the uncertain interval. The consistency branch and the bias removal branch work together under the Mean Teacher framework, which can achieve better segmentation performance than existing methods in the case of scarce labels. (2) In the semi-supervised remote sensing image semantic segmentation system and method described in this invention, the attention enhancement module is more robust to noise through median pooling. Combined with multi-scale spatial context, it can better recover small targets and complex boundaries, enhance boundary and texture expression, improve noise resistance, and perform more stably in high-resolution remote sensing image noise or small target interference scenarios. Attached Figure Description

[0013] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic diagram of the semi-supervised remote sensing image semantic segmentation system described in the embodiments of the present invention; Figure 2 A schematic diagram of the consistent branch described in the embodiment of the present invention; Figure 3 A schematic diagram of the attention enhancement module described in an embodiment of the present invention; Figure 4 A flowchart illustrating the semi-supervised remote sensing image semantic segmentation method described in an embodiment of the present invention; Figure 5 A comparison diagram showing the results of the method provided by this invention with existing methods. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0016] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0017] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0018] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] like Figure 1 and Figure 2As shown in the embodiments of the present invention, the semi-supervised remote sensing image semantic segmentation system includes a fully supervised network and a semi-supervised network. The fully supervised network segments the input first remote sensing image to obtain a predicted segmentation result and shares the network weights in the fully supervised network with the semi-supervised network. The semi-supervised network includes a bias correction branch, a student branch, a teacher branch, and a consistency branch. The semi-supervised network uses a Mean Teacher framework as the training backbone. The bias correction branch estimates the prior class bias of the unlabeled image and corrects the pseudo-labels, thereby improving the class balance and quality of the pseudo-labels. The consistency branch introduces adversarial perturbations and applies consistency regularization to the multi-branch outputs, enhancing the model's robustness to unlabeled samples and the quality of the pseudo-labels. The bias correction branch obtains the content-free input image of the unlabeled remote sensing image and extracts the class bias term of the remote sensing image from the content-free input image. The student branch performs hard enhancement on the second remote sensing image and segments the hard-enhanced image according to the shared weights to obtain pseudo-labels. The teacher branch performs soft enhancement on the second remote sensing image, extracts features from the enhanced image based on shared weights, subtracts a bias term from the extracted features, and decodes the features after subtracting the bias term to obtain the predicted segmentation image corresponding to the second remote sensing image. The consistency branch inputs the second remote sensing image with added random perturbations into the student branch, generates the optimal adversarial perturbation through a virtual adversarial training strategy, inputs the second remote sensing image with added adversarial perturbations into the student branch to obtain adversarial perturbation pseudo-labels, and calculates the consistency loss between the adversarial perturbation pseudo-labels and the predicted segmentation image. The consistency branch is as follows: Figure 2 As shown, random perturbations are introduced into the unlabeled image input, and an adversarial perturbation r with optimal orientation is generated through a virtual adversarial training strategy. adv Then input the perturbed image into the model and compare it with the original image. Figure 1 The consistency loss Loss_vat is calculated using the consistency output, thereby enhancing the model's robustness to uncertain regions in pseudo-labels.

[0020] In some embodiments, the fully supervised network includes a first encoder, an attention enhancement module, and a first decoder, wherein: the first encoder performs feature encoding on the input remote sensing image, inputs the obtained encoded features into the attention enhancement module for feature enhancement, and the first decoder decodes the features from the attention enhancement module to obtain the corresponding labels. In this embodiment of the invention, the first encoder uses a ResNet101 structure for feature encoding, and the first decoder uses a structure symmetrical to the ResNet101 structure for upsampling decoding.

[0021] In some embodiments, both the teacher and student branches include a second encoder, an attention enhancement module, and a second decoder. The weight parameters of the second encoder and the second decoder are shared by the first encoder and the first decoder. In the student branch, the second encoder extracts features from the hard-enhanced image, inputs the extracted features into the attention enhancement module, and the enhanced features are input into the second decoder to obtain pseudo-labels. In the teacher branch, the second encoder extracts features from the soft-enhanced image, inputs the extracted features into the attention enhancement module, subtracts the bias term from the enhanced features, and then inputs them into the second decoder for decoding to obtain the predicted segmentation image. In some embodiments, soft enhancement includes conventional data augmentation operations, including rotating, resizing, and translating the second remote sensing image. Hard enhancement uses the Cutmix method, specifically including cropping and then stitching the second remote sensing image.

[0022] In some embodiments, the debiasing branch includes a content removal module, a second encoder, and an attention enhancement module, wherein: the content removal module performs full-image averaging on the unlabeled remote sensing image to obtain a content-free input image; the second encoder extracts features from the content-free input image and inputs the extracted features into the attention enhancement module to obtain the bias term. Since debiasing weight sharing can only be performed between identical models, in this embodiment of the invention, the second encoder adopts the same ResNet101 structure as the first encoder, and the second decoder adopts a symmetrical decoding structure of the same ResNet101 structure as the first decoder.

[0023] In some embodiments, the attention enhancement module introduces three feature extraction methods—average pooling, max pooling, and median pooling—in the channel attention branch, then generates three sets of attention weights through a shared multilayer perceptron and fuses them. The spatial attention branch extracts contextual information through multi-scale dilated convolution kernels to generate a spatial attention map, which is used to enhance the modeling ability of ground object edges and texture regions. Specifically, the feature processing in the attention enhancement module includes channel attention operations and spatial attention operations. In the channel attention operation: the input features are subjected to max pooling, average pooling, and median pooling; the features after the three pooling operations are processed by a multilayer perceptron with shared weights to obtain channel weights, and then the channel weights are added to the input features to obtain channel attention features; in the spatial attention operation: the channel attention features are subjected to multi-scale dilated convolution, and the obtained multi-scale features are fused to obtain the spatial weights of the channel attention features; the spatial weights are added to the channel attention features to obtain the output features.

[0024] The attention enhancement module in this embodiment of the invention is as follows: Figure 3As shown, specifically, in the channel attention operation, after performing a 1×1 convolution and GeLU activation on the input feature X, the resulting feature is simultaneously subjected to three pooling methods: max pooling, average pooling, and median pooling. The core idea of ​​median pooling is to use median filtering to replace the current local region with the median, thereby achieving feature dimensionality reduction. This method can reduce noise while preserving edge and texture information, and resist salt-and-pepper noise. The features after the three pooling methods are input into a multilayer perceptron, and the weights of the three multilayer perceptrons are mutually exclusive. In each multilayer perceptron, the input feature is subjected to a 1×1 convolution, GeLU activation, and then another 1×1 convolution, and the resulting feature is then activated with sigmoid to obtain the corresponding channel weights extracted by the three pooling methods. The three channel weights are then summed pixel-by-pixel to obtain the complete channel weight F. c Then adjust the channel weight F c Appended to the input feature X, the channel attention feature X is obtained. c In the spatial attention operation, after performing a 5×5 convolution on the channel attention features, dilated convolutions of various scales including 1×7, 7×1, 1×21, and 21×1 are performed. The output features of the dilated convolutions of various scales are then summed pixel by pixel, followed by a 1×1 convolution to obtain the spatial weights F. s Spatial weights are added to the channel attention features, and the resulting features are convolved with a 1×1 convolution to obtain the output feature X of the attention enhancement module. s .

[0025] This invention also provides a semi-supervised remote sensing image semantic segmentation method, combining... Figure 1 and Figure 4 ,include: S1: Obtain labeled and unlabeled remote sensing image datasets, preprocess the remote sensing image datasets to obtain the training set. To ensure the generalization of the method, i.e., to cope with remote sensing images in different scenarios, in this embodiment of the invention, the ISPRS Potsdam dataset, the ISPRS Vaihingen dataset, and the LoveDA dataset are integrated to obtain the training set.

[0026] S2: Construct a semi-supervised remote sensing image semantic segmentation system as provided in this invention. Train the semantic segmentation system using the training set obtained in step S1 to obtain a semantic segmentation model. In this embodiment of the invention, the cross-entropy loss function is used to train the semantic segmentation system.

[0027] S3: Repeat step S2 and adjust the hyperparameters during training until the optimal semantic segmentation model is obtained. In this embodiment of the invention, the preferred hyperparameters include: a training batch size of 8, an optimizer using SGD with weight decay of 0.0001, and an initial learning rate of 0.001.

[0028] S4: Input the remote sensing image to be semantically labeled into the optimal semantic segmentation model obtained in step S3 to obtain semantic segmentation labels.

[0029] To verify the segmentation effect of the method provided by this invention on remote sensing images, embodiments of this invention utilize existing methods and the method provided by this invention to segment remote sensing images from the Potsdam and Vaihingen datasets. The segmentation results are compared. Figure 5 As shown, existing methods include LSST, WSCL, DWL, and UniMatch. Blue represents building tags, cyan represents vegetation tags, green represents tree tags, and yellow represents vehicle tags. By comparing the area within the red box, it can be seen that the method provided by this invention can more accurately recover the target edge and has a stronger ability to recognize small targets (such as cars).

[0030] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0031] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A semi-supervised remote sensing image semantic segmentation system, characterized in that, Including fully supervised networks and semi-supervised networks, among which: The fully supervised network segments the first input remote sensing image to obtain the predicted segmentation result, and shares the network weights in the fully supervised network with the semi-supervised network; Semi-supervised networks include: Remove the biased branches, obtain the content-free input image of the unlabeled remote sensing image, and extract the category bias term of the remote sensing image from the content-free input image; The student branch performs hard enhancement on the second remote sensing image, and segments the hard-enhanced image according to the shared weights to obtain pseudo-labels; The teacher branch performs soft enhancement on the second remote sensing image, extracts features from the soft-enhanced image based on the shared weights, subtracts the bias term from the extracted features, and decodes the features after subtracting the bias term to obtain the predicted segmentation image corresponding to the second remote sensing image. The consistency branch inputs a second remote sensing image with added random perturbations into the student branch, and generates the optimal adversarial perturbation through a virtual adversarial training strategy; the second remote sensing image with added adversarial perturbations is input into the student branch to obtain adversarial perturbation pseudo-labels, and the consistency loss between the adversarial perturbation pseudo-labels and the predicted segmentation image is calculated.

2. The semi-supervised remote sensing image semantic segmentation system according to claim 1, characterized in that, The fully supervised network consists of a first encoder, an attention enhancement module, and a first decoder, wherein: The first encoder encodes the features of the input remote sensing image, and inputs the obtained encoded features into the attention enhancement module for feature enhancement. The first decoder decodes the features from the attention enhancement module to obtain the corresponding labels.

3. The semi-supervised remote sensing image semantic segmentation system according to claim 2, characterized in that, Both the teacher branch and the student branch include a second encoder, an attention enhancement module, and a second decoder. The weight parameters of the second encoder and the second decoder are shared by the first encoder and the first decoder. In the student branch, the second encoder extracts features from the hard-enhanced image, inputs the extracted features into the attention enhancement module, and inputs the enhanced features into the second decoder to obtain pseudo-labels. In the teacher branch, the second encoder extracts features from the soft-enhanced image, inputs the extracted features into the attention enhancement module, subtracts the bias term from the enhanced features, and then inputs them into the second decoder for decoding to obtain the predicted segmentation image.

4. The semi-supervised remote sensing image semantic segmentation system according to claim 3, characterized in that, The de-biasing pathway includes a content removal module, a second encoder, and an attention enhancement module, where: The content removal module performs full-image mean normalization on the unlabeled remote sensing image to obtain a content-free input image. The second encoder extracts features from the empty input image and inputs the extracted features into the attention enhancement module to obtain the bias term.

5. The semi-supervised remote sensing image semantic segmentation system according to claim 4, characterized in that, The attention enhancement module includes channel attention and spatial attention, among which: In the process of channel attention: the input features are subjected to max pooling, average pooling and median pooling; the features after the three pooling are processed by a multilayer perceptron with weight sharing to obtain channel weights, and then the channel weights are attached to the input features to obtain channel attention features. In the process of spatial attention: multi-scale dilated convolution is performed on the channel attention features, and the resulting multi-scale features are fused to obtain the spatial weights of the channel attention features; the spatial weights are added to the channel attention features accordingly to obtain the output features.

6. The semi-supervised remote sensing image semantic segmentation system according to claim 1, characterized in that, Soft enhancement includes rotating, resizing, and translating the second remote sensing image; Hard enhancement involves cropping the second remote sensing image and then stitching it together.

7. A semi-supervised semantic segmentation method for remote sensing images, characterized in that, include: S1: Obtain labeled and unlabeled remote sensing image datasets, preprocess the remote sensing image datasets to obtain the training set; S2: Construct a semi-supervised remote sensing image semantic segmentation system as described in any one of claims 1 to 6, and train the semantic segmentation system using the training set obtained in step S1 to obtain a semantic segmentation model; S3: Repeat step S2 and adjust the hyperparameters during training until the optimal semantic segmentation model is obtained; S4: Input the remote sensing image to be semantically labeled into the optimal semantic segmentation model obtained in step S3 to obtain semantic segmentation labels.

Images