A Semi-Supervised Remote Sensing Image Semantic Segmentation Method Based on Double Consistency

Abstract

This invention relates to a semi-supervised remote sensing image semantic segmentation method based on double consistency. The semi-supervised semantic segmentation method includes: establishing a semi-supervised learning model based on a teacher-student architecture; using labeled data to perform supervised learning training on the teacher model and optimizing the teacher model parameters; the student model consists of a single-encoder, double-decoder deep learning model; using a sharpening function to reduce the information entropy of the student model's predictions; firstly, using the teacher model to generate pseudo-labels from unlabeled data to semi-supervise the prediction results of the student model; secondly, cross-supervision through two sets of pseudo-labels converted from the prediction results of the two sets of student models; and alternatingly training the student network through these two consistency tasks, enabling the model to fully understand the large amount of information in the unlabeled data; after training, the test data directly passes through the student network to obtain the predicted semantic label map, and the semantic label map is mapped onto the original image to visualize the segmentation results.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a semi-supervised remote sensing image semantic segmentation method based on double consistency. Background Technology

[0002] With the continuous development of computer science and technology in the 21st-century information age, the sensor technologies and space science technologies involved in remote sensing imaging are also constantly advancing, making remote sensing imaging technology increasingly crucial for Earth observation. Remote sensing images provide a wealth of information for observation missions, and the development of remote sensing image technology has promoted the development of downstream fields such as military, meteorology, and transportation. In recent years, satellite and UAV imaging have developed rapidly, making the acquisition of remote sensing images more convenient, continuously improving image quality, and leading to a dramatic increase in remote sensing image data, ushering in the era of remote sensing big data.

[0003] Deep learning has freed up the manual labor for image analysis tasks, and real-time intelligent image processing technology has provided a foundation for the development of downstream tasks. However, efficient deep learning architectures rely on supervised learning on large manually labeled datasets. A huge labeled dataset requires significant time and manpower costs. For remote sensing datasets, the higher spatial resolution and semantic overlap make the situation more complex, leading to even higher costs for labeling remote sensing datasets. Furthermore, labels used for semantic segmentation tasks require pixel-level annotations, and the high labeling cost has become one of the main problems restricting the development of semantic segmentation of remote sensing images. To make the vast amount of unlabeled remote sensing data useful and reduce image labeling costs, many researchers have begun to mine the information hidden in unlabeled data and explore using unlabeled data to train segmentation models, making semi-supervised learning models a hot topic in image segmentation development. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing remote sensing image semantic segmentation techniques that rely on a large amount of label data, by providing a semi-supervised remote sensing image semantic segmentation method based on double consistency.

[0005] To address the aforementioned technical problems, one technical solution adopted by this invention is to provide a semi-supervised remote sensing image semantic segmentation method based on double consistency, wherein the semi-supervised semantic segmentation method includes the following steps:

[0006] S1. Divide the dataset into labeled datasets D. l Unlabeled dataset D u ;

[0007] S2. There will be a labeled dataset D. l Optimize teacher model parameters by calculating supervised loss using the teacher model and its real labels;

[0008] S3. For unlabeled data, the consistency principle states that the prediction results for the same pixel should remain consistent after different perturbations, and the prediction results should remain consistent after the input undergoes different perturbations. The semi-supervised training of the student model is divided into two groups: first, the pseudo-labels from the teacher model provide semi-supervision for both student models; second, the pseudo-labels generated by the two student models provide cross-supervision. Both semi-supervised tasks are trained alternately under the consistency principle. Unlabeled data generates prediction results through the teacher model, and the corresponding pseudo-label Y is generated by performing a Hadamard product between the prediction result and the original image. t Unlabeled data needs to be transformed into feature vectors through the common encoding network of the student models, and then predicted through two sets of student decoding networks. For the prediction results of the two sets of student decoding networks, the semi-supervised loss of the pseudo-labels generated by the teacher model for the two sets of student models needs to be calculated first, and the parameters of the two sets of student models are updated using the two sets of semi-supervised losses. Finally, the pseudo-labels Y corresponding to the two sets of student models are generated by performing a Hadamard product between the predictions of the two sets of student models and the original image. s1 Y s2 The semi-supervised loss was calculated using cross-supervision and the parameters of the two sets of student models were updated.

[0009] S4. Based on all the supervised losses obtained in step S3 above, train the semi-supervised segmentation model separately to obtain the trained semi-supervised semantic segmentation model; then, perform semantic segmentation of the remote sensing image through the trained semi-supervised semantic segmentation model.

[0010] Preferably, in step S1, the teacher model consists of a deep neural network architecture with an encoding network and a decoding network. A channel attention mechanism is added to the encoding network to enhance the accuracy of feature extraction by the teacher model, thereby providing high-quality pseudo-labels for the student model. The student model consists of a single encoding network and a dual decoding network architecture. The two student decoding networks share a single encoding network to work. At the same time, after the predictions of the two sets of student models, a sharpening function is used to reduce the entropy of the unlabeled data and improve the confidence of the edge contours. The semi-supervision of the student model is divided into two groups: the first is the semi-supervision of the pseudo-labels of the teacher model for the two sets of student models, and the second is the cross-supervision of the pseudo-labels generated by the two sets of student models. The two semi-supervision tasks are trained alternately under the consistency principle.

[0011] Preferably, in step S2, two publicly available high-resolution remote sensing image datasets, Potsdam and Vaihingen, are used to evaluate the model. Both datasets are cropped to 512×512 pixels. 10% of the datasets are randomly selected as the test set, and the rest are used as the training set. For the data in the training set, 50% of the data is randomly selected and its labels are removed for semi-supervised learning training.

[0012] Preferably, the model training in step S3 requires calculating the predictions of labeled data under the teacher model and calculating the supervised loss with their true labels to optimize the teacher model parameters; for unlabeled data, the teacher model is first used to generate prediction results, and the corresponding pseudo-label Y is generated by performing a Hadamard product between the prediction results and the original image. t Secondly, the unlabeled data needs to be transformed into feature vectors through the common encoding network of the student models, and then predicted through two sets of student decoding networks. For the prediction results of the two sets of student decoding networks, firstly, the semi-supervised loss of the pseudo-labels generated by the teacher model for the two sets of student models needs to be calculated, and the parameters of the two sets of student models are updated using the two sets of semi-supervised losses. Secondly, the pseudo-labels Y corresponding to the two sets of student models need to be generated by performing a Hadamard product between the predictions of the two sets of student models and the original image. s1 Y s2 The semi-supervised loss was calculated using cross-supervision, and the parameters of the two student models were updated. The semi-supervision of the student models was divided into two groups: the first was the semi-supervision of the pseudo-labels generated by the teacher model for the two student models, and the second was the cross-supervision of the pseudo-labels generated by the two student models. The two semi-supervised tasks were trained alternately under the consistency principle. The training parameters were set as follows: the number of iterations (epochs) was 100, the learning rate was initialized to 0.00001, the weight decay was set to 0.0001, the batch size was set to 4, the loss function was the cross-entropy loss function which is commonly used in semantic segmentation, and the model was trained using the Adam optimizer with default parameters. Attached Figure Description

[0013] Figure 1 This is a schematic flowchart of the method of the present invention;

[0014] Figure 2 This is a schematic diagram of the overall network model of the present invention;

[0015] Figure 3 Visualize the results on the Potsdam dataset;

[0016] Figure 4 Visualize the results on the Vaihingen dataset; Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0018] All features disclosed in this specification, or steps in all methods or processes disclosed herein, may be combined in any way, except for mutually exclusive features or steps.

[0019] Any feature disclosed in this specification (including any appended claims and abstract) may, unless specifically stated otherwise, be replaced by other equivalent or similar features. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features. Figure 1 As shown, in this embodiment of the present invention, the semi-supervised remote sensing image semantic segmentation method includes the following steps:

[0020] S1. Establish a deep neural network model based on a semi-supervised remote sensing image semantic segmentation architecture with double consistency.

[0021] S2. Obtain publicly available remote sensing image datasets;

[0022] S3. Input the trained dataset into the semi-supervised network model for training to obtain the optimal network weights;

[0023] S4. Input the original test image into the student model to predict and obtain semantic tags of the same size as the original image;

[0024] S5. Map the semantic tag map onto the original image to visualize the segmentation results.

[0025] In this example, the overall structure of the network is as follows: Figure 2 As shown, step S1 specifically includes:

[0026] S101: The teacher model consists of a deep neural network architecture with an encoding network and a decoding network. A channel attention mechanism is added to the encoding network to enhance the accuracy of feature extraction by the teacher model, thereby providing high-quality pseudo-labels for the student model.

[0027] S102: The student model consists of a single encoder network and a dual decoder network architecture. The two student decoder networks share a single encoder network to work. At the same time, after the predictions of the two student models, a sharpening function is used to reduce the entropy of the unlabeled data and improve the confidence of the edge contours. The semi-supervision of the student model is divided into two groups. The first is the semi-supervision of the pseudo-labels of the teacher model for the two student models. The second is the cross-supervision of the pseudo-labels generated by the two student models. Both semi-supervision tasks are trained alternately under the consistency principle.

[0028] Step S2 obtains a dataset of publicly available urban road images, specifically including:

[0029] This invention uses two high-resolution remote sensing image datasets published by ISPRS, the Potsdam dataset and the Vaihingen dataset, to evaluate the model. These two datasets are high-resolution datasets obtained through aerial photography using airborne remote sensing equipment. Both datasets include very high-resolution true orthophotos and corresponding digitally labeled images derived from dense image matching techniques. Both datasets cover urban scenes. The Potsdam dataset contains 38 images, all in RGB 3-channel TIFF format, with a spatial resolution of 5 cm and a pixel size of [missing value]. Each image in this dataset is accompanied by a corresponding label image, with the same size and format as the original image. Figure 1 The Vaihingen dataset contains 33 images, all in RGB 3-channel TIFF format with a spatial resolution of 9cm. However, the pixel size varies, with an average pixel size similar to the Potsdam dataset. Each image is accompanied by a corresponding label image of the same size and format as the original dataset. The labels are divided into six categories: impermeable surfaces, buildings, low-lying plants, trees, vehicles, and others. Figure 1 The labels are divided into 6 categories: impermeable surfaces, buildings, low-lying plants, trees, vehicles, and others.

[0030] Both datasets were cropped to 512×512 pixels; 10% of the datasets were randomly selected as the test set; for the data in the training set, 50% of the data were randomly selected and their labels were removed for semi-supervised training.

[0031] Step S3 specifically includes:

[0032] S301. Divide the dataset into labeled datasets D. l Unlabeled dataset D u ;

[0033] S302. The labeled dataset D will be... l Optimize teacher model parameters by calculating supervised loss using the teacher model and its real labels;

[0034] S303. For unlabeled data, the consistency principle states that the prediction results for the same pixel should remain consistent after different perturbations, and the prediction results should remain consistent after the input undergoes different perturbations. First, let it generate prediction results through the teacher model, and then generate the corresponding pseudo-label Y by performing a Hadamard product between the prediction results and the original image. tSecondly, the unlabeled data needs to be transformed into feature vectors through the common encoding network of the student models, and then predicted through two sets of student decoding networks. For the prediction results of the two sets of student decoding networks, firstly, the semi-supervised loss of the pseudo-labels generated by the teacher model for the two sets of student models needs to be calculated, and the parameters of the two sets of student models are updated using the two sets of semi-supervised losses. Secondly, the pseudo-labels Y corresponding to the two sets of student models need to be generated by performing a Hadamard product between the predictions of the two sets of student models and the original image. s1 Y s2 The semi-supervised loss was calculated and the parameters of the two student models were updated through cross-supervision. The semi-supervision of the student models was divided into two groups: the first was the semi-supervision of the pseudo-labels of the teacher model for the two student models, and the second was the cross-supervision of the pseudo-labels generated by the two student models. The two semi-supervised tasks were trained alternately under the consistency principle.

[0035] S304. Parameter settings: The number of iterations (epochs) is set to 100, the learning rate is initialized to 0.00001, and the weight decay is set to 0.0001; the batch size is set to 4; the loss function is the cross-entropy loss function, which is commonly used in semantic segmentation; the model is trained using the Adam optimizer with default parameters.

[0036] In step S4, the original test image is input to obtain a semantic label map with the same size as the original image. Specifically, in the testing phase, the input image does not need to be preprocessed, and the segmentation label map is obtained directly.

[0037] Step S5 maps the semantic tag map onto the original image to visualize the segmentation results. Specifically, different semantic categories are mapped to different colors and overlaid on the original image, thus providing an intuitive visualization of the segmentation results. Figure 3 Visualize the results on the Potsdam dataset. Figure 4 Visualization results on the Vaihingen dataset.

[0038] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A method for semi-supervised semantic segmentation of remote sensing images based on dual consistency, characterized in that, The semi-supervised remote sensing image semantic segmentation method comprises the following steps; The semi-supervised learning model of the teacher-student architecture is established, and for the student model, a single-encoding and double-decoding model architecture is adopted. S1 : divide the dataset into a labeled dataset D l , and an unlabeled dataset D u ; S2: obtaining a labeled data set D l optimizing the parameters of the teacher model by the teacher model and calculating a supervision loss with the real labels thereof. S3: For the unlabeled data, the consistency principle indicates that the prediction results should remain consistent after different perturbations for the same pixel point, and the prediction results remain consistent after different disturbances to the input. First, let the unlabeled data pass through the teacher model to generate prediction results, and then do Hadamard product operation with the original image to generate corresponding pseudo labels Y t Second, the unlabeled data also needs to be converted into a feature vector through the public coding network of the student model, and predicted through two sets of student decoding networks; for the prediction results of the two sets of student decoding networks, first, the pseudo labels generated by the teacher model need to be calculated for the semi-supervised loss of the two sets of student models, and the parameters of the two sets of student models are updated through the two sets of semi-supervised loss; second, the Hadamard product operation is performed between the prediction results of the two sets of student models and the original image to generate the corresponding pseudo labels Y s1 s2 of the two sets of student models, and the parameters of the two sets of student models are updated; the semi-supervised of the student model is divided into two groups, the first is the semi-supervised of the pseudo labels of the teacher model for the two sets of student models, and the second is the cross-supervised of the pseudo labels generated by the two sets of student models, and the two sets of semi-supervised tasks are alternately trained under the consistency principle.​ S4: According to all the supervised losses obtained in step S3, the semi-supervised segmentation model is trained respectively to obtain the trained semi-supervised semantic segmentation model; and then, the trained semi-supervised semantic segmentation model is used for semantic segmentation of remote sensing images. 2.The method of claim 1, wherein, The step S1 specifically comprises: Randomly sampling the label set corresponding to the data set, and for the remaining data, dividing it into a labeled data set D l , an unlabeled data set D u . 3.The method of claim 1, wherein, The step S2 specifically comprises: The teacher model is composed of a deep neural network architecture of an encoding network and a decoding network, and a channel attention mechanism is added to the encoding network to enhance the accuracy of feature extraction of the teacher model, thereby providing high-quality pseudo labels for the student model. 4.The method of claim 1, wherein, The step S3 specifically comprises: The student model is composed of a single-encoding network and a double-decoding network architecture, and the two student decoding networks share one encoding network for work, and a sharpening function is used after the prediction of the two student models to reduce the entropy of the unlabeled data and improve the confidence of the edge contour; the semi-supervised of the student model is divided into two groups, the first is the pseudo label of the teacher model for the semi-supervised of the two student models, and the second is the cross supervision of the pseudo labels generated by the two student models, and the two semi-supervised tasks are alternately trained under the consistency principle.

5. The semi-supervised remote sensing image semantic segmentation method based on double consistency according to any one of claims 1-4, characterized in that: The semi-supervised remote sensing image semantic segmentation method further comprises a training step arranged after the step of establishing the network model based on the teacher-student architecture; the training step comprises inputting the trained data into the network model in batches for iterative training to obtain optimal network weights.

6. The semi-supervised remote sensing image semantic segmentation method based on dual consistency according to claim 5, characterized in that: The training step specifically comprises: The data set adopts two public high-resolution remote sensing image data sets: Potsdam and Vaihingen to evaluate the model; both data sets are cropped to 512x512 size; 10% of the data set is randomly selected for testing, and the rest is used for training; for the data in the training set, 50% of the label data is randomly selected and deleted; the parameter setting iteration number epoch is 100, the learning rate initialization is set to 0.00001, the weight decay is set to 0.0001; the batch processing is set to 4; the loss function selects the cross-entropy loss function commonly used in semantic segmentation; the model is trained with the Adam optimizer with default parameters.

7. A semi-supervised remote sensing image semantic segmentation system based on double consistency, characterized in that: It includes a model construction unit, a training unit and a visualization unit; the model construction unit is used to establish a semi-supervised semantic segmentation model composed of a teacher-student architecture, the teacher model is composed of an encoding-decoding semantic segmentation network with channel attention mechanism, the channel attention can help the teacher model to strengthen the feature extraction capability and optimize the teacher model parameters, so as to better generate pseudo labels, the student model is composed of a single encoding double decoding architecture, two decoding networks share an encoding network, and the prediction results of the two decoding networks are processed through a sharpening function to reduce the information entropy of the unsupervised prediction of the student model and improve the confidence of the edge contour; the training unit calculates the supervised loss and updates the teacher model parameters for the labeled data set D l inputting the semi-supervised semantic segmentation model, allowing it to pass through the teacher model and be supervised by the corresponding real label data, calculating the supervised loss and updating the teacher model parameters, and for the unlabeled data set D u , first, let it pass through the teacher model for prediction and generate the pseudo label Y t of the teacher model prediction by taking the Hadamard product of the original image and the prediction, second, let the unlabeled data pass through the student model to generate two sets of prediction results of the student model, for the prediction results of the two student decoding networks, first, the pseudo label generated by the teacher model needs to be calculated for the semi-supervised loss of the two student models, and the two student model parameters are updated through the two semi-supervised losses, second, the two student model predictions are taken Hadamard product with the original image to generate the corresponding pseudo label Y s1 of the two student models, Y s2 , and the semi-supervised loss is calculated by cross supervision and the two student model parameters are updated, the semi-supervised of the student model is divided into two groups, the first is the semi-supervised of the teacher model pseudo label for the two student models, and the second is the cross supervision of the pseudo labels generated by the two student models, both semi-supervised tasks are alternately trained under the consistency principle; the training unit dataset adopts two public high-resolution remote sensing image datasets: Potsdam and Vaihingen to evaluate the model; both datasets are cropped to 512x512 size, 10% of the data in the dataset are randomly selected as the test set, and the rest are used as the training set; for the data in the training set, 50% of the label data is randomly selected and deleted, the parameter settings are that the number of iterations epoch is 100, the learning rate initialization is set to 0.00001, the weight decay is set to 0.0001, the batch processing is set to 4, the loss function selects the commonly used cross-entropy loss function for semantic segmentation, and the model is trained with the Adam optimizer with default parameters; the visualization unit is used to map the semantic label map to the original image to realize the visualization of the segmentation result.

Images