Semi-supervised remote sensing image semantic segmentation method and system based on region contrastive learning

Abstract

The present invention discloses a semi-supervised remote sensing image semantic segmentation method and system based on region-contrast learning. The method employs a hierarchical enhancement module (HA), a region-aware contrast module (RC), three student networks each consisting of an encoder and a segmentation network, and a teacher network to perform semi-supervised semantic segmentation training on the encoder and segmentation networks. The method includes: hierarchically enhancing the labeled or unlabeled image using HA and then inputting it into the student network; updating the teacher network parameters based on the student network parameters and generating pseudo-labels; and calculating a total loss function consisting of region-aware contrast loss and pixel-level cross-entropy loss using the region-aware contrast module (RC) based on the region centers predicted by the student network and the pseudo-labels, thereby updating the student network parameters. The present invention aims to accurately and effectively identify and segment different categories of objects in remote sensing images, improving the performance and training efficiency of the semantic segmentation network composed of the encoder and segmentation networks.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a semi-supervised remote sensing image semantic segmentation method and system based on region contrast learning. Background Art

[0002] Remote sensing images generally refer to records of electromagnetic wave information of surface objects acquired through aerial or satellite platforms, usually in the form of film or digital photographs. Remote sensing images can be acquired through various imaging methods, including aerial photography, aerial scanning, and microwave radar. Remote sensing images have a wide range of applications, including but not limited to geographic information systems, environmental monitoring, urban planning, agricultural management, and disaster assessment. With technological advancements, the resolution and quality of remote sensing images continue to improve. Furthermore, their integration with artificial intelligence and data analysis technologies has further deepened and expanded their applications.

[0003] Remote sensing image semantic segmentation is an advanced image processing technique that aims to assign each pixel in a remote sensing image to a specific class, thereby enabling accurate identification and classification of different ground objects. This process is crucial for fields such as geographic information systems, environmental monitoring, and urban planning. However, despite its enormous potential and application value, this technology faces a series of technical issues and challenges: Intra-class and inter-class variance: High-resolution remote sensing imagery contains a large amount of ground object information, which is diverse and complex. Objects within the same class may exhibit large variations (intra-class variance), while variations between different classes may be small (inter-class variance). This results in poor class discrimination and makes semantic segmentation difficult. Scale variation: In remote sensing imagery, geospatial objects can exhibit significant scale variation, meaning that the same ground object may appear different in size and shape in different images. This scale variation poses challenges to accurate segmentation. Data fusion strategies: To improve segmentation accuracy, multiple data sources and network architectures can be combined. For example, combining geometric and spectral information or using different data as auxiliary features can improve segmentation. The problem of limited samples: In some cases, there may be few labeled samples available for training. This requires using data synthesis or data augmentation techniques to expand the training set, or adopting semi-supervised learning methods to utilize unlabeled data. In summary, remote sensing image semantic segmentation is a complex and multifaceted technical field that requires continuous research and innovation to solve the above problems in order to better serve related application scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems in the prior art, a semi-supervised remote sensing image semantic segmentation method and system based on region contrast learning is provided. The present invention aims to accurately and effectively identify and segment objects of different categories in remote sensing images, and improve the performance and training efficiency of the semantic segmentation network composed of the encoder and the segmentation network.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A semi-supervised remote sensing image semantic segmentation method based on region contrast learning includes using a hierarchical enhancement module HA, a region-aware contrast module RC, three student networks each consisting of an encoder and a segmentation network, and a teacher network to perform semi-supervised semantic segmentation training on the encoder and segmentation network. A semi-supervised semantic segmentation training includes:

[0007] S101, there will be label images The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. , the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations ;

[0008] S102, the image Input the first branch’s student network to extract labeled images Features , and according to the characteristics Predict labeled images The prediction results , according to the prediction results Get labeled images The center of the region will have the label image The center of the area is saved to the central storage part; the image Input the student network of the second branch to extract unlabeled images Features , according to the characteristics Predict unlabeled images The prediction results , according to the prediction results Get unlabeled images The center of the area; the image Input to the student network of the third branch to extract features and obtain the predicted value of the unlabeled remote sensing image , according to the predicted value Get the center of the region; update the parameters of the teacher network after the exponential moving average EMA operation on the parameters of the student network, and use the new teacher network to perform the hierarchical enhancement module HA on the unlabeled image The extracted unlabeled image generation only simulates spatial variations to produce unlabeled images Pseudo labels ;

[0009] S103, through the regional awareness comparison module RC according to the regional center and pseudo label Calculate the region-aware contrast loss and pixel-level cross entropy loss, and update the parameters of the student network based on the total loss function composed of the region-aware contrast loss and pixel-level cross entropy loss.

[0010] Optionally, in step S101, there will be a label image The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. When including images with labels Weak enhancement using layered enhancement module HA w Get an image that only simulates spatial changes , the weak enhancement A w This includes partial or complete translation, rotation, scaling, and flipping.

[0011] Optionally, in step S101, the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations When including the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A w Then perform strong enhancement A s Get an image that simulates spatial variation and uniform spectral variation , the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A' w Then perform strong enhancement A' s , and then the obtained image and image Splice A c Get an image that simulates spatial variation and non-uniform spectral variation , where weak enhancement A w and weakly enhanced A' w Including translation, rotation, scaling, flipping of part or all, strong enhancement A s and strong enhancement A' s This includes randomly changing the image color, changing the image brightness and contrast, and adding some or all of Gaussian noise.

[0012] Optionally, the calculation function expression of the region center is:

[0013] ,

[0014] In the above formula, For a regional center of category i, Pixels in the image Features, is the pixel in the prediction result Tags, A binary indicator indicating whether the pixel belongs to class i.

[0015] Optionally, in step S102, there will be a label image x l The function expression of the regional center saved to the central storage part is:

[0016] ,

[0017] In the above formula, For the central storage part, For a labeled dataset, is the i-th labeled image in the labeled dataset, is the i-th labeled image in the labeled dataset the entire regional center.

[0018] Optionally, the function expression of the total loss function based on the region-aware contrast loss and the pixel-level cross entropy loss in step S102 is:

[0019] ,

[0020] In the above formula, is the total loss function, is the pixel-level cross entropy loss, is the weight parameter, is the region-aware contrast loss, and the pixel-level cross entropy loss is composed of the supervised learning part and the unsupervised learning part Additive composition, region-aware contrast loss has supervised learning part and the unsupervised learning part Add together and we have:

[0021] ,

[0022] ,

[0023] ,

[0024] ,

[0025] In the above formula, is the number of labeled images, is the number of unlabeled images, For labeled images, For a labeled dataset, is the i-th category label of the labeled image, To simulate only spatially varying images Through the student network The result obtained is, is an unlabeled dataset, To simulate the image of spatial variation and uniform spectral variation Through the teacher network The result obtained is, To simulate the image of spatial variation and uniform spectral variation Through the student network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the teacher network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the student network The result obtained is, is the number of categories, is the regional perception of category i, and has:

[0026] ,

[0027] In the above formula, The central storage part All regional centers belonging to category i in for The feature vectors of the same category in yes The feature vectors of different categories in is the feature vector of the current image, is a hyperparameter.

[0028] Optionally, in step S102, the function expression for updating the parameters of the teacher network after performing the exponential moving average EMA operation on the parameters of the student network is:

[0029] ,

[0030] In the above formula, is the weight or bias in the parameters of the teacher network at the current moment, is the smoothing coefficient, is the weight or bias in the parameters of the teacher network at the previous moment, is the weight or bias in the parameters of the student network at the current moment.

[0031] In addition, the present invention also provides a semi-supervised remote sensing image semantic segmentation system based on region contrast learning, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning.

[0032] In addition, the present invention also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning through a processor.

[0033] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning through a processor.

[0034] Compared with the prior art, the present invention mainly has the following advantages:

[0035] 1. The semantic segmentation of the present invention can accurately and effectively identify and segment different types of objects in an image. It first performs layered enhancement on the image to further simulate the spatial changes and spectral distribution changes in real situations, and then uses the region-aware contrast learning strategy to extract the regional centers of the image. The segmentation results of the unlabeled image are supervised based on the set of regional centers of the labeled image.

[0036] 2. The present invention proposes a hierarchical enhancement strategy for semi-supervised semantic segmentation of remote sensing images based on regional contrast learning, that is, through the hierarchical enhancement module HA, three data enhancement branches simulate three situations in remote sensing images, namely: (1) only simulating spatial changes, (2) spatial changes and uniform spectral changes, and (3) spatial changes and non-uniform spectral changes. These common data enhancements can simulate remote sensing images in different scenarios through different combinations, enabling the model to learn from remote sensing data in various situations, which can greatly improve the performance of the network.

[0037] 3. This paper proposes a region-aware comparative learning strategy. This strategy uses the features and predictions of labeled data to determine the region centers of remote sensing images through the region-aware comparative module (RC). This strategy then uses the region centers of the labeled data to supervise the segmentation results of the unlabeled data. This strategy can obtain reliable class-level feature distributions, improve the performance of the segmentation network, increase the utilization of unlabeled data, and effectively reduce the impact of pseudo-labels on model performance.

[0038] 4. The present invention uses a self-training model, utilizing a model trained on labeled images to generate pseudo-labels for unlabeled remote sensing images. The generated pseudo-labels are then used to supervise the unlabeled image segmentation. The accuracy and quantity of the pseudo-labels determine the performance of the model. When the labeled images are not sufficiently trained, the pseudo-labels may contain noise. However, the method of this embodiment retains all pseudo-labels, allowing the model to fully understand the true feature distribution of the unlabeled data, and class labels are used to reduce the impact of false pseudo-labels. This solves the problems of semantic information loss and potential distribution bias in self-training.

[0039] 5. The present invention adopts deep learning to determine the training weights and pseudo-label confidences. The obtained semantic segmentation images can more accurately identify, classify and label objects, and the segmentation effect is better when there is less training data and the training difficulty is higher.

[0040] 6. When training network parameters, the present invention has low requirements for training data and is easy to use. After the network is trained, the applicable scenarios are diverse and can meet various needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0042] Figure 2 Schematic diagram of the network structure used in the embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the network structure of the regional awareness comparison module RC used in an embodiment of the present invention.

[0044] Figure 4 These are the imaging effects caused by spatial variation, uniform spectral variation, and non-uniform spectral variation in the embodiments of the present invention.

[0045] Figure 5 The figure shows the comparison between the method of the embodiment of the present invention and WSCL on the Potsdam test set.

[0046] Figure 6 The figure shows the comparison between the method of the embodiment of the present invention and WSCL on the Vaihingen test set.

[0047] Figure 7 Schematic diagram of the weak-to-strong network structure used in the ablation experiment of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] like Figure 1 and Figure 2As shown, the semi-supervised remote sensing image semantic segmentation method based on regional contrast learning in this embodiment includes using a hierarchical enhancement module HA (Hierarchical Augmentation), a region-aware contrast module RC (Region-aware Contrastive), three student networks each consisting of an encoder and a segmentation network, and a teacher network to perform semi-supervised semantic segmentation training on the encoder and the segmentation network. It should be noted that the encoder and the segmentation network can adopt the required network type as needed. For example, as an optional implementation, the encoder in this embodiment adopts the resnet-50 model pre-trained by ImageNet, and the segmentation network adopts the FCN model. Therefore, the network model composed of the encoder and the segmentation network is referred to as the FCN-resnet50 model. The method of this embodiment performs a semi-supervised semantic segmentation training including:

[0049] S101, there will be label images The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. , the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations ;

[0050] S102, the image Input the first branch’s student network to extract labeled images Features , and according to the characteristics Predict labeled images The prediction results , according to the prediction results Get labeled images The center of the region will have the label image The center of the area is saved to the central storage part; the image Input the student network of the second branch to extract unlabeled images Features , according to the characteristics Predict unlabeled images The prediction results , according to the prediction results Get unlabeled images The center of the area; the image Input to the student network of the third branch to extract features and obtain the predicted value of the unlabeled remote sensing image , according to the predicted value Get the center of the region; update the parameters of the teacher network after the exponential moving average EMA operation on the parameters of the student network, and use the new teacher network to perform the hierarchical enhancement module HA on the unlabeled image The extracted unlabeled image generation only simulates spatial variations to produce unlabeled images Pseudo labels ;

[0051] S103, through the regional awareness comparison module RC according to the regional center and pseudo label Calculate the region-aware contrast loss and pixel-level cross entropy loss, and update the parameters of the student network based on the total loss function composed of the region-aware contrast loss and pixel-level cross entropy loss, such as Figure 3 As shown, It is the region-aware contrast loss.

[0052] The input labeled and unlabeled remote sensing image data first pass through the hierarchical enhancement module HA to obtain images with three different enhancement strategies, which simulate spatial changes, uniform spectral changes, and non-uniform spectral changes respectively. The labeled data simulates spatial changes, and the unlabeled data simulates uniform and non-uniform spectral changes. The images of the three branches are then input into the student network respectively. The features and prediction results are processed in the regional awareness comparison module RC. Finally, the final segmented image is obtained under the supervision of pseudo labels. Figure 1 and Figure 2 As shown, in step S101 of this embodiment, there will be a label image The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. When including images with labels Weak enhancement using layered enhancement module HA w Get an image that only simulates spatial changes , the weak enhancement A w This includes some or all of the translation, rotation, scaling, and flipping, which can be expressed as:

[0053] ,

[0054] In step S101, the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations When including the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A w Then perform strong enhancement A s Get an image that simulates spatial variation and uniform spectral variation , which can be expressed as:

[0055] ,

[0056] Unlabeled images Use the layered enhancement module HA to first perform weak enhancement A' w Then perform strong enhancement A' s , and then the obtained image and image Splice A c Get an image that simulates spatial variation and non-uniform spectral variation , which can be expressed as:

[0057] ,

[0058] Among them, weak enhancement A w and weakly enhanced A' w Including translation, rotation, scaling, flipping of part or all, strong enhancement A s and strong enhancement A' s This includes randomly changing the image color, changing the image brightness and contrast, and adding some or all of Gaussian noise. Figure 4 The imaging effects caused by spatial variation, uniform spectral variation, and non-uniform spectral variation in this embodiment are shown in Figure 1, where (a) is the original image, (b)-(d) simulate the spatial variation of the original image, including flipping, rotation, and cropping, (e)-(h) simulate the uniform spectral variation of the original image, including blur, grayscale, noise, and distortion, and (i)-(l) simulate the non-uniform spectral variation of the original image. Figure 1 、 Figure 2 As can be seen from the above description, the hierarchical enhancement module HA in this embodiment includes three branches. The first branch is used to transform the labeled image into Weak enhancement using layered enhancement module HA w Get an image that only simulates spatial changes , the second branch is used to convert the unlabeled images Use the layered enhancement module HA to first perform weak enhancement A w Then perform strong enhancement A s Get an image that simulates spatial variation and uniform spectral variation , the third branch is used to convert the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A' w Then perform strong enhancement A' s , and then the obtained image and image Splice A c Get an image that simulates spatial variation and non-uniform spectral variation , the data enhancement processing performed on each branch is different, and the weak enhancement and strong enhancement on the three branches are also different (random). During network training, three data processed with different data enhancements are input into three different student networks respectively. The labeled and unlabeled images are input into the hierarchical enhancement module HA respectively, and then input into three different student networks after three different data enhancement processes. The obtained features and prediction results are used as the input of the regional awareness comparison module, and the regional centers of the three branch images are obtained respectively. The set of regional centers of the labeled images is used as the central storage part to supervise the prediction results of the unlabeled images. After the network training is completed, this embodiment only needs to input the unlabeled remote sensing image x u The final segmentation result can be obtained. In the obtained result, each type of object will be identified and marked with the same color, and different types of objects will use different colors. Since the structure of the student network and the teacher network are the same, the parameters of the student network processed by EMA can be directly used in the teacher network. Together with the parameters, the teacher network is also fed with the weakly enhanced A w The unlabeled image is generated by the teacher network, and the prediction results of two images are generated respectively. These two prediction results are then subjected to A c Operation, generating an unlabeled image x u The pseudo label y pseudo , the image processed by uniform spectral change and the image processed by non-uniform spectral change use their respective regional centers and prediction results p us 、p uc To calculate the central storage part and pseudo label y pseudo Contrast loss and pixel-level cross entropy loss. Since the pseudo labels cannot be completely accurate, the central storage part is not updated in the learning of unlabeled images.

[0059] The region awareness comparison module (RC) uses the image features and predictions obtained using the three data augmentation methods to calculate the region center and stores the region center of the labeled image in the center storage. The region center of category i is the average of the features of all pixels belonging to category i in a single image. In this embodiment, the calculation function of the region center is expressed as:

[0060] ,

[0061] In the above formula, For a regional center of category i, Pixels in the image Features, is the pixel in the prediction result Tags, A binary indicator indicating whether the pixel belongs to class i.

[0062] In this embodiment, in step S102, there will be a label image The function expression of the regional center saved to the central storage part is:

[0063] ,

[0064] In the above formula, For the central storage part, For a labeled dataset, is the i-th labeled image in the labeled dataset, is the i-th labeled image in the labeled dataset Then calculate the region center of the unlabeled image, calculate the regional contrast loss between the region centers of the unlabeled image, and update the center storage part M during the training of the labeled image.

[0065] In this embodiment, in the final region awareness comparison module RC, in order to obtain more accurate prediction results, pixel-level cross entropy loss and region-aware contrast loss are used. The total loss is the sum of the two. According to the obtained loss, the parameters in the student network and the teacher network are adjusted until the appropriate parameters are adjusted to minimize the loss. The function expression of the total loss function composed of region-aware contrast loss and pixel-level cross entropy loss in step S102 of this embodiment is:

[0066] ,

[0067] In the above formula, is the total loss function, is the pixel-level cross entropy loss, is the weight parameter, is the region-aware contrast loss, and the pixel-level cross entropy loss is composed of the supervised learning part and the unsupervised learning part Additive composition, region-aware contrast loss has supervised learning part and the unsupervised learning part Add together and we have:

[0068] ,

[0069] ,

[0070] ,

[0071] ,

[0072] In the above formula, is the number of labeled images, is the number of unlabeled images, For labeled images, For a labeled dataset, is the i-th category label of the labeled image, To simulate only spatially varying images Through the student network The result obtained is, is an unlabeled dataset, To simulate the image of spatial variation and uniform spectral variation Through the teacher network The result obtained is, To simulate the image of spatial variation and uniform spectral variation Through the student network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the teacher network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the student network The result obtained is, is the number of categories, is the regional perception of category i, and has:

[0073] ,

[0074] In the above formula, The central storage part All regional centers belonging to category i in for The feature vectors of the same category in yes The feature vectors of different categories in is the feature vector of the current image, is a hyperparameter.

[0075] In step S102 of this embodiment, the function expression for updating the parameters of the teacher network after the parameters of the student network are subjected to the exponential moving average EMA operation is:

[0076] ,

[0077] In the above formula, is the weight or bias in the parameters of the teacher network at the current moment, is the smoothing coefficient, is the weight or bias in the parameters of the teacher network at the previous moment, is the weight or bias in the parameters of the student network at the current moment.

[0078] In this example, each batch of training contains four labeled images and four unlabeled images. Stochastic gradient descent (SGD) with a weight decay of 5e-4 and a momentum of 0.9 is used to optimize the segmentation network. Experiments were conducted on the Potsdam and Vaihingen datasets using two Nvidia RTX2080Ti GPUs, and on the LoveDA dataset using a single Nvidia RTX3090 GPU.

[0079] In order to further verify the effectiveness of the proposed method, this embodiment uses average F1 and mean IoU as evaluation indicators, where the calculation function expression of average F1 is:

[0080] ,

[0081] In the above formula, For accuracy, is the recall rate, and:

[0082] ,

[0083] ,

[0084] In the above formula, 、 and are the number of true positive, false positive, and false negative samples, respectively.

[0085] The calculation function expression of mean IoU is:

[0086] ,

[0087] In the above formula, is the mean IoU, is the number of categories, is the intersection and ratio, and we have:

[0088] ,

[0089] In the above formula, this embodiment uses a commonly used semi-supervised method and the proposed semi-supervised remote sensing image semantic segmentation method based on multi-level enhancement and region contrast learning to compare on the three datasets of Potsdam, Vaihingen and LoveDA. The results are shown in Tables 1, 2, and 3.

[0090] Table 1 Performance comparison of this embodiment and other semi-supervised methods on the Potsdam dataset

[0091]

[0092] Table 2 Performance comparison of this embodiment and other semi-supervised methods on the Vaihingen dataset

[0093]

[0094] Table 3 Performance comparison of this embodiment and other semi-supervised methods on the urban-rural adaptive land cover dataset

[0095]

[0096] In Tables 1 through 3, the baseline method used for comparison directly uses FCN-resnet50 and is supervised by labeled data. Hung et al. uses an existing self-training method for comparison; for details, see WC Hung, YHTsai, YT Liou, YY Lin, and MH Yang, “Adversarial learning for semi-supervised semantic segmentation,” in 29th British Machine Vision Conference, BMVC 2018, 2019. S4-GAN uses an existing self-training method for comparison; for details, see S. Mittal, M.Tatarchenko, and T. Brox, “Semi-supervised semantic segmentation with high-and low-level consistency,” IEEE transactions on pattern analysis and machine intelligence, vol. 43, no. 4, pp. 1369–1379, 2019. CCT is an existing consistency regularization-based method used for comparison. For details, see the literature: Y. Ouali, C. Hudelot, and M. Tami, “Semi-supervised semantic segmentation with cross-consistency training,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12674–12684. Li et al. is an existing self-training method used for comparison. For details, see the literature: J. Li, B. Sun, S.Li, and X. Kang, “Semisupervised semantic segmentation of remote sensing images with consistency self-training,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–11, 2021.WSCL is an existing consistency regularization-based method used for comparison. For details, see X. Lu, L. Jiao, L. Li, F. Liu, X. Liu, S. Yang, Z. Feng, and P. Chen, “Weak-to-strong consistency learning for semisupervised image segmentation,” IEEE Transactions on Geoscience and Remote Sensing, 2023. HARC represents the method used in this embodiment. As shown in Tables 1 to 3:

[0097] Compared to the baseline method, the method of this embodiment (HARC) achieves significant improvements on all three test sets. On the Potsdam test set, for 1 / 8 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 5.1% and 7.48%, respectively; on 1 / 4 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 4.31% and 5.05%, respectively. On the Vaihingen test set, for 1 / 8 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 8.85% and 10.52%, respectively; on 1 / 4 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 6.65% and 8.66%, respectively. The segmentation performance of the car category is significantly improved on the Potsdam and Vaihingen test sets. The evaluation metrics of the baseline methods show that the car category is the most difficult to segment, and in both datasets, it has the fewest training samples. This result demonstrates that the method (HARC) of this embodiment performs better on small and difficult categories. On the LoveDA test set, for 1 / 8 labeled data, the method (HARC) of this embodiment improves the average F1 and average IoU by 2.96% and 3.28%, respectively; for 1 / 4 labeled data, the method (HARC) of this embodiment improves the average F1 and average IoU by 2.97% and 3.09%, respectively. Across all three datasets, the method of this embodiment outperforms the baseline on 1 / 8 labeled data compared to the baseline on 1 / 4 labeled data. While the Vaihingen dataset has the fewest training samples, the method (HARC) of this embodiment achieves the greatest improvement over Vaihingen. This demonstrates that the method (HARC) of this embodiment can better handle the challenges of small amounts of labeled samples compared to other baseline methods.

[0098] Compared with other consistency regularization-based methods (such as CCT and WSCL), on the Potsdam test set, in 1 / 8 labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 1.07%-37.1% and 1.73%-5.7% respectively; in 1 / 4 labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 1.03%-2.81% and 1.67%-4.33% respectively. This embodiment also made a visual comparison with WSCL, which has the second highest evaluation index, as shown in the figure below. Figure 5 As shown, 1 / 8 and 1 / 4 labeled data of the Potsdam dataset are compared, where (a) is the input original image, (b) is the true label, (c) and (d) are the results of WSCL and the method of this embodiment (HARC) on 1 / 8 labeled data of the Potsdam dataset, respectively, and (e) and (f) are the results of WSCL and the method of this embodiment (HARC) on 1 / 4 labeled data of the Potsdam dataset, respectively. On the Vaihingen test set, in 1 / 8 labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 4.68%-9.77% and 5.88%-11.27% respectively; in 1 / 4 labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 3.78%-60.1% and 5.10%-7.33% respectively. This embodiment also made a visual comparison with WSCL, which has the second highest evaluation index, as shown in the figure. Figure 6 The figure shows a comparison of the 1 / 8 and 1 / 4 labeled data of the Vaihingen dataset, where (a) is the input original image, (b) is the true label, (c) and (d) are the results of WSCL and the method of this embodiment (HARC) on the 1 / 8 labeled data of the Vaihingen dataset, respectively, and (e) and (f) are the results of WSCL and the method of this embodiment (HARC) on the 1 / 4 labeled data of the Vaihingen dataset, respectively. On the LoveDA test set, on the 1 / 8 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 14.48% and 13.05%, respectively; on the 1 / 4 labeled data, the method of this embodiment (HARC) improves the average F1 and average IoU by 11.76% and 11.13%, respectively.

[0099] Compared with other self-training methods such as Hung et al., s4-GAN, and Li et al., on the Potsdam test set, the method of this embodiment (HARC) improved the average F1 and average IoU by 0.84%-2.05% and 1.38%-2.68%, respectively, on 1 / 8 of the labeled data; on 1 / 4 of the labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 1.32%-2.64% and 2.03%-2.80%, respectively. On the Vaihingen test set, on 1 / 8 of the labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 3.4%-15.55% and 4.58%-12.89%, respectively; on 1 / 4 of the labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 3.06%-17.41% and 4.03%-15.58%, respectively. On the LoveDA test set, in 1 / 8 of the labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 1.00% and 1.23% respectively; in 1 / 4 of the labeled data, the method of this embodiment (HARC) improved the average F1 and average IoU by 3.21% and 3.30% respectively.

[0100] In addition, this embodiment also completed an ablation experiment on the Vaihingen dataset to explore the respective roles of the hierarchical enhancement module HA and the regional awareness comparison module RC. Table 4 shows the performance comparison results of the ablation experiment, mainly comparing weak to strong, weak to strong + RC, HA and the baseline method. The weak-to-strong network structure is as follows: Figure 7 As shown, the contrast-based class-level supervision in weak-to-strong+RC is weak-to-strong; HA is HARC without contrast-based class-level supervision.

[0101] Table 4 Performance comparison results of the performance ablation experiment on the Vaihingen dataset

[0102]

[0103] As shown in Table 4, compared with weak-to-strong, HA improves average F1 and average IoU by 0.21% and 0.20% on 1 / 8 labeled data, and by 1.2% and 1.69% on 1 / 4 labeled data. Compared with weak-to-strong+RC, the method of this embodiment (HARC) improves average F1 and average IoU by 0.22% and 0.30% on 1 / 8 labeled data. The hierarchical enhancement module HA improves average F1 and average IoU by 0.28% and 0.39%, respectively, on 1 / 4 labeled data. These improvements demonstrate the effectiveness of HA in semi-supervised semantic segmentation using RS. Compared with weak-to-strong, weak-to-strong+RC improves average F1 and average IoU by 0.27% and 0.55% on 1 / 8 labeled data, and by 2.02% and 2.86%, respectively, on 1 / 4 labeled data. Compared to HA, the method in this example (HARC) improved the average F1 and average IoU by 0.48% and 0.65% on 1 / 8 of the labeled data, and by 1.10% and 1.56% respectively on 1 / 4 of the labeled data. These improvements demonstrate that contrast-based class-level supervision is highly effective for semi-supervised semantic segmentation in RS. Furthermore, the improvements achieved by the region-aware contrast module (RC) are more significant than those achieved by the hierarchical enhancement module (HA).

[0104] In summary, based on the analysis of consistency regularization and self-training, this embodiment proposes a semi-supervised remote sensing image semantic segmentation method based on region contrast learning, which is a semi-supervised remote sensing image semantic segmentation method based on hierarchical enhancement and region-aware contrast, referred to as the HARC (Hierarchical Augmentation and Region-aware Contrastive) method. Its key features include a hierarchical enhancement module HA and a region-aware contrast module RC. The hierarchical enhancement module HA can simulate more situations than weak to strong consistency regularization. It allows the segmentation network to learn from images affected by uneven spectral changes, thereby improving the performance of the segmentation network; the region-aware contrast module RC allows the segmentation network to fully learn the class-level feature distribution of labeled data, and can also improve the utilization rate of unlabeled data, effectively reducing the impact of pseudo labels on model performance. The hierarchical enhancement module (HA) consists of three components, each simulating three conditions in remote sensing imagery. The first branch simulates only the effects of spatial variation; the second branch simulates the effects of both spatial variation and uniform spectral variation; and the third branch uses a splice between the second and third branches, which are affected by both spatial variation and uniform spectral variation, to make the spectral distribution of the third branch non-uniform, thereby simulating the effects of both spatial variation and non-uniform spectral variation. In the method of this embodiment, the first branch is used to train on labeled images and use the trained network to generate pseudo-labels for unlabeled image samples. The second and third branches are used to train on unlabeled samples. To balance the quality and quantity of pseudo-labels, this embodiment does not focus on experimenting to find the optimal confidence level, but instead retains all pseudo-labels to fully utilize the unlabeled data. This makes the model susceptible to the influence of a large number of noisy labels during the unlabeled data training phase. Therefore, it is crucial for the model to understand the true feature distribution of the unlabeled data. To address this issue, this embodiment designs a region-aware comparison module (RC). First, the region-aware contrast module (RC) uses region-aware contrastive learning to learn the class feature distribution during labeled data training. Then, the region-aware contrast module (RC) constrains the unlabeled data to have the same class distribution characteristics as the labeled data. Specifically, the region-aware contrast module converts each image into n region centers based on the image-based prediction results, where n is the number of categories in the dataset. Contrastive learning is then used to allow feature centers of the same category to be clustered together. In this way, the class feature memory is found and updated during labeled data training. Thanks to accurate label guidance, the categorical features of labeled data have a more accurate distribution than the categorical features of unlabeled data. Therefore, the class feature memory can be a good class label for unlabeled data to mitigate the impact of false pseudo-labels.The HARC method of this embodiment uses region-aware contrastive learning to learn the class-level feature distribution of all unlabeled samples, rather than some screened unlabeled samples, which solves the problem that the consistency regularization method is insufficient in simulating remote sensing images and the self-training method causes loss of semantic information. In the HARC method of this embodiment, a hierarchical enhancement strategy is used to simulate three levels of imaging condition changes, which include spatial changes, uniform spectral changes, and non-uniform spectral changes. The first layer of enhancement strategy only includes spatial changes, the second level includes spatial changes and uniform spectral changes, and the third level includes spatial changes and non-uniform spectral changes. Experimental results show that the HARC method of this embodiment is superior to the most advanced semi-supervised methods, and can accurately and effectively realize the semantic segmentation of remote sensing images. It is also proved through ablation experiments that the robust features of the data can be fully learned under the interference of noisy pseudo-labels.

[0105] In addition, this embodiment also provides a semi-supervised remote sensing image semantic segmentation system based on region contrast learning, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning.

[0106] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning through a processor.

[0107] In addition, this embodiment also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning through a processor.

[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0109] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A semi-supervised remote sensing image semantic segmentation method based on region contrast learning, characterized in that: It includes using a hierarchical enhancement module HA, a region-aware contrast module RC, three student networks consisting of an encoder and a segmentation network, and a teacher network to perform semi-supervised semantic segmentation training on the encoder and segmentation network. A semi-supervised semantic segmentation training includes: S101, there will be label images The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. , the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations ; S102, the image Input the first branch’s student network to extract labeled images Features , and according to the characteristics Predict labeled images The prediction results , according to the prediction results Get labeled images The center of the region will have the label image The center of the area is saved to the central storage part; the image Input the student network of the second branch to extract unlabeled images Features , according to the characteristics Predict unlabeled images The prediction results , according to the prediction results Get unlabeled images The center of the area; the image Input to the student network of the third branch to extract features and obtain the predicted value of the unlabeled remote sensing image , according to the predicted value Get the center of the region; update the parameters of the teacher network after the exponential moving average EMA operation on the parameters of the student network, and use the new teacher network to perform the hierarchical enhancement module HA on the unlabeled image The extracted unlabeled image generation only simulates spatial variations to produce unlabeled images Pseudo labels ; S103, through the regional awareness comparison module RC according to the regional center and pseudo label Calculate the region-aware contrast loss and pixel-level cross entropy loss, and update the parameters of the student network based on the total loss function composed of the region-aware contrast loss and pixel-level cross entropy loss.

2. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: In step S101, there will be a label image The hierarchical enhancement module HA is used to enhance the image to simulate only spatial changes. When including images with labels Weak enhancement using layered enhancement module HA w Get an image that only simulates spatial changes , the weak enhancement A w This includes partial or complete translation, rotation, scaling, and flipping.

3. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: In step S101, the unlabeled image The hierarchical enhancement module HA is used to enhance the image to simulate spatial variation and uniform spectral variation. and images that simulate spatial variations and non-uniform spectral variations When including the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A w Then perform strong enhancement A s Get an image that simulates spatial variation and uniform spectral variation , the unlabeled image Use the layered enhancement module HA to first perform weak enhancement A' w Then perform strong enhancement A' s , and then the obtained image and image Splice A c Get an image that simulates spatial variation and non-uniform spectral variation , where weak enhancement A w and weakly enhanced A' w Including translation, rotation, scaling, flipping of part or all, strong enhancement A s and strong enhancement A' s This includes randomly changing the image color, changing the image brightness and contrast, and adding some or all of Gaussian noise.

4. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: The calculation function expression of the regional center is: ， In the above formula, For a regional center of category i, Pixels in the image Features, is the pixel in the prediction result Tags, A binary indicator indicating whether the pixel belongs to class i.

5. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: In step S102, there will be a label image x l The function expression of the regional center saved to the central storage part is: ， In the above formula, For the central storage part, For a labeled dataset, is the i-th labeled image in the labeled dataset, is the i-th labeled image in the labeled dataset the entire regional center.

6. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: The function expression of the total loss function based on region-aware contrast loss and pixel-level cross entropy loss in step S102 is: ， In the above formula, is the total loss function, is the pixel-level cross entropy loss, is the weight parameter, is the region-aware contrast loss, and the pixel-level cross entropy loss is composed of the supervised learning part and the unsupervised learning part Additive composition, region-aware contrast loss has supervised learning part and the unsupervised learning part Add together and we have: ， ， ， ， In the above formula, is the number of labeled images, is the number of unlabeled images, For labeled images, For a labeled dataset, is the i-th category label of the labeled image, To simulate only spatially varying images Through the student network The result obtained is, is an unlabeled dataset, To simulate the image of spatial variation and uniform spectral variation Through the teacher network The result obtained is, To simulate the image of spatial variation and uniform spectral variation Through the student network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the teacher network The result obtained is, To simulate the spatial variation and non-uniform spectral variation of the image Through the student network The result obtained is, is the number of categories, is the regional perception of category i, and has: ， In the above formula, The central storage part All regional centers belonging to category i in for The feature vectors of the same category in yes The feature vectors of different categories in is the feature vector of the current image, is a hyperparameter.

7. The semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to claim 1, characterized in that: In step S102, the function expression for updating the parameters of the teacher network after the parameters of the student network are subjected to the exponential moving average EMA operation is: ， In the above formula, is the weight or bias in the parameters of the teacher network at the current moment, is the smoothing coefficient, is the weight or bias in the parameters of the teacher network at the previous moment, is the weight or bias in the parameters of the student network at the current moment.

8. A semi-supervised remote sensing image semantic segmentation system based on region contrast learning, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instruction stored therein, characterized in that: The computer program or instruction is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to any one of claims 1 to 7 through a processor.

10. A computer program product comprising a computer program or instructions, characterized in that The computer program or instruction is programmed or configured to execute the semi-supervised remote sensing image semantic segmentation method based on region contrast learning according to any one of claims 1 to 7 through a processor.

Images