Semi-supervised landslide image semantic segmentation method and system

Abstract

The invention discloses a semi-supervised landslide image semantic segmentation method and system, and belongs to the technical field of deep learning. When the confidence of a pseudo label is smaller than a confidence threshold, marked sample image information is injected into an unmarked sample with the confidence smaller than the confidence threshold through participating in the clipping and pasting process; constructing a semantic similarity graph of the marked samples, performing weighted diffusion on a prediction result of the marked samples by determining a semantic similarity perception graph, and determining a high-confidence region mask; randomly sampling a plurality of anchor points from the semantic similarity graph to generate a shape mask of a position corresponding to each anchor point, and injecting the shape feature of each anchor point into an unmarked sample of which the confidence coefficient is greater than or equal to a confidence coefficient threshold value; and generating a mixed sample by updating the unmarked sample to serve as a training set, and training the segmentation model. The method can effectively improve the accuracy of the semantic segmentation result of the semi-supervised landslide image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, more particularly to a semi-supervised landslide image semantic segmentation method and system. BACKGROUND

[0002] In early landslide remote sensing interpretation research, traditional machine learning methods such as support vector machine and random forest dominate, and deep neural network technology has not been used. In addition to landslide area identification, disaster susceptibility assessment is also a research focus, aiming to establish a disaster probability model to divide risk levels by analyzing the spatial distribution of historical landslides, topographic features and related environmental triggering factors. However, traditional methods require manual feature design and use of high-precision labeled data, which has the problems of low efficiency and strong subjectivity, especially in the application of large-scale complex terrain areas.

[0003] In recent years, deep learning semantic segmentation technology has gradually replaced traditional methods in landslide remote sensing detection, showing higher accuracy and automation.

[0004] However, the generalization ability of such models is insufficient due to large geographical environmental differences, limited size of existing landslide data sets, and non-uniform labeling standards, resulting in insufficient cross-regional adaptability of the model during identification and detection, and easy misjudgment of landslide boundaries in plateau and canyon areas with less training data, affecting the robustness and horizontal comparison research of the model, thereby affecting the semantic segmentation accuracy of landslides. SUMMARY

[0005] To solve the above problems, the present application provides a semi-supervised landslide image semantic segmentation method and system, which compares the confidence of the obtained random number and the pseudo label. When the confidence of the pseudo label is less than the random number, the real label information of the labeled sample is injected into the unlabeled sample with confidence less than the random number through the process of cutting and pasting. When the confidence of the pseudo label is greater than or equal to the random number, the semantic similarity graph of the labeled sample is constructed, and the semantic similarity perception and region propagation of the landslide shape are realized according to the semantic information and geometric feature information of the semantic similarity graph. The shape features of each anchor point are injected into the unlabeled sample with confidence greater than or equal to the random number, and the unlabeled sample is updated to generate a mixed sample. The mixed sample is used as a training set to train the segmentation model, effectively suppressing the error propagation of the misjudged area, thereby improving the accuracy of the semi-supervised landslide image semantic segmentation result.

[0006] To solve the above technical problems, the present application discloses a semi-supervised landslide image semantic segmentation method, comprising the following steps: Obtain unlabeled samples and labeled samples of landslide images; The prediction result of the unlabeled sample is obtained by the segmentation model, the prediction result is used as a pseudo label, and the confidence of the pseudo label is determined. A semantic similarity graph of the labeled sample is constructed, the prediction result of the labeled sample is weighted and diffused by determining a semantic similarity perception graph, a high confidence area mask is determined, and a shape mask corresponding to a position of each anchor point is generated by randomly sampling a plurality of anchor points from the semantic similarity graph. The shape mask and the high confidence area mask are used to determine an overlap rate, when the overlap rate is greater than an overlap rate threshold, a dominant class in the shape area is calculated through the semantic similarity perception graph, when the proportion of the dominant class exceeds a dominant class threshold, the shape feature of each anchor point is injected into the unlabeled sample with a confidence greater than or equal to the random number. The mixed sample is generated by updating the unlabeled sample, and the mixed sample is used as a training set to train the segmentation model.

[0007] Preferably, the confidence of the pseudo label is obtained by calculating a score of the confidence of the pseudo label, and the score formula of the confidence of the pseudo label is: ; Wherein, H is the height of the feature map, W is the width of the feature map, is the first i the prediction result generated by the segmentation model in the feature map of the first J indicates the first i pixel in the first J unlabeled sample, and C is the classification category in the landslide segmentation task.

[0008] Preferably, the pseudo label of the unlabeled sample with a confidence less than a confidence threshold is specifically: The prediction result of the unlabeled sample obtained by the segmentation model is used as a pseudo label, and the score of the confidence of the pseudo label is determined. A random number between 0 and 1 is obtained, and when the score of the confidence of the pseudo label of the unlabeled sample is less than the random number, the square cutting contour of the labeled sample image is pasted onto the unlabeled sample image to generate a preliminary mixed sample. The square cutting contour of the preliminary mixed sample is pasted onto the unlabeled sample by randomly selecting the square cutting contour of the preliminary mixed sample.

[0009] Preferably, the semantic similarity perception graph is specifically determined as follows: A feature extraction network for obtaining a semantic similarity perception graph is constructed, and the feature extraction network includes an encoder and two convolutional layers. The multi-stage feature interaction is performed by the encoder to construct a semantic similarity graph of the labeled sample: ; In the formula, is a feature map obtained after the first stage of the encoder; is a feature map obtained after the first stage of the encoder; n is a feature map obtained after the first stage of the encoder; D is the number of channels; HW is the number of pixels of the feature map; Conv indicates a 1x1 convolution; After the encoder extracts the features, the semantic similarity graph of the labeled sample is projected by two convolutional layers, the projected features are multiplied by a matrix and normalized to obtain a semantic similarity perception graph matrix: ; In the formula, T is a transposition operation of a matrix; is a semantic similarity perception graph matrix; each element C in the matrix c ( i , j ) quantifies the semantic association degree between the position i and j .

[0010] Preferably, the determination of the high-confidence region mask specifically comprises: obtaining an unnormalized initial prediction result x i of the labeled sample by the segmentation model; ; Multiplying the unnormalized initial prediction result of the labeled sample by the semantic similarity perception graph matrix. The unnormalized initial prediction result of the labeled sample is weighted and diffused by the semantic similarity perception graph matrix: ; In the formula, is a bilinear interpolation operation for adjusting the feature size, and the interpolation operation is equivalent to neighborhood aggregation of the prediction result by the similarity weight; Comparing the maximum value in the unnormalized initial prediction result of the labeled sample with the confidence threshold value, when the maximum value in the unnormalized initial prediction result of the labeled sample is greater than the confidence threshold value, the high-confidence region mask of the labeled sample is: ; In the formula, is a preset confidence threshold value; M is a high-confidence region mask, is an indicator function.

[0011] Preferably, the determining the overlap rate by the shape mask and the high-confidence region mask comprises: generating a binary shape mask corresponding to the position of each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph; and performing an intersection operation between the binary shape mask and the high-confidence region to obtain the overlap rate.

[0012] Preferably, the method further comprises constructing a loss function integrating the labeled samples and the unlabeled samples, and specifically comprising: performing a cross-entropy loss between the prediction result of the unlabeled sample and the pseudo label to obtain a semi-supervised unlabeled sample loss function; performing a cross-entropy loss between the unnormalized initial prediction result of the labeled sample and the real label to obtain a semi-supervised labeled sample loss function; weighting and summing the cross-entropy loss of the updated pseudo label, the semi-supervised unlabeled sample loss function and the semi-supervised labeled sample loss function to obtain a total loss function integrating the labeled samples and the unlabeled samples.

[0013] Preferably, the cross-entropy loss function of the updated pseudo label is a calculation result of weighting and diffusing the unnormalized initial prediction result of the labeled sample, and pixel-by-pixel multiplication with the high-confidence region mask of the labeled sample.

[0014] Preferably, the method further comprises a semi-supervised landslide image semantic segmentation system, comprising: a data acquisition module for acquiring unlabeled samples and labeled samples of landslide images; a pseudo label mixing module for obtaining a prediction result of the unlabeled sample by a segmentation model, taking the prediction result as a pseudo label, and determining the confidence of the pseudo label, wherein when the confidence of the pseudo label is less than a random number, the labeled sample image information is injected into the unlabeled sample with confidence less than the random number by participating in the process of cutting and pasting; a pseudo label mixing module with confidence not less than a random number, comprising a landslide semantic segmentation module and a landslide region propagation module, wherein: the landslide semantic segmentation module is configured to construct a semantic similarity graph of the labeled sample, to perform weighted diffusion on the prediction result of the labeled sample by determining a semantic similarity perception graph, and to determine a high-confidence region mask; the landslide region propagation module is configured to generate a shape mask corresponding to the position of each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph, to determine an overlap rate by the shape mask and the high-confidence region mask, to calculate a dominant class within the shape region by the semantic similarity perception graph when the overlap rate is greater than an overlap rate threshold, and to inject the shape feature of each anchor point into the unlabeled sample with confidence greater than or equal to the random number when the proportion of the dominant class exceeds a dominant class threshold. The segmentation module is used for generating a mixed sample by updating the unlabeled sample, taking the mixed sample as a training set, and training the segmentation model.

[0015] Compared with the prior art, the present application has the following beneficial effects: The semi-supervised mountain landslide image semantic segmentation method provided by the present application injects high-confidence labeled sample image information into the unlabeled sample with confidence less than the random number, enhances the pseudo-label confidence through a hybrid enhancement process when the pseudo-label confidence of the unlabeled sample is less than the random number. When the pseudo-label confidence of the unlabeled sample is greater than or equal to the random number, a semantic similarity graph of the labeled sample is constructed, the labeled information provides a clear training target for the model, so that the model can more fully utilize the semantic information and geometric feature information of the labeled sample, utilize the semantic association and shape implicit information in the similarity perception graph to enhance the perception ability of the landslide area, update the unlabeled sample and expand the high-confidence area. The confidence of the pseudo-label is used as the probability of avoiding triggering data enhancement, that is, the higher the confidence, the less likely it is to trigger such enhancement, which reduces the deviation caused by strong dependence on the pseudo-label, the generated mixed sample can effectively suppress the error propagation caused by the pseudo-label noise in the early training, reduce the uncertainty in the training process, and thus effectively improve the segmentation performance of the model under a small amount of labeled data. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the semi-supervised mountain landslide image semantic segmentation method provided by the present application is provided. Figure 2 The pixel semantic propagation process graph provided by the embodiment of the present application is provided. Figure 3 The hybrid enhancement flowchart provided by the embodiment of the present application is provided. Figure 4 The landslide area shape perception graph provided by the embodiment of the present application is provided. Figure 5 The segmentation comparison results of the mountain landslide area of a certain province data set identified by different methods provided by the embodiment of the present application are provided. Figure 6 The segmentation comparison results of the mountain landslide area of a certain province data set identified by different methods provided by the embodiment of the present application are provided. Figure 7 The segmentation comparison results of the mountain landslide area of a certain province data set identified by different methods provided by the embodiment of the present application are provided. Figure 8 The evolution process of the semantic segmentation result provided by the embodiment of the present application with the increase of the labeled data is provided. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application. Figures 1-8 The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application.

[0018] Embodiments As shown in the embodiments of the present application, a semi-supervised landslide image semantic segmentation method is provided, comprising the following steps: Figure 1 S1: obtaining unmarked samples and marked samples of landslide images; The prediction result of the unmarked sample is obtained through the segmentation model, the prediction result is used as a pseudo-label, and the confidence of the pseudo-label is determined; when the confidence of the pseudo-label is less than a random number, the image information of the marked sample is injected into the unmarked sample with a confidence less than the random number by participating in the cutting and pasting process; S2: constructing a semantic similarity graph of the marked sample, performing weighted diffusion on the prediction result of the marked sample through the semantic similarity perception graph, determining a high-confidence area mask, and generating a shape mask corresponding to the position of each anchor point through random sampling of multiple anchor points from the semantic similarity graph; the overlap rate is determined through the shape mask and the high-confidence area mask; when the overlap rate is greater than an overlap rate threshold, the dominant class in the shape area is calculated through the semantic similarity perception graph; when the proportion of the dominant class exceeds a dominant class threshold, the shape features of each anchor point are injected into the unmarked sample with a confidence greater than or equal to the random number; S3: generating mixed samples by updating the unmarked samples, and training the segmentation model by taking the mixed samples as a training set.

[0019] Specifically, the unmarked sample data is mixed and enhanced through an adaptive anchoring mixed enhancement strategy. The adaptive anchoring mixed enhancement strategy merges the weak enhancement and strong enhancement data streams of the unmarked sample data in the previous semi-supervised learning into one data stream, and determines the enhancement scale of each unmarked sample instance according to the adaptive anchoring mixed enhancement strategy.

[0020] In step S2, the adaptive anchoring mixed enhancement strategy uses the labels of the marked sample images to participate in the cutting and pasting process to enhance the credibility of the pseudo-labels, and can also make more full use of the semantic information and geometric feature information of the marked samples. Anchoring means that when the pseudo-label quality of the unmarked data is insufficient, the marked data is injected into the mixed process as a reliable knowledge anchor.

[0021] As shown in the embodiments of the present application, a semi-supervised landslide image semantic segmentation method is provided, comprising the following steps: Figure 3 As shown in the process diagram of the adaptive anchoring mixed enhancement strategy, in the figure u 1, u 2, u ​3 refers to the unmarked image number that needs to be enhanced, x 1, x 2, x 3 refers to the marked image number participating in enhancement, , , is u 1, u 2, u 3 corresponds to the confidence score of the pseudo label calculated by calculation, , , is u 1, u 2, u 3 corresponds to a random number in the range of [0, 1].

[0022] At the same time, considering mixing the marked information with high credibility and the pseudo label of unmarked data, while improving the credibility and the amount of information, the proportion of unmarked data will also be reduced, and even some parts of some unmarked sample data may be covered without participating in training. Therefore, the application specially calculates the confidence score formula of the pseudo label to determine whether the anchor marked sample is needed through adaptive detection. The confidence score formula of the pseudo label is: ; Among them, H is the height of the feature map, W is the width of the feature map, is the i th unmarked sample feature map generated by the segmentation model, J represents the i th pixel in the J th unmarked sample, and C is the classification category in the landslide segmentation task. The value of the application is N 2.

[0023] Particularly, in the training stage, the segmentation model is a traditional student model; in the prediction stage, the segmentation model is a traditional teacher model.

[0024] The process of the adaptive anchor mixed enhancement strategy specifically includes: First, number each unmarked sample image u m , and through random assignment of another unmarked sample image u n and a marked sample image x n to form a pair.

[0025] The current model is calculated by the confidence score formula of the pseudo labeli Pseudo-label confidence scores for unlabeled samples , the pseudo label confidence score is obtained by comparing the student model to the first i The prediction values ​​of the unlabeled samples are logarithmized and then weighted averaged to estimate the value.

[0026] Obviously, the confidence score of the pseudo label It is targeted at each unlabeled sample and is closely related to the generalization ability of the current student model. The confidence score of the pseudo-label is used as the probability of avoiding triggering data augmentation. That is, the higher the confidence score, the less likely this augmentation will be triggered.

[0027] A random number between 0 and 1 obtained by comparison The size of the score with the confidence of the pseudo label determines whether to perform hybrid enhancement. Assuming that the unlabeled sample image u m If enhancement is required: There will be labeled sample images x n Cutting contour M n Paste to unlabeled sample image u n Get a mixed sample , so that the unlabeled samples used for cut-paste contain the true label information of the labeled samples with high credibility, and obtain the preliminary mixed samples ; Then randomly select the initial mixed sample The square cut outline M n , the initial mixed sample Square cut outline M n Paste it onto the unlabeled sample and obtain a mixed sample when the confidence of the pseudo label is less than the random number obtained .

[0028] The mixed sample The generation formula is as follows: ; ; in, M n For labeled sample images x n The corresponding mixed sample is cut by randomly selecting a square contour Send it to the student model for training.

[0029] In step S3, when the confidence of the pseudo-label is greater than or equal to the obtained random number, a semantic similarity graph is constructed to capture the semantic association and shape implicit information of the landslide area, specifically: By using the multi-stage feature interaction of the encoder, a semantic similarity graph is constructed. Based on the semantic similarity graph and the landslide area shape perception map sampled from the graph, the semantic association and shape implicit information of the landslide area are captured.

[0030] Based on the semantic similarity perception map, a pixel-level semantic propagation strategy is constructed, and the invisible information of the landslide shape contained in the similarity perception map is sampled. A landslide area propagation strategy is constructed to facilitate the use of the high-level semantic information and hidden shape information contained in the semantic similarity perception map.

[0031] like Figure 3 As shown in FIG, the calculation process of the semantic similarity perception map and the landslide area shape perception map is shown. The present invention constructs a feature extraction network for obtaining the semantic similarity perception map, and the feature extraction network includes an encoder and two convolutional layers; Through the encoder, multi-stage feature interaction is performed to construct a semantic similarity graph of labeled samples: ; in, is the feature map obtained in the first stage of the encoder, is the feature map obtained after the fourth stage of the encoder, D is the number of channels, HW is the number of pixels in the feature map, refer to convolution.

[0032] After the encoder extracts the features, it projects the features of the semantic similarity graph of the labeled samples through two convolutional layers. The projected features are matrix multiplied and normalized to obtain the semantic similarity perception graph matrix: ; in, T is the transpose operation of the matrix; is the semantic similarity perception map matrix, which C Each element in c ( i , j ) quantifies the position i and j The degree of semantic association between i and j express C Elements in the semantic similarity perception graph matrix cThe horizontal and vertical coordinates, and the unique advantage is embodied in the fusion of multi-scale information through feature product, effectively fusing deep semantic information and shallow geometric features, capturing details and textures while maintaining the shape of the landslide main body.

[0033] The shape information of the calculated graph is hidden, and the spatial correlation network is implicitly constructed through matrix operation, and the landslide shape is modeled. The matrix multiplication operation of the semantic similarity graph essentially constructs the topological correlation network between pixels, so that the positions within the landslide area have higher interconnection strength. This feature will help the model maintain the geometric continuity of the prediction results in the subsequent, and strengthen the model to naturally maintain the geometric integrity of the prediction results through feature interaction.

[0034] The pixel-level semantic propagation strategy is to realize the semantic propagation of the landslide area through the semantic similarity perception graph. The specific method of this pixel-level semantic propagation strategy is: through the similarity graph, the unnormalized prediction is weighted and propagated, thereby enhancing the model's perception ability of the landslide area. The implementation is very simple, through the teacher model to obtain the labeled sample x i Unnormalized initial prediction result of the labeled sample x i Unnormalized initial prediction result Multiply the semantic similarity perception graph matrix to get: ; Where, is a bilinear interpolation operation for adjusting feature size. The interpolation operation is equivalent to using similarity weights to perform neighborhood aggregation on the prediction results, so that the predictions of similar pixels tend to be consistent.

[0035] After completing the weighted diffusion of the semantic similarity perception graph matrix on the prediction results, the high-confidence region mask and the updated cross-entropy loss function of the pseudo-label are obtained respectively: ; ; Where, is a manually set confidence threshold, M is a high-confidence region mask, is a pixel-by-pixel multiplication, is a pseudo-label of an unlabeled sample i , and N is the number of feature image pixels, L CE is a cross-entropy loss function, is an indicator function.

[0036] This cross-entropy loss enables the propagation process to focus on optimizing reliable areas. In this way, the pixel-level semantic propagation module can effectively propagate the semantic information contained in the similarity map to the entire image, thereby improving the model's perception of landslide areas.

[0037] A landslide area propagation strategy is constructed, which can realize the perception and propagation of landslide shapes through the geometric characteristics of the semantic similarity graph.

[0038] Semantic Similarity Perception Map C Each row c represents the similarity between a single feature vector and all vectors in the entire feature map, which implicitly encapsulates shape information.

[0039] This paper proposes a landslide area propagation strategy that enhances pseudo-labels using this shape information. The core idea of ​​this landslide area propagation strategy is to use the landslide shape information implicit in the correlation graph to enhance the geometric integrity of the pseudo-labels. The specific implementation includes three steps: Shape information extraction. By using semantic similarity perception graph C Random sampling m anchor points, generate a binary shape mask at the corresponding position: ; in, C 𝑘 The first k This operation converts the continuous similarity map into a discrete shape mask.

[0040] Trusted region fusion. Shape mask is combined with high confidence regions Perform intersection and union operations to obtain the overlap ratio: ; When the overlap rate r Greater than the preset overlap rate threshold Compute the dominant category within the shape region when : ; in, c Semantic similarity perception graph C Any row in .

[0041] Label enhancement propagation. When the proportion of the dominant category exceeds the preset dominant category threshold When , update the shape area to the pseudo label: ; Also expand the high confidence region: ; Update the high confidence region mask of the labeled samples by expanding the high confidence region.

[0042] The landslide area propagation strategy generates a smooth target contour through the local continuity of the correlation map, and effectively suppresses the error propagation of the misdetected area through the statistics of the dominant category, and reduces the computational complexity through random sampling. down to , avoiding the high cost of full-graph calculation.

[0043] The design of the landslide area propagation strategy enables the model to fully utilize shape information in the landslide segmentation task and enhance the perception of the landslide area.

[0044] The present invention also includes constructing a total loss function that integrates labeled samples and unlabeled samples according to a batch-based stochastic gradient descent method, using the mixed samples enhanced by adaptive anchor mixing as a training set, and training the segmentation model through the total loss function.

[0045] Construct a loss function that combines labeled samples and unlabeled samples. The specific process includes: Obtain prediction results for unlabeled samples through the segmentation model P ( U i ), the prediction results of unlabeled samples P ( U i ) and pseudo labels Perform cross entropy loss to obtain the semi-supervised unlabeled sample loss function: ; Obtaining prediction results for labeled samples through segmentation models P ( L i ), there will be prediction results for labeled samples P ( L i ) and the true label y i Perform cross entropy loss to obtain the semi-supervised labeled sample loss function: ; The cross entropy loss function after pseudo label update is: ; The cross entropy loss after pseudo label update, the semi-supervised unlabeled sample loss function and the semi-supervised labeled sample loss function are weighted summed to obtain the overall target loss function, that is, the total loss function of the combined labeled and unlabeled samples: ; in, is the overall target loss function; is a semi-supervised unlabeled sample loss function; is a cross-entropy loss of the prediction result of the labeled sample and the true value label; is the cross-entropy loss of the updated pseudo label through the landslide region propagation strategy; and are hyperparameters, respectively controlling the weights of the three loss functions.

[0046] The overall target loss function is composed of three loss functions, represents the traditional semi-supervised unlabeled sample loss function, and the cross-entropy loss of the prediction result of the unlabeled sample and the pseudo label; represents the cross-entropy loss of the prediction result of the labeled sample and the true value label, and is the cross-entropy loss of the updated pseudo label. In the practical examples of the present application, and are respectively set to 0.25, 0.25 and 0.5.

[0047] By updating the high-confidence region mask of the unlabeled sample and the pseudo label, an adaptive enhanced mixed sample is generated. The generated adaptive enhanced mixed sample is used as a training set to train the segmentation model (student model).

[0048] Correspondingly, the present application also provides a semi-supervised landslide image semantic segmentation system, comprising: A data acquisition module is used to acquire unlabeled samples and labeled samples of landslide images. A pseudo label mixing module with confidence less than a random number is used to obtain the prediction result of the unlabeled sample through the segmentation model, take the prediction result as the pseudo label, and determine the confidence of the pseudo label. When the confidence of the pseudo label is less than the obtained random number, the labeled sample image information is injected into the unlabeled sample with confidence less than the random number by participating in the clipping and pasting process. A pseudo label mixing module with confidence not less than a random number comprises a landslide semantic segmentation module and a landslide region propagation module, wherein: The landslide semantic segmentation module is used to construct a semantic similarity graph of the labeled sample, to perform weighted diffusion on the prediction result of the labeled sample by determining a semantic similarity perception graph matrix, and to determine a high-confidence region mask. ​​The mountain landslide region propagation module is configured to generate a shape mask corresponding to a position of each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph, determine an overlap rate by the shape mask and the high-confidence region mask, calculate a dominant class within the shape region by the semantic similarity perception graph when the overlap rate is greater than an overlap rate threshold, and inject shape features of each anchor point into unlabeled samples with a confidence greater than or equal to a random number when a proportion of the dominant class exceeds a dominant class threshold. The segmentation module is configured to generate mixed samples by updating the unlabeled samples, and train the segmentation model by taking the mixed samples as a training set.

[0049] To solve the problem of lack of labeled data in the mountain landslide remote sensing image segmentation task and the urgent need for a high-precision segmentation model under a small amount of labeled scene, the present application provides a semi-supervised semi-supervised mountain landslide image semantic segmentation method based on semantic similarity perception. The method introduces an adaptive anchor mixed enhancement strategy, and integrates the real label information of high-confidence labeled samples into the mixed enhancement process of unlabeled samples, effectively inhibiting the error propagation caused by pseudo-label noise in early training. By constructing a semantic similarity graph, a pixel-level semantic propagation strategy and a landslide region propagation strategy are constructed, and the semantic information and the implicit shape information in the similarity perception graph are used to effectively improve the segmentation performance of the model under a small amount of labeled data.

[0050] Experimental analysis The present embodiment is tested on a mountain landslide data set of a certain province, and the experimental results show that the present application significantly improves the mountain landslide region recognition ability of the model.

[0051] As shown in Table 1, the main index values of a small number of label proportions on a certain province mountain landslide face data set are compared, as shown in Table 2, the main index values of a medium number of label proportions on a certain province mountain landslide face data set are compared, by comparing the precision (Precision), recall (Recall), F1 index (F1-score), Dice similarity coefficient (Dice Similarity Coefficient, DSC) and average Hausdorff distance (Hausdorff Distance, HD) on the certain province mountain landslide face data set. The numerical comparison of the values of the invention and other benchmark methods on the mountain landslide data set is shown in Table 1 and Table 2. The benchmark methods compared include FixMatch, UniMatch, Mean Teacher, Cross Teaching between CNN and Transformer, Uncertainty Rectified Pyramid Consistency, Interpolation Consistency Training, Cross Pseudo Supervision and ABD.

[0052] From the quantitative results of Table 1 and Table 2, it can be seen that the main indicators of the method of the present application are significantly higher than those of other methods. The loess landslide data set of a certain province in China is collected from optical remote sensing images of multiple regions in a certain province in China, and a total of 1025 labeled samples are included. Among them, 820 training samples, accounting for about 80% of the total, 96 examples in the validation set and 109 examples in the test set are reserved, accounting for 10% respectively. As shown in Table 1 and Table 2, the labels set in this embodiment, of which 5% is called the label proportion, 10% is called a small amount of proportion, and 20%, 30%, and 50% are called medium proportions.

[0053] Table 1 Comparison of main index values of a small number of label proportions on a certain province mountain landslide face data set

[0054] Table 2 Comparison of main index values of a medium number of label proportions on a certain province mountain landslide face data set

[0055] Figure 5 、 Figure 6 、 Figure 7 The above benchmark methods and the semi-supervised mountain landslide image semantic segmentation method based on semantic similarity perception proposed by the present application are respectively shown in the segmentation comparison results of the mountain landslide region of a certain province data set in the mountain forest, gobi and farmland scenes.

[0056] From Figure 5 、Figure 6 、 Figure 7 As can be seen, the semi-supervised landslide image semantic segmentation method proposed by the present application has very good segmentation effect on landslides of different scales in various geographical environments, with clear edges and accurate recognition. From the segmentation results, it can be observed that the present application has the best control over the shape of the landslide area. The other comparative methods have overlarge segmentation areas due to misjudgment of landslides, and the adhesion phenomenon is also serious. The present application is closest to the label in shape.

[0057] The visualization comparison results show that the present application exhibits robust segmentation capability on the certain province dataset, and the comprehensive performance is better than that of the existing comparative methods. Under complex environmental conditions, the model can still maintain good segmentation effect, and has strong generalization ability and anti-interference ability.

[0058] The visualization comparison results of the above Figures 5-7 show that the present application exhibits robust segmentation capability on the certain province dataset, and the comprehensive performance is better than that of the existing comparative methods. Under complex environmental conditions, the model can still maintain good segmentation effect, and has strong generalization ability and anti-interference ability.

[0059] As Figure 8 shown, the evolution process of the segmentation results with the growth of the labeled data is shown. The first column is the original remote sensing image, the second column is the true value result labeled by artificial labeling, and the subsequent five columns correspond to the prediction effect when the proportion of labeled data is 5%, 10%, 20%, 30% and 50% respectively.

[0060] From the observation of the first row in Figure 8 , it can be seen that the present application has strong segmentation capability with a small amount of labeling, and from 5% to 50%, the segmentation result is highly consistent with the true value and can remain stable.

[0061] From the observation of the second row in Figure 8 , it can be seen that when the background around the landslide is similar to the optical signal, the present application can still segment out the shape consistent with the label at 5%.

[0062] From the observation of the third row in Figure 8 , it can be seen that the present application can also accurately segment the landslide in a background with a small proportion of 5% and similar color.

[0063] From the observation of the fourth row in Figure 8 , it can be seen that the present application is not disturbed by the road. It is worth noting that in the extreme case where the labeled sample data is only 5%, the model can already accurately capture the outline of the landslide, but a larger proportion of labeled sample data is needed to better handle it. When the proportion of labeled sample data is 20%, the model output is significantly improved in terms of fine degree and ability to capture the outline of the landslide.

[0064] When the labeled sample data is greater than 20%, the model has a relatively accurate segmentation result for landslides in all complex terrains and environments. When the labeling ratio reaches 50%, the segmentation result of the method of the present application is extremely close to the true value label. It can be seen that the ability of the method of the present application to understand the global context of landslides is significantly enhanced as the proportion of labeled data increases.

[0065] By analyzing the semantic segmentation results of different methods provided by the present embodiment, it is shown that the method proposed by the present application achieves excellent segmentation results on the loess landslide data set of a certain province, and is superior to all other comparative models in key indicators. Ablation experiments also prove the effectiveness of each module, and the visualization effect diagram shows good landslide feature extraction capability and complex scene noise resistance capability, verifying the generalization capability and robustness of the method in various terrain scenes.

[0066] The above describes only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can make equivalent replacements or changes to the technical solutions and inventive concepts of the present application within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

[0067] In addition, unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art. All documents mentioned in the specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of the specification shall prevail.

Claims

1. A semi-supervised semantic segmentation method for landslide images, characterized in that, The method comprises the following steps: Obtain the unmarked sample and the marked sample of the landslide image; Obtain the prediction result of the unmarked sample through the segmentation model, take the prediction result as a pseudo label, and determine the confidence of the pseudo label; when the confidence of the pseudo label is less than a random number, inject the image information of the marked sample into the unmarked sample with the confidence less than the random number through the participation of the cutting and pasting process; Construct a semantic similarity graph of the marked sample, perform weighted diffusion on the prediction result of the marked sample through the semantic similarity perception graph, and determine a high-confidence region mask; Randomly sample multiple anchor points from the semantic similarity graph, generate a shape mask corresponding to the position of each anchor point, determine an overlap rate through the shape mask and the high-confidence region mask, calculate the dominant class in the shape region through the semantic similarity perception graph when the overlap rate is greater than an overlap rate threshold, and inject the shape features of each anchor point into the unmarked sample with the confidence greater than or equal to the random number when the proportion of the dominant class exceeds a dominant class threshold; Generate a mixed sample by updating the unmarked sample, take the mixed sample as a training set, and train the segmentation model.

2. The semi-supervised semantic segmentation method of mountain landslide image according to claim 1, characterized in that, The confidence of the pseudo label is obtained by calculating the confidence score of the pseudo label, and the confidence score formula of the pseudo label is: ； wherein, H is a height of the feature map, W is a width of the feature map, is a first i a prediction result generated by the segmentation model in the feature map of the unlabeled sample, J represents a first i a first J pixel in the unlabeled sample, and C is a classification category in the landslide segmentation task.

3. The semi-supervised landslide image semantic segmentation method according to claim 2, characterized in that: The pseudo label of the unmarked sample with the confidence less than the confidence threshold is specifically as follows: Take the prediction result of the unmarked sample obtained through the segmentation model as a pseudo label, and determine the confidence score of the pseudo label; Obtain a random number between 0 and 1, and when the confidence score of the pseudo label of the unmarked sample is less than the random number, cut the square contour of the marked sample image and paste it on the unmarked sample image to generate a preliminary mixed sample; Randomly select the square cutting contour of the preliminary mixed sample, and paste the square cutting contour of the preliminary mixed sample on the unmarked sample.

4. The semi-supervised semantic segmentation method of mountain landslide image according to claim 1, characterized in that, The determination of the semantic similarity perception graph specifically comprises: Construct a feature extraction network for obtaining the semantic similarity perception graph, and the feature extraction network comprises an encoder and two convolutional layers; Perform multi-stage feature interaction through the encoder to construct a semantic similarity graph of the marked sample: ； In the formula, is the feature map obtained after the first stage of the encoder; is the feature map obtained after the first stage of the encoder; n is the feature map obtained after the first stage of the encoder; D is the number of channels; HW is the number of pixels of the feature map; Conv indicates a 1x1 convolution; After the encoder extracts the features, perform feature projection on the semantic similarity graph of the marked sample through the two convolutional layers, multiply the projected features by a matrix, and normalize to obtain a semantic similarity perception graph matrix: ； where T is a transpose operation of a matrix; is a semantic similarity-aware graph matrix; this matrix C each element c ( i , j ) quantifies the degree of semantic association between a location i and j .

5. The semi-supervised semantic segmentation method of mountain landslide image according to claim 4, characterized in that, The determination of the high-confidence region mask specifically comprises: acquiring labeled samples by segmentation model x i unnormalized initial prediction results ; Multiply the unnormalized initial prediction result of the marked sample by the semantic similarity perception graph matrix to perform weighted diffusion on the unnormalized initial prediction result of the marked sample through the semantic similarity perception graph matrix: ； In the formula, For the bilinear interpolation operation to adjust the feature size, the interpolation operation is equivalent to neighborhood aggregation of the prediction result with similarity weights. Compare the maximum value in the unnormalized initial prediction result of the marked sample with a confidence threshold, and when the maximum value in the unnormalized initial prediction result of the marked sample is greater than the confidence threshold, generate a high-confidence region mask of the marked sample: ； wherein is a preset confidence threshold; M is a high confidence region mask, is an indicator function.

6. The semi-supervised semantic segmentation method of mountain landslide image according to claim 5, characterized in that, The determination of the overlap rate through the shape mask and the high-confidence region mask comprises: The binary shape mask corresponding to the position of each anchor point is generated by randomly sampling a plurality of anchor points from the semantic similarity graph.

7. The semi-supervised semantic segmentation method of mountain landslide image according to claim 1, characterized in that, Also comprising constructing a loss function integrating labeled samples and unlabeled samples, specifically comprising: Cross-entropy loss is made between the prediction result of the unlabeled sample and the pseudo label to obtain a semi-supervised unlabeled sample loss function; Cross-entropy loss is made between the non-normalized initial prediction result of the labeled sample and the real label to obtain a semi-supervised labeled sample loss function; The cross-entropy loss function after updating the pseudo label, the semi-supervised unlabeled sample loss function and the semi-supervised labeled sample loss function are weighted and summed to obtain a total loss function integrating labeled samples and unlabeled samples.

8. The semi-supervised semantic segmentation method of landslides according to claim 7, characterized in that, The cross-entropy loss function after updating the pseudo label is the pixel-by-pixel multiplication of the calculation result of the weighted diffusion of the non-normalized initial prediction result of the labeled sample and the high-confidence area mask of the labeled sample. 9.A semi-supervised semantic segmentation system for landslide images, characterized in that, Comprising: The data acquisition module is used for acquiring the unlabeled sample and the labeled sample of the mountain landslide image; The pseudo label mixing module with confidence less than a random number is used for obtaining the prediction result of the unlabeled sample through the segmentation model, taking the prediction result as the pseudo label, and determining the confidence of the pseudo label. When the confidence of the pseudo label is less than the obtained random number, the labeled sample image information is injected into the unlabeled sample with confidence less than the random number by participating in the process of cutting and pasting; The pseudo label mixing module with confidence not less than a random number comprises a mountain landslide semantic segmentation module and a mountain landslide region propagation module, wherein: The mountain landslide semantic segmentation module is used for constructing a semantic similarity graph of the labeled sample, performing weighted diffusion on the prediction result of the labeled sample by determining a semantic similarity perception graph, and determining a high-confidence area mask; The mountain landslide region propagation module is used for generating a shape mask corresponding to the position of each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph, determining an overlap rate through the shape mask and the high-confidence area mask, calculating the dominant class in the shape region through the semantic similarity perception graph when the overlap rate is greater than an overlap rate threshold, and injecting the shape features of each anchor point into the unlabeled sample with confidence greater than or equal to the random number when the proportion of the dominant class exceeds a dominant class threshold; The segmentation module is used for generating mixed samples by updating the unlabeled samples, taking the mixed samples as a training set, and training the segmentation model.

Images