A semi-supervised landslide image semantic segmentation method and system

Abstract

The application discloses a kind of semi-supervised mountain landslide image semantic segmentation method and system, belong to deep learning technical field, when the confidence of pseudo label is less than confidence threshold, with marked sample image information is injected by participating in cutting and pasting process in unmarked sample whose confidence is less than confidence threshold;The semantic similarity graph of marked sample is constructed, the prediction result of marked sample is weighted diffusion by determining semantic similarity perception graph, determine high confidence area mask;And by randomly sampling multiple anchor points from semantic similarity graph, the shape mask of the position corresponding to each anchor point is generated, the shape feature of each anchor point is injected in unmarked sample whose confidence is greater than or equal to confidence threshold;By updating unmarked sample to generate mixed sample, as training set, the segmentation model is trained.The method can effectively improve the accuracy of semi-supervised mountain landslide image semantic segmentation result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, more particularly to a semi-supervised landslide image semantic segmentation method and system. BACKGROUND

[0002] In early landslide remote sensing interpretation research, traditional machine learning methods such as support vector machine and random forest dominate, and deep neural network technology has not been used. In addition to landslide area identification, disaster susceptibility assessment is also a research focus, aiming to establish a disaster probability model to divide risk levels by analyzing the spatial distribution of historical landslides, topographic features and related environmental triggering factors. However, traditional methods require manual feature design and use of high-precision labeled data, which has the problems of low efficiency and strong subjectivity, especially in the application of large-scale complex terrain areas.

[0003] In recent years, deep learning semantic segmentation technology has gradually replaced traditional methods in landslide remote sensing detection, showing higher accuracy and automation.

[0004] However, the generalization ability of such models is insufficient due to large geographical environmental differences, limited size of existing landslide data sets, and non-uniform labeling standards, resulting in insufficient cross-regional adaptability of the model during identification and detection, and easy misjudgment of landslide boundaries in plateau and canyon areas with less training data, affecting the robustness and horizontal comparison research of the model, thereby affecting the semantic segmentation accuracy of landslides. SUMMARY

[0005] To solve the above problems, the present application provides a semi-supervised landslide image semantic segmentation method and system, which compares the confidence of the obtained random number and the pseudo label. When the confidence of the pseudo label is less than the random number, the real label information of the labeled sample is injected into the unlabeled sample with confidence less than the random number through the process of cutting and pasting. When the confidence of the pseudo label is greater than or equal to the random number, the semantic similarity graph of the labeled sample is constructed, and the semantic similarity perception and region propagation of the landslide shape are realized according to the semantic information and geometric feature information of the semantic similarity graph. The shape features of each anchor point are injected into the unlabeled sample with confidence greater than or equal to the random number, and the unlabeled sample is updated to generate a mixed sample. The mixed sample is used as a training set to train the segmentation model, effectively suppressing the error propagation of the misjudged area, thereby improving the accuracy of the semi-supervised landslide image semantic segmentation result.

[0006] To solve the above technical problems, the present application discloses a semi-supervised landslide image semantic segmentation method, comprising the following steps:

[0007] Obtain unlabeled samples and labeled samples of landslide images;

[0008] The prediction results of unlabeled samples are obtained by segmentation model, and the prediction results are used as pseudo labels. The confidence of the pseudo labels is determined. When the confidence of the pseudo label is less than the obtained random number, the image information of labeled samples is injected into the unlabeled samples with a confidence of less than the random number by participating in the cutting and pasting process.

[0009] A semantic similarity map of labeled samples is constructed. The prediction results of labeled samples are weighted and diffused by the semantic similarity map to determine the high-confidence region mask. Multiple anchor points are randomly sampled from the semantic similarity map to generate shape masks for each anchor point. The overlap rate is determined by the shape mask and the high-confidence region mask. When the overlap rate is greater than the overlap rate threshold, the dominant category in the shape region is calculated by the semantic similarity map. When the proportion of the dominant category exceeds the dominant category threshold, the shape features of each anchor point are injected into unlabeled samples with a confidence greater than or equal to a random number.

[0010] By updating unlabeled samples to generate mixed samples, the mixed samples are used as the training set to train the segmentation model.

[0011] Preferably, the confidence level of the pseudo-label is determined by calculating a confidence score for the pseudo-label, and the formula for the confidence score of the pseudo-label is:

[0012] ;

[0013] in, H The height of the feature map, W The width of the feature map. For the first i The prediction results generated by the segmentation model in the feature map of Zhang's unlabeled samples. J Indicates the first i The first unlabeled sample J Pixels, C represents the classification category in the landslide segmentation task.

[0014] Preferably, the injection of pseudo-labels for unlabeled samples with a confidence level less than a confidence threshold specifically includes:

[0015] The prediction results of unlabeled samples obtained by the segmentation model are used as pseudo-labels, and the confidence score of the pseudo-labels is determined.

[0016] Get a random number between 0 and 1. When the confidence score of the pseudo-label of the unlabeled sample is less than the random number, paste the square cutout outline of the labeled sample image onto the unlabeled sample image to generate a preliminary mixed sample.

[0017] By randomly selecting a square cutout profile from the preliminary mixed sample, the square cutout profile of the preliminary mixed sample is pasted onto the unmarked sample.

[0018] Preferably, the determining the semantic similarity perception map specifically comprises:

[0019] A feature extraction network for obtaining the semantic similarity perception map is constructed, and the feature extraction network comprises an encoder and two convolutional layers;

[0020] Multi-stage feature interaction is performed through the encoder to construct the semantic similarity map of the labeled sample:

[0021] ;

[0022] In the formula, is the feature map obtained after the first stage of the encoder; is the feature map obtained after the second stage of the encoder; n is the feature map obtained after the second stage of the encoder; D is the number of channels; HW is the number of pixels of the feature map; Conv indicates a 1x1 convolution;

[0023] After the encoder extracts the features, the semantic similarity map of the labeled sample is projected through the two convolutional layers, the projected features are multiplied by a matrix and normalized to obtain a semantic similarity perception map matrix:

[0024] ;

[0025] In the formula, T is a transposition operation of a matrix; is the semantic similarity perception map matrix; each element C in the matrix c quantifies the degree of semantic association between the position i and j . i j Preferably, the determining the high-confidence region mask specifically comprises:

[0026] The unlabeled initial prediction result of the labeled sample is obtained through a segmentation model

[0027] ; x i ; ;

[0028] The unlabeled initial prediction result of the labeled sample is multiplied by the semantic similarity perception map matrix. The unlabeled initial prediction result of the labeled sample is weighted and diffused through the semantic similarity perception map matrix:

[0029] ;

[0030] In the formula, For the bilinear interpolation operation of adjusting the feature size, the interpolation operation is equivalent to neighborhood aggregation of the prediction result with similarity weight;

[0031] The maximum value in the non-normalized initial prediction result of the labeled sample is compared with the confidence threshold, and when the maximum value in the non-normalized initial prediction result of the labeled sample is greater than the confidence threshold, the high-confidence region mask of the labeled sample is generated as:

[0032] ;

[0033] In the formula, The preset confidence threshold is: M The high-confidence region mask is: The indicator function is.

[0034] Preferably, the determination of the overlap rate through the shape mask and the high-confidence region mask comprises:

[0035] A binary shape mask corresponding to the position of each anchor point is generated by randomly sampling a plurality of anchor points from the semantic similarity graph; the binary shape mask is subjected to an intersection and union operation with the high-confidence region to obtain the overlap rate.

[0036] Preferably, it further comprises constructing a loss function integrating the labeled sample and the unlabeled sample, specifically comprising:

[0037] The prediction result of the unlabeled sample is subjected to a cross-entropy loss with the pseudo label to obtain a semi-supervised unlabeled sample loss function;

[0038] The non-normalized initial prediction result of the labeled sample is subjected to a cross-entropy loss with the real label to obtain a semi-supervised labeled sample loss function;

[0039] The cross-entropy loss after updating the pseudo label, the semi-supervised unlabeled sample loss function and the semi-supervised labeled sample loss function are weighted and summed to obtain a total loss function integrating the labeled sample and the unlabeled sample.

[0040] Preferably, the cross-entropy loss function after updating the pseudo label is the calculation result of weighted diffusion of the non-normalized initial prediction result of the labeled sample multiplied by the high-confidence region mask of the labeled sample pixel by pixel.

[0041] Preferably, it further comprises a semi-supervised landslide image semantic segmentation system, comprising:

[0042] A data acquisition module for acquiring unlabeled samples and labeled samples of landslide images;

[0043] The pseudo-label mixing module with a confidence less than a random number is used for obtaining a prediction result of an unlabeled sample by a segmentation model, taking the prediction result as a pseudo-label, and determining the confidence of the pseudo-label, wherein when the confidence of the pseudo-label is less than a random number, the labeled sample image information is injected into the unlabeled sample with the confidence less than the random number by participating in the process of cutting and pasting.

[0044] The pseudo-label mixing module with a confidence not less than a random number comprises a landslide semantic segmentation module and a landslide region propagation module, wherein:

[0045] The landslide semantic segmentation module is used for constructing a semantic similarity graph of a labeled sample, performing weighted diffusion on a prediction result of the labeled sample by determining a semantic similarity perception graph, and determining a high-confidence region mask.

[0046] The landslide region propagation module is used for generating a shape mask of a position corresponding to each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph, determining an overlap rate by the shape mask and the high-confidence region mask, calculating a dominant class in a shape region by the semantic similarity perception graph when the overlap rate is greater than an overlap rate threshold, and injecting shape features of each anchor point into the unlabeled sample with the confidence greater than or equal to the random number when the dominant class accounts for more than a dominant class threshold.

[0047] The segmentation module is used for generating a mixed sample by updating the unlabeled sample, taking the mixed sample as a training set, and training the segmentation model.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] The semi-supervised landslide image semantic segmentation method provided by the present application injects high-confidence labeled sample image information into the unlabeled sample with the confidence less than the random number when the pseudo-label confidence of the unlabeled sample is less than the random number, and enhances the pseudo-label confidence through a mixed enhancement process. When the pseudo-label confidence of the unlabeled sample is greater than or equal to the random number, the semantic similarity graph of the labeled sample is constructed, the labeled information provides a clear training target for the model, the model can more fully utilize the semantic information and geometric feature information of the labeled sample, the semantic association and shape implicit information in the similarity perception graph are utilized to enhance the perception ability of the landslide region, and the unlabeled sample is updated and the high-confidence region is expanded. The confidence of the pseudo-label is taken as the probability of avoiding triggering of data enhancement, that is, the higher the confidence, the less likely the enhancement is triggered, the deviation caused by the strong dependence on the pseudo-label is reduced, the mixed sample generated can effectively inhibit the error propagation caused by the pseudo-label noise in the early training, the uncertainty in the training process is reduced, and thus the segmentation performance of the model under a small amount of labeled data is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 A semi-supervised landslide image semantic segmentation method flowchart is provided for the present application.

[0051] Figure 2 A pixel semantic propagation process diagram is provided for the embodiment of the present application.

[0052] Figure 3 A mixed enhancement flowchart is provided for the embodiment of the present application.

[0053] Figure 4 A landslide area shape perception diagram is provided for the embodiment of the present application.

[0054] Figure 5 The segmentation comparison results of the landslide area of a certain province data set identified by different methods in the embodiment of the present application are respectively in the mountain forest.

[0055] Figure 6 The segmentation comparison results of the landslide area of a certain province data set identified by different methods in the embodiment of the present application are respectively in the gobi.

[0056] Figure 7 The segmentation comparison results of the landslide area of a certain province data set identified by different methods in the embodiment of the present application are respectively in the farmland scene.

[0057] Figure 8 The evolution process of the semantic segmentation result with the growth of the labeled data is provided for the embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Figures 1-8 The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application.

[0059] EMBODIMENT

[0060] As shown in the figure, a semi-supervised landslide image semantic segmentation method is provided for the present application, which includes the following steps: Figure 1

[0061] S1: Obtain unmarked samples and marked samples of landslide images;

[0062] Obtain the prediction result of the unmarked sample through the segmentation model, use the prediction result as a pseudo label, and determine the confidence of the pseudo label. When the confidence of the pseudo label is less than the obtained random number, inject the unmarked sample with a confidence less than the random number into the marked sample image information by participating in the clipping and pasting process;

[0063] ​S2: Construct a semantic similarity map of labeled samples. Weighted diffusion is applied to the prediction results of labeled samples using the semantic similarity map to determine a high-confidence region mask. Multiple anchor points are randomly sampled from the semantic similarity map to generate shape masks for each anchor point. The overlap rate is determined between the shape mask and the high-confidence region mask. When the overlap rate exceeds a threshold, the dominant category within the shape region is calculated using the semantic similarity map. When the proportion of the dominant category exceeds a dominant category threshold, the shape features of each anchor point are injected into unlabeled samples with a confidence level greater than or equal to a random number.

[0064] S3: Generate mixed samples by updating unlabeled samples, and use the mixed samples as the training set to train the segmentation model.

[0065] Specifically, an adaptive anchored hybrid augmentation strategy is used to perform hybrid augmentation on unlabeled sample data. This adaptive anchored hybrid augmentation strategy merges the weakly augmented and strongly augmented data streams that coexist in the unlabeled sample data in the previous semi-supervised learning into a single data stream, and determines the augmentation scale for each unlabeled sample instance according to the adaptive anchored hybrid augmentation strategy.

[0066] In step S2, the adaptive anchoring hybrid enhancement strategy uses the labels of labeled sample images in the cut-and-paste process to enhance the credibility of pseudo-labels, while also making fuller use of the semantic and geometric features of the labeled samples. Anchoring refers to injecting labeled data as a reliable knowledge anchor into the hybrid process when the quality of the pseudo-labels of the unlabeled data is insufficient.

[0067] like Figure 3 The diagram shows a schematic of the adaptive anchoring hybrid enhancement strategy. u 1, u 2, u 3 refers to the number of the unlabeled image that needs enhancement. x 1, x 2, x 3 refers to the number of the labeled image that participated in the enhancement. , , for u 1, u 2, u The corresponding pseudo-label is determined by calculating the confidence score of the pseudo-label. , , for u 1, u 2, u 3 corresponds to a random number in the range [0, 1].

[0068] Meanwhile, considering that the high-reliability marked information is mixed with the pseudo label of the unmarked data, the reliability and the information amount are improved, and the proportion of the unmarked data is reduced, and even some parts of some unmarked sample data may be covered and not participate in training. Therefore, the application specially calculates the confidence score formula of the pseudo label to determine whether the anchor marked sample is needed through self-adaptive detection, and the confidence score formula of the pseudo label is:

[0069] ;

[0070] wherein, H is the height of the feature map, W is the width of the feature map, is the first i feature map of the unmarked sample, J represents the first i pixel in the first J unmarked sample, and C is the classification category in the landslide segmentation task, and the application takes the value N is 2.

[0071] Particularly, in the training stage, the segmentation model is a conventional student model; and in the prediction stage, the segmentation model is a conventional teacher model.

[0072] The process of the self-adaptive anchor mixed enhancement strategy specifically includes:

[0073] First, each unmarked sample image is numbered u m , and another unmarked sample image u n and a marked sample image x n are randomly specified to form a pair.

[0074] The pseudo label confidence score of the current model for the first i unmarked sample is calculated by the confidence score formula of the pseudo label , and the pseudo label confidence score is estimated by logarithmic processing and weighted average value of the prediction value of the student model for the first i unmarked sample.

[0075] Obviously, the pseudo label confidence score is for each unmarked sample, and is closely related to the generalization ability of the current student model. The value of the pseudo label confidence score is taken as the probability of avoiding triggering of data enhancement, that is, the higher the confidence, the less likely to trigger such enhancement.

[0076] A random number between 0 and 1 is obtained by comparison The size of the score of the confidence of the pseudo label determines whether to perform mixed enhancement. It is assumed that the unlabeled sample image u m If enhancement is needed, then:

[0077] The labeled sample image x n is cut out M n and pasted onto the unlabeled sample image u n to obtain a mixed sample so that the unlabeled sample used for cutting and pasting contains real label information of the labeled sample with high credibility, and a preliminary mixed sample is obtained ;

[0078] The square cutting contour of the preliminary mixed sample is randomly selected M n , and the square cutting contour of the preliminary mixed sample is pasted onto the unlabeled sample to obtain a mixed sample M n when the confidence of the pseudo label is less than the obtained random number. .

[0079] The generation formula of the mixed sample is as follows:

[0080] ;

[0081] ;

[0082] wherein, M n the labeled sample image x n is sent to the student model for training through the square cutting contour randomly selected.

[0083] In step S3, when the confidence of the pseudo label is greater than or equal to the obtained random number, a semantic similarity graph for capturing the semantic association and shape implicit information of the landslide area is constructed, specifically:

[0084] The semantic similarity graph is constructed by using the multi-stage feature interaction of the encoder, and the semantic association and shape implicit information of the landslide area are captured according to the semantic similarity graph and the landslide area shape perception graph obtained by sampling from the graph.

[0085] ​Using a semantic similarity-aware map, a pixel-level semantic propagation strategy is constructed, based on the implicit landslide shape information contained in the sampling portion of the similarity-aware map; a landslide region propagation strategy is also constructed to facilitate the use of the high-level semantic information and hidden shape information contained in the semantic similarity-aware map.

[0086] like Figure 3 The diagram shows the calculation process of the semantic similarity perception map and the landslide area shape perception map. This invention constructs a feature extraction network for obtaining the semantic similarity perception map, which includes an encoder and two convolutional layers.

[0087] A semantic similarity graph of labeled samples is constructed through multi-stage feature interaction using an encoder.

[0088] ;

[0089] in, This is the feature map obtained in the first stage of the encoder. This is the feature map obtained after the fourth stage of the encoder. D For the number of channels, HW The number of pixels in the feature map. refer to The convolution.

[0090] After the encoder extracts features, it projects the semantic similarity map of labeled samples using two convolutional layers. The projected features are then multiplied by matrices and normalized to obtain the semantic similarity-aware map matrix.

[0091] ;

[0092] in, T This is the transpose operation of a matrix; This is a semantic similarity-aware graph matrix. C Each element in c ( i , j ) Quantified location i and j The degree of semantic connection between them i and j express C Elements in the semantic similarity-aware graph matrix c The unique advantage of the horizontal and vertical coordinates lies in the fusion of multi-scale information through feature product, which effectively integrates deep semantic information with shallow geometric features, capturing detailed textures while maintaining the main shape of the landslide.

[0093] The shape information of the calculated graph is hidden, and the spatial correlation network is implicitly constructed by matrix operation, so that the landslide shape is modeled. The matrix multiplication operation of the semantic similarity graph essentially constructs the topological correlation network between pixels, so that the positions within the landslide region have higher interconnection strength. This feature will help the model maintain the geometric continuity of the prediction results in the subsequent process, and strengthen the model's natural ability to maintain the geometric integrity of the prediction results through feature interaction.

[0094] The pixel-level semantic propagation strategy is used to propagate the semantics of the landslide region through the semantic similarity perception graph. The specific method of this pixel-level semantic propagation strategy is to weight the non-normalized prediction through the similarity graph, thereby enhancing the model's perception ability of the landslide region. The implementation is very simple. The teacher model is used to obtain the non-normalized initial prediction result of the labeled sample x i ;The non-normalized initial prediction result of the labeled sample i is multiplied by the semantic similarity perception graph matrix to obtain: x

[0095] ;

[0096] wherein, is a bilinear interpolation operation for adjusting the feature size. The interpolation operation is equivalent to using the similarity weight to perform neighborhood aggregation on the prediction result, so that the predictions of similar pixels tend to be consistent.

[0097] After the weighted diffusion of the semantic similarity perception graph matrix on the prediction result is completed, the high-confidence region mask and the updated cross-entropy loss function of the pseudo-label are obtained as follows:

[0098] ;

[0099] ;

[0100] wherein, is a confidence threshold set by human, M is a high-confidence region mask, is a pixel-by-pixel multiplication, is a pseudo-label of an unlabeled sample i , N is the number of feature image pixels, L CE is a cross-entropy loss function, is an indicator function.

[0101] ​The cross-entropy loss makes the propagation process focus on optimizing reliable areas. In this way, the pixel-level semantic propagation module can effectively propagate the semantic information contained in the similarity graph to the entire image, thereby improving the model's perception ability of the landslide area.

[0102] The landslide area propagation strategy is constructed, which can realize the perception and propagation of the landslide shape through the geometric features of the semantic similarity graph.

[0103] Semantic similarity perception graph C Each row c represents the similarity between a single feature vector and all vectors in the entire feature map, which implicitly encapsulates shape information.

[0104] The present application proposes a landslide area propagation strategy for enhancing pseudo-labels through these shape information. The core idea of the landslide area propagation strategy is to use the landslide shape information implicitly contained in the correlation graph to enhance the geometric integrity of the pseudo-labels, which includes three steps:

[0105] Shape information extraction. By randomly sampling C an anchor point in the semantic similarity perception graph m , a binary shape mask corresponding to the position is generated:

[0106] ;

[0107] Wherein, C 𝑘 is the i-th row of the correlation graph, and the operation converts the continuous similarity map into a discrete shape mask. k

[0108] Confidence area fusion. Perform intersection operation on the shape mask and the high confidence area to obtain the overlap rate:

[0109] ;

[0110] When the overlap rate r is greater than the preset overlap rate threshold , calculate the dominant class in the shape area:

[0111] ;

[0112] Wherein, c is any row in the semantic similarity perception graph C .

[0113] Label enhancement propagation. When the dominant class proportion exceeds the preset dominant class threshold , update the shape area to the pseudo-label:

[0114] ​ ;

[0115] Meanwhile, the high-confidence region is expanded:

[0116] ;

[0117] By expanding the high-confidence region, the high-confidence region mask of the labeled sample is updated.

[0118] The landslide region propagation strategy generates a smooth target contour through the local continuity of the correlation map, suppresses the error propagation of the false detection region through the statistical effectiveness of the dominant class, reduces the computational complexity through random sampling, and reduces to , avoids the high overhead of full map calculation.

[0119] The design of the landslide region propagation strategy enables the model to fully utilize the shape information in the landslide segmentation task and enhances the perception ability of the landslide region.

[0120] The application also includes constructing a total loss function integrating labeled samples and unlabeled samples according to a batch-based stochastic gradient descent method, taking the mixed sample enhanced by adaptive anchor mixing as a training set, and training the segmentation model through the total loss function.

[0121] The loss function integrating labeled samples and unlabeled samples is constructed, and the specific process specifically includes:

[0122] The prediction result of the unlabeled sample is obtained by the segmentation model P ( U i ), and the prediction result of the unlabeled sample P ( U i ) is cross-entropy loss with the pseudo label , to obtain the semi-supervised unlabeled sample loss function:

[0123] ;

[0124] The prediction result of the labeled sample is obtained by the segmentation model P ( L i ), and the prediction result of the labeled sample P ( L i ) is cross-entropy loss with the real label y i , to obtain the semi-supervised labeled sample loss function:

[0125] ;

[0126] The cross-entropy loss function after updating the pseudo label is:

[0127]

[0128] The total target loss function is obtained by weighted sum of the cross-entropy loss function after updating the pseudo label, the semi-supervised unlabeled sample loss function and the semi-supervised labeled sample loss function, that is, the total loss function of the integrated labeled sample and unlabeled sample:

[0129]

[0130] wherein, is the total target loss function; is the semi-supervised unlabeled sample loss function; is the cross-entropy loss of the prediction result of the labeled sample and the true value label; is the cross-entropy loss of the pseudo label after updating by the landslide region propagation strategy; , and are hyperparameters, respectively controlling the weights of the three loss functions.

[0131] The total target loss function is composed of three loss functions, which represents the traditional semi-supervised unlabeled sample loss function, the cross-entropy loss of the prediction result of the unlabeled sample and the pseudo label; which represents the cross-entropy loss of the prediction result of the labeled sample and the true value label, and which is the cross-entropy loss of the updated pseudo label. In the practical examples, the , and are respectively set to 0.25, 0.25 and 0.5.

[0132] By updating the high-confidence region mask of the unlabeled sample and the pseudo label, the self-adaptive enhanced mixed sample is generated. The generated self-adaptive enhanced mixed sample is used as a training set to train the segmentation model (student model).

[0133] Correspondingly, the application also provides a semi-supervised landslide image semantic segmentation system, comprising:

[0134] A data acquisition module is configured to acquire unlabeled samples and labeled samples of the landslide image.

[0135] ​​The pseudo-label mixing module with confidence less than the random number is used for obtaining a prediction result of an unlabeled sample through a segmentation model, taking the prediction result as a pseudo-label, and determining the confidence of the pseudo-label, when the confidence of the pseudo-label is less than a random number, injecting the unlabeled sample with confidence less than the random number through the participation of the clipping and pasting process by using the labeled sample image information;

[0136] The pseudo-label mixing module with confidence not less than the random number comprises a landslide semantic segmentation module and a landslide region propagation module, and wherein:

[0137] The landslide semantic segmentation module is used for constructing a semantic similarity graph of the labeled sample, performing weighted diffusion on the prediction result of the labeled sample by determining a semantic similarity perception graph matrix, and determining a high-confidence region mask.

[0138] The landslide region propagation module is used for generating a shape mask corresponding to the position of each anchor point by randomly sampling a plurality of anchor points from the semantic similarity graph, determining an overlap rate through the shape mask and the high-confidence region mask, when the overlap rate is greater than an overlap rate threshold, calculating a dominant class in the shape region through the semantic similarity perception graph, and when the proportion of the dominant class exceeds a dominant class threshold, injecting the shape features of each anchor point into the unlabeled sample with confidence greater than or equal to the random number.

[0139] The segmentation module is used for generating a mixed sample by updating the unlabeled sample, taking the mixed sample as a training set, and training the segmentation model.

[0140] In order to solve the problem of lack of labeled data in landslide remote sensing image segmentation tasks and the urgent need for high-precision segmentation models in a small amount of labeled scenarios, the present application proposes a semi-supervised landslide image semantic segmentation method based on semantic similarity perception, which introduces an adaptive anchor mixing enhancement strategy to integrate the real label information of high-confidence labeled samples into the mixing enhancement process of unlabeled samples, effectively suppressing the error propagation caused by pseudo-label noise in early training. By constructing a semantic similarity graph, constructing a pixel-level semantic propagation strategy and a landslide region propagation strategy, and using the semantic information and implicit shape information in the similarity perception graph, the segmentation performance of the model under a small amount of labeled data is effectively improved.

[0141] Experimental analysis

[0142] The present embodiment is tested on a certain provincial landslide data set, and the experimental results show that the present application significantly improves the landslide region recognition ability of the model.

[0143] As shown in Table 1, the main index values of a small number of labeled proportions on a certain province mountain landslide face data set are compared, as shown in Table 2, the main index values of a medium number of labeled proportions on a certain province mountain landslide face data set are compared, by comparing the precision (Precision), recall (Recall), F1 index (F1-score), Dice similarity coefficient (Dice Similarity Coefficient, DSC) and average Hausdorff distance (Hausdorff Distance, HD) on the certain province mountain landslide face data set. The numerical comparison of the values of the invention and other benchmark methods on the mountain landslide data set is shown in Table 1 and Table 2. The benchmark methods compared include FixMatch, UniMatch, Mean Teacher, Cross Teaching between CNN and Transformer, Uncertainty Rectified Pyramid Consistency, Interpolation Consistency Training, Cross Pseudo Supervision and ABD.

[0144] From the quantitative results of Table 1 and Table 2, it can be seen that the main indicators of the method of the present application are significantly higher than those of other methods. The certain province loess landslide data set is collected from optical remote sensing images of multiple regions in a certain province of China, and a total of 1025 labeled samples are included. Among them, 820 training samples, accounting for about 80% of the total, 96 examples in the validation set and 109 examples in the test set are reserved, accounting for 10% respectively. As shown in Table 1 and Table 2, the labels set in this embodiment, of which 5% is called labeled proportion, 10% is called small proportion, 20%, 30%, and 50% are called medium proportion.

[0145] Table 1 Comparison of main index values of a small number of labeled proportions on a certain province mountain landslide face data set

[0146]

[0147] Table 2 Comparison of main index values of a medium number of labeled proportions on a certain province mountain landslide face data set

[0148]

[0149] Figure 5 、 Figure 6 、 Figure 7 The above benchmark methods and the semi-supervised mountain landslide image semantic segmentation method based on semantic similarity perception proposed by the present application are respectively shown in the segmentation comparison results of the mountain landslide region of the certain province data set in the mountain forest, gobi and farmland scenes.

[0150] By Figure 5 、 Figure 6 、 Figure 7 As can be seen, the semi-supervised landslide image semantic segmentation method proposed by the present application has very good segmentation effect on landslides of different scales in various geographical environments, with clear edges and accurate recognition. From the segmentation results, it can be observed that the present application method has the best control over the shape of the landslide area. The other comparative methods have overlarge segmentation areas due to misjudgment of the landslide, and the adhesion phenomenon is also serious. The present application method is closest to the label in shape.

[0151] The visualization comparison results show that the present application method exhibits robust segmentation capability on the certain province dataset, and the comprehensive performance is better than that of the existing comparative methods. Under complex environmental conditions, the model can still maintain good segmentation effect, and has strong generalization ability and anti-interference ability.

[0152] The visualization comparison results of the above Figures 5-7 show that the present application method exhibits robust segmentation capability on the certain province dataset, and the comprehensive performance is better than that of the existing comparative methods. Under complex environmental conditions, the model can still maintain good segmentation effect, and has strong generalization ability and anti-interference ability.

[0153] As shown in Figure 8 , the evolution process of the segmentation result with the growth of the labeled data is shown. The first column is the original remote sensing image, the second column is the true value result labeled by artificial labeling, and the subsequent five columns correspond to the prediction effect when the proportion of labeled data is 5%, 10%, 20%, 30% and 50% respectively.

[0154] From the observation of the first row in Figure 8 , it can be known that the present application method has extremely strong segmentation capability with a small amount of labeling, and from 5% to 50%, the segmentation result is highly consistent with the true value and can remain stable.

[0155] From the observation of the second row in Figure 8 , it can be known that when the background around the landslide is an optical signal similar to the label, the shape consistent with the label can still be segmented at 5%.

[0156] From the observation of the third row in Figure 8 , it can be known that the present application method can also accurately segment the landslide in a background with a small proportion of 5% and similar color.

[0157] From the observation of the fourth row in Figure 8 , it can be known that the present application method is not disturbed by the road. It is worth noting that in the extreme case where the proportion of labeled sample data is only 5%, the model can already accurately capture the outline of the landslide, but a larger proportion of labeled sample data is needed to better process it. When the proportion of labeled sample data is 20%, the output of the model is significantly improved in terms of fineness and the ability to capture the outline of the landslide.

[0158] When the labeled sample data is greater than 20%, the model has a relatively accurate segmentation result for landslides under all complex topographies and environments. When the labeling ratio reaches 50%, the segmentation result of the method of the present application is extremely close to the true value label. It can be seen that the ability of the method of the present application to understand the global context of landslides is significantly enhanced as the proportion of labeled data increases.

[0159] By analyzing the semantic segmentation results of different methods provided in the present embodiment, it is shown that the method proposed in the present application achieves excellent segmentation results on the loess landslide data set of a certain province, and is superior to all other comparative models in key indicators. The ablation experiment also proves the effectiveness of each module, and the visualization effect diagram shows good landslide feature extraction capability and complex scene noise resistance capability, verifying the generalization capability and robustness of the method in various topography scenes.

[0160] The above description is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art, according to the technical solution and the inventive concept of the present application, can make equivalent replacement or change within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

[0161] In addition, unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art. All documents mentioned in the specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of conflict with any incorporated document, the content of the specification shall prevail.

Claims

1. A semi-supervised semantic segmentation method for landslide images, characterized in that, Includes the following steps: Obtain unlabeled and labeled samples of landslide images; The prediction results of unlabeled samples are obtained by segmentation model, and the prediction results are used as pseudo labels. The confidence of the pseudo labels is determined. When the confidence of the pseudo label is less than the obtained random number, the image information of labeled samples is injected into the unlabeled samples with a confidence of less than the random number by participating in the cutting and pasting process. Construct a semantic similarity map of labeled samples, and use the semantic similarity perception map to perform weighted diffusion on the prediction results of labeled samples to determine the high-confidence region mask; The shape mask for each anchor point is generated by randomly sampling multiple anchor points from the semantic similarity map. The overlap rate is determined by the shape mask and the high confidence region mask. When the overlap rate is greater than the overlap rate threshold, the dominant category in the shape region is calculated by the semantic similarity perception map. When the proportion of the dominant category exceeds the dominant category threshold, the shape features of each anchor point are injected into unlabeled samples with a confidence level greater than or equal to a random number. By updating unlabeled samples to generate mixed samples, the mixed samples are used as the training set to train the segmentation model; The confidence level of the pseudo-label is determined by calculating a confidence score for the pseudo-label, and the formula for the confidence score of the pseudo-label is as follows: ； in, H The height of the feature map, W The width of the feature map. For the first i The prediction results generated by the segmentation model in the feature map of Zhang's unlabeled samples. J Indicates the first i The first unlabeled sample J 1 pixel, C represents the classification category in the landslide segmentation task; The determination of the semantic similarity-aware map specifically includes: A feature extraction network for obtaining semantic similarity-aware maps is constructed, which includes an encoder and two convolutional layers; A semantic similarity graph of labeled samples is constructed through multi-stage feature interaction using an encoder. ； In the formula, This is the feature map obtained after the first stage of the encoder; For encoder number n Feature maps obtained after the phase; D Number of channels; HW The number of pixels in the feature map; Conv Refers to 1×1 convolution; After the encoder extracts features, it projects the semantic similarity map of labeled samples using two convolutional layers. The projected features are then multiplied by matrices and normalized to obtain the semantic similarity-aware map matrix. ； In the formula, T represents the matrix transpose operation; This is a semantic similarity-aware graph matrix; this matrix C Each element in c ( i , j ) Quantified location i and j The degree of semantic connection between them.

2. The semi-supervised semantic segmentation method for landslide images according to claim 1, characterized in that, The injected pseudo-labels for unlabeled samples with a confidence level less than the confidence threshold specifically include: The prediction results of unlabeled samples obtained by the segmentation model are used as pseudo-labels, and the confidence score of the pseudo-labels is determined. Get a random number between 0 and 1. When the confidence score of the pseudo-label of the unlabeled sample is less than the random number, paste the square cutout outline of the labeled sample image onto the unlabeled sample image to generate a preliminary mixed sample. By randomly selecting a square cutout profile from the preliminary mixed sample, the square cutout profile of the preliminary mixed sample is pasted onto the unmarked sample.

3. The semi-supervised semantic segmentation method for landslide images according to claim 1, characterized in that, The determination of the high-confidence region mask specifically includes: Obtain labeled samples using a segmentation model. x i Unnormalized initial prediction results ; The unnormalized initial prediction results of labeled samples are multiplied by the semantic similarity-aware map matrix, and then the unnormalized initial prediction results of labeled samples are weighted and diffused using the semantic similarity-aware map matrix: ； In the formula, The bilinear interpolation operation, which adjusts the feature size, is equivalent to performing neighborhood aggregation on the prediction results using similarity weights. The maximum value in the unnormalized initial prediction results of the labeled samples is compared with a confidence threshold. When the maximum value in the unnormalized initial prediction results of the labeled samples is greater than the confidence threshold, a high-confidence region mask for the labeled samples is generated as follows: ； In the formula, The preset confidence threshold; M This is a high-confidence region mask. This is an indicator function.

4. The semi-supervised semantic segmentation method for landslide images according to claim 3, characterized in that, The method of determining the overlap rate by using a shape mask and a high-confidence region mask includes: By randomly sampling multiple anchor points from the semantic similarity map, a binary shape mask is generated for each anchor point; the binary shape mask is then intersected and merged with the high-confidence region to obtain the overlap rate.

5. The semi-supervised semantic segmentation method for landslide images according to claim 1, characterized in that, It also includes constructing a loss function that integrates labeled and unlabeled samples, specifically including: The prediction results of unlabeled samples are subjected to cross-entropy loss with pseudo-labels to obtain the semi-supervised unlabeled sample loss function. The unnormalized initial prediction results of labeled samples are compared with the true labels to obtain the semi-supervised labeled sample loss function by performing cross-entropy loss. The weighted sum of the cross-entropy loss after pseudo-label update, the semi-supervised unlabeled sample loss function, and the semi-supervised labeled sample loss function is used to obtain the total loss function that combines labeled and unlabeled samples.

6. The semi-supervised semantic segmentation method for landslide images according to claim 5, characterized in that, The cross-entropy loss function after the pseudo-label update is calculated by multiplying the weighted diffusion result of the unnormalized initial prediction result of the labeled sample with the high-confidence region mask of the labeled sample pixel by pixel.

7. A semi-supervised semantic segmentation system for landslide images, characterized in that, include: The data acquisition module is used to acquire both unlabeled and labeled samples of landslide images; The pseudo-label mixing module with confidence less than the random number is used to obtain the prediction results of unlabeled samples through the segmentation model, use the prediction results as pseudo-labels, and determine the confidence of the pseudo-labels. When the confidence of the pseudo-labels is less than the obtained random number, the labeled sample image information is injected into the unlabeled samples with confidence less than the random number by participating in the cutting and pasting process. A pseudo-label hybrid module with a confidence level not less than a random number, comprising a landslide semantic segmentation module and a landslide regional propagation module, wherein: The semantic segmentation module for landslides is used to construct a semantic similarity map of labeled samples. By determining the semantic similarity perception map, the prediction results of labeled samples are weighted and diffused to determine the high-confidence region mask. The landslide area propagation module is used to generate shape masks for each anchor point by randomly sampling multiple anchor points from the semantic similarity map. The overlap rate is determined by the shape mask and the high-confidence area mask. When the overlap rate is greater than the overlap rate threshold, the dominant category in the shape area is calculated by the semantic similarity perception map. When the proportion of the dominant category exceeds the dominant category threshold, the shape features of each anchor point are injected into unlabeled samples with a confidence greater than or equal to a random number. The segmentation module is used to generate mixed samples by updating unlabeled samples, and uses the mixed samples as the training set to train the segmentation model. The confidence level of the pseudo-label is determined by calculating a confidence score for the pseudo-label, and the formula for the confidence score of the pseudo-label is as follows: ； in, H The height of the feature map, W The width of the feature map. For the first i The prediction results generated by the segmentation model in the feature map of Zhang's unlabeled samples. J Indicates the first i The first unlabeled sample J 1 pixel, C represents the classification category in the landslide segmentation task; The determination of the semantic similarity-aware map specifically includes: A feature extraction network for obtaining semantic similarity-aware maps is constructed, which includes an encoder and two convolutional layers; A semantic similarity graph of labeled samples is constructed through multi-stage feature interaction using an encoder. ； In the formula, This is the feature map obtained after the first stage of the encoder; For encoder number n Feature maps obtained after the phase; D Number of channels; HW The number of pixels in the feature map; Conv Refers to 1×1 convolution; After the encoder extracts features, it projects the semantic similarity map of labeled samples using two convolutional layers. The projected features are then multiplied by matrices and normalized to obtain the semantic similarity-aware map matrix. ； In the formula, T represents the matrix transpose operation; This is a semantic similarity-aware graph matrix; this matrix C Each element in c ( i , j ) Quantified location i and j The degree of semantic connection between them.

Images