Ultra-high resolution remote sensing image semi-supervised change detection method based on improvement of pseudo label boundary quality

Abstract

The invention discloses an ultra-high resolution remote sensing image semi-supervised change detection method based on pseudo label boundary quality improvement. The change detection aims to explore a change region of an image at a later moment compared with an image at a previous moment in a dual-time image; according to the method, semi-supervised learning guidance is enhanced through a high-quality pseudo tag, and the method comprises the following steps: 1, preparing a semi-supervised learning training data set; 2, building a teacher model and a student model; 3, starting training, reasoning data of the training data set by the student model, and generating corresponding rough and false labels by utilizing the teacher model; 4, refining the rough and false labels by using a boundary optimization method, and calculating the consistency loss of the labels and the reasoning result loss of the student model; 5, calculating the total loss, and updating the parameters of the student model and the teacher model; 6, finishing training, and performing change detection on the remote sensing double-time image by the student model; according to the method, the semi-supervised change detection performance which is limited by rough false label boundaries and cannot be further improved is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision applications, and in particular relates to a semi-supervised change detection method for ultra-high resolution remote sensing images based on improving pseudo-label boundary quality. Background Art

[0002] Remote sensing image change detection plays a key role in earth observation and environmental monitoring. With the continuous development of remote sensing sensors and the advancement of satellite technology, more and more ultra-high resolution remote sensing images can be obtained. These data have great potential in fields such as agriculture, urban planning and natural resource management.

[0003] With the development of deep learning, mainstream change detection methods have shown remarkable performance with the help of fully supervised deep neural networks. However, accurate change detection using deep neural networks usually requires a large amount of labeled data for training. Since image annotation for change detection requires manual alignment and detailed annotations, the process of annotating change detection datasets is extremely expensive and time-consuming.

[0004] In the early research on semi-supervised change detection, the mainstream methods were virtual adversarial training and generative adversarial networks. These methods train by applying random perturbations to unlabeled images or generating corresponding adversarial samples. Subsequently, semi-supervised change detection methods based on teacher-student models have also made some progress. However, these generation-based methods lack the stability of generating reliable training data. For the teacher-student model, as the training progresses, the parameters of the teacher model and the student model tend to converge, which limits the ability of the student model to learn new knowledge. In recent years, some consistency regularization methods have been introduced into semi-supervised change detection. This method enforces consistency between strong and weak views. Some methods evaluate the quality of pseudo-labels and propose pseudo-label selection schemes. However, these methods do not fundamentally improve the quality of pseudo-labels, which is crucial for semi-supervised change detection.

[0005] Through the research and experiments on semi-supervised change detection methods, it is found that the current advanced semi-supervised change detection methods mainly follow consistency regularization. These methods can achieve relatively complete predictions in the central area of ​​the pseudo-label contour, but it is still difficult to align with the actual target in the boundary part. Therefore, the present invention studies a semi-supervised change detection method based on a teacher-student model. Making full use of the boundary knowledge of unlabeled data and generating boundary-refined change detection prediction results are important for land planning and resource utilization. Summary of the invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to propose a semi-supervised change detection method for ultra-high resolution remote sensing images based on improving the quality of pseudo-label boundaries. The present invention protects the boundary knowledge of remote sensing dual-time images, stabilizes the semi-supervised learning state, and improves the semi-supervised change detection performance that cannot be further improved due to the rough pseudo-label boundaries.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A semi-supervised change detection method for ultra-high resolution remote sensing images based on improving pseudo-label boundary quality comprises the following steps:

[0009] Step A: Divide the remote sensing dual-time image dataset into a labeled dataset and an unlabeled dataset; apply weak data augmentation to the labeled dataset, and first apply weak data augmentation to the unlabeled dataset and then apply strong data augmentation and feature augmentation; splice the labeled dataset and the unlabeled dataset to obtain a semi-supervised learning training dataset;

[0010] Step B: Build the teacher model and the student model, and initialize the parameters of the teacher model and the student model;

[0011] Step C: Training starts, using the student model obtained in step B to infer the labeled weak augmented data, unlabeled strong augmented data, and unlabeled feature augmented data in the training data set generated in step A; using the teacher model obtained in step B to infer the unlabeled weak augmented data in the training data set generated in step A to generate rough pseudo labels;

[0012] Step D: Calculate the labeled supervised loss for the student model obtained in step C on the labeled weak augmented data inference results and the corresponding labels in the training data set generated in step A; Use the boundary optimization method designed for remote sensing dual-time image change detection to refine the coarse pseudo-labels obtained in step C, and first calculate the consistency loss between the refined pseudo-labels and the coarse pseudo-labels; Then calculate the semi-supervised loss for the refined pseudo-labels and the student model obtained in step C on the unlabeled strong augmented and unlabeled feature augmented data inference results;

[0013] Step E: Calculate the total loss using the labeled supervision loss, consistency loss, and semi-supervised loss obtained in step D, and update the student model parameters using stochastic gradient descent; calculate the stability of the student model and update the teacher model parameters;

[0014] Step F: After the training is completed, the student model with updated parameters obtained in step E is used to infer the remote sensing dual-time image to obtain the change detection result.

[0015] The specific steps of step A are as follows:

[0016] Step A01: Divide the remote sensing dual-time image dataset into four semi-supervised learning ratios: ①5% labeled dataset, 95% unlabeled dataset; ②10% labeled dataset, 90% unlabeled dataset; ③20% labeled dataset, 80% unlabeled dataset; ④40% labeled dataset, 60% unlabeled dataset;

[0017] Step A02: Set a weak data augmentation pool, including scaling, cropping and horizontal inversion; set a strong data augmentation pool, including color injection, blurring, cropping and splicing, and grayscale adjustment; set feature augmentation to the Dropout strategy; perform weak data augmentation on the labeled data set, and first perform weak data augmentation on the unlabeled data and then perform strong data augmentation and feature augmentation; splice the labeled data set and the unlabeled data set to obtain a semi-supervised learning training data set.

[0018] The specific steps of step B are as follows:

[0019] Step B01: First, build a change detection network, use ResNet50 and DeepLabv3+ as the encoder and decoder of the change detection network respectively; use ResNet50 to extract the features of the remote sensing dual-time image, and send the features to DeepLabv3+ for change detection:

[0020]

[0021] Where A and B are the previous and next time images of the input remote sensing dual-time image to be tested, ε(·) is the constructed change detection network encoder, (·) * The multi-dimensional features extracted by the encoder; the multi-dimensional features of the remote sensing dual-time image are subtracted and the absolute value is taken and sent to the decoder In the prediction of change detection

[0022] Step B02: Use the change detection network built in step B01 to construct a student model and a teacher model. The student model and the teacher model use the same change detection network structure, but the parameters between the student model and the teacher model are not shared; the parameters of the teacher model and the student model are initialized using random initial values.

[0023] The specific steps of step C are as follows:

[0024] Step C01: Training starts, using the student model initialized with parameters in step B to infer the labeled weak augmented data, unlabeled strong augmented data, and unlabeled feature augmented data in the semi-supervised learning training data set:

[0025]

[0026] In the formula, and They represent weak data augmentation, strong data augmentation and feature augmentation respectively; x1 and x2 are the images of the previous moment and the next moment in the input labeled remote sensing dual-time image, respectively; u1 and u2 are the images of the previous moment and the next moment in the input unlabeled remote sensing dual-time image, respectively; represents the student model, and They represent the labeled prediction, unlabeled strongly augmented prediction, and feature augmented prediction of the student model respectively;

[0027] Step C02: Use the teacher model initialized with the parameters in step B to infer the unlabeled weak augmented data in the semi-supervised learning training data set to generate rough pseudo labels:

[0028]

[0029] In the formula, represents the teacher model, Represents the generated rough pseudo label.

[0030] The specific steps of step D are as follows:

[0031] Step D01: Calculate the loss of the student model obtained in step C for the predicted results of the labeled weakly augmented data and the corresponding labels in the semi-supervised learning training data set generated in step A:

[0032]

[0033] In the formula, denotes label-supervised loss, and M represent the labeled dataset and its number respectively; CE represents the cross entropy loss, and i They respectively represent the prediction of the student model for the i-th labeled weak augmented data and its corresponding label;

[0034] Step D02: Use the boundary optimization method designed for remote sensing image change detection to refine the coarse pseudo-labels generated in step C: first, apply a 3×3 two-dimensional convolution to the coarse pseudo-label change map to extract the coordinate set of the boundary area points of the change map; first, use the convolution operation to obtain the coordinates of these boundary points, and the formula is expressed as follows:

[0035]

[0036] In the formula, x i and i are the horizontal and vertical coordinates of the boundary point; is the mask corresponding to the i-th boundary block; K is a 3×3 convolution kernel, s is the convolution step size, and p is the convolution padding; w represents the predicted width of the boundary; the operation conv is a two-dimensional convolution operation, and the operation where comes from NumPy;

[0037] The boundary points are expanded along the upper left corner and the lower right corner into 48×48 boundary blocks; these boundary blocks are distributed along the boundary, depicting the changed part and its surrounding part, and the formula is expressed as:

[0038]

[0039] In the formula, the generated is an array that records the coordinates of the upper left corner and lower right corner of the boundary block; bs represents the block size, which is set to 48; t is used to represent the previous moment or the next moment in the remote sensing dual-time image;

[0040] According to the characteristics of boundary blocks, a series of boundary block filters are designed; first, an "overlap" filter is designed to remove highly overlapping boundary blocks; overlapping boundary blocks are removed to reduce the computational burden; since the convolution operation may cause boundary blocks to appear in the change map and be located in the center of the change map, an "inside" filter is introduced to discard boundary blocks that appear in the change map; in order to ensure that the boundary partition prediction is aligned with the change detection prediction and avoid completely wrong classification, a "distribution" filter is designed to perform a mask acquisition operation on each boundary partition prediction to generate a mask corresponding to each boundary; then, the intersection-and-union ratio with the original change map is calculated at the corresponding boundary block position; boundary blocks with significant intersection-and-union ratio differences are considered unreliable and discarded. The "distribution" filter ensures that the boundary partition prediction distribution is aligned with the change detection prediction as much as possible;

[0041] Finally, the boundary segmentation prediction is integrated with the change detection prediction:

[0042]

[0043] Where σ(·) is the sigmoid function, and sigmoid is used to obtain the confidence level of the prediction; then, the change detection prediction Prediction with boundary demarcation The decision boundary is pushed to the side with higher confidence. The final pseudo-label with refined boundary is obtained by applying argmax. The consistency loss of the refined pseudo-label and the rough pseudo-label obtained in step C is calculated:

[0044]

[0045] In the formula, is the consistency loss; and N represent the unlabeled dataset and its number, respectively; represents the jth coarse pseudo label generated, and represents the pseudo-label after the jth boundary refinement; the semi-supervised loss is calculated by combining the refined pseudo-label with the unlabeled strong augmentation prediction and the feature augmentation prediction:

[0046]

[0047] In the formula, is a semi-supervised loss used to constrain the consistency between the student model's predictions and the refined pseudo-labels for strong data augmentation and feature augmentation; and They represent the j-th unlabeled strong augmented prediction and the j-th feature augmented prediction, respectively.

[0048] Preferably, the predicted width w of the boundary is set to 3.

[0049] The specific steps of step E are as follows:

[0050] Step E01: Calculate the total loss using the obtained labeled supervision loss, consistency loss, and semi-supervised loss, and update the student model parameters using the loss calculation results and stochastic gradient descent:

[0051]

[0052] In the formula, is the total loss, which is the sum of the constraint losses of each part;

[0053] Step E02: Calculate the stability of the student model and update the parameters of the teacher model; use the student model to calculate the intersection and union ratio of the output of the unlabeled strongly augmented data and the mask after the boundary refinement of the unlabeled data:

[0054]

[0055] In the formula, represents the predicted value of the student model for the jth unlabeled remote sensing dual-time image change detection, and express Mask after boundary refinement; Calculate the argmax operation and The average distance between The average distance is measured using IoU, or intersection over union; if the stability score If the model is improved, it is considered that the current student model has reached a more stable state, and its parameters are updated to the teacher model by copying the overall parameters.

[0056] The specific steps of step F are as follows:

[0057] Step F01: After the training is completed, the student model with updated parameters is used to perform change detection on the remote sensing dual-time image to be tested, and the change detection prediction result is obtained:

[0058]

[0059] In the formula, For student models, is the change detection prediction result; through the arg max operation, the binary image of change detection, i.e., the change image, is obtained

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] First, in the process of semi-supervised learning, the method of the present invention abandons the traditional pseudo-label filtering strategy based on fixed thresholds and improves the boundary quality of pseudo-labels by means of boundary refinement. High-quality pseudo-labels are of great significance for the guidance of semi-supervised learning, enabling the student model to utilize and learn better boundary information and enhance its boundary improvement for remote sensing dual-time image change detection. This makes the boundary details of the change detection results complete and sharp.

[0062] Second, the present invention can achieve the most advanced semi-supervised change detection performance by using ResNet50 and DeepLabv3+ as encoder and decoder respectively.

[0063] Third, the present invention proposes a carefully designed change graph boundary optimization method based on remote sensing image change detection. This boundary optimization can improve the performance of pseudo labels in semi-supervised learning and fully activate the boundary knowledge in unlabeled data.

[0064] Fourth, the present invention proposes a new teacher parameter update method, and the parameter update strategy of the present invention method can measure the stability of the current student model, judge its stability, and choose whether to update the teacher model parameters.

[0065] Fifth, a semi-supervised change detection method for ultra-high resolution remote sensing images based on improving the quality of pseudo-label boundaries is designed in the present invention, achieving new state-of-the-art performance on the semi-supervised setting of LEVIR-CD, WHU-CD and DSIFN-CD datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is an overall flow chart of the method of the present invention.

[0067] Figure 2 It is a flowchart of semi-supervised learning proposed by the present invention.

[0068] Figure 3 This is a boundary optimization effect diagram proposed by the present invention.

[0069] Figure 4 It is the boundary optimization flow chart proposed by the present invention.

[0070] Figure 5 This is a flow chart of change matching in boundary optimization proposed by the present invention.

[0071] Figure 6 It is a schematic diagram of overlapping and internal filtering in the boundary optimization proposed by the present invention.

[0072] Figure 7 It is a schematic diagram of distributed filtering in boundary optimization proposed in the present invention.

[0073] Figure 8 It is a schematic diagram of prediction integration in boundary optimization proposed in the present invention. DETAILED DESCRIPTION

[0074] The specific details of each step of the present invention are described in detail below with reference to the accompanying drawings.

[0075] The present invention proposes a semi-supervised change detection method for ultra-high resolution remote sensing images based on improving the quality of pseudo-label boundaries. The entire process of the method is as follows: Figure 1 As shown, the specific steps include:

[0076] Step A: Divide the remote sensing dual-time image dataset into a labeled dataset and an unlabeled dataset, apply weak data augmentation to the labeled dataset, and first apply weak data augmentation to the unlabeled dataset and then apply strong data augmentation and feature augmentation. Concatenate the labeled dataset and the unlabeled dataset to obtain the semi-supervised learning training dataset.

[0077] Step A: The specific steps are as follows:

[0078] Step A01: The remote sensing dual-time image dataset is divided into four semi-supervised learning ratios: ①5% labeled dataset, 95% unlabeled dataset; ②10% labeled dataset, 90% unlabeled dataset; ③20% labeled dataset, 80% unlabeled dataset; ④40% labeled dataset, 60% unlabeled dataset.

[0079] Step A02: Figure 2 As shown in the lower left, set weak data augmentation pool, including scaling, cropping and horizontal inversion. Set strong data augmentation pool, including color injection, blurring, cropping and splicing, and grayscale adjustment. Set feature augmentation to Dropout strategy. Perform weak data augmentation on the labeled dataset, and perform weak data augmentation on the unlabeled data first, and then perform strong data augmentation and feature augmentation. Concatenate the labeled dataset and the unlabeled dataset to obtain the semi-supervised learning training dataset. Figure 2The middle left part shows the labeled data and unlabeled data being augmented and input into the student model or teacher model, which is the data preprocessing process.

[0080] Step B: Build the teacher model and the student model, and initialize the parameters of the teacher model and the student model.

[0081] Step B: The specific steps are as follows:

[0082] Step B01: First, build a change detection network, using ResNet50 and DeepLabv3+ as the encoder and decoder of the change detection network respectively. Figure 2 As shown in the middle part, the student model and the teacher model share the same network, and the student model parameters are updated to the teacher model through the proposed teacher parameter update strategy. ResNet50 is used to extract the features of remote sensing dual-time images and send them to DeepLabv3+ for change detection:

[0083]

[0084] Where A and B are the previous and next time images of the input remote sensing dual-time image to be tested, respectively. ε(·) is the constructed change detection network encoder, (·) * The multi-dimensional features extracted by the encoder. The multi-dimensional features of the remote sensing dual-time image are subtracted and the absolute value is taken, which is sent to the decoder g to obtain the prediction of change detection

[0085] Step B02: Use the change detection network built in step B01 to build a student model and a teacher model. The student model and the teacher model use the same change detection network structure, but the parameters between the student model and the teacher model are not shared. The teacher model and the student model parameters are initialized using random initial values.

[0086] Step C: Training starts, using the student model obtained in step B to infer the labeled weak augmented data, unlabeled strong augmented data, and unlabeled feature augmented data in the training data set generated in step A. Using the teacher model obtained in step B to infer the unlabeled weak augmented data in the training data set generated in step A to generate rough pseudo labels.

[0087] Step C: The specific steps are as follows:

[0088] Step C01: Training starts. The student model after the parameters are initialized in step B performs inference on the labeled weak augmented data, unlabeled strongly augmented data, and unlabeled feature augmented data in the semi-supervised learning training data set:

[0089]

[0090] In the formula, and They represent weak data augmentation, strong data augmentation and feature augmentation respectively. x1 and x2 are the images of the previous moment and the next moment in the input labeled remote sensing dual-time image, respectively. u1 and u2 are the images of the previous moment and the next moment in the input unlabeled remote sensing dual-time image, respectively. represents the student model, and They represent the labeled prediction, unlabeled strongly augmented prediction and feature augmented prediction of the student model respectively.

[0091] Step C02: Use the teacher model initialized with the parameters in step B to infer the unlabeled weak augmented data in the semi-supervised learning training data set to generate rough pseudo labels:

[0092]

[0093] In the formula, represents the teacher model, Represents the generated rough pseudo label. Figure 2 As shown on the right side of the figure, the four predictions arranged from top to bottom are the labeled predictions of the student model. Unlabeled Strong Augmentation Prediction Feature augmentation prediction and the rough pseudo labels generated by the teacher model

[0094] Step D: Calculate the labeled supervised loss for the student model inference results of the labeled weak augmented data obtained in step C and the corresponding labels in the training data set generated in step A. Use the boundary optimization method designed for remote sensing dual-time image change detection to refine the coarse pseudo-labels obtained in step C, and first calculate the consistency loss between the refined pseudo-labels and the coarse pseudo-labels. Then calculate the semi-supervised loss for the refined pseudo-labels and the student model inference results of the unlabeled strong augmented and unlabeled feature augmented data obtained in step C.

[0095] Step D The specific steps are as follows:

[0096] Step D01: Figure 2 As shown in the upper right corner, the loss of the student model obtained in step C is first calculated for the prediction results of the labeled weakly augmented data and the corresponding labels in the training data set generated in step A:

[0097]

[0098] In the formula, denotes label-supervised loss, and M represent the labeled dataset and its number respectively. CE represents the cross entropy loss, and iThey respectively represent the student model's prediction for the i-th labeled weakly augmented data and its corresponding label.

[0099] Step D02: Use the boundary optimization method designed for remote sensing image change detection to refine the rough pseudo-labels generated in step C. Figure 4 As shown in the boundary optimization process shown in , a 3×3 two-dimensional convolution is first applied to the coarse pseudo-label change map to extract the coordinate set of the boundary area points of the change map. Figure 5 The matching relationship between each part of the change detection result and the remote sensing dual-time image is shown. First, the coordinates of these boundary points are obtained by convolution operation, and the formula is expressed as:

[0100]

[0101] In the formula, x i and i are the horizontal and vertical coordinates of the boundary points. is the mask corresponding to the i-th boundary block. K is a 3×3 convolution kernel, s is the convolution step size, and p is the convolution padding. w represents the predicted width of the boundary, and w is set to 3. The operation conv is a two-dimensional convolution operation, and the operation where comes from NumPy;

[0102] The boundary points are expanded along the upper left corner and the lower right corner to form 48×48 boundary blocks. These boundary blocks are distributed along the boundary, depicting the changed part and its surrounding environment, and the formula is expressed as:

[0103]

[0104] In the formula, the generated is an array that records the coordinates of the upper left corner and lower right corner of the boundary block. bs represents the block size and is set to 48. t represents the previous or next moment in the dual-phase image.

[0105] According to the characteristics of the boundary blocks, a series of boundary block filters are designed. Figure 4 As shown in the pre-processing and post-processing pipeline, an “overlap” filter is first designed to remove highly overlapping boundary blocks. Figure 6 The upper middle part visualizes the process of removing overlapping boundary blocks by the "overlap" filter to reduce the computational burden. Since the convolution operation may cause boundary blocks to appear in the change map and be located in the center of the change map, an "inner" filter is introduced to discard boundary blocks that appear in the change map. Figure 6 The flow of the “inside” filter is visualized in the middle and bottom. This is to ensure that the boundary demarcation predictions are aligned with the change detection predictions and avoid completely wrong classifications. Figure 7As shown, the present invention designs a "distribution" filter to perform a mask acquisition operation on each boundary partition prediction to generate a mask corresponding to each boundary. Then, the intersection-and-union ratio with the original change map is calculated at the corresponding boundary block position. Boundary blocks with significant intersection-and-union ratio differences are considered unreliable and discarded. The "distribution" filter ensures that the boundary partition prediction distribution is aligned with the change detection prediction as much as possible. Figure 8 The improved results of the change detection boundary after boundary optimization are shown. Compared with the original prediction, the optimized boundary can better depict the outline of the object and provide clearer and sharper boundary prediction.

[0106] Finally, the boundary segmentation prediction is integrated with the change detection prediction:

[0107]

[0108] Where σ(·) is the sigmoid function, and sigmoid is used to obtain the confidence level of the prediction. Then, the change detection prediction Prediction with boundary demarcation The decision boundary is pushed to the side with higher confidence. By applying argmax, the final pseudo label with refined boundary is obtained. The consistency loss of the refined pseudo-label and the rough pseudo-label obtained in step C is calculated:

[0109]

[0110] In the formula, is the consistency loss. and N represent the unlabeled dataset and its number, respectively. represents the jth coarse pseudo label generated, and represents the pseudo-label after the jth boundary refinement. The semi-supervised loss is then calculated by combining the refined pseudo-label with the unlabeled strong augmentation prediction and feature augmentation prediction:

[0111]

[0112] In the formula, is a semi-supervised loss used to constrain the consistency between the student model's predictions and the refined pseudo-labels for strong data augmentation and feature augmentation. and They represent the j-th unlabeled strong augmented prediction and the j-th feature augmented prediction, respectively.

[0113] Figure 3It is the visualization of the boundary optimization method on different datasets. Before and after the change represent the previous and next moments in the remote sensing dual-time image, respectively, and the change map represents the output of change detection. The boundary optimization method helps to separate the boundaries of the adhered buildings in the WHU-CD dataset, making the edges clearer and sharper. In the results of LEVIR-CD, the refined boundaries are even closer to the actual edges than the true values. For the overpass in the DSIFN-CD dataset, the boundary optimization method can also provide more refined segmentation results.

[0114] Step E: Calculate the total loss using the labeled supervision loss, consistency loss, and semi-supervised loss obtained in step D, and update the student model parameters using stochastic gradient descent. Calculate the stability of the student model and update the teacher model parameters.

[0115] Step E: The specific steps are as follows:

[0116] Step E01: Calculate the total loss using the obtained labeled supervision loss, consistency loss, and semi-supervised loss, and update the student model parameters using the loss calculation results and stochastic gradient descent:

[0117]

[0118] In the formula, is the total loss, which is the sum of the constraint losses of each part.

[0119] Step E02: Calculate the stability of the student model and update the parameters of the teacher model. Use the output of the student model for the unlabeled strongly augmented data and the mask after the boundary refinement of the unlabeled data to calculate the intersection and union ratio:

[0120]

[0121] In the formula, represents the predicted value of the student model for the jth unlabeled remote sensing dual-time image change detection, and express The mask after boundary refinement. Calculate the argmax operation and The average distance between The average distance is measured using the IoU (Intersection over Union) ratio. If the model is improved, it is considered that the current student model has reached a more stable state, and its parameters are updated to the teacher model by copying the overall parameters.

[0122] Step F: After the training is completed, the student model with updated parameters obtained in step E is used to infer the remote sensing dual-time image to obtain the change detection result.

[0123] Step F The specific steps are as follows:

[0124] Step F01: After the training is completed, the student model with updated parameters is used to perform change detection on the remote sensing dual-time image to be tested, and the change detection prediction result is obtained:

[0125]

[0126] In the formula, For student models, The prediction result of change detection is obtained by arg max operation.

[0127] Furthermore, in order to quantify the effectiveness of the method, the present invention measures the performance of the present invention by using intersection over union (IoU), F1-score, precision (Pre) and recall (Rec):

[0128]

[0129] Among them, TP represents the number of samples that are actually positive but predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative but predicted to be negative.

[0130] The remote sensing change test results of the present invention are shown in Table 1, Table 2 and Table 3.

[0131] Table 1: Experimental test results of WHU-CD dataset

[0132]

[0133]

[0134] Table 2: Experimental test results of LEVIR-CD dataset

[0135]

[0136]

[0137] Table 3: Experimental test results of DSIFN-CD dataset

[0138]

[0139]

[0140] Table 1 shows the semi-supervised change detection results of the present invention on the WHU-CD dataset. Using only 5% of the labeled dataset, the intersection over union (IoU) of the present invention in change detection reaches 83.19%. Table 2 shows the results on the LEVIR-CD dataset. Under the settings of 5%, 10%, 20% and 40% labeled datasets, the present invention improves the C2FNet method by 2.36%, 1.76%, 1.05% and 0.86% respectively. It is worth noting that using only 5% of the labeled dataset, the detection results of the present invention exceed the UniMatch method using 40% of the labeled dataset, reaching 83.32% of the intersection over union (IoU). Even with only 5% of the sample data, the present invention can still effectively learn boundary information from high-quality pseudo labels. Table 3 shows the semi-supervised change detection results of the present invention on the DSIFN-CD dataset. Under the settings of 5%, 10%, 20% and 40% labeled data sets, the present invention improves 6.88%, 3.59%, 2.06% and 2.34% respectively over the C2FNet method. Thanks to the high-quality pseudo-labels, the present invention significantly improves the guidance of semi-supervised learning.

[0141] In summary, the present invention solves the problem that the teacher model generates low-quality pseudo labels and the student model cannot acquire new knowledge. The present invention proposes a new semi-supervised change detection method and a boundary refinement for a dual-time image change map. By refining the pseudo labels, new state-of-the-art performance is achieved in semi-supervised change detection. Experiments were conducted on three data sets to verify the effectiveness of the method of the present invention. The present invention protects the boundary knowledge of remote sensing dual-time images, stabilizes the semi-supervised learning state, and improves the semi-supervised change detection performance that is constrained by the rough pseudo-label boundaries and cannot be further improved.

Claims

1. A semi-supervised change detection method for ultra-high resolution remote sensing images based on improving the quality of pseudo-label boundaries, characterized by: The steps include: Step A: Divide the remote sensing dual-time image dataset into a labeled dataset and an unlabeled dataset; apply weak data augmentation to the labeled dataset, and first apply weak data augmentation to the unlabeled dataset and then apply strong data augmentation and feature augmentation; Concatenate the labeled dataset and the unlabeled dataset to obtain the semi-supervised learning training dataset; Step B: Build the teacher model and the student model, and initialize the parameters of the teacher model and the student model; Step C: Training starts, using the student model obtained in step B to infer the labeled weak augmented data, unlabeled strong augmented data, and unlabeled feature augmented data in the training data set generated in step A; using the teacher model obtained in step B to infer the unlabeled weak augmented data in the training data set generated in step A to generate rough pseudo labels; Step D: Calculate the labeled supervised loss for the student model obtained in step C on the labeled weak augmented data inference results and the corresponding labels in the training data set generated in step A; Use the boundary optimization method designed for remote sensing dual-time image change detection to refine the coarse pseudo-labels obtained in step C, and first calculate the consistency loss between the refined pseudo-labels and the coarse pseudo-labels; Then calculate the semi-supervised loss for the refined pseudo-labels and the student model obtained in step C on the unlabeled strong augmented and unlabeled feature augmented data inference results; Step E: Calculate the total loss using the labeled supervision loss, consistency loss, and semi-supervised loss obtained in step D, and update the student model parameters using stochastic gradient descent; calculate the stability of the student model and update the teacher model parameters; Step F: After the training is completed, the student model with updated parameters obtained in step E is used to infer the remote sensing dual-time image to obtain the change detection result.

2. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step A are as follows: Step A01: Divide the remote sensing dual-temporal image dataset into four semi-supervised learning ratios: ① 5% labeled dataset, 95% unlabeled dataset; ②10% labeled data set, 90% unlabeled data set; ③20% labeled data set, 80% unlabeled data set; ④40% labeled dataset, 60% unlabeled dataset; Step A02: setting a weak data augmentation pool, including scaling, cropping, and horizontal inversion; setting a strong data augmentation pool, including color injection, blurring, cropping, splicing, and grayscale adjustment; Set feature augmentation to Dropout strategy; perform weak data augmentation on labeled data sets, and first perform weak data augmentation on unlabeled data and then perform strong data augmentation and feature augmentation; concatenate labeled data sets and unlabeled data sets to obtain a semi-supervised learning training data set.

3. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step B are as follows: Step B01: First, build a change detection network, use ResNet50 and DeepLabv3+ as the encoder and decoder of the change detection network respectively; use ResNet50 to extract the features of the remote sensing dual-time image, and send the features to DeepLabv3+ for change detection: Where A and B are the previous and next time images of the input remote sensing dual-time image to be tested, ε(·) is the constructed change detection network encoder, (·) * Multi-dimensional features extracted for the encoder; The multi-dimensional features of the remote sensing dual-time image are subtracted and the absolute value is taken and sent to the decoder In the prediction of change detection Step B02: Use the change detection network built in step B01 to construct a student model and a teacher model. The student model and the teacher model use the same change detection network structure, but the parameters between the student model and the teacher model are not shared; the parameters of the teacher model and the student model are initialized using random initial values.

4. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step C are as follows: Step C01: Training starts, using the student model initialized with parameters in step B to infer the labeled weak augmented data, unlabeled strong augmented data, and unlabeled feature augmented data in the semi-supervised learning training data set: In the formula, and They represent weak data augmentation, strong data augmentation and feature augmentation respectively; x1 and x2 are the images of the previous moment and the next moment in the input labeled remote sensing dual-time image, respectively; u1 and u2 are the images of the previous moment and the next moment in the input unlabeled remote sensing dual-time image, respectively; represents the student model, and They represent the labeled prediction, unlabeled strongly augmented prediction, and feature augmented prediction of the student model respectively; Step C02: Use the teacher model initialized with the parameters in step B to infer the unlabeled weak augmented data in the semi-supervised learning training data set to generate rough pseudo labels: In the formula, represents the teacher model, Represents the generated rough pseudo label.

5. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step D are as follows: Step D01: Calculate the loss of the student model obtained in step C for the predicted results of the labeled weakly augmented data and the corresponding labels in the semi-supervised learning training data set generated in step A: In the formula, denotes label-supervised loss, and M represent the labeled dataset and its number respectively; CE stands for cross entropy loss, and i They represent the prediction of the student model for the i-th labeled weak augmented data and its corresponding label respectively; Step D02: Use the boundary optimization method designed for remote sensing image change detection to refine the coarse pseudo-labels generated in step C: first, apply a 3×3 two-dimensional convolution to the coarse pseudo-label change map to extract the coordinate set of the boundary area points of the change map; first, use the convolution operation to obtain the coordinates of these boundary points, and the formula is expressed as follows: In the formula, x i and i are the horizontal and vertical coordinates of the boundary point; is the mask corresponding to the i-th boundary block; K is a 3×3 convolution kernel, s is the convolution step size, and p is the convolution padding; W represents the predicted width of the boundary; the operation conv is a two-dimensional convolution operation, and the operation where comes from NumPy; The boundary points are expanded along the upper left corner and the lower right corner into 48×48 boundary blocks; these boundary blocks are distributed along the boundary, depicting the changed part and its surrounding part, and the formula is expressed as: In the formula, the generated is an array that records the coordinates of the upper left corner and lower right corner of the boundary block; bs represents the block size, which is set to 48; t is used to represent the previous moment or the next moment in the remote sensing dual-time image; According to the characteristics of boundary blocks, a series of boundary block filters are designed. First, an "overlap" filter is designed to remove highly overlapping boundary blocks. Overlapping boundary blocks are removed to reduce the computational burden. Since the convolution operation may cause boundary blocks to appear in the change map and be located in the center of the change map, an "inside" filter is introduced to discard boundary blocks that appear in the change map. In order to ensure that the boundary partition prediction is aligned with the change detection prediction and avoid completely wrong classification, a "distribution" filter is designed to perform a mask acquisition operation on each boundary partition prediction to generate a mask corresponding to each boundary. Then, the intersection-and-union ratio with the original change map is calculated at the corresponding boundary block position. Boundary blocks with significant intersection-and-union ratio differences are considered unreliable and discarded. The "distribution" filter ensures that the boundary partition prediction distribution is aligned with the change detection prediction as much as possible. Finally, the boundary segmentation prediction is integrated with the change detection prediction: Where σ(·0) is the sigmoid function, and sigmoid is used to obtain the confidence level of the prediction; then, the change detection prediction Prediction with boundary demarcation The decision boundary is pushed to the side with higher confidence. The final pseudo-label with refined boundary is obtained by applying arg max. The consistency loss of the refined pseudo-label and the rough pseudo-label obtained in step C is calculated: In the formula, is the consistency loss; and N represent the unlabeled dataset and its number, respectively; represents the jth coarse pseudo label generated, and represents the pseudo-label after the jth boundary refinement; the semi-supervised loss is calculated by combining the refined pseudo-label with the unlabeled strong augmentation prediction and the feature augmentation prediction: In the formula, is a semi-supervised loss used to constrain the consistency between the student model's predictions and the refined pseudo-labels for strong data augmentation and feature augmentation; and They represent the j-th unlabeled strong augmented prediction and the j-th feature augmented prediction, respectively.

6. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 5, characterized in that: The predicted width w of the boundary is set to 3.

7. The semi-supervised change detection method for ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step E are as follows: Step E01: Calculate the total loss using the obtained labeled supervision loss, consistency loss, and semi-supervised loss, and update the student model parameters using the loss calculation results and stochastic gradient descent: In the formula, is the total loss, which is the sum of the constraint losses of each part; Step E02: Calculate the stability of the student model and update the parameters of the teacher model; The student model is used to calculate the intersection and union ratio of the output of the unlabeled strongly augmented data and the mask after the boundary refinement of the unlabeled data: In the formula, represents the predicted value of the student model for the jth unlabeled remote sensing dual-time image change detection, and express Mask after boundary refinement; Calculate the argmax after operation and The average distance between The average distance is measured using IoU, or intersection over union; if the stability score If the model is improved, it is considered that the current student model has reached a more stable state, and its parameters are updated to the teacher model by copying the overall parameters.

8. The method for semi-supervised change detection of ultra-high resolution remote sensing images according to claim 1, characterized in that: The specific steps of step F are as follows: Step F01: After the training is completed, the student model with updated parameters is used to perform change detection on the remote sensing dual-time image to be tested, and the change detection prediction result is obtained: In the formula, For student models, is the change detection prediction result; through the arg max operation, the binary image of change detection, i.e., the change image, is obtained

Images