A semi-supervised multi-temporal satellite image time-varying information extraction method

Abstract

The application discloses a kind of semi-supervised multi-temporal satellite image time-varying information extraction method, belongs to remote sensing image processing technical field.To improve the semantic change detection performance of model and the detection accuracy of complex shape change ground object.The application includes constructing semantic change detection model, using a small amount of labeled double temporal remote sensing image to semantic change detection model using binary change detection supervision loss function and semantic segmentation supervision loss function is fully supervised training to obtain initial model, obtains the semantic change detection prediction result of each image pair;Using pseudo-label optimization strategy to the semantic change detection prediction result of each image pair is optimized;The high confidence pseudo-label data obtained is merged with a small amount of labeled double temporal remote sensing image as new training set, uses new training set in semantic change detection model to semi-supervised training to initial model in, using consistency regularization combined loss function supervision training, obtains new model;Until the pre-set iteration round number is reached.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a semi-supervised method for extracting time-varying information from multi-temporal satellite images. Background Technology

[0002] Changes in ground features in multi-temporal remote sensing satellite imagery often exhibit randomness and complexity, resulting in various irregular shapes of changed areas. In urban scenes, changes caused by building expansion and demolition typically do not conform to rectangular or regular polygonal forms; in natural scenes, erosion and siltation changes in river morphology are manifested as tortuous, bifurcated, and irregular boundaries. These irregularly shaped change areas not only require change detection models to simultaneously perform two major tasks—region detection and internal semantic classification—but also place higher demands on the model's structural design. The model must extract global semantics while accurately depicting detailed boundaries; otherwise, it is prone to false changes such as discontinuities, holes, or noise. The complex landscape and spatial heterogeneity in remote sensing imagery are among the main factors contributing to the difficulty of change detection.

[0003] Meanwhile, deep learning-based semantic change detection methods are highly dependent on the quality and quantity of labeled data, especially the accurate labeling of irregularly shaped change regions, which often requires manual pixel-level segmentation or object-oriented multi-class labeling. Labeling high-resolution, multi-temporal remote sensing data not only requires specialized domain knowledge but also consumes significant manpower and time, significantly increasing project costs. The low efficiency of manual labeling of large-scale remote sensing data has become one of the bottlenecks restricting the research and application of change detection.

[0004] Meanwhile, for irregular water system changes such as rivers and wetlands, the number of real change samples is limited, making it difficult to form a sufficient training set. This results in generally low detection accuracy of the model in these low-sample categories. On the one hand, the lack of diverse training samples leads to a decrease in the model's generalization ability; on the other hand, because the spatiotemporal changes of natural landscapes such as rivers are affected by multiple factors such as climate and hydrological conditions, their morphological changes exhibit high heterogeneity, making it even more difficult for a small number of samples to cover typical change patterns, thus exacerbating the phenomenon of missed detections and false detections. Furthermore, research on the detection of damage to large buildings also shows that it is difficult to learn stable local and neighborhood relationships in low-sample scenarios, affecting the change discrimination effect.

[0005] Semi-supervised strategies, through joint training with a small amount of precisely labeled data and a large amount of unlabeled data, have shown potential in reducing labeling requirements and accelerating model training. Based on consistency regularization, pseudo-labels are generated using unlabeled bi-temporal images to improve model stability and robustness. Further combining graph convolution, self-attention, and multi-scale network structures achieves feature enhancement and iterative pseudo-label updates under weak supervision, significantly alleviating the label scarcity problem. However, the performance of semi-supervised strategies in regions with complex and irregular shapes remains unsatisfactory: pseudo-labels struggle to accurately depict curved boundaries, and reducing the amount of labeling exacerbates the loss of shape details. Therefore, how to enhance the feature representation and background discriminability of irregularly shaped regions within a semi-supervised framework remains a key direction for future research. Summary of the Invention

[0006] The problem this invention aims to solve is to improve the semantic change detection performance of the model and the detection accuracy of features with complex shape changes. It proposes a semi-supervised method for extracting time-varying information from multi-temporal satellite images.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A semi-supervised method for extracting time-varying information from multi-temporal satellite imagery includes the following steps:

[0009] S1. Collect a small number of labeled bi-temporal remote sensing images and corresponding semantic change detection label data, collect a large number of unlabeled bi-temporal remote sensing image data, preprocess the collected bi-temporal remote sensing images, and obtain a set of preprocessed bi-temporal remote sensing images.

[0010] S2. Construct a semantic change detection model, including two sets of encoders and decoders, a multi-scale feature fusion module, a dual-temporal feature interaction module, a change detection module, and two sets of semantic segmentation modules;

[0011] S3. Using the small number of preprocessed labeled dual-temporal remote sensing images obtained in step S1, the semantic change detection model constructed in step S2 is trained under full supervision using the binary change detection supervised loss function and the semantic segmentation supervised loss function to obtain the initial model model_1.

[0012] S4. Input the large amount of preprocessed unlabeled dual-temporal remote sensing image data obtained in step S1 into the initial model model_1 obtained in step S3 for prediction, and obtain the semantic change detection prediction results for each pair of images.

[0013] S5. The semantic change detection and prediction results of each pair of images obtained in step S4 are optimized using a pseudo-label optimization strategy. The pseudo-label optimization strategy includes confidence consistency correction and confidence threshold filtering to generate pseudo-label data with high confidence.

[0014] S6. Combine the high-confidence pseudo-label data obtained in step S5 with the small amount of labeled dual-temporal remote sensing images obtained in step S1 to form a new training set. Use the new training set to perform semi-supervised training on the initial model model_1 in the semantic change detection model, and use the consistency regularized combined loss function to supervise the training to obtain a new model model_2.

[0015] S7. Iterate through steps S4-S6 until the preset number of iterations is reached, and finally output the model model_N. Use the model model_N to extract time-varying information from multi-temporal satellite images.

[0016] Furthermore, the preprocessing method in step S1 is to match the sizes of the two-temporal remote sensing images and detect whether the two-temporal remote sensing images are three-channel images. The two-temporal remote sensing images of the previous time and the next time with the same size and three channels are set as image pairs. The image pairs are converted into NumPy format files to obtain the preprocessed two-temporal remote sensing images.

[0017] Furthermore, the specific implementation method of step S2 includes the following steps:

[0018] S2.1. The semantic change detection model is set as a twin-structure encoder-decoder architecture. The multi-scale feature fusion module is embedded at the end of the encoder-decoder architecture. The output of the multi-scale feature fusion module is connected to the dual-temporal feature interaction module. The dual-temporal feature interaction module is connected to the change detection module and two sets of semantic segmentation modules respectively.

[0019] S2.2. The multi-scale feature fusion module is configured to consist of two complementary sub-modules: a global semantic enhancement sub-module and a local edge optimization sub-module. The output expression of the global semantic enhancement sub-module is:

[0020]

[0021] in, It is the output of the global semantic enhancement submodule. This indicates a feature concatenation operation along the channel dimension. This represents the sigmoid activation function. Indicates the expansion rate The dilated convolution operation, Indicates pixel-by-pixel multiplication. This represents the multi-scale features of the encoder's 4th or 3rd layer. This represents the multi-scale feature of the encoder's first or second layer, where i is a three-dimensional value from 1 to 3.

[0022] The output expression of the local edge optimization submodule is:

[0023]

[0024] in, It is the output of the local edge optimization submodule;

[0025] S2.3. The dual-temporal feature interaction module is set up to realize the collaborative modeling of cross-temporal information through a spatiotemporal coupling mechanism. The core process includes two stages: obtaining the change weight map and obtaining spatial semantic features.

[0026] The change weight map of the dual-temporal feature interaction module and the generation process of spatial semantic features with spatiotemporal information are shown in the following formulas:

[0027]

[0028]

[0029] in, For the changing weights graph, For local edge optimization features at time 1, For local edge optimization features at time 2, To carry spatial semantic features rich in spatiotemporal information, Indicates convolutional layer operations. This indicates a concatenation operation at the channel dimension.

[0030] Furthermore, during the training process in step S3, the loss function is changed to multivariate cross-entropy loss. and binary change cross-entropy loss Composition, the expression is:

[0031]

[0032] in, Let the loss function be for fully supervised training. For time 1, the multivariate cross-entropy loss, For the multivariate cross-entropy loss at time 2, This represents the cross-entropy loss due to binary changes.

[0033] Furthermore, step S4 uses the initial model model_1 to make predictions, obtaining the semantic change detection prediction results for each pair of images, and uses the results as initial pseudo-labels. .

[0034] Furthermore, the method for confidence consistency correction in step S5 is as follows:

[0035] For the first round of model_1, which predicts a large amount of unlabeled data, instead of confidence consistency correction, confidence threshold filtering is directly applied to filter the initial pseudo-labels. Pixels with low confidence levels are masked.

[0036] For the (i-1)th round of model_i update, retain the pseudo-labels used to train the (i-1)th round of model_i. Meanwhile, the predicted pseudo-labels of model_i for unlabeled data are retained. ; through integration and Correcting some erroneous pixels in the pseudo-labels of the same unlabeled remote sensing image pair, and generating optimized pseudo-labels for the i-th round. Used for training model_i+1, fusion and The expression is:

[0037]

[0038]

[0039]

[0040] in, It represents the confidence level at pixel position (x, y) in the mask images of round i-1 and round i. This is the probability distribution of the nth land feature generated in the (i-1)th round of pseudo-labeling. This represents the probability distribution of the nth land feature generated in the i-th round of pseudo-labeling. It is any one of N, representing the nth type of land cover. It is the probability that the object at (x, y) is of type n in the i-th training round. It represents the total number of land cover categories.

[0041] Furthermore, the confidence threshold filtering method in step S5 involves setting the confidence level below a preset threshold. The pixels are masked, and the masked pixel samples do not participate in the calculation of the loss function and the optimization of model parameters in the current round, but are retained. and medium confidence level higher than The pixel samples are used as optimized pseudo-labels for the next round of model training;

[0042] The expression for confidence level is:

[0043]

[0044] in, yes Confidence of samples at each pixel Let represent the entropy of the probability distribution of pixel values ​​at (x, y). This represents the normalization factor.

[0045] Furthermore, the consistency regularization combined loss function in step S6... The expression is:

[0046]

[0047] in, It is a consistency loss function. It's a hyperparameter;

[0048] The consistency loss function is calculated using KL divergence loss, and the formula is as follows:

[0049]

[0050]

[0051] Where M represents the number of unlabeled remote sensing image pairs, Indicates a pseudo tag. This represents the prediction result for an unlabeled image, where Num represents the number of change cover categories of interest in the dataset. This represents the probability of category j in the pseudo-label probability distribution. This represents the probability of category j in the probability distribution of the predicted result; This represents the difference between pseudo-labels with applied random perturbations and predictions from remote sensing images.

[0052] The beneficial effects of this invention are:

[0053] This invention presents a semi-supervised multi-temporal satellite image time-varying information extraction method. Addressing the limitations of expensive, time-consuming, and highly specialized pixel-level semantic change detection annotation, as well as the scarcity of samples of complex shape-changing features, this invention proposes a semi-supervised training strategy for semantic change detection models. The aim is to train a high-performance model using a small amount of labeled data and a large amount of unlabeled data. First, a confidence consistency correction strategy and a confidence threshold filtering strategy are proposed to optimize the quality of pseudo-labels used to help the model learn the distribution information of features in unlabeled data. Furthermore, KL divergence calculation is introduced into the consistency regularization loss function to calculate the loss between unlabeled data and optimized pseudo-labels. During this process, image and label enhancement operations are performed to impose consistency regularization constraints on the model, enhancing its resistance to noise and other interference factors. This improves the semantic change detection performance and the detection accuracy of complex shape-changing features even with insufficient labels. Attached Figure Description

[0054] Figure 1 This is a flowchart of a semi-supervised multi-temporal satellite image time-varying information extraction method according to the present invention;

[0055] Figure 2 This is a flowchart illustrating the extraction of time-varying information from time-phase satellite imagery according to the present invention.

[0056] Figure 3 This is a flowchart illustrating the training process of the semantic change detection network under the semi-supervised strategy of this invention.

[0057] Figure 4 This is a schematic diagram of the semantic change detection network of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0059] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0060] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 -Appendix Figure 4 Detailed explanation is as follows:

[0061] Example 1:

[0062] A semi-supervised method for extracting time-varying information from multi-temporal satellite imagery includes the following steps:

[0063] S1. Collect a small number of labeled bi-temporal remote sensing images and corresponding semantic change detection label data, collect a large number of unlabeled bi-temporal remote sensing image data, preprocess the collected bi-temporal remote sensing images, and obtain a set of preprocessed bi-temporal remote sensing images.

[0064] Furthermore, the preprocessing method in step S1 is to match the sizes of the two-temporal remote sensing images and detect whether the two-temporal remote sensing images are three-channel images. The two-temporal remote sensing images of the previous time and the next time with the same size and three channels are set as image pairs. The image pairs are converted into NumPy format files to obtain the preprocessed two-temporal remote sensing images.

[0065] S2. Construct a semantic change detection model, including two sets of encoders and decoders, a multi-scale feature fusion module, a dual-temporal feature interaction module, a change detection module, and two sets of semantic segmentation modules;

[0066] Furthermore, the specific implementation method of step S2 includes the following steps:

[0067] S2.1. The semantic change detection model is set as a twin-structure encoder-decoder architecture. The multi-scale feature fusion module is embedded at the end of the encoder-decoder architecture. The output of the multi-scale feature fusion module is connected to the dual-temporal feature interaction module. The dual-temporal feature interaction module is connected to the change detection module and two sets of semantic segmentation modules respectively.

[0068] S2.2. The multi-scale feature fusion module is configured to consist of two complementary sub-modules: a global semantic enhancement sub-module and a local edge optimization sub-module. The output expression of the global semantic enhancement sub-module is:

[0069]

[0070] in, It is the output of the global semantic enhancement submodule. This indicates a feature concatenation operation along the channel dimension. This represents the sigmoid activation function. Indicates the expansion rate The dilated convolution operation, Indicates pixel-by-pixel multiplication. This represents the multi-scale features of the encoder's 4th or 3rd layer. This represents the multi-scale feature of the encoder's first or second layer, where i is a three-dimensional value from 1 to 3.

[0071] The output expression of the local edge optimization submodule is:

[0072]

[0073] in, It is the output of the local edge optimization submodule;

[0074] Furthermore, a multi-scale feature fusion module is embedded at the end of the encoder-decoder architecture, responsible for integrating the spatial-semantic information of cross-level features. The encoder outputs four layers of feature tensors with dimensions of 64×W / 2×H / 2, 128×W / 4×H / 4, 256×W / 8×H / 8, and 512×W / 8×H / 8 (input size is 3×W×H), where deep features maintain a W / 8×H / 8 resolution to preserve high-dimensional semantic information. After upsampling and reconstruction by the decoder, an incrementally increasing feature sequence is obtained, which is divided into two groups of heterogeneous features: a high-level semantic feature group (F3, F4) and a low-level detail feature group (F1, F2), which are then input into the dual-path fusion structure.

[0075] Furthermore, the global semantic enhancement submodule extracts multi-scale contextual features through three parallel dilated convolutions (dilation rates r=1, 2, 4), and generates a spatial attention weight map after sigmoid activation. This weight map is then fused with the underlying high-resolution features using channel weighting to highlight the shape features and spatial positioning information of irregular targets.

[0076] S2.3. The dual-temporal feature interaction module is set up to realize the collaborative modeling of cross-temporal information through a spatiotemporal coupling mechanism. The core process includes two stages: obtaining the change weight map and obtaining spatial semantic features.

[0077] Furthermore, the initial change features are obtained by concatenating the outputs of the multi-scale feature fusion modules on both branches, a common method for obtaining change features in most SCD methods. Next, the initial change features are processed through a series of convolutional layers to fully extract spatiotemporal information, which is crucial for guiding the generation of the final mask. Then, the sigmoid function is used to normalize the change features enhanced in the spatiotemporal information dimension to highlight the changing target in the feature map. Subsequently, reverse attention is applied to further amplify the difference between the changing target and the background, extracting detailed edge information. This series of processes generates a change weight map, which not only highlights the overall position of the changing target but also pays special attention to its detailed edges. The change weight map is then used to weight the output features of the previous module on both branches, resulting in a two-branch spatial semantic feature result carrying rich spatiotemporal information, achieving modeling of the correlation between temporal, spatial, and semantic information. Finally, these two enhanced spatial semantic features are concatenated along the channel dimension to obtain the enhanced change features.

[0078] The change weight map of the dual-temporal feature interaction module and the generation process of spatial semantic features with spatiotemporal information are shown in the following formulas:

[0079]

[0080]

[0081] in, For the changing weights graph, For local edge optimization features at time 1, For local edge optimization features at time 2, To carry spatial semantic features rich in spatiotemporal information, Indicates convolutional layer operations. This indicates a concatenation operation at the channel dimension.

[0082] Furthermore, the enhanced spatial semantic features are fed into the dual-temporal feature interaction module, and the spatiotemporal change features obtained through concatenation are used as auxiliary information. First, the multi-scale fusion features extracted from the dual branches of the Siamese network are concatenated along the channel dimension to form an initial spatiotemporal joint representation. Then, a series of convolutional layers are used to extract spatiotemporal context features, extracting discriminative change features. This feature is normalized by the sigmoid function to generate a spatial attention weight map, highlighting change-sensitive regions and suppressing background interference. Subsequently, reverse attention is applied to further amplify the difference between the changing target and the background, extracting detailed edge information and achieving secondary optimization of the feature map. The change weight map is used to weight the output features of the previous module on the dual branches to obtain dual-branch spatial semantic features carrying rich spatiotemporal information, realizing the modeling of the correlation between temporal, spatial, and semantic information. Finally, the two enhanced spatial semantic features are concatenated along the channel dimension to obtain enhanced change features.

[0083] Furthermore, the dual-temporal feature interaction module integrates multi-layer features carrying semantic and spatial information from different time branches to obtain change features carrying spatiotemporal information, and uses these change features to guide the identification of land cover types in changed areas. Before performing the above operations, the outputs of the preceding modules need to be connected to a single time branch. The 4-1 fusion feature and the 3-2 fusion feature on a single branch are concatenated along the channel dimension to obtain the single-branch spatial semantic enhancement feature. The single-branch spatial semantic enhancement feature and the fourth layer feature of the encoder-decoder are concatenated again to ensure that important semantic information in the image is not lost. After both branches have completed the operations, the final dual-branch spatial semantic feature can be input into the dual-branch feature interaction module to execute the process of generating a semantic change mask guided by change information as described above.

[0084] Furthermore, a twin encoder-decoder architecture is employed to process dual-temporal remote sensing images. In the feature extraction stage, a ResNet34-based backbone network extracts multi-level spatial features from both temporal images. Through a designed cross-scale feature fusion mechanism, the network integrates low-level fine-grained features with high-level abstract features, achieving effective coupling between local texture details and global semantic information. The fused enhanced features are then input into a temporal interaction module, which constructs spatiotemporal correlations through feature concatenation and information interaction mechanisms, providing a joint spatial-spectral representation for the identification of change areas. Finally, a classifier composed of deconvolutional and convolutional layers outputs in parallel a binarized change area detection map and a multi-class land cover change semantic map, achieving joint analysis from pixel-level change detection to semantic-level change interpretation.

[0085] Furthermore, the temporal satellite imagery semantic change detection dataset is divided into a training set and a test set. Training groups are extracted from the training set, and each training group is used as a training batch. The semantic change detection network is trained using the training set to obtain a trained semantic change detection model. The trained semantic change detection network is then evaluated using the test set to obtain an assessment of the semantic change detection model.

[0086] S3. Using the small number of preprocessed labeled dual-temporal remote sensing images obtained in step S1, the semantic change detection model constructed in step S2 is trained under full supervision using the binary change detection supervised loss function and the semantic segmentation supervised loss function to obtain the initial model model_1.

[0087] Furthermore, during the training process in step S3, the loss function is changed to multivariate cross-entropy loss. and binary change cross-entropy loss Composition, the expression is:

[0088]

[0089] in, Let the loss function be for fully supervised training. For time 1, the multivariate cross-entropy loss, For the multivariate cross-entropy loss at time 2, This represents the cross-entropy loss due to binary changes.

[0090] S4. Input the large amount of preprocessed unlabeled dual-temporal remote sensing image data obtained in step S1 into the initial model model_1 obtained in step S3 for prediction, and obtain the semantic change detection prediction results for each pair of images.

[0091] Furthermore, step S4 uses the initial model model_1 to make predictions, obtaining the semantic change detection prediction results for each pair of images, and uses the results as initial pseudo-labels. .

[0092] S5. The semantic change detection and prediction results of each pair of images obtained in step S4 are optimized using a pseudo-label optimization strategy. The pseudo-label optimization strategy includes confidence consistency correction and confidence threshold filtering to generate pseudo-label data with high confidence.

[0093] Furthermore, the confidence consistency correction modifies some erroneous pixels in the initial pseudo-labels based on the changes in ground features in the real remote sensing images, in order to improve the accuracy of their semantic change identification; the confidence threshold filtering selects samples with confidence scores higher than a preset threshold from the initial pseudo-labels as pseudo-label data to expand the new training set, thereby improving the quality of the new training set.

[0094] Furthermore, the method for confidence consistency correction in step S5 is as follows:

[0095] For the first round of model_1, which predicts a large amount of unlabeled data, instead of confidence consistency correction, confidence threshold filtering is directly applied to filter the initial pseudo-labels. Pixels with low confidence levels are masked.

[0096] For the (i-1)th round of model_i update, retain the pseudo-labels used to train the (i-1)th round of model_i. Meanwhile, the predicted pseudo-labels of model_i for unlabeled data are retained. ; through integration and Correcting some erroneous pixels in the pseudo-labels of the same unlabeled remote sensing image pair, and generating optimized pseudo-labels for the i-th round. Used for training model_i+1, fusion and The expression is:

[0097]

[0098]

[0099]

[0100] in, It represents the confidence level at pixel position (x, y) in the mask images of round i-1 and round i. This is the probability distribution of the nth land feature generated in the (i-1)th round of pseudo-labeling. This represents the probability distribution of the nth land feature generated in the i-th round of pseudo-labeling. It is any one of N, representing the nth type of land cover. It is the probability that the object at (x, y) is of type n in the i-th training round. It represents the total number of land cover categories.

[0101] Furthermore, and The fusion strategy is as follows: First, generate corresponding probability distribution maps for the (i-1)th round pseudo-label and the i-th round pseudo-label, respectively. and , where Num represents the probability set of the pixel at position (x, y) on the mask map belonging to each land cover category, and Num represents the number of land cover categories in the dataset. Then, the probability distribution maps from the two rounds are aligned spatially, and the probability values ​​of each pixel belonging to the Num land cover categories in the two rounds are compared. The maximum probability value is taken as the category probability distribution for that pixel. Finally, the pseudo-labels are corrected based on this category. The pixel value at (x, y);

[0102] Furthermore, the confidence threshold filtering method in step S5 involves setting the confidence level below a preset threshold. The pixels are masked, and the masked pixel samples do not participate in the calculation of the loss function and the optimization of model parameters in the current round, but are retained. and medium confidence level higher than The pixel samples are used as optimized pseudo-labels for the next round of model training;

[0103] The expression for confidence level is:

[0104]

[0105] in, yes Confidence of samples at each pixel Let represent the entropy of the probability distribution of pixel values ​​at (x, y). This represents the normalization factor.

[0106] Furthermore, pseudo-labels corrected for confidence consistency. Some erroneous pixels were corrected, essentially by selecting the category with the higher predicted probability from two probability maps as the category for that pixel. However, the confidence level of the selected category may still be low because some ground features in remote sensing image pairs are difficult to detect and identify due to factors such as occlusion and shadows, resulting in similar probability values ​​for the ground feature category in that area. Therefore... There are still noisy samples that can mislead the model training. To reduce the impact of these noisy pixels, a confidence threshold filtering strategy is used to filter them. Noisy pixels in the image. Updated pseudo-labels. The confidence level of each pixel sample is calculated using its probability distribution.

[0107] S6. Combine the high-confidence pseudo-label data obtained in step S5 with the small amount of labeled dual-temporal remote sensing images obtained in step S1 to form a new training set. Use the new training set to perform semi-supervised training on the initial model model_1 in the semantic change detection model, and use the consistency regularized combined loss function to supervise the training to obtain a new model model_2.

[0108] Furthermore, the consistency regularization combined loss function in step S6... The expression is:

[0109]

[0110] in, It is a consistency loss function. It's a hyperparameter;

[0111] The consistency loss function is calculated using KL divergence loss, and the formula is as follows:

[0112]

[0113]

[0114] Where M represents the number of unlabeled remote sensing image pairs, Indicates a pseudo tag. This represents the prediction result for an unlabeled image, where Num represents the number of change cover categories of interest in the dataset. This represents the probability of category j in the pseudo-label probability distribution. This represents the probability of category j in the probability distribution of the predicted result; This represents the difference between the pseudo-labels with applied random perturbations and the predictions from remote sensing images. During model training and parameter optimization, minimizing this difference constrains the model's response to random perturbations and improves the model's robustness and generalization.

[0115] Furthermore, in the iterative semi-supervised phase, in each iteration, a new training set consisting of labeled data and pseudo-labeled data is generated. Input model_1 / i, where Generate prediction results and with The loss is calculated using a formula. First, random combination augmentation is performed, then the data is input into the model to generate prediction results. . Enhancement operations through the same random combination, then with Calculate the consistency regularization loss. This is done by analyzing the image... and tags Apply the same noise and minimize the prediction results. and tags The difference in pixel probability distribution between them allows for the imposition of consistency constraints on the model and enhances the model's ability to resist different disturbances and noise.

[0116] The loss in the semi-supervised stage is determined by the prediction results. and pseudo-tags The calculation yielded, where From unlabeled images after applying enhanced perturbations Pseudo-tags generated from model_1 / i The prediction results are generated by the model_0 / i-1 after confidence consistency correction, confidence threshold filtering, and the above combined enhanced perturbation.

[0117] S7. Iterate through steps S4-S6 until the preset number of iterations is reached, and finally output the model model_N. Use the model model_N to extract time-varying information from multi-temporal satellite images.

[0118] In this embodiment, the initial learning rate is set to 1×10. -3 The training continues until a total of 200 training cycles are completed. Furthermore, the SGD optimizer is used for training to accelerate the model convergence process. This embodiment effectively achieves high-precision recognition of time-varying information under semi-supervised training, with its mIoU consistently maintained above 70%.

[0119] Example 2:

[0120] Based on Example 1, this embodiment further illustrates the training process of the semantic change detection network under the semi-supervised strategy as follows:

[0121] A large number of dual-temporal remote sensing images were acquired, and training and testing sets were created. The semi-supervised training strategy was divided into a supervised part using a small amount of labeled data and an iterative semi-supervised part using a large amount of unlabeled data. The dataset was also divided into two parts, namely (1) labeled dataset. ,in Represents image pairs, (2) Unlabeled dataset ,in Let M represent image pairs, and M represent the dataset size. M >> N. Training sets are extracted from the training set, with each training set considered a training batch. The semantic change detection network is trained using the training set to obtain a trained semantic change detection model. The trained semantic change detection network is then evaluated using a test set to obtain an assessment of the model.

[0122] In the supervised part, the initial model is trained using 20% ​​of the labeled data, and the prediction results are calculated and narrowed down. and tags The parameters of the initial model are optimized using the loss between the two methods to obtain model_1. The loss function used here is a combined loss function that includes semantic class multivariate cross-entropy loss and binary change cross-entropy loss.

[0123] The iterative semi-supervised part is divided into four steps. The following is a detailed description of each step: (1) Input 80% of the unlabeled data pairs into model_1 / model_i to obtain the prediction results. (2) A pseudo-label optimization strategy is employed, using confidence consistency correction to adjust low-confidence predictions, and then filtering out high-confidence predictions using a confidence threshold to generate pseudo-labels. (3) Merging 20% ​​of the labeled data and its labels and 80% of the unlabeled data and its pseudo-labels into a new training set can be represented as follows: Where N + M is the number of all remote sensing image pairs, then a new training set is used. Retrain the initial model. Narrow down the prediction results. , And tags (pseudo tags) , The model parameters are updated using the loss between the two, so that the model fits the semantic change detection task and obtains the model model_i, where i is the number of rounds of model update. (4) In order to continuously optimize the quality of pseudo-labels and the fit of model parameters to the task, steps 1, 2 and 3 are repeated until the model completes the specified number of rounds of update. In step 1, the latest updated model model_i is used to replace model_i-1, and the new model is used to generate the prediction results of unlabeled data; in step 2, the pseudo-labels of round i-1 are retained to correct the pseudo-labels of round i; in step 3, supervision is carried out. and The loss function is a combination of binary change cross-entropy loss and semantic class multivariate cross-entropy loss, supervised. and The loss function is the consistency regularization loss.

[0124] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0125] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A semi-supervised method for extracting time-varying information from multi-temporal satellite images, characterized in that, Includes the following steps: S1. Collect a small number of labeled bi-temporal remote sensing images and corresponding semantic change detection label data, collect a large number of unlabeled bi-temporal remote sensing image data, preprocess the collected bi-temporal remote sensing images, and obtain a set of preprocessed bi-temporal remote sensing images. S2. Construct a semantic change detection model, including two sets of encoders and decoders, a multi-scale feature fusion module, a dual-temporal feature interaction module, a change detection module, and two sets of semantic segmentation modules; S3. Using the small number of preprocessed labeled dual-temporal remote sensing images obtained in step S1, the semantic change detection model constructed in step S2 is trained in full supervision using the binary change cross-entropy loss function and the semantic class multivariate cross-entropy loss function to obtain the initial model model_1. S4. Input the large amount of preprocessed unlabeled dual-temporal remote sensing image data obtained in step S1 into the initial model model_1 obtained in step S3 for prediction, and obtain the semantic change detection prediction results for each pair of images. S5. The semantic change detection and prediction results of each pair of images obtained in step S4 are optimized using a pseudo-label optimization strategy. The pseudo-label optimization strategy includes confidence consistency correction and confidence threshold filtering to generate pseudo-label data with high confidence. S6. Combine the high-confidence pseudo-label data obtained in step S5 with the small amount of labeled dual-temporal remote sensing images obtained in step S1 to form a new training set. Use the new training set to perform semi-supervised training on the initial model model_1 in the semantic change detection model, and use the consistency regularized combined loss function to supervise the training to obtain a new model model_2. S7. Iterate through steps S4-S6 until the preset number of iterations is reached, and finally output the model model_N. Use the model model_N to extract time-varying information from multi-temporal satellite images.

2. The semi-supervised multi-temporal satellite image time-varying information extraction method according to claim 1, characterized in that, The preprocessing method in step S1 is to match the sizes of the two-temporal remote sensing images and detect whether the two-temporal remote sensing images are three-channel images. The two-temporal remote sensing images of the previous time and the next time with the same size and three channels are set as image pairs. The image pairs are converted into NumPy format files to obtain the preprocessed two-temporal remote sensing images.

3. A semi-supervised method for extracting time-varying information from multi-temporal satellite images according to claim 1 or 2, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. The semantic change detection model is set as a twin-structure encoder-decoder architecture. The multi-scale feature fusion module is embedded at the end of the encoder-decoder architecture. The output of the multi-scale feature fusion module is connected to the dual-temporal feature interaction module. The dual-temporal feature interaction module is connected to the change detection module and two sets of semantic segmentation modules respectively. S2.

2. The multi-scale feature fusion module is configured to consist of two complementary sub-modules: a global semantic enhancement sub-module and a local edge optimization sub-module. The output expression of the global semantic enhancement sub-module is: in, It is the output of the global semantic enhancement submodule. This indicates a feature concatenation operation along the channel dimension. This represents the sigmoid activation function. Indicates the expansion rate The dilated convolution operation, Indicates pixel-by-pixel multiplication. This represents the multi-scale features of the encoder's 4th or 3rd layer. This represents the multi-scale feature of the encoder's first or second layer, where i is a three-dimensional value from 1 to 3. The output expression of the local edge optimization submodule is: in, It is the output of the local edge optimization submodule; S2.

3. The dual-temporal feature interaction module is set up to realize the collaborative modeling of cross-temporal information through a spatiotemporal coupling mechanism. The core process includes two stages: obtaining the change weight map and obtaining spatial semantic features. The change weight map of the dual-temporal feature interaction module and the generation process of spatial semantic features with spatiotemporal information are shown in the following formulas: in, For the changing weights graph, For local edge optimization features at time 1, For local edge optimization features at time 2, To carry spatial semantic features rich in spatiotemporal information, Indicates convolutional layer operations. This indicates a concatenation operation at the channel dimension.

4. The semi-supervised multi-temporal satellite image time-varying information extraction method according to claim 3, characterized in that, During the training process in step S3, the loss function is composed of multivariate cross-entropy loss. and binary change cross-entropy loss Composition, the expression is: in, Let the loss function be for fully supervised training. For time 1, the multivariate cross-entropy loss, For the multivariate cross-entropy loss at time 2, This represents the cross-entropy loss due to binary changes.

5. The semi-supervised multi-temporal satellite image time-varying information extraction method according to claim 4, characterized in that, Step S4 uses the initial model model_1 to make predictions, obtaining the semantic change detection prediction results for each pair of images, and uses the results as initial pseudo-labels. .

6. The semi-supervised multi-temporal satellite image time-varying information extraction method according to claim 5, characterized in that, The confidence consistency correction method in step S5 is as follows: For the first round of model_1, which predicts a large amount of unlabeled data, instead of confidence consistency correction, confidence threshold filtering is directly applied to filter the initial pseudo-labels. Pixels with low confidence levels are masked. For the (i-1)th round of model_i update, retain the pseudo-labels used to train the (i-1)th round of model_i. Meanwhile, the predicted pseudo-labels of model_i for unlabeled data are retained. ; through integration and Correcting some erroneous pixels in the pseudo-labels of the same unlabeled remote sensing image pair, and generating optimized pseudo-labels for the i-th round. Used for training model_i+1, fusion and The expression is: in, It represents the confidence level at pixel position (x, y) in the mask images of round i-1 and round i. This is the probability distribution of the nth land feature generated in the (i-1)th round of pseudo-labeling. This represents the probability distribution of the nth land feature generated in the i-th round of pseudo-labeling. It is any one of N, representing the nth type of land cover. It is the probability that the object at (x, y) is of type n in the i-th training round. It represents the total number of land cover categories.

7. The semi-supervised method for extracting time-varying information from multi-temporal satellite images according to claim 6, characterized in that, The confidence threshold filtering method in step S5 is to set the confidence level below a preset threshold. The pixels are masked, and the masked pixel samples do not participate in the calculation of the loss function and the optimization of model parameters in the current round, but are retained. and medium confidence level higher than The pixel samples are used as optimized pseudo-labels for the next round of model training; The expression for confidence level is: in, yes Confidence of samples at each pixel The entropy represents the probability distribution of pixel values ​​at (x, y). This represents the normalization factor.

8. The semi-supervised method for extracting time-varying information from multi-temporal satellite images according to claim 7, characterized in that, Step S6 Consistency Regularization Combination Loss Function The expression is: in, It is a consistency loss function. It's a hyperparameter; The consistency loss function is calculated using KL divergence loss, and the formula is as follows: Where M represents the number of unlabeled remote sensing image pairs, Indicates a pseudo tag. This represents the prediction result for an unlabeled image, where Num represents the number of change cover categories of interest in the dataset. This represents the probability of category j in the pseudo-label probability distribution. This represents the probability of category j in the probability distribution of the predicted result; This represents the difference between pseudo-labels with applied random perturbations and predictions from remote sensing images.

Images