Method and system for constructing large-scale remote sensing change detection dataset based on diffusion model

Abstract

This invention discloses a method and system for constructing a large-scale remote sensing change detection dataset based on a diffusion model. The invention organizes text-image training data from a large-scale remote sensing domain, fine-tunes the text-image model and instigation model of a stable diffusion model to adapt to the remote sensing domain, and generates high-quality change detection samples with balanced sample categories based on the fine-tuned stable diffusion model. First, in the fine-tuning of the text-image model, CLIP is used to identify new remote sensing vocabulary and establish visual priors for various land cover types. Then, the instigation model is fine-tuned to obtain additional local image editing capabilities, which are applied to change sample generation. Furthermore, the automatic change sample generation algorithm designed according to the principle of sample category balance allows for the rapid construction of a large-scale change detection dataset while effectively controlling label quality and sample balance. In addition, the introduction of negative samples effectively improves the model's ability to identify false changes.

Description

Technical Field

[0001] This invention belongs to the field of dataset construction and relates to a method for constructing a large-scale remote sensing change detection dataset based on an open-vocabulary generative visual language model. It realizes the process of constructing a large-scale, high-quality generative remote sensing change detection dataset through emerging artificial intelligence content generation technology, which solves a major obstacle in the field of modern optical remote sensing image change detection: the lack of large-scale, high-quality training data. It can promote the application of change detection methods and their applications in fields such as geographic information system (GIS) map updates, ecological monitoring, and urban change monitoring. Background Technology

[0002] Information on land cover change is crucial for understanding natural and anthropogenic processes over time, supporting decision-making in areas such as natural resource management, disaster response, and land use planning. However, detecting land cover change using remote sensing technologies currently faces significant challenges. First, modern deep learning methods require large-scale, high-quality change samples, which are much rarer than samples of static land cover types. Consequently, small-scale change detection datasets are scarce, and each dataset has limited scenarios and categories. For example, the LEVIR-CD dataset focuses solely on building changes, while the GVLM dataset includes natural land cover types for disaster monitoring, and the HTCD dataset focuses on urban changes while ignoring natural changes. The scarcity of training data currently poses a significant obstacle to developing effective models for practical applications.

[0003] Secondly, deep learning models heavily rely on high-quality datasets with accurate labels, but accurately annotating change labels is challenging due to differences in individual interpretations, variations in photographic conditions, and phenological differences in natural land cover (e.g., seasonal variations in farmland growth and harvest). However, correcting mislabeled samples has shown significant performance improvements across a variety of scenarios. For example, correcting mislabeled samples in the ImageNet test set resulted in an average accuracy improvement of 11.743% across 10 models. Similarly, correcting errors in the Inria architecture dataset improved the Intersection over Union (IoU) score by 19.44%. These examples highlight the importance of accurate labeling for achieving reliable deep learning models to detect changes in remotely sensed images.

[0004] Finally, many current change detection datasets only provide labels for the changed pixels, which is insufficient for semantic change detection (SCD), as the task involves identifying "from-to" changes. This limitation makes it difficult to meet diverse application needs. Currently, only a few studies have developed semantic change detection datasets that include both before and after the change. Furthermore, datasets such as HRSCD, SECOND, and Hi-UCD mini suffer from severe class imbalance. For example, in Hi-UCD mini, the smallest class accounts for only 0.09% of the labeled pixels. This imbalance severely impacts the learning process of deep learning models. Currently, large-scale, high-quality datasets that effectively support semantic change tasks are extremely scarce.

[0005] Therefore, researching and implementing the construction of large-scale, high-quality generative remote sensing change detection datasets using emerging artificial intelligence-generated content technologies is of great significance for improving the development of the field of remote sensing image change detection. Summary of the Invention

[0006] This invention addresses the severe shortage of training data for deep learning in change detection. Drawing upon advanced Artificial Intelligence Generated Content (AIGC) technology, it proposes a method for constructing a large-scale remote sensing change detection dataset based on an open-vocabulary generative visual language model. This method includes organizing large-scale text-to-image training data across the remote sensing domain, fine-tuning the text-to-image model and inpainting model of a stable diffusion model to adapt them to the remote sensing domain, generating high-quality change detection samples with balanced class distributions based on the fine-tuned stable diffusion model, and training the constructed large-scale generative change detection dataset under multiple deep learning frameworks for change detection. The dataset is then used to predict changes on real-world dual-time remote sensing imagery, achieving pixel-level change detection and class prediction before and after changes in remote sensing areas. This method provides a general, high-quality, automated, and low-cost solution for developing data engines for change detection tasks.

[0007] The technical solution adopted to achieve the purpose of this invention is: a method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model, comprising the following steps:

[0008] Step 1: Collect publicly available remote sensing datasets and attach corresponding text titles to remote sensing images with classification or segmentation labels to organize large-scale remote sensing training data.

[0009] Step 2: Introduce a generative visual language big model with open vocabulary generation capabilities, namely the stable diffusion model. Use the collected image and text training data to fine-tune the text-to-image model and the repair model of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples.

[0010] Step 3: Based on the fine-tuned stable diffusion model SD-Inpainting-RS, and according to the principles of reasonable change direction and balanced sample categories, new land cover content is generated within the specified change area in the remote sensing image to construct a large-scale generative change detection dataset for binary change detection and semantic change detection tasks.

[0011] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0012] Step 1.1: First, design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels. The template should be uniformly in the form of [domain identifier][class name], where [domain identifier] identifies the special bird's-eye view style in the remote sensing field, and [class name] describes the corresponding land cover category.

[0013] Step 1.2: Collect scene classification datasets as scene-level training data, select available scenes and group them into the corresponding land cover categories, and attach text titles;

[0014] Step 1.3: Collect the semantic segmentation dataset as pixel-level training data. Use semantic labels to extract one land cover type at a time, filter out other categories, and attach text titles to each type.

[0015] Furthermore, the stable diffusion model consists of two core modules. The first module is the image-aware compression module, which uses variational autoencoder technology to compress the input image into the latent space. The second module is the diffusion module, which uses a U-Net network to perform the denoising process in the latent space.

[0016] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0017] Step 2.1: First, the text-to-image model is fine-tuned using the collected scene-level and pixel-level image-text data to obtain the SD-T2I-RS model. During the fine-tuning of the text-to-image model, the text encoder of the text-image discrimination model CLIP is used to learn new words in the remote sensing image. The obtained text embedding is used as a conditional input to the weight-adjustable diffusion model to denoise the randomly sampled noisy images in the latent space in an iterative manner. The weights of the pre-trained CLIP text encoder are kept frozen throughout the training process.

[0018] Step 2.2: The repair model is initialized with the weights of the SD-T2I-RS model, and then trained with the collected pixel-level image and text data to give it additional region-based image editing capabilities, thus obtaining the SD-Inpainting-RS model.

[0019] Furthermore, in the repair model training process described in step 2.2, a random polygon m is created using a synthetic mask generator on the original stable diffusion model. r Based on the image missing parts and the trained model's ability to repair them, a mask for a single land cover type is extracted from the collected pixel-level image and text data using image semantic tags. i and the random sample m generated by the synthetic mask generator r Superimpose the layers to generate a single-class mask m. s Training is performed, and the superposition cases include l i (p=0.5), l i ∩m r (p=0.25), and l i -m r (p=0.25) means no superposition, intersection, and difference, respectively, where p represents the probability of each superposition case.

[0020] Furthermore, the large-scale generative change detection dataset in step 3 consists of two parts: 1) Generated from a publicly available semantic segmentation dataset: The algorithm automatically selects suitable categories and objects for change on an image as change masks. Based on the designed reasonable change direction, the SD-Inpainting-RS model is used to modify the change mask region of the image to another land cover type. The generated image and the original image constitute a pair of change samples; 2) Short-term dual-temporal images sampled from Google Earth. An image from any one temporal phase is selected, and the land cover object mask on the image is extracted using the all-segmentation model as a change mask. The SD-Inpainting-RS model is then used to modify the change mask region of the image to another land cover type. This image and the image from the other temporal phase constitute a pair of change samples.

[0021] Furthermore, the process of generating changed samples in dataset 1) includes the following sub-steps:

[0022] 1.1) Select appropriate semantic categories and objects in the image before the change; first, filter out categories with pixels below a specified threshold, then segment the masks of the effective categories into individual objects, and filter out small objects according to another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select a few instances for the change. For continuous features such as forests and farmland, combine the synthetic mask generator in the repair model training process to generate random polygons and overlay them with large-area masks, and then change them according to the result of the overlay.

[0023] 1.2) Design reasonable change directions: Introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge features, define all feasible change directions, and set an area metric to filter sufficiently large square masks for overlay with the mask library below.

[0024] 1.3) Establish a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and its expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask.

[0025] 1.4) Using the SD-Inpainting-RS model, the land cover type of the selected change mask area is changed to another land cover type, and the generated image and the original image constitute a pair of change samples.

[0026] Furthermore, step 4 involves training the constructed large-scale generative change detection dataset under multiple deep learning frameworks for change detection, and then using the weights of the trained network model to make predictions on real-world dual-time remote sensing images, thereby achieving pixel-level change detection and category prediction before and after the change area in the remote sensing images.

[0027] The specific implementation includes the following sub-steps:

[0028] Step 4.1: Train the neural network using the constructed large-scale generative change detection dataset. In the binary change detection task, only binary change labels are used as ground truth supervision for model training. In the semantic change detection task, binary change labels are used to supervise the output of the change detection head, and semantic labels are used to supervise the output of the semantic segmentation head. Iterate the training until the model converges.

[0029] Step 4.2: Given real-world dual-time remote sensing images to be predicted, input them into the network model with loaded network model weights to obtain pixel-level prediction results for change detection on each pair of dual-time remote sensing images. In the semantic change detection task, prediction results of land cover type before and after change can also be obtained, i.e., from-to prediction maps.

[0030] This invention also provides a system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection dataset based on a diffusion model as described above.

[0031] The present invention has the following advantages:

[0032] (1) It has strong scalability. The method for constructing a large-scale remote sensing change detection dataset based on an open-vocabulary generative visual language model has the ability to generate open vocabularies. By identifying new remote sensing vocabularies through CLIP and fully fine-tuning the diffusion model to establish visual priors for corresponding land cover categories, this method can be used to generate remote sensing land cover types or objects of any category.

[0033] (2) It is pioneering. The proposed fine-tuning strategy for the stable diffusion model is the only method currently available specifically for generating change detection samples and constructing large-scale datasets in the field of remote sensing. The resulting SD-Inpainting-RS model weights with additional image local editing capabilities are the first diffusion model in the field of remote sensing that can be used to create change detection samples.

[0034] (3) It has advanced and superior features. This method has significant advantages in controlling the size of the dataset, label quality, diversity of land cover change samples and class balance.

[0035] (4) Strong generalization ability: The large-scale, high-quality generative change detection dataset constructed using this method supports both binary change detection and semantic change detection tasks. It has been verified that deep learning models pre-trained on this dataset can outperform real-world datasets, achieving state-of-the-art generalization performance. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the Wensheng graph model in the fine-tuned stable diffusion model proposed in this invention.

[0037] Figure 2 This is a flowchart illustrating the repair model in the fine-tuned stable diffusion model proposed in this invention.

[0038] Figure 3 This is a schematic diagram of the generated sample of the fine-tuned text-based image model proposed in this invention.

[0039] Figure 4 This is a sampling diagram of the fine-tuning repair model process proposed in this invention. Detailed Implementation

[0040] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0041] This invention proposes a method for constructing a large-scale remote sensing change detection dataset based on an open-vocabulary generative visual language model, comprising the following steps:

[0042] Step 1: Collect publicly available remote sensing datasets and attach corresponding text titles to remote sensing images with classification or segmentation labels to organize large-scale remote sensing training data.

[0043] Step 1 includes the following sub-steps:

[0044] Step 1.1: First, design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels. The template should be uniformly in the form of "[domain identifier][class name]", where [domain identifier] identifies the special bird's-eye view style of the remote sensing field and [class name] describes the corresponding land cover category, such as remote sensing building (RS building) and remote sensing water (RS water).

[0045] Step 1.2: Collect scene classification datasets as scene-level training data, select available scenes and group them into the corresponding land cover categories, and attach text titles;

[0046] Step 1.3: Collect the semantic segmentation dataset as pixel-level training data. Use semantic labels to extract one land cover type at a time, filter out other categories, and attach text titles to each type.

[0047] Step 2: Introduce a generative visual language model with open vocabulary generation capabilities, namely the stable diffusion (SD) model. Use the collected text-image training data to fine-tune the text-to-image model and the repair model (the repair model is a type of text-to-image model) of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples.

[0048] The stable diffusion model described is a generative vision-language model (VLM) that learns the probability distribution of data from samples through a stepwise denoising process. It consists of two core modules. The first module is an image-aware compression module, which uses a variational autoencoder (VAE) technique to compress the input image into a latent space. This significantly reduces computational and memory costs compared to operating in a high-dimensional image space. Its network structure is similar to VQGAN, consisting of an encoder ε and a decoder. Both are pre-trained on a large number of images. Given an image in RGB space... The encoder ε compresses I into a latent representation. decoder Image recovery from latent space makes in The default downsampling factor f = H / h = W / w = 8. The second component is the diffusion module, which uses a U-Net network to perform the denoising process in the latent space. Stable support allows for tuning the denoising model using various types of auxiliary information (e.g., text, semantic graphs, or depth images) and shows particularly promising results in text-conditional image generation (also known as text-generated image models). The text-generated image model of stable diffusion integrates a jointly trained contrastive visual language model, using the text encoder of the contrastive language-image pre-trained model CLIP to process open-vocabulary text cues to obtain text embeddings as conditions for image generation. For example, when using text conditions, the stable diffusion model applies a conditional denoising autoencoder. t = 1, ..., T is used to control the synthesis process, where It is the input of time step t. The noisy version, p is the text prompt guiding the denoiser, τ θ It projects p onto the text embedding. An open-vocabulary text encoder is used, which is then processed by a noise predictor through a cross-attention layer. Based on image-text conditional pairs, a denoising score matching algorithm is employed. Denoising is then performed. The objective of the text-conditional stable diffusion model can be expressed as follows: Where, ∈ θ and τ θ Joint optimization is performed, where t is uniformly sampled from {1,...,T}, and ∈ is Gaussian noise given at time t.

[0049] The stable diffusion model described supports multiple conditional modes. Besides text-to-image models, it can also be used as an efficient general image-to-image conversion model, including semantic synthesis, super-resolution, and inpainting models. Two of these conditional models are used in this invention. One is a text-to-image model with open-vocabulary capability, meaning it can learn and integrate remote sensing vocabulary to generate images reflecting specific remote sensing features and scenes; the other is an inpainting model with local editing capability, allowing the filling of given masked regions in remote sensing images with new semantic content, enabling the generation of rich and diverse variation samples.

[0050] Step 2 includes the following sub-steps:

[0051] Step 2.1: First, the text-to-image model is fine-tuned using collected scene-level and pixel-level image-text data to obtain the SD-T2I-RS model, a stable diffusion model for text-to-image data adapted to the remote sensing domain. During the fine-tuning of the text-to-image model, the text encoder of the contrastive language-image pre-training (CLIP) model is used to learn new words in the remote sensing image. The obtained text embeddings are used as conditional inputs to the weighted diffusion model to denoise randomly sampled noisy images in the latent space in an iterative manner. The weights of the pre-trained CLIP text encoder are kept frozen throughout the training process. Step 2.2: The inpainting model is initialized using the weights of the SD-T2I-RS model, and then trained using collected pixel-level image-text data to give it additional region-based image editing capabilities, resulting in the SD-Inpainting-RS model.

[0052] In the training process of the repaired stable diffusion model described in sub-step 2.2, a random polygon m is created using a synthetic mask generator on the original stable diffusion model. r Based on the image missing parts and the trained model's ability to repair them, a mask for a single land cover type is extracted from the collected pixel-level image and text data using image semantic tags. i and the random sample m generated by the synthetic mask generator r Superimpose the layers to generate a single-class mask m. s Training is performed. The superposition cases include l i (p=0.5), l i ∩m r (p=0.25), and l i -m r (p=0.25) means no superposition, intersection, and difference, respectively, where p represents the probability of each superposition case.

[0053] Step 3: Based on the fine-tuned stable diffusion model, and according to the principles of reasonable change direction and balanced sample categories, new land cover content is generated in the specified change area in the remote sensing image to construct a large-scale generative change detection dataset for binary change detection (BCD) and semantic change detection (SCD) tasks.

[0054] The large-scale generative change detection dataset described in step 3 consists of two parts: 1) Generated from publicly available semantic segmentation datasets. An algorithm automatically selects suitable change categories and objects on an image as change masks. Based on a reasonably designed change direction, the SD-Inpainting-RS model is used to modify the change mask region of the image to another land cover type. The generated image and the original image constitute a pair of change samples; 2) Short-term dual-temporal images sampled from Google Earth. An image from any one temporal phase is selected, and the segment anything model (SAM) is used to extract the land cover object mask on the image as a change mask. Then, the SD-Inpainting-RS model is used to modify the change mask region of the image to another land cover type, forming a pair of change samples with the image from the other temporal phase.

[0055] Furthermore, the automatic generation process of data change samples in 1) includes the following sub-steps:

[0056] 1.1) Select appropriate semantic categories and objects in the image before the change; first, filter out categories with pixels below a specified threshold, then segment the masks of the effective categories into individual objects, and filter out small objects according to another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select a few instances for the change. For continuous features such as forests and farmland, which usually have large pixel areas, combine the synthetic mask generator mentioned in the training process of the repair stable diffusion model to generate random polygons and superimpose them with large area masks, and then change them according to the superimposed result.

[0057] 1.2) Design reasonable change directions; introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge characteristics, and define all feasible change directions. Set an area metric to filter sufficiently large square masks for overlay with the mask library below;

[0058] 1.3) Establish a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and its expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask.

[0059] 1.4) Using the SD-Inpainting-RS model, the land cover type of the selected change mask area is changed to another land cover type, and the generated image and the original image constitute a pair of change samples.

[0060] Furthermore, in 2), part of the data is used to generate change samples. The segmentanything model (SAM) is used to interactively extract the masks of various land cover objects as change masks by manually marking points on the objects of interest. Then, the SD-Inpainting-RS model is used to modify the mask region of one image in the two-time image pair to another land cover type, and together with the other image, they form a pair of change samples.

[0061] Step 4 involves training the constructed large-scale generative change detection dataset within multiple deep learning frameworks for change detection. By loading the weights of the trained network model, predictions are made on real-world dual-time remote sensing images, enabling pixel-level change detection and category prediction before and after changes in remote sensing images.

[0062] Step 4 includes the following sub-steps:

[0063] Step 4.1: Train the neural network using the constructed large-scale generative change detection data. In the binary change detection task, only binary change labels are used as ground truth supervision for model training. In the semantic change detection task, binary change labels are used to supervise the output of the change detection head, and semantic labels are used to supervise the output of the semantic segmentation head. Iterative training continues until the model converges.

[0064] Step 4.2: Given real-world dual-time remote sensing images to be predicted, input them into the network model with loaded training weights to obtain pixel-level prediction results for change detection on each pair of dual-time remote sensing images. In the semantic change detection task, prediction results of land cover type before and after change can also be obtained, i.e., "from-to" prediction maps.

[0065] The effects of the present invention are illustrated below through specific experiments:

[0066] First, text-image training data for a large-scale remote sensing domain were organized according to the method of this invention. In this embodiment, seven publicly available remote sensing datasets were collected, including two large-scale scene classification datasets: AID and Million-AID, and five semantic segmentation datasets: LoveDA, Evlab-SS, LandCover.ai, and the Berlin and Chicago subsets from CITY-OSM. Six common land cover categories with abundant samples were selected: buildings, roads, water bodies, bare land, forest, and agricultural land. These categories were prefixed with "RS" as specific aerial scene descriptors. Specifically, for the two scene classification datasets, in the AID dataset, 10,000 images of 600×600 pixels were processed, and 16 of the 30 aerial scene types were selected and classified into the aforementioned six land cover types. Similarly, from the Million-AID dataset containing 51 scene types, five relevant types were selected, including forest and four types of transportation land. After resizing to 512×512 pixels, this process yielded a total of 6,212 scene-level training samples. For the semantic segmentation dataset, various preprocessing steps were employed, including downsampling, cropping, resizing, and manual annotation. The LoveDA dataset consists of 5,987 high-resolution images with a spatial resolution of 0.3m (1024×1024 pixels), with class labels consistent with the selected six types. The LoveDA training set (2522 images) was cropped into non-overlapping 512×512 pixel patches as training data. The Evlab-SS dataset contains 60 images with a resolution of 4500×4500 pixels, with ground sampling distances (GSDs) of 0.5m, 0.25m, and 0.1m, covering 10 annotation categories other than background. Forty-five images were selected and scaled to 0.5m resolution, retaining six relevant categories, while the remaining four categories were classified as background. The LandCover.ai dataset includes 33 images of 9000×9500 pixels and 8 images of 4200×4700 pixels, with GSDs of 0.25 meters and 0.5 meters, respectively, and is annotated with four labeled land cover types. We manually annotated agricultural land and bare land categories to ensure coverage of all relevant land cover types. The Berlin and Chicago datasets contain aerial images from Google Maps, along with pixel-by-pixel annotations of buildings, roads, and backgrounds from OpenStreetMap. These images were originally approximately 2500×2500 pixels with a GSD of 0.1 meters and were resized to 512×512 pixels. Pixel-level training data, totaling 17,037 512×512 tiles, was extracted from these five semantic segmentation datasets, with ground sampling distances ranging from 0.3 meters to 0.5 meters.

[0067] Then, the text-based image model and the repair model of the stable diffusion model were fine-tuned sequentially using organized training data. First, the text-based image model was fine-tuned using scene-level and pixel-level datasets. We initialized the text-based image model with Stable-Diffusion-v1-4 (SDv1.4) checkpoint weights, which had previously been fine-tuned on the laion-aestheticsv2 dataset. After 66,000 iterations of fine-tuning on two scene classification datasets (scene-level) and an additional 30,000 iterations on training data from five semantic segmentation datasets (pixel-level), the text-based image model checkpoint (referred to as SD-T2I-RS) was obtained. During training, the text conditions were reduced by 10% to enhance classifier-free guidance (CFG) sampling. Figure 3 An example sample generated using the fine-tuned model weights is shown, with a sampling step of 50 and a CFG ratio of 5.0. Following the text-based image fine-tuning stage, the inpainting model is then fine-tuned. The inpainting model recovers from the text-based image checkpoint SD-T2I-RS and adds five additional input channels from the diffusion model (U-Net architecture), initialized to zero, with four channels used for encoding the mask image and one for the mask image itself. 42,000 training iterations were performed using data from five semantic segmentation datasets, with the text condition reduced by 10%. This yielded the final inpainting model checkpoint (called SD-Inpainting-RS), capable of generating diverse semantic samples. Figure 4 An example of the repair training process is provided. Both phases of fine-tuning were performed on two NVIDIA A6000 GPUs with a batch size of 4. The learning rates used were 1e-4 for the raw graph model and 7.5e-5 for the repair model.

[0068] A large-scale generative change detection dataset, WHU-GCD, was constructed based on the fine-tuned inpainting model SD-Inpainting-RS and an automatic change sample generation strategy. First, change samples were created on three publicly available semantic segmentation datasets: LoveDA, Evlab-SS, and LandCover.ai, all containing the same six land cover types. Twenty-five semantic change types were generated, with a relatively balanced distribution. Second, real-world invariant samples were introduced to improve the model's robustness to false changes caused by different photographic conditions. Twenty-two pairs of short-term bi-temporal images, covering a total area of ​​approximately 513 square kilometers, were collected from Google Earth worldwide. Some changed pixels in these images were manually annotated. Simultaneously, point-based cue-based SAM was applied to select homogeneous regions, and the SD-Inpainting-RS model was used to generate changes within these regions. These two datasets, totaling 28,067 pairs of bi-temporal change samples, were divided into training, validation, and test sets. Finally, to evaluate the generalization ability of the change detection model, we merged the test sets of five real-world change detection datasets, including DSIFN, LEVIR-CD, SECOND, CLCD, and CNAM-CD, as Test Set II, totaling 3906 change samples.

[0069] To verify the effectiveness and advancement of our method, we compared our constructed large-scale change detection dataset with other real-world change detection datasets on state-of-the-art change detection methods, focusing particularly on the model's generalization performance. We measured the generalization performance of dataset A on dataset B by training on the training set of dataset A and testing on the test set of dataset B. We selected five commonly used real-world datasets—DSIFN, LEVIR-CD, SYSU-CD, ECOND, and Hi-UCD mini—for comparison. Using the Changeformer change detection model, we statistically analyzed the IoU score (IoU1) and mean F1 score (mean F1, mF1) of the change categories as the main metrics for accuracy evaluation. After ignoring the results of the dataset on its own test set (diagonal values), we averaged the results of its tests on all other datasets to measure the overall generalization performance of a dataset. As shown in Table 1, the model trained on the generative dataset constructed using our method exhibits superior generalization performance compared to other real-world change detection datasets. In terms of overall generalization performance, our dataset demonstrates at least a 5.68% advantage in IoU1 compared to other datasets, and at least a 3.43% advantage in mF1 compared to other existing methods. Furthermore, our model achieves state-of-the-art performance on four out of five real-world datasets. This demonstrates the superiority of our method and its significant practical engineering value.

[0070] Table 1. Comparison of the dataset generated by this invention with existing datasets using advanced change detection methods.

[0071]

[0072] Table 1 compares the generalization performance of the ChangeFormer model with multiple real-world change detection datasets; bold and underlined results represent the best and second-best results, respectively, and strikethrough indicates results not included in the average calculation.

[0073] On the other hand, embodiments of the present invention also provide a system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, characterized in that: it includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection dataset based on a diffusion model as described in the above technical solution.

[0074] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A method for constructing a large-scale remote sensing change detection dataset based on a diffusion model, characterized in that, Includes the following steps: Step 1: Collect publicly available remote sensing datasets and attach corresponding text titles to remote sensing images with classification or segmentation labels to organize large-scale remote sensing training data. Step 2: Introduce a generative visual language big model with open vocabulary generation capabilities, namely the stable diffusion model. Use the collected image and text training data to fine-tune the text-to-image model and the repair model of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples. Step 2 includes the following sub-steps: Step 2.1: First, the text-to-image model is fine-tuned using the collected scene-level and pixel-level image-text data to obtain the SD-T2I-RS model. During the fine-tuning of the text-to-image model, the text encoder of the text-image discrimination model CLIP is used to learn new words in the remote sensing image. The obtained text embedding is used as a conditional input to the weight-adjustable diffusion model to denoise the randomly sampled noisy images in the latent space in an iterative manner. The weights of the pre-trained CLIP text encoder are kept frozen throughout the training process. Step 2.2: The repair model is initialized with the weights of the SD-T2I-RS model, and then trained with the collected pixel-level image and text data to give it additional region-based image editing capabilities, thus obtaining the SD-Inpainting-RS model. Step 3: Based on the fine-tuned stable diffusion model SD-Inpainting-RS, and according to the principles of reasonable change direction and balanced sample categories, new land cover content is generated in the specified change area in the remote sensing image to construct a large-scale generative change detection dataset for binary change detection tasks and semantic change detection tasks. The large-scale generative change detection dataset in step 3 consists of two parts: 1) Generated from a publicly available semantic segmentation dataset: The algorithm automatically selects suitable categories and objects for change on an image as change masks. Based on a reasonable change direction, the SD-Inpainting-RS model is used to modify the change mask region of the image to another land cover type. The generated image and the original image constitute a pair of change samples; 2) Short-term dual-temporal images sampled from Google Earth. An image from any one temporal phase is selected, and the land cover object mask on the image is extracted using the AllSegmentation model as a change mask. The SD-Inpainting-RS model is then used to modify the change mask region of the image to another land cover type. This image and the image from the other temporal phase constitute a pair of change samples.

2. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1.1: First, design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels. The template should be uniformly in the form of [domain identifier][class name], where [domain identifier] identifies the special bird's-eye view style in the remote sensing field, and [class name] describes the corresponding land cover category. Step 1.2: Collect scene classification datasets as scene-level training data, select available scenes and group them into the corresponding land cover categories, and attach text titles; Step 1.3: Collect the semantic segmentation dataset as pixel-level training data. Use semantic labels to extract one land cover type at a time, filter out other categories, and attach text titles to each type.

3. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The stable diffusion model consists of two core modules. The first module is the image-aware compression module, which uses variational autoencoder technology to compress the input image into the latent space. The second module is the diffusion module, which uses a U-Net network to perform the denoising process in the latent space.

4. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: In the repair model training process described in step 2.2, random polygons are created using a synthetic mask generator on the original stable diffusion model. m r Based on the image missing parts and the trained model's ability to repair them, a mask for a single land cover type is extracted from the collected pixel-level image and text data using image semantic tags. l i and the random samples generated by the synthetic mask generator. m r Superimpose the layers to generate a single-class mask. m s Training is conducted, and the superposition of cases includes... l i ( p =0.5), l i ∩ m r ( p =0.25), and l i - m r ( p =0.25), which respectively mean non-overlapping, intersection, and difference. p This represents the probability of each possible superposition scenario.

5. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The process of generating changed samples in dataset 1) includes the following sub-steps: 1.1) Select appropriate semantic categories and objects in the image before the change; first, filter out categories with pixels below a specified threshold, then segment the masks of the effective categories into individual objects, and filter out small objects according to another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select a few instances for the change. For continuous features such as forests and farmland, combine the synthetic mask generator in the repair model training process to generate random polygons and overlay them with large-area masks, and then change them according to the result of the overlay. 1.2) Design reasonable change directions: Introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge features, define all feasible change directions, and set an area metric to filter sufficiently large square masks for overlay with the mask library below. 1.3) Establish a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and its expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask. 1.4) Using the SD-Inpainting-RS model, the land cover type of the selected change mask area is changed to another land cover type, and the generated image and the original image constitute a pair of change samples.

6. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: It also includes step 4, which uses the constructed large-scale generative change detection dataset to train under multiple deep learning frameworks for change detection, and by loading the weights of the trained network model, it makes predictions on real-world dual-time remote sensing images to achieve pixel-level change detection and category prediction before and after the change area in the remote sensing images. The specific implementation includes the following sub-steps: Step 4.1: Train the neural network using the constructed large-scale generative change detection dataset. In the binary change detection task, only binary change labels are used as ground truth supervision for model training. In the semantic change detection task, binary change labels are used to supervise the output of the change detection head, and semantic labels are used to supervise the output of the semantic segmentation head. Iterate the training until the model converges. Step 4.2: Given real-world dual-time remote sensing images to be predicted, input them into the network model with loaded network model weights to obtain pixel-level prediction results for change detection on each pair of dual-time remote sensing images. In the semantic change detection task, prediction results of land cover type before and after change can also be obtained, i.e., from-to prediction maps.

7. A system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection dataset based on a diffusion model as described in any one of claims 1-6.

Images