Large-scale remote sensing change detection data set construction method and system based on diffusion model

Abstract

The invention discloses a large-scale remote sensing change detection data set construction method and system based on a diffusion model. According to the method, text-image training data of a large-scale remote sensing domain is organized, a text graph model and a repair model of a stable diffusion model are finely adjusted to adapt to the remote sensing domain, and a high-quality change detection sample with balanced sample categories is generated based on the fine-adjusted stable diffusion model. The method comprises the following steps: firstly, in fine tuning of a text graph model, using CLIP to identify new remote sensing vocabularies and establish visual priori of various land coverage types, and then performing fine tuning on a repair model to obtain extra image local editing capability which is applied to change sample generation; besides, in the method, a change sample automatic generation algorithm is designed according to a sample category equalization principle, so that a large-scale change detection data set can be quickly constructed, and the label quality and the sample balance can be well controlled. In addition, the introduction of the negative sample can effectively improve the recognition capability of the model for false changes.

Description

Technical Field

[0001] The present invention belongs to the field of dataset construction, and relates to a method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model. The present invention realizes a process of constructing a large-scale high-quality generative remote sensing change detection dataset through an emerging artificial intelligence content generation technology, and solves a major obstacle in the field of modern optical remote sensing image change detection, which is the lack of large-scale, high-quality training data. The present invention can promote change detection methods and their applications in the fields of geographic information system (GIS) map updating, ecological monitoring, and urban change monitoring. Background Art

[0002] Information about land surface changes is essential for understanding natural and anthropogenic processes over time, and can support decision-making in natural resource management, disaster response, and land use planning. However, there are currently significant challenges in detecting land cover changes through remote sensing technology. First, modern deep learning methods require large-scale, high-quality change samples, which are rarer than samples of static land cover types. As a result, there are currently few small-scale change detection datasets available, and each dataset has limited scenes and categories. For example, the LEVIR-CD dataset only focuses on building changes, while the GVLM dataset includes natural land cover types for disaster monitoring, and the HTCD dataset focuses on urban changes and ignores natural changes. Currently, the scarcity of training data poses a major obstacle to the development of effective models for practical applications.

[0003] Second, deep learning models rely heavily on high-quality datasets with accurate labels, but accurately annotating change labels is challenging due to differences in individual interpretations, changes in photographic conditions, and differences in the phenology of natural land cover (such as seasonal changes in cropland growth and harvesting). But in fact, resolving mislabeled samples has shown significant improvements in model performance in a variety of situations. For example, correcting mislabeled samples in the ImageNet test set resulted in an average improvement of 11.743% in the accuracy of 10 models. Similarly, correcting errors in the Inria building dataset increased the intersection over union (IoU) score by 19.44%. These examples highlight the importance of accurate labeling for implementing reliable deep learning models to detect changes in remote sensing imagery.

[0004] Finally, many current change detection datasets only provide labels for changed pixels, which is insufficient for semantic change detection (SCD) because the task of semantic change detection is to identify "from-to" changes. This limitation makes it difficult to meet the diverse application needs. Currently, only a small number of studies have developed semantic change detection datasets that include before and after categories. However, datasets such as HRSCD, SECOND, and Hi-UCD mini have serious class imbalance problems. For example, the smallest category in Hi-UCD mini only accounts for 0.09% of the labeled pixels. This imbalance seriously affects the learning process of deep learning models. At present, large-scale, high-quality datasets that can effectively support semantic change tasks are very scarce.

[0005] Therefore, researching and implementing the construction of large-scale high-quality generative remote sensing change detection datasets using emerging artificial intelligence content generation technologies is of great significance for promoting the development of remote sensing image change detection. Summary of the invention

[0006] Aiming at the serious shortage of existing deep learning training data for change detection, the present invention combines the current advanced artificial intelligence generated content (AIGC) technology and proposes a method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model. The method includes organizing large-scale text-image training data in the remote sensing domain, fine-tuning the text-to-image model and inpainting model of the stable diffusion model to adapt to the remote sensing domain, generating high-quality change detection samples with balanced sample categories based on the fine-tuned stable diffusion model, and using the constructed large-scale generative change detection dataset to train under multiple deep learning frameworks for change detection, and predicting on real-world dual-time remote sensing images, to achieve pixel-level change detection of remote sensing images and category prediction before and after the change area. The method provides a general, high-quality, automated and low-cost solution for the development of change detection task data engines.

[0007] The technical solution used to achieve the purpose of the present invention is: a method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model, comprising the following steps:

[0008] Step 1: Collect publicly available remote sensing datasets and attach corresponding text captions to remote sensing images with classification or segmentation labels to organize large-scale remote sensing domain image and text training data;

[0009] Step 2: introduce a generative visual language model with open vocabulary generation capability, namely the stable diffusion model, and use the collected text-image training data to fine-tune the text-image model and restoration model of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples;

[0010] Step 3, based on the fine-tuned stable diffusion model SD-Inpainting-RS, according to the principles of reasonable change direction and balanced sample categories, a large-scale generative change detection dataset is constructed by generating new land cover content in the change area specified in the remote sensing image for binary change detection tasks and semantic change detection tasks.

[0011] Furthermore, the specific implementation of step 1 includes the following sub-steps:

[0012] Step 1.1, first design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels, uniformly using the form of [domain identifier][class name], where [domain identifier] identifies the special bird's-eye view style in the remote sensing field, and [class name] describes the corresponding land cover category;

[0013] Step 1.2, collect scene classification datasets as scene-level training data, select available scenes and group them into corresponding land cover categories, and attach text titles;

[0014] Step 1.3, collect semantic segmentation datasets as pixel-level training data, extract one land cover type at a time using semantic labels, filter out other categories, and attach text titles to each.

[0015] Furthermore, the stable diffusion model consists of two core modules, the first module is the image-aware compression module, which compresses the input image into the latent space using the variational autoencoder technique, and the second module is the diffusion module, which uses the U-Net network to perform the denoising process in the latent space.

[0016] Furthermore, the specific implementation of step 2 includes the following sub-steps:

[0017] Step 2.1, first use the collected scene-level and pixel-level image and text data to fine-tune the text-image model in turn to obtain the SD-T2I-RS model. During the fine-tuning of the text-image model, the text encoder of the text-image discriminant model CLIP is used to learn new words in remote sensing images. The obtained text embedding is used as a conditional input into the weight-adjustable diffusion model to iteratively denoise the randomly sampled noisy images in the latent space. The pre-trained CLIP text encoder weights are always kept frozen during the training process;

[0018] In step 2.2, the inpainting model is initialized using the SD-T2I-RS model weights and then trained using the collected pixel-level image data to give it additional region-based image editing capabilities, resulting in the SD-Inpainting-RS model.

[0019] Furthermore, in the repair model training process described in step 2.2, a random polygon m is created using a synthetic mask generator in the original stable diffusion model. r As the basis for training the model's repair capabilities and the missing parts of the image, the image semantic labels are used to extract the mask of a single land cover type for the collected pixel-level image data. i , and the random sample m generated by the synthetic mask generator r Superposition is performed to generate a single-class mask m s For training, the superposition situation includes l i (p=0.5), l i ∩m r (p = 0.25), and l i -m r (p=0.25), which means no superposition, intersection and difference respectively, where p represents the probability of each superposition situation.

[0020] Furthermore, the large-scale generative change detection dataset in step 3 includes two parts: 1) generated from a public semantic segmentation dataset: using an algorithm to automatically select categories and objects suitable for change on an image as change masks, and using the SD-Inpainting-RS model to modify the change mask area of ​​the image to another land cover type according to the designed reasonable change direction, the generated image and the original image constitute a pair of change samples; 2) from short-term dual-phase images sampled from Google Earth, select an image of any one phase, use the segmentation model to extract the mask of the land cover object on the image as the change mask, and then use the SD-Inpainting-RS model to modify the change mask area of ​​the image to another land cover type, and form a pair of change samples with the image of the other phase.

[0021] Furthermore, in the process of generating the variation samples of the data set in 1), the following sub-steps are included:

[0022] 1.1) Select appropriate semantic categories and objects in the image before the change; first filter out categories with pixels below a specified threshold, then segment the masks of valid categories into separate objects, and filter out small objects based on another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select several instances for change. For continuous features such as forests and farmland, combine the synthetic mask generator in the restoration model training process to generate random polygons and overlay them with large area masks, and make changes based on the overlay results.

[0023] 1.2) Design a reasonable change direction: Introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge features, define all feasible change directions, and set an area metric to screen a square mask that is large enough to be superimposed with the mask library described below;

[0024] 1.3) Create a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and the expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask;

[0025] 1.4) The SD-Inpainting-RS model is used to change the land cover type of the selected change mask area to another land cover type. The generated image and the original image constitute a pair of change samples.

[0026] Furthermore, the method further includes step 4, using the constructed large-scale generative change detection dataset to train under multiple deep learning frameworks for change detection, and predicting on real-world dual-temporal remote sensing images by loading the trained network model weights, thereby achieving pixel-level change detection of remote sensing images and category prediction before and after the changed area;

[0027] The specific implementation includes the following sub-steps:

[0028] Step 4.1, use the constructed large-scale generative change detection dataset to train the neural network. In the binary change detection task, only the binary change label is used as the true value to supervise the model training. In the semantic change detection task, the binary change label is used to supervise the output of the change detection head, and the semantic label is used to supervise the output of the semantic segmentation head. Iterate the training until the model converges.

[0029] Step 4.2: Given the real-world dual-temporal remote sensing images to be predicted, they are input into the network model with loaded network model weights to obtain the pixel-level prediction result map of change detection on each pair of dual-temporal remote sensing images. In the semantic change detection task, the prediction results of the land cover type before and after the change, i.e., the from-to prediction map, can also be obtained.

[0030] The present invention also provides a system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection dataset based on a diffusion model as described in the above scheme.

[0031] The present invention has the following advantages:

[0032] (1) Strong scalability: The method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model has the ability to generate open vocabulary. New remote sensing vocabulary identified by CLIP and fully fine-tuning the diffusion model to establish visual priors for the corresponding land cover categories can be used to generate any category of remote sensing land cover types or objects.

[0033] (2) The proposed fine-tuning strategy for the stable diffusion model is the only method currently designed specifically for change detection sample generation and large-scale dataset construction in the remote sensing field. The obtained SD-Inpainting-RS model weights with additional image local editing capabilities are the first diffusion model that can be used for change detection sample production in the remote sensing field.

[0034] (3) This method is advanced and superior. It has significant advantages in controlling dataset size, label quality, diversity of land cover change samples, and category balance.

[0035] (4) Strong generalization: The large-scale, high-quality generative change detection dataset constructed by this method supports both binary change detection and semantic change detection tasks. It has been verified that the deep learning model pre-trained on it can surpass the real-world dataset and achieve the most advanced generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the Vincent graph model in the fine-tuning stable diffusion model proposed in the present invention.

[0037] Figure 2 It is a flow chart of the repair model in the fine-tuning stable diffusion model proposed in the present invention.

[0038] Figure 3 It is a schematic diagram of the generation samples of the fine-tuned text graph model proposed in the present invention.

[0039] Figure 4 It is a sampling schematic diagram of the fine-tuning repair model process proposed in the present invention. DETAILED DESCRIPTION

[0040] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0041] The present invention proposes a method for constructing a large-scale remote sensing change detection dataset based on an open vocabulary generative visual language model, comprising the following steps:

[0042] Step 1: Collect publicly available remote sensing datasets and attach corresponding text captions to remote sensing images with classification or segmentation labels to organize large-scale remote sensing domain image and text training data;

[0043] The specific implementation of step 1 includes the following sub-steps:

[0044] Step 1.1, first design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels, uniformly in the form of "[domain identifier][class name]", where [domain identifier] identifies the special bird's-eye view style in the remote sensing field, and [class name] describes the corresponding land cover category, such as remote sensing building (RS building), remote sensing water (RS water), etc.;

[0045] Step 1.2, collect scene classification datasets as scene-level training data, select available scenes and group them into corresponding land cover categories, and attach text titles;

[0046] Step 1.3, collect semantic segmentation datasets as pixel-level training data, extract one land cover type at a time using semantic labels, filter out other categories, and attach text titles to each.

[0047] Step 2: introduce a generative visual language model with open vocabulary generation capability, namely the stable diffusion model (SD). Use the collected text-image training data to fine-tune the text-generated graph model and the repair model (the repair model is a type of image-generated graph) of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples.

[0048] The stable diffusion model is a generative vision-language model (VLMs) that learns the probability distribution of data from samples through a stepwise denoising process. It consists of two core modules. The first module is the image-aware compression module, which uses the variational autoencoder (VAE) technique to compress the input image into a latent space. This greatly reduces the computational and memory costs compared to operating in a high-dimensional image space. Its network structure is similar to VQGAN, consisting of an encoder ε and a decoder Both are pre-trained on a large number of images. Given an image in RGB space The encoder ε compresses I into a latent representation Decoder Recover the image from the latent space such that in The default downsampling factor f=H / h=W / w=8. The second component is the diffusion module, which uses a U-Net network to perform the denoising process in the latent space. Stable supports using various types of auxiliary information (such as text, semantic maps, or depth images) to condition the denoising model, and has shown particularly promising results in text-conditioned image generation (also known as the Vincent graph model). The Stable Diffusion Vincent graph model integrates a jointly trained contrastive visual-language model, processing open vocabulary text cues with a text encoder from a contrastive language-image pre-trained model CLIP to obtain text embeddings as conditions for image generation. For example, when using text conditioning, the Stable Diffusion model applies a conditional denoising autoencoder t=1,...,T, to control the synthesis process, where is the input at time step t is the noisy version of the denoiser, p is the textual cue that guides the denoiser, and τ θ is to project p into the text embedding The open vocabulary text encoder is then processed by a noise predictor through a cross-attention layer. Based on the image-text condition pair, a denoising score matching algorithm is used to match The corresponding objective of the stable diffusion model conditioned on text can be expressed as, Among them, ∈ θ and τ θ Perform joint optimization, t is uniformly sampled from {1,...,T}, ∈ is a Gaussian noise given at time t.

[0049] The stable diffusion model supports multiple conditional modes. In addition to the text-to-image model, it can also be used as an efficient general image-to-image conversion model, including semantic synthesis, super-resolution and restoration models. In the invention, two of the conditional models are used. One is a text-based graph model with open vocabulary capabilities, that is, it can learn and integrate remote sensing vocabulary to generate images that reflect specific remote sensing features and scenes; the other is a restoration model with local editing capabilities, that is, it allows a given mask area in a remote sensing image to be filled with new semantic content, and can generate rich and diverse change samples.

[0050] The specific implementation of step 2 includes the following sub-steps:

[0051] Step 2.1, first use the collected scene-level and pixel-level text and image data to fine-tune the language-image model in turn, and obtain the SD-T2I-RS model, which is a language-image stable diffusion model adapted to the remote sensing domain. During the fine-tuning of the language-image model, the text encoder of the contrastive language-image pre-training (CLIP) model is used to learn new vocabulary in remote sensing images. The obtained text embedding is used as a conditional input into the weight-adjustable diffusion model to denoise the randomly sampled noisy images in the latent space in an iterative manner. The weights of the pre-trained CLIP text encoder are always frozen during the training process; Step 2.2, the inpainting model is initialized with the SD-T2I-RS model weights, and then trained using the collected pixel-level text and image data to give it additional region-based image editing capabilities, and obtain the SD-Inpainting-RS model.

[0052] In the repaired stable diffusion model training process described in sub-step 2.2, a random polygon m is created using a synthetic mask generator in the original stable diffusion model. r As the basis for training the model's repair capabilities and the missing parts of the image, the image semantic labels are used to extract the mask of a single land cover type for the collected pixel-level image data. i , and the random sample m generated by the synthetic mask generator r Superposition is performed to generate a single-class mask m s Training is performed. The superposition case includes l i (p=0.5), l i ∩m r (p = 0.25), and l i -m r (p=0.25), which means no superposition, intersection and difference respectively, where p represents the probability of each superposition situation.

[0053] Step 3: Based on the fine-tuned stable diffusion model, according to the principles of reasonable change direction and balanced sample categories, a large-scale generative change detection dataset is constructed by generating new land cover content in the change area specified in the remote sensing image for binary change detection (BCD) and semantic change detection (SCD) tasks;

[0054] The large-scale generative change detection dataset described in step 3 includes two parts: 1) generated from a public semantic segmentation dataset. The algorithm automatically selects the categories and objects suitable for change on an image as the change mask. According to the designed reasonable change direction, the SD-Inpainting-RS model is used to modify the change mask area of ​​the image to another land cover type. The generated image and the original image constitute a pair of change samples; 2) short-term dual-phase images sampled from Google Earth. Select an image of any phase, use the segment anything model (SAM) to extract the mask of the land cover object on the image as the change mask, and then use the SD-Inpainting-RS model to modify the change mask area of ​​the image to another land cover type, and form a pair of change samples with the image of the other phase.

[0055] Furthermore, in the process of automatically generating the data change samples in 1), the following sub-steps are included:

[0056] 1.1) Select appropriate semantic categories and objects in the image before the change; first filter out categories with pixels below a specified threshold, then segment the masks of valid categories into separate objects, and filter out small objects based on another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select several instances for change. For continuous features such as forests and farmland, which usually have large pixel areas, combine the synthetic mask generator described in the training process of the inpainting stable diffusion model to generate random polygons and superimpose them with large area masks, and make changes based on the superimposed results;

[0057] 1.2) Design a reasonable change direction; introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge features, and define all feasible change directions. Set an area metric to screen a square mask that is large enough to overlay with the mask library described below;

[0058] 1.3) Build a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and the expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask.

[0059] 1.4) The SD-Inpainting-RS model is used to change the land cover type of the selected change mask area to another land cover type. The generated image and the original image constitute a pair of change samples.

[0060] Furthermore, part of the data in 2) is used to generate change samples. The segmentation anything model (SAM) is used to interactively extract masks of various land cover objects as change masks by manually marking points on the objects of interest. Then the SD-Inpainting-RS model is used to modify the mask area of ​​one image in the dual-time image pair to another land cover type, and form a pair of change samples with the other image.

[0061] Step 4: Use the constructed large-scale generative change detection dataset to train under multiple deep learning frameworks for change detection. By loading the trained network model weights and predicting on real-world dual-temporal remote sensing images, pixel-level change detection of remote sensing images and category prediction before and after the change area can be achieved.

[0062] The specific implementation of step 4 includes the following sub-steps:

[0063] Step 4.1, use the constructed large-scale generative change detection data to train the neural network. In the binary change detection task, only the binary change label is used as the true value to supervise the model training. In the semantic change detection task, the binary change label is used to supervise the output of the change detection head, and the semantic label is used to supervise the output of the semantic segmentation head. Iterate the training until the model converges.

[0064] Step 4.2: Given the real-world dual-temporal remote sensing images to be predicted, they are input into the network model with loaded training weights to obtain the pixel-level prediction result map of change detection on each pair of dual-temporal remote sensing images. In the semantic change detection task, the prediction results of the land cover type before and after the change, namely the “from-to” prediction map, can also be obtained.

[0065] The effect of the present invention is described below by specific experiments:

[0066] First, the text-image training data of large-scale remote sensing domain is organized according to the method of the present invention. Seven public remote sensing datasets are collected in the embodiment, including two large-scale scene classification datasets: AID and Million-AID, and five semantic segmentation datasets: LoveDA, Evlab-SS, LandCover.ai, and sub-datasets Berlin and Chicago from CITY-OSM. Six common land cover categories with rich samples are selected: buildings, roads, water bodies, bare land, forests, and agricultural land. We add the "RS" prefix to these categories as special bird's-eye view scene descriptors. Specifically, for these two scene classification datasets, in the AID dataset, 10,000 images of 600×600 pixels were processed, 16 of which were selected from 30 aerial scene types and classified into the above six land cover types. Similarly, from the Million-AID dataset containing 51 scene types, 5 relevant types were screened, including forests and four types of transportation land. This process yields a total of 6,212 scene-level training samples after resizing to 512×512 pixels. For the semantic segmentation datasets, various preprocessing steps were used, including downsampling, cropping, resizing, and manual annotation. The LoveDA dataset consists of 5,987 high-resolution images with a spatial resolution of 0.3m (1024×1024 pixels), and the class labels are consistent with the six selected categories. The training set of LoveDA (2522 images) was cropped into non-overlapping 512×512 pixel patches as training data. The Evlab-SS dataset contains 60 images with a resolution of 4500×4500 pixels, with ground sampling distances (GSD) of 0.5m, 0.25m, and 0.1m, covering 10 annotated categories other than background. 45 of the images scaled to 0.5m resolution were selected, retaining the six relevant categories, while the remaining four categories were classified as background. The LandCover.ai dataset includes 33 images of 9000×9500 pixels and 8 images of 4200×4700 pixels with a GSD of 0.25 meters and 0.5 meters, respectively, annotated with 4 labeled land cover types. We manually annotated the agricultural land and bare land categories to ensure coverage of all relevant land cover types. The Berlin and Chicago datasets contain aerial images from Google Maps, as well as pixel-by-pixel annotations of buildings, roads, and backgrounds from OpenStreetMap. These images were originally approximately 2500×2500 pixels with a GSD of 0.1 meters and resized to 512×512 pixels. The pixel-level training data extracted from these five semantic segmentation datasets totals 17,037 512×512 tiles with ground sampling distances ranging from 0.3 meters to 0.5 meters.

[0067] The organized training data is then used to fine-tune the Wensheng graph model and the inpainting model of the stable diffusion model in sequence. First, the Wensheng graph model is fine-tuned using scene-level and pixel-level datasets. We initialize the Wensheng graph model using the Stable-Diffusion-v1-4 (SDv1.4) checkpoint weights, which was previously fine-tuned on the laion-aestheticsv2 dataset. The Wensheng graph model checkpoint (called SD-T2I-RS) is obtained after 66,000 iterations of fine-tuning on two scene classification datasets (scene-level) and an additional 30,000 iterations on training data from five semantic segmentation datasets (pixel-level). During training, the text conditioning is reduced by 10% to enhance classifier-free guidance (CFG) sampling. Figure 3 Examples of samples generated using the fine-tuned model weights are shown, with the sampling step set to 50 and the CFG scale to 5.0. After the Vincent map fine-tuning stage, the inpainting model is fine-tuned. The inpainting model is restored from the Vincent map checkpoint SD-T2I-RS and adds five additional input channels from the diffusion model (U-Net structure), initialized to zero, four of which are for the encoded mask image and one for the mask map itself. 42,000 training iterations were performed using data from five semantic segmentation datasets, and the text conditioning was reduced by 10%. This produced the final inpainting model checkpoint (called SD-Inpainting-RS), which is able to generate diverse semantic samples. Figure 4 An example of the inpainting training process is provided. Both stages of fine-tuning are performed on two NVIDIA A6000 GPUs with a batch size of 4. The learning rates used are 1e-4 for the Vincent graph model and 7.5e-5 for the inpainting model.

[0068] Based on the fine-tuned inpainting model SD-Inpainting-RS and the automatic change sample generation strategy, a large-scale generative change detection dataset WHU-GCD is constructed. First, change samples are created on three public semantic segmentation datasets LoveDA, Evlab-SS and LandCover.ai, which have the same six land cover types. 25 semantic change types are generated with a relatively balanced distribution. Second, real-world invariant samples are introduced to improve the robustness of the model to spurious changes caused by different photographic conditions. 22 pairs of short-term bi-temporal images with a total area of ​​about 513 square kilometers are collected from Google Earth around the world. Some change pixels in these images are manually annotated. At the same time, point-based cueing SAM is applied to select homogeneous areas, and the SD-Inpainting-RS model is used to generate changes within these areas. These two parts of data total 28,067 pairs of bi-temporal change samples, which are divided into training set, validation set and test set I. Finally, in order to evaluate the generalization ability of the change detection model, we merge the test sets of five real-world change detection datasets, including DSIFN, LEVIR-CD, SECOND, CLCD, and CNAM-CD, as test set II, with a total of 3906 pairs of change samples.

[0069] In order to verify the effectiveness and advancement, we compare the constructed large-scale change detection dataset with other real-world change detection datasets on advanced change detection methods, paying special attention to the generalization performance of the model. By training on the training set of dataset A and testing on the test set of dataset B, the generalization performance of dataset A on dataset B can be measured. Five commonly used real-world datasets including DSIFN, LEVIR-CD, SYSU-CD, ECOND and Hi-UCD mini are selected for comparison. Using the Changeformer change detection model, the IoU score (IoU1) and the mean F1 score (mean F1, mF1) of the change category are statistically used as the main indicators of accuracy evaluation. After ignoring the results of the dataset tested on its own test set (diagonal value), the results of its test on all other datasets are averaged, and this is used to measure the comprehensive generalization performance of a dataset. From the main indicators of IoU1 score and mean F1 score in Table 1, the generalization performance of the model trained on the generative dataset constructed by the method of the present invention is better than that of several other real-world change detection datasets. In terms of comprehensive generalization performance, our dataset has at least 5.68% advantage over other datasets in terms of IoU1 indicator, and our method has at least 3.43% advantage over other existing methods in terms of mF1 indicator. Compared with other datasets, the model achieved the best performance on four of the five real datasets. This proves the superiority of the method of the present invention and has good engineering practical value.

[0070] Table 1 Comparison results of the dataset generated by the present invention and the existing datasets in advanced change detection methods

[0071]

[0072] Table 1 compares the generalization performance of the ChangeFormer model with multiple real-world change detection datasets; the bold and underlined ones are the best and second-best results, respectively, and the deleted ones are not included in the calculation of the average.

[0073] On the other hand, an embodiment of the present invention also provides a system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, which is characterized by comprising a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection dataset based on a diffusion model as described in the above technical solution.

[0074] The specific implementations described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for constructing a large-scale remote sensing change detection dataset based on a diffusion model, characterized in that: The following steps are involved: Step 1: Collect publicly available remote sensing datasets and attach corresponding text captions to remote sensing images with classification or segmentation labels to organize large-scale remote sensing domain image and text training data; Step 2: introduce a generative visual language model with open vocabulary generation capability, namely the stable diffusion model, and use the collected text-image training data to fine-tune the text-image model and restoration model of the stable diffusion model to adapt to the remote sensing domain and generate diverse change detection samples; Step 3, based on the fine-tuned stable diffusion model SD-Inpainting-RS, according to the principles of reasonable change direction and balanced sample categories, a large-scale generative change detection dataset is constructed by generating new land cover content in the change area specified in the remote sensing image for binary change detection tasks and semantic change detection tasks.

2. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The specific implementation of step 1 includes the following sub-steps: Step 1.1, first design a template for the text title attached to the corresponding remote sensing image with classification or segmentation labels, uniformly using the form of [domain identifier][class name], where [domain identifier] identifies the special bird's-eye view style in the remote sensing field, and [class name] describes the corresponding land cover category; Step 1.2, collect scene classification datasets as scene-level training data, select available scenes and group them into corresponding land cover categories, and attach text titles; Step 1.3, collect semantic segmentation datasets as pixel-level training data, extract one land cover type at a time using semantic labels, filter out other categories, and attach text titles to each.

3. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The stable diffusion model consists of two core modules, the first module is the image-aware compression module, which uses the variational autoencoder technique to compress the input image into the latent space, and the second module is the diffusion module, which uses the U-Net network to perform the denoising process in the latent space.

4. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The specific implementation of step 2 includes the following sub-steps: Step 2.1, first use the collected scene-level and pixel-level image and text data to fine-tune the text-image model in turn to obtain the SD-T2I-RS model. During the fine-tuning of the text-image model, the text encoder of the text-image discriminant model CLIP is used to learn new words in remote sensing images. The obtained text embedding is used as a conditional input into the weight-adjustable diffusion model to iteratively denoise the randomly sampled noisy images in the latent space. The pre-trained CLIP text encoder weights are always kept frozen during the training process; In step 2.2, the inpainting model is initialized using the SD-T2I-RS model weights and then trained using the collected pixel-level image data to give it additional region-based image editing capabilities, resulting in the SD-Inpainting-RS model.

5. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: In the repair model training process described in step 2.2, a random polygon m is created using a synthetic mask generator in the original stable diffusion model. r As the basis for training the model's repair capabilities and the missing parts of the image, the image semantic labels are used to extract the mask of a single land cover type for the collected pixel-level image data. i , and the random sample m generated by the synthetic mask generator r Superposition is performed to generate a single-class mask m s For training, the superposition situation includes l i (p=0.5), l i ∩m r (p = 0.25), and l i -m r (p=0.25), which means no superposition, intersection and difference respectively, where p represents the probability of each superposition situation.

6. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The large-scale generative change detection dataset in step 3 consists of two parts: 1) generated from a public semantic segmentation dataset: the algorithm is used to automatically select the categories and objects suitable for change on an image as the change mask, and the SD-Inpainting-RS model is used to modify the change mask area of ​​the image to another land cover type according to the designed reasonable change direction. The generated image and the original image constitute a pair of change samples; 2) from the short-term dual-phase images sampled from Google Earth, select any one of the phase images, use the segmentation model to extract the mask of the land cover object on the image as the change mask, and then use the SD-Inpainting-RS model to modify the change mask area of ​​the image to another land cover type, and form a pair of change samples with the image of the other phase.

7. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 6, characterized in that: In the process of generating the variation samples of the data set in 1), the following sub-steps are included: 1.1) Select appropriate semantic categories and objects in the image before the change; first filter out categories with pixels below a specified threshold, then segment the masks of valid categories into separate objects, and filter out small objects based on another predefined pixel threshold. Finally, for countable objects such as buildings and pools, randomly select several instances for change. For continuous features such as forests and farmland, combine the synthetic mask generator in the restoration model training process to generate random polygons and overlay them with large area masks, and make changes based on the overlay results. 1.2) Design a reasonable change direction: Introduce a roundness index to measure the interchangeability between land cover types with different shapes and edge features, define all feasible change directions, and set an area metric to screen a square mask that is large enough to be superimposed with the mask library described below; 1.3) Create a mask library for objects with special shapes such as buildings, roads, ponds and rivers; when a large pre-change mask is selected and the expected post-change category belongs to these types, randomly select several masks from the mask library, then intersect each mask with the pre-change mask, and finally select the mask that produces the largest intersection area as the final change mask; 1.4) The SD-Inpainting-RS model is used to change the land cover type of the selected change mask area to another land cover type. The generated image and the original image constitute a pair of change samples.

8. The method for constructing a large-scale remote sensing change detection dataset based on a diffusion model according to claim 1, characterized in that: The method also includes step 4, using the constructed large-scale generative change detection dataset to train under multiple deep learning frameworks for change detection, and predicting on real-world dual-temporal remote sensing images by loading the trained network model weights to achieve pixel-level change detection of remote sensing images and category prediction before and after the change area; The specific implementation includes the following sub-steps: Step 4.1, use the constructed large-scale generative change detection dataset to train the neural network. In the binary change detection task, only the binary change label is used as the true value to supervise the model training. In the semantic change detection task, the binary change label is used to supervise the output of the change detection head, and the semantic label is used to supervise the output of the semantic segmentation head. Iterate the training until the model converges. Step 4.2: Given the real-world dual-temporal remote sensing images to be predicted, they are input into the network model with loaded network model weights to obtain the pixel-level prediction result map of change detection on each pair of dual-temporal remote sensing images. In the semantic change detection task, the prediction results of the land cover type before and after the change, i.e., the from-to prediction map, can also be obtained.

9. A system for constructing a large-scale remote sensing change detection dataset based on a diffusion model, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for constructing a large-scale remote sensing change detection data set based on a diffusion model as described in any one of claims 1 to 8.

Images