Remote sensing open vocabulary change detection sample data synthesis method and system

Abstract

The present application provides a remote sensing open vocabulary change detection sample data synthesis method and system, belonging to the technical field of remote sensing image processing and computer vision. The principle of the present application is to use the global layout reasoning ability of the pre-trained visual language model to ensure the rationality of the change position and category conversion, combine the local shape refinement ability of the diffusion model to ensure the spatial compatibility of the change target, and use the open vocabulary representation ability of the CLIP text embedding to break through the limitation of the fixed category vocabulary, to realize the synthesis data generation of diversified change types in disaster scenes. The present application can generate change detection data sets covering building collapse, road interruption, ground flooding, vegetation degradation and other disaster types, effectively alleviating the problems of scarcity of labeled data and variety of change types in disaster scenes, and is suitable for remote sensing emergency monitoring tasks of natural disasters such as earthquakes, floods, landslides and fires.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically relating to a method and system for synthesizing remote sensing open vocabulary change detection sample data for disaster scenarios, which is applied to remote sensing emergency monitoring and analysis of natural disasters. Background Technology

[0002] The development of deep learning has significantly improved the accuracy of change detection in remote sensing images, but it still faces two key challenges in disaster monitoring scenarios. Firstly, data is extremely scarce in disaster scenarios. Disasters are sudden and random, making it costly to acquire large-scale pre- and post-disaster remote sensing images, and pixel-level annotation is difficult. Different disaster types (such as earthquakes, floods, landslides, and fires) exhibit significantly different change patterns, lacking a unified annotation standard, further increasing the difficulty of data construction and limiting the training effect and cross-scenario generalization ability of models. Secondly, existing change detection data synthesis methods are mostly based on fixed-category vocabularies, limiting the types of changes and making it difficult to cover the diverse and unpredictable changes in disaster scenarios. Existing methods typically rely on rule-driven region operations (such as copy-paste or noise perturbation), which struggle to guarantee the rationality and authenticity of changes under open-category conditions. The distribution and detail representation of the generated data are also insufficient, making it difficult to effectively improve model performance. Therefore, there is an urgent need for a remote sensing change detection data synthesis method that can break through the limitations of fixed categories and flexibly generate various types of disaster changes (such as building collapse, road interruption, surface flooding, vegetation degradation, etc.) to construct high-quality and diversified change detection data, thereby alleviating the problem of data scarcity in disaster scenarios. Summary of the Invention

[0003] To address the limitations of existing remote sensing change detection sample data synthesis methods, such as category restrictions and unrealistic change simulations, this invention proposes a remote sensing open-vocabulary change detection data synthesis method for disaster scenarios. This method enables the flexible generation of various disaster change types, thereby improving the generalization ability and application efficiency of change detection models in complex disaster scenarios. It can be used for remote sensing emergency monitoring tasks such as earthquakes, floods, landslides, and fires.

[0004] The present invention provides a method for synthesizing sample data for detecting changes in remote sensing open vocabulary, which mainly includes the following steps: Given a previous time-phase remote sensing image and its semantic mask, combined with the user-defined category change type, a pre-trained visual language model is used to infer the reasonable change location and category transformation relationship to obtain a shape-preserving semantic mask; then, combined with the candidate change location set, the shape of the ground object in the change area in the shape-preserving semantic mask is locally refined to generate a shape change semantic mask that has both global layout rationality and local context consistency. The shape-preserving semantic mask and the shape-changing semantic mask are fused pixel by pixel to obtain the final transformed semantic mask. The transformed semantic embedding is extracted and combined with the masked input image and the pseudo-transformation mask. The mask-to-image diffusion model is used to generate the post-temporal remote sensing image. The changed semantic mask is paired with the subsequent time-phase remote sensing image and output as a change detection synthetic sample.

[0005] Furthermore, the specific implementation of obtaining the shape-preserving semantic mask is as follows: Shape-preserving semantic mask transformation reasoning: transforming previous temporal images semantic mask and user-defined category change types The input is a pre-trained visual language model, and the output is a shape-preserving transformation scheme; specifically, the visual language model starts from... Existing categories and Select appropriate conversion source categories from the target categories. With conversion target category And estimate a conversion ratio. This indicates that it belongs to the previous phase image. In all connected regions of the category, the proportion is The region will undergo a category conversion to become The output of the shape-preserving transformation is a triplet. ; Semantic label replacement: based on the triples output by the visual language model From the previous temporal semantic mask Extract all elements belonging to the source conversion category. The connected components constitute the set of connected components. Each connected component Composed of several pixels, with all adjacent pixels having the same category label. , To determine the maximum number of connected components, select from the preset rules. The sampling ratio is Connected regions are used as target regions for shape-preserving transformation. Specific selection strategies include: sorting by connected region area in descending order and then selecting the first region. Find the largest connected component; or select the top [number] connected components after sorting by area in ascending order. A minimum connected component; or proportionally within the entire set of connected components. Randomly and uniformly sampled; the set of selected connected components is denoted as . For each of its connected components The category labels of all pixels within it are determined by Replace with At the same time, it maintains the contour shape and spatial distribution of each connected component completely unchanged, generating a shape-preserving semantic mask. : ; in, Indicates pixel position, The semantic mask that preserves shape after transformation is in Category labels at the location; This indicates the position of the preceding temporal semantic mask at the pixel location. Category labels at the location; Indicates the target category label after conversion; This represents a subset of connected components selected from the original set of connected components; express Any connected component in; symbol It means union. This indicates a union operation on the selected connected components; that is, the pixel region formed by all the selected connected components; therefore, the condition... Represents pixels It lies inside any of the selected connected components.

[0006] Furthermore, the specific implementation method for obtaining the shape change semantic mask is as follows: Shape change transformation reasoning: transforming previous phase images semantic mask Candidate change location set and category change types Simultaneously, the pre-trained visual language model is input, and the model is required to complete the following two tasks: First, analyze the visual features of various object regions in the image and determine which candidate locations... Suitable for the appearance of new category targets, and from Select a subset from First, change the position as the final shape; second, for each selected position... ,from Assign a reasonable target category Generate a set of target categories whose shapes change. ,in The maximum number of target categories to change the shape; the output of the shape change transformation is a subset of candidate locations. With the corresponding target category set ; Shape-changing region mask construction: Take The union of pixels of all bounding boxes in the region constitutes the binary mask of the shape-changing region. : ; in Indicates pixel position; Semantic mask for the preceding phase Perform the occlusion operation to obtain the occluded semantic graph: ; in This represents element-wise multiplication, where the occluded region is... The corresponding position in the middle is cleared to zero, while the original category labels of the remaining areas are preserved; at the same time, a mapping relationship from color to category is constructed. Establish a mapping between each color value in the semantic mask and its corresponding category name text, for subsequent layout into the mask diffusion model; Local shape refinement: occluded semantic graphs Category-color mapping relationship and shape change target category As a conditional input, it is fed into the layout of the mask diffusion model. Generate shape-change semantic masks: .

[0007] Furthermore, the layout-to-mask diffusion model adopts the FLUX.1 padding architecture, whose built-in T5 text encoder receives data from... The text description, composed of concatenated category names, serves as semantic conditions. The layout-to-mask diffusion model is trained on a large-scale remote sensing semantic segmentation dataset, learning prior statistical associations between different semantic categories and target shape contours. During training, the input image is masked in two ways: category-aware instance masking (masking the target area of ​​a specified category using the minimum bounding rectangle or the exact connected region boundary) and random region masking (masking areas that may span multiple categories using random rectangles). In the inference phase, the layout-to-mask diffusion model, given... Under the condition of the target category text, The coverage area generates a target shape that is compatible with the surrounding ground features and outputs the distribution of category labels for the region in the semantic mask after the transformation.

[0008] Furthermore, the previous phase images and semantic mask Input a pre-trained visual language model and require the model to identify a set of natural feature categories in an image that may exhibit pseudo-variations. And estimate a perturbation ratio for each category. Subsequently, regarding China belongs to Various categories The connected regions, according to proportion Generating pseudo-change masks by randomly selecting a portion of the region .

[0009] Furthermore, regarding the modified semantic mask... Each unique category appearing in Generate a text fragment of the category name, then input it into the CLIP text encoder. Encode the changes and output the semantic embeddings: ; in Spatial resolution and The same applies to each spatial location. The embedding vector encodes the semantic information of the category to which the corresponding pixel belongs.

[0010] Furthermore, the previous phase images By masking the pixels corresponding to the changed and pseudo-changed regions, a masked input image for image synthesis is obtained: ; Where M is the binary change mask corresponding to the shape change semantic mask, in It is a pseudo-change mask.

[0011] Furthermore, the mask-to-image diffusion model is an improvement on the Stable Diffusion architecture, replacing the original image conditional encoder with a CLIP text encoder to receive the changed semantic embedding. As a semantic condition, the modified semantics are embedded through the ControlNet structure. The spatial semantic information is fused layer by layer into the denoising U-Net to accurately guide the generation of the appearance of the corresponding category of ground objects at each spatial location; in the pseudo-change mask Coverage area, mask to image diffusion model The image is re-rendered based on the same category label as a condition, only introducing the affected category label. Guided texture and color differences ensure semantic consistency.

[0012] Furthermore, the mask-to-image diffusion model processing procedure is as follows: Mask-to-image diffusion model training: acquiring remote sensing images and its corresponding semantic annotation graph The category labels in the text are mapped to category names or category descriptions in the open vocabulary, and then input into the CLIP text encoder to obtain the corresponding category semantic vectors; then, according to the semantic annotation map... The spatial location fills the category semantic vector into the corresponding pixel or region, forming a transformed semantic embedding that is spatially aligned with the remote sensing image. ; Multi-granularity mask construction strategy: During training, a multi-granularity mask strategy is used to construct the training mask. The multi-granularity masking strategy includes at least one of instance-level masking, region-level masking, and global-level masking; wherein, instance-level masking is based on the semantic annotation graph. After extracting semantic connected components and filtering connected components with an area smaller than a preset threshold, one or more connected components are randomly selected as masking regions, and a mask is generated using the precise boundaries of the selected connected components. Regional masking involves randomly sampling rectangular regions of different scales and aspect ratios in remote sensing images as masking regions. These rectangular regions can cover one or more semantic categories, enabling the model to learn visual transition relationships across category boundaries. Global-level masking masks the entire remote sensing image with a preset probability, allowing the model to reconstruct the entire scene based solely on semantic embedding, thereby enhancing the model's ability to model global spatial layout and scene consistency. The current training mask is generated based on the instance-level mask, region-level mask, or global mask described above. and using the current training mask For remote sensing images Perform masking to obtain the mask image. :

[0013] in Indicates pixel-by-pixel multiplication; Randomly discard semantic vectors: During training, the modified semantic embeddings are processed. The semantic vector is randomly discarded, that is, the semantic embedding vectors corresponding to some spatial locations, some semantic locations, or some categories are randomly set to zero with a preset probability, resulting in perturbed semantic embeddings. :

[0014] in, The semantic vector preserves the mask, which is dimensional and Alignment, whose value consists of 0 and 1. When the value is 1, the semantic embedding vector corresponding to the corresponding position or region is retained; when the value is 0, the semantic embedding vector corresponding to the corresponding position or region is set to zero. Based on diffusion denoising training model: the original remote sensing image Encoding into the latent space of Stable Diffusion yields latent variables. ; at random time steps Towards Add Gaussian noise Noisy latent variables are obtained. :

[0015] in This represents the diffuse noise scheduling parameter, used to control the time step. The mixing ratio of the original latent variable and the noise component; Noisy latent variables Time step Mask image Training mask and the perturbated semantic embedding Input mask to image diffusion model Noise predicted by the model And it is trained using diffusion denoising loss;

[0016] in, This represents the diffusion denoising loss. Represents the latent variable Time step Gaussian noise Disrupted semantic embedding and training mask Calculate the expectation from the joint sampling; Indicates ControlNet based on The generated multi-scale spatial semantic control features The squared norm (L2) is used to measure true noise. With prediction noise The differences between them; Post-temporal remote sensing image: Image after masking Semantic embedding after changes and pseudo-change mask As a conditional signal, the trained mask is input into the image diffusion model. Generated temporal remote sensing images: .

[0017] The present invention also provides a remote sensing open vocabulary change detection sample data synthesis system, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute the method described in the above technical solution.

[0018] The principle of this invention is to use the global layout reasoning ability of a pre-trained visual language model to ensure the rationality of changes in location and category transformation, combine the local shape refinement ability of a diffusion model to ensure the spatial compatibility of the changing target, and use the open vocabulary representation ability of CLIP text embedding to break through the limitations of a fixed category vocabulary, thereby realizing the generation of synthetic data with diverse change types in disaster scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall framework of the method of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the effect generated by the method of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings.

[0022] This invention provides a method for synthesizing remote sensing open vocabulary change detection data for disaster scenarios, which mainly includes the following aspects: 1. Shape-preserving semantic mask construction. The construction method is based on the semantic mask of the previous phase image, combined with the change position and category transformation relationship obtained by the pre-trained visual language model, directly reuses the shape of the existing ground object in the previous phase image, and only replaces the semantic category label of the specified change area with the target category, so as to achieve shape preservation of the same type of ground object between different disaster types and generate the changed semantic mask.

[0023] 2. Shape Change Semantic Mask Construction. The construction method uses previous temporal images and their semantic masks as input, combined with the change location and target category inferred from a pre-trained visual language model, to generate ground object shapes with new categories on the specified change area through a layout-to-mask diffusion model: a pseudo-mask is constructed using the union of the bounding boxes of the shape change transformation area; the occluded previous temporal semantic mask, the category-color mapping relationship, and the target category text description are used as conditional inputs to drive the diffusion model to refine the local shape of the change area based on surrounding context information, generating a refined semantic mask for the change area. The diffusion model is trained on a large-scale remote sensing semantic segmentation dataset, learning the prior associations between different semantic categories and target shapes to ensure the spatial compatibility of the newly generated ground object with the surrounding environment, and finally outputting the changed semantic mask.

[0024] 3. Pseudo-change simulation. To enhance the realism of the synthesized image, the module introduces a pseudo-change simulation mechanism to simulate visual appearance changes caused by differences in imaging conditions in unchanged areas, including changes in vegetation density and surface water content. The pre-trained visual language model identifies semantic categories of possible pseudo-changes from the input image and estimates the perturbation ratio, thereby generating a pseudo-change mask. This mask, together with the real change mask, is used to guide the generation of subsequent phase images.

[0025] 4. Mask-to-Image Diffusion Model. This model uses a transformed semantic mask as a conditional signal to generate a corresponding post-temporal remote sensing image. Unlike traditional methods that use RGB-encoded color masks as control signals, this model uses a CLIP text encoder to transform the semantic mask into a transformed semantic embedding as a condition for image generation. This replaces the traditional RGB color-category binding encoding scheme, supports open-vocabulary image synthesis at the scale of thousands of categories, and eliminates the rigid binding constraint between fixed colors and semantic categories.

[0026] 5. Multi-granularity masking training strategy. The open vocabulary masking to image model is trained using an instance-level, region-level, and global-level multi-granularity masking strategy: instance-level masks erase connected components in the semantic mask, prompting the model to learn prior appearances of targets of specific categories; region-level masks use rectangles of random size and aspect ratio to occlude regions spanning multiple semantic categories, prompting the model to learn cross-category visual transitions and contextual reasoning abilities; global-level masks completely occlude the input image in extreme cases, requiring the model to reconstruct the entire image solely based on semantic conditional signals, thus strengthening the model's global scene modeling capabilities. Furthermore, a random semantic vector discarding strategy is introduced, randomly setting the semantic embeddings of local regions to zero during training, enhancing the model's robustness to semantic guidance loss.

[0027] like Figure 1 As shown, this embodiment of the invention provides a method for synthesizing remote sensing open vocabulary change detection sample data for disaster scenarios, including the following steps: Step 1: Input Preparation. Given previous temporal remote sensing images. Its corresponding semantic mask and a user-defined set of category change types ,in The semantic category of each pixel in the previous phase image is labeled in the form of color encoding, and each category corresponds to a predefined category label (such as building, road, vegetation, water body, etc.). Each element in Textual descriptions of the target semantic category names that the user wants to generate in later-phase images (such as "buildings", "vegetation", "water bodies", etc.). The scale and content are flexibly specified by the user according to the target application scenario (such as building collapse, surface flooding, etc. in disaster monitoring scenarios), without being limited by any pre-trained dataset category set.

[0028] Step 2: Candidate change location generation. Within the spatial range of the previous temporal image, candidate changes are randomly generated according to a preset strategy. A set of candidate change locations Each candidate position Represented as a rectangular bounding box, its coordinates are determined by the pixel coordinates of the top-left and bottom-right corners. The size and location of candidate bounding boxes are randomly sampled within a certain range to cover ground target areas of different scales and spatial locations in the image.

[0029] Step 3: Shape-preserving semantic mask construction, including the following sub-steps: Step 3.1: Shape-preserving semantic mask transformation inference. Transform the previous temporal image... semantic mask and user-defined category change types The input is a pre-trained visual language model (such as Qwen3-VL), and the model is required to output a shape-preserving transformation scheme. Specifically, the visual language model starts from... Existing categories and Select appropriate conversion source categories from the target categories. With conversion target category And estimate a conversion ratio. This indicates that it belongs to the previous phase image. In all connected regions of the category, the proportion is The region will undergo a category conversion to become Among them, the conversion source category With target category The mapping relationship between elements must conform to natural logic (e.g., vegetation → bare soil is reasonable, but vegetation → building is not), and this is determined by the visual language model based on its world knowledge learned from large-scale visual data. The output of the shape-preserving transformation is a triple. The transformation preserves the original outline shape of the ground features, changing only their semantic labels.

[0030] Step 3.2: Semantic label replacement. Based on the triples output by the visual language model in Step 3.1. From the previous temporal semantic mask Extract all elements belonging to the source conversion category. The connected components constitute the set of connected components. Each connected component Composed of several pixels, with all adjacent pixels having the same category label. , This represents the maximum number of connected components in this category. Based on this, selections are made according to preset rules. The sampling ratio is Connected regions are used as target regions for shape-preserving transformation. Specific selection strategies include: sorting by connected region area in descending order and then selecting the first region. The largest connected region (prioritizing larger buildings, bodies of water, and other prominent targets); or sorted by connected region area in ascending order and selecting the largest connected region. The smallest connected component (preferably small vegetation patches, fragmented patches, etc.); or proportionally within the entire set of connected components. Randomly and uniformly sampled. The set of selected connected components is denoted as . For each of these connected components The category labels of all pixels within it are determined by Replace with At the same time, the outline shape and spatial distribution of each connected component remain completely unchanged, generating a semantic mask fragment with shape preservation after transformation.

[0031] ; in, Indicates the pixel position. This indicates the semantic mask after shape preservation transformation. Category labels at the location; This indicates the position of the preceding temporal semantic mask at the pixel location. Category labels at the location; Indicates the target category label after conversion; This represents a subset of connected components selected from the original set of connected components; express Any connected component in; symbol It means union. This indicates a union operation on the selected connected components; that is, the pixel region formed by all the selected connected components. Therefore, the condition... Represents pixels It lies inside any of the selected connected components.

[0032] Step 4: Shape change semantic mask construction, including the following sub-steps: Step 4.1: Shape Change Transformation Reasoning. Transform the previous temporal image... semantic mask Candidate change location set and category change types Simultaneously, the pre-trained visual language model is input, requiring the model to complete the following two tasks: First, analyze the visual features of various object regions in the image and determine which candidate locations... Suitable for the appearance of new category targets, and from Select a subset from First, change the position as the final shape; second, for each selected position... ,from Assign a reasonable target category Generate a set of target categories whose shapes change. ,in This represents the maximum number of target categories for shape change. Shape change inference must satisfy the following constraints: when the model outputs, no more than two categories should be selected in the same image, and background categories (such as grass or bare soil) should be prioritized as the locations where new targets (such as buildings or trees) appear to ensure the rationality of spatial logic. The output of the shape change transformation is a subset of candidate locations. With the corresponding target category set This transformation introduces new ground features into the later-phase image, whose shapes are not directly related to the shapes of the corresponding areas in the earlier-phase image.

[0033] Step 4.2: Constructing the shape-changing region mask. (Take...) The union of pixels of all bounding boxes in the region constitutes the binary mask of the shape-changing region. : ; in Indicates the pixel position. Semantic mask for the preceding phase Perform the occlusion operation to obtain the occluded semantic graph: ; in This represents element-wise multiplication, where the occluded region is... The corresponding position in the middle is cleared to zero, while the original category labels of the remaining areas are preserved. Simultaneously, a mapping relationship between color and category is constructed. Establish a mapping between each color value in the semantic mask and its corresponding category name text for use by the subsequent diffusion model.

[0034] Step 4.3: Local Shape Refinement. The occluded semantic map obtained in Step 4.1 is then refined. Mapping relationship between color and category and shape change target category As a conditional input, it is fed into the layout of the mask diffusion model. Generate a fine-grained semantic mask for the shape-changing region: ; The layout-to-mask diffusion model uses a FLUX.1 padding architecture, whose built-in T5 text encoder receives data from... The text description, composed of concatenated category names, serves as semantic conditions. The model is trained on a large-scale remote sensing semantic segmentation dataset, learning prior statistical associations between different semantic categories and target shape contours. During training, the input image is masked in two ways: category-aware instance masking (masking the target region of a specified category with a minimum bounding rectangle or exact connected region boundary) and random region masking (masking regions that may span multiple categories with random rectangles). During the inference phase, the model, given... Under the condition of the target category text, The coverage area generates a target shape that is spatially compatible with the surrounding ground features, and outputs the distribution of category labels for that area in the transformed semantic mask.

[0035] Step 5: Semantic mask fusion after transformation. The mask generated by the shape-preserving transformation is then used. Masks generated by shape transformation Perform pixel-by-pixel fusion to obtain the final transformed semantic mask: ; Among these, if both transformations cover the same pixel, the shape change transformation has higher priority. (This is achieved through comparison.) and The differences between the labels of each category are used to generate a binary transformation mask pixel by pixel: ; Step 6: Generating a pseudo-change mask. To simulate the visual appearance changes in real remote sensing images caused by differences in imaging conditions such as illumination, season, and sensor parameters, the previous phase image is used as a mask. and semantic mask Input the pre-trained visual language model and require the model to identify a set of natural land cover categories in an image that may exhibit pseudo-variations. (Categories such as vegetation and water bodies that are significantly affected by sunlight and seasons), and estimate a perturbation ratio for each category. Subsequently, regarding China belongs to Various categories The connected regions, according to proportion Generating pseudo-change masks by randomly selecting a portion of the region The area identified by the mask will be re-rendered with the same category label when the subsequent phase image is generated, but it introduces different textures and color variations in visual appearance.

[0036] Step 7: Semantic Mask Text Embedding Extraction. Extract the transformed semantic mask text. The category label for each pixel location in the mask is converted into a corresponding category name text description and organized into a text annotation sequence with the same spatial resolution as the original image. Specifically, for each unique category appearing in the mask... This generates a text segment in the form of "category name", which is then input into the CLIP text encoder. Encode the changes and output the semantic embeddings: ; in Spatial resolution and The same applies to each spatial location. The embedding vector encodes the semantic information of the category to which the corresponding pixel belongs. This embedding serves as a conditional signal for the image generation model, representing arbitrary semantic labels without the need for color-category binding.

[0037] Step 8: Input image masking, converting previous phase images... By masking the pixels corresponding to the changed and pseudo-changed regions, a masked input image for image synthesis is obtained: ; Where M is the binary transformation mask obtained in step 5 after semantic mask fusion. For areas of change in coverage, the model needs to generate entirely new appearances of ground features whose semantic categories have altered; In the pseudo-change areas covered, the model needs to keep the category label unchanged but introduce subtle visual differences (such as changes in vegetation texture density, changes in water hue, etc.); the remaining unchanged areas directly retain the original pixel values ​​of the previous phase image.

[0038] Step 9: Open vocabulary post-temporal image synthesis, including the following sub-steps: Step 9.1, Mask-to-Image Diffusion Model Training: Before synthesis, a mask-to-image diffusion model needs to be trained. After one training iteration, the model can support subsequent generation tasks. This is used to acquire remote sensing images from large-scale single-temporal remote sensing images. and its corresponding semantic annotation graph The category labels in the text are mapped to category names or category descriptions in the open vocabulary, and then input into the CLIP text encoder to obtain the corresponding category semantic vectors; then, according to the semantic annotation map... The spatial location fills the category semantic vector into the corresponding pixel or region, forming a semantic embedding that is spatially aligned with the remote sensing image. .

[0039] Step 9.2, Constructing the Training Mask Using a Multi-Granularity Strategy: During training, a multi-granularity mask strategy is used to construct the training mask. The multi-granularity masking strategy includes at least one of instance-level masking, region-level masking, and global-level masking. Instance-level masking extracts semantic connected components from the semantic annotation graph S, filters connected components with areas smaller than a preset threshold, randomly selects one or more connected components as masking regions, and generates a mask using the precise boundaries of the selected connected components.

[0040] Regional masking involves randomly sampling rectangular regions of different scales and aspect ratios in remote sensing images as masking regions. These rectangular regions can cover one or more semantic categories, enabling the model to learn visual transition relationships across category boundaries.

[0041] Global-level masking masks the entire remote sensing image with a preset probability, enabling the model to reconstruct the entire scene based solely on semantic embedding, thereby enhancing the model's ability to model global spatial layout and scene consistency.

[0042] The current training mask is generated based on the instance-level mask, region-level mask, or global mask described above. and using the training mask For remote sensing images Perform masking to obtain the mask image. :

[0043] in This indicates pixel-by-pixel multiplication.

[0044] Step 9.3, perform random discarding of semantic vectors: During training, the semantic embeddings are... The semantic vector is randomly discarded, that is, the semantic embedding vectors corresponding to some spatial locations, some semantic locations, or some categories are randomly set to zero with a preset probability, resulting in perturbed semantic embeddings. :

[0045] in, The semantic vector preserves the mask, which is dimensional and Alignment, with values ​​consisting of 0 and 1. Positions with a value of 1 retain the semantic embedding vector corresponding to that position or region; when the value is 0, the semantic embedding vector corresponding to that position or region is set to zero, and the semantic vector is randomly discarded. This allows the model to infer a reasonable appearance by combining the semantic information of the surrounding region and the context of the visible image when the semantic conditions are incomplete, thus improving the stability of generation.

[0046] Step 9.4, Training the model based on diffusion denoising: The original remote sensing image... Encoding into the latent space of Stable Diffusion yields latent variables. ; at random time steps Towards Add Gaussian noise Noisy latent variables are obtained. :

[0047] in This represents the diffuse noise scheduling parameter, used to control the time step. The mixing ratio of the original latent variable and the noise component.

[0048] Noisy latent variables Time step Mask image Training mask and the perturbated semantic embedding Input mask to image diffusion model Noise predicted by the model And it is trained using diffusion denoising loss.

[0049]

[0050] in, This represents the diffusion denoising loss. Represents the latent variable Time step Gaussian noise Disrupted semantic embedding and training mask The expectation is obtained from the joint sampling. Indicates ControlNet based on The generated multi-scale spatial semantic control features. The squared norm (L2) is used to measure true noise. With prediction noise The differences between them.

[0051] During training, the diffusion denoising loss is minimized. This enables the model to learn to recover remote sensing image content consistent with semantic conditions from noisy latent variables, given a masked image, a training mask, and semantic control conditions. After training, the mask-to-image diffusion model has the ability to generate post-temporal remote sensing images based on pseudo-transformation masks and transformed semantic embeddings.

[0052] Step 9.5, Generate post-temporal remote sensing image: Image after masking. Semantic embedding after changes and pseudo-change mask As a conditional signal, the input mask is fed into the trained image diffusion model. Generated temporal remote sensing images: ; The modified semantic embedding The modified semantic embedding obtained in step 7 is compared with that in step 9.3. different, Use only during training.

[0053] The mask-to-image diffusion model is an improvement on the Stable Diffusion architecture, replacing the original image conditional encoder with a CLIP text encoder to receive... As a semantic condition, and through the ControlNet structure... The spatial semantic information is fused layer by layer into the denoising U-Net to accurately guide the generation of the appearance of the corresponding category of ground objects at each spatial location. In the pseudo-change mask... Coverage area, model with The image is re-rendered based on the same category label as a condition, only introducing the affected category label. Guided texture and color differences ensure semantic consistency.

[0054] Step 10: Pair the changed semantic mask with the subsequent time-phase remote sensing image and output it as a change detection synthetic sample.

[0055] Step 11: Batch generation of open-vocabulary synthetic data. Users define arbitrary combinations of change types (including parameters such as source category, target category, and change scale). Following the complete workflow from Steps 1 to 10, synthetic change detection datasets for different disaster application scenarios are generated in batches after a single model training iteration. The synthetic data includes: pre-temporal remote sensing images. Post-temporal remote sensing images Previous phase semantic mask Semantic mask after transformation Binary transformation mask pseudo-change mask And a set of semantic category transformation label pairs. Users do not need to retrain the model for each new combination of category change types; they can flexibly expand the category coverage of synthetic data simply by adjusting the input parameters.

[0056] To verify the effectiveness of the proposed method, experiments were conducted on two tasks: binary change detection and semantic change detection. Specifically, ChangeFormer was used as the evaluation model for binary change detection, and Change3D was used as the evaluation model for semantic change detection. After training on the dataset synthesized in this invention, the models were transferred to the test set of real benchmark datasets for evaluation. The binary change detection experiments used four publicly available benchmark datasets: LEVIR-CD, WHU-CD, DSIFN-CD, and SEC-BCD. The semantic change detection experiments used two publicly available benchmark datasets: SECOND and HRSCD. The synthesized data were constructed based on two remote sensing semantic segmentation datasets, OpenEarthMap and FLAIR, maintaining consistency with the comparison datasets. Three synthesized datasets were generated for verification: CoVo-BCD for building change detection, CoVo-SEC for semantic change detection (based on the SECOND category system), and CoVo-HR for semantic change detection (based on the HRSCD category system). Each synthesized dataset contained 10,000 samples. Figure 2The presentation includes three datasets that qualitatively demonstrate that the proposed method can obtain both the altered remote sensing image and the altered semantic map in a single step, with the semantic map exhibiting good quality. The CoVo-BCD dataset illustrates the complex changes to buildings caused by the proposed method, while the CoVo-SEC and CoVo-HR datasets demonstrate the method's ability to model changes in multiple land cover categories and fine-grained changes in high-resolution scenes, respectively. Furthermore, the objective experimental results are shown in Tables 1 and 2.

[0057] Table 1. Experimental results of binary change detection

[0058] Table 2. Experimental results of semantic change detection

[0059] Experimental Validation: For the binary change detection task, zero-shot cross-domain transfer (generalization) experiments were conducted using four public datasets: LEVIR-CD, WHU-CD, DSIFN-CD, and SECOND. Experimental results show that, compared to existing methods such as WHU-GCD, FSC-180K, and Changen2-S1, the proposed method (CoVo-BCD) exhibits superior generalization performance across all experimental settings, achieving performance improvements of 6.64% in mean Intersection over Union (mIoU) and 8.90% in mean F1 score.

[0060] For the semantic change detection task, zero-shot cross-domain transfer (generalization) experiments were conducted using two public datasets, SECOND and HRSCD. Experimental results show that, compared to existing methods such as WHU-GCD, FSC-180k, and Changen2-S9, the proposed method (CoVo-SEC / HR) exhibits superior generalization performance across all experimental settings, achieving performance improvements of 6.78%, 0.78%, and 11.74% in average F1 score, average intersection-over-union ratio (mIoU), and average semantic change score (SCS), respectively.

[0061] On the other hand, embodiments of the present invention also provide a remote sensing open vocabulary change detection sample data synthesis system, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a remote sensing open vocabulary change detection sample data synthesis method as described in the above technical solution.

[0062] Finally, it should be noted that those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, the intent of this invention is also encompassed within these modifications and variations.

Claims

1. A method for synthesizing sample data for remote sensing open vocabulary change detection, characterized in that, Includes the following steps: Given a previous time-phase remote sensing image and its semantic mask, combined with the user-defined category change type, a pre-trained visual language model is used to infer the reasonable change location and category transformation relationship to obtain a shape-preserving semantic mask; then, combined with the candidate change location set, the shape of the ground object in the change area in the shape-preserving semantic mask is locally refined to generate a shape change semantic mask that has both global layout rationality and local context consistency. The specific implementation of obtaining the shape-preserving semantic mask is as follows: Shape-preserving semantic mask transformation reasoning: transforming previous temporal images semantic mask and user-defined category change types Input a pre-trained visual language model and output a shape-preserving transformation scheme; the visual language model starts from... Existing categories and Select appropriate conversion source categories from the target categories. With conversion target category And estimate a conversion ratio. This indicates that it belongs to the previous phase image. In all connected regions of the category, the proportion is The region will undergo a category conversion to become The output of the shape-preserving transformation is a triplet. ; Semantic label replacement: based on the triples output by the visual language model From the previous temporal semantic mask Extract all elements belonging to the source conversion category. The connected components constitute the set of connected components. Each connected component Composed of several pixels, with all adjacent pixels having the same category label. , To determine the maximum number of connected components, select from the preset rules. The sampling ratio is Connected regions are used as target regions for shape-preserving transformation. Specific selection strategies include: sorting by connected region area in descending order and then selecting the first region. Find the largest connected component; or select the top [number] connected components after sorting by area in ascending order. A minimum connected component; or proportionally within the entire set of connected components. Randomly and uniformly sampled; the set of selected connected components is denoted as . For each of its connected components The category labels of all pixels within it are determined by Replace with At the same time, it maintains the contour shape and spatial distribution of each connected component completely unchanged, generating a shape-preserving semantic mask. : ； in, Indicates pixel position, The semantic mask that preserves shape after transformation is in Category labels at the location; This indicates the position of the preceding temporal semantic mask at the pixel location. Category labels at the location; Indicates the target category label after conversion; This represents a subset of connected components selected from the original set of connected components; express Any connected component in; symbol It means union. This represents the pixel region formed by performing a union operation on the selected connected components; condition. Represents pixels It lies within any of the selected connected components; The specific implementation method for obtaining the shape change semantic mask is as follows: Shape change transformation reasoning: transforming previous phase images semantic mask Candidate change location set and category change types Simultaneously, the pre-trained visual language model is input, and the model is required to complete the following two tasks: First, analyze the visual features of various object regions in the image and determine which candidate locations... Suitable for the appearance of new category targets, and from Select a subset from First, change the position as the final shape; second, for each selected position... ,from Assign a reasonable target category Generate a set of target categories whose shapes change. ,in The maximum number of target categories to change the shape; the output of the shape change transformation is a subset of candidate locations. With the corresponding target category set ; Shape-changing region mask construction: Take The union of pixels of all bounding boxes in the region constitutes the binary mask of the shape-changing region. : ； in Indicates pixel position; Semantic mask for the preceding phase Perform the occlusion operation to obtain the occluded semantic graph: ； in This represents element-wise multiplication, where the occluded region is... The corresponding position in the middle is cleared to zero, while the original category labels of the remaining areas are preserved; at the same time, a mapping relationship from color to category is constructed. Establish a mapping between each color value in the semantic mask and its corresponding category name text, for subsequent layout into the mask diffusion model; Local shape refinement: occluded semantic graphs Category-color mapping relationship and shape change target category As a conditional input, it is fed into the layout of the mask diffusion model. Generate shape-change semantic masks: ； The shape-preserving semantic mask and the shape-changing semantic mask are fused pixel by pixel to obtain the final transformed semantic mask. The transformed semantic embedding is extracted and combined with the masked input image and the pseudo-transformation mask. The mask-to-image diffusion model is used to generate the post-temporal remote sensing image. The changed semantic mask is paired with the subsequent time-phase remote sensing image and output as a change detection synthetic sample.

2. The method according to claim 1, characterized in that: The layout-to-mask diffusion model uses a FLUX.1 padding architecture, whose built-in T5 text encoder receives data from... The text description, composed of concatenated category names, serves as semantic conditions. The layout-to-mask diffusion model is trained on a large-scale remote sensing semantic segmentation dataset, learning prior statistical associations between different semantic categories and target shape contours. During training, the input image is masked in two ways: category-aware instance masking and random region masking, which respectively mask the target area of ​​a specified category using the minimum bounding rectangle or the exact connected region boundary and mask areas that may span multiple categories using random rectangles. In the inference phase, the layout-to-mask diffusion model, given... Under the condition of the target category text, The coverage area generates a target shape that is compatible with the surrounding ground features and outputs the distribution of category labels for the region in the semantic mask after the transformation.

3. The method according to claim 1, characterized in that: Previous phase images and semantic mask Input a pre-trained visual language model and require the model to identify a set of natural feature categories in an image that may exhibit pseudo-variations. And estimate a perturbation ratio for each category. ;right China belongs to Various categories The connected regions, according to proportion Generating pseudo-change masks by randomly selecting a portion of the region .

4. The method according to claim 1, characterized in that: Transformed semantic mask Each unique category appearing in Generate a text fragment of the category name and input it into the CLIP text encoder. Encode and output the transformed semantic embedding. : ； in Spatial resolution and The same applies to each spatial location. The embedding vector encodes the semantic information of the category to which the corresponding pixel belongs.

5. The method according to claim 1, characterized in that: Previous phase images By masking the pixels corresponding to the changed and pseudo-changed regions, a masked input image for image synthesis is obtained. : ； Where M is the binary change mask corresponding to the shape change semantic mask, in It is a pseudo-change mask.

6. The method according to claim 1, characterized in that: The mask-to-image diffusion model is an improvement on the StableDiffusion architecture, replacing the original image conditional encoder with a CLIP text encoder to receive the changed semantic embeddings. As a semantic condition, the modified semantics are embedded through the ControlNet structure. The spatial semantic information is fused layer by layer into the denoising U-Net to accurately guide the generation of the appearance of the corresponding category of ground objects at each spatial location; in the pseudo-change mask Coverage area, mask to image diffusion model The image is re-rendered based on the same category label as a condition, only introducing the affected category label. Guided texture and color differences ensure semantic consistency.

7. The method according to claim 1, characterized in that: The process of masking to image diffusion model is as follows: Mask-to-image diffusion model training: acquiring remote sensing images and its corresponding semantic annotation graph The category labels in the text are mapped to category names or category descriptions in the open vocabulary, and then input into the CLIP text encoder to obtain the corresponding category semantic vectors; then, according to the semantic annotation map... The spatial location fills the category semantic vector into the corresponding pixel or region, forming a transformed semantic embedding that is spatially aligned with the remote sensing image. ; Multi-granularity mask construction strategy: During training, a multi-granularity mask strategy is used to construct the training mask. The multi-granularity masking strategy includes at least one of instance-level masking, region-level masking, and global-level masking; wherein, instance-level masking is based on the semantic annotation graph. After extracting semantic connected components and filtering connected components with an area smaller than a preset threshold, one or more connected components are randomly selected as masking regions, and a mask is generated using the precise boundaries of the selected connected components. Regional masking involves randomly sampling rectangular regions of different scales and aspect ratios in remote sensing images as masking regions. These rectangular regions can cover one or more semantic categories, enabling the model to learn visual transition relationships across category boundaries. Global-level masking masks the entire remote sensing image with a preset probability, allowing the model to reconstruct the entire scene based solely on semantic embedding, thereby enhancing the model's ability to model global spatial layout and scene consistency. The current training mask is generated based on the instance-level mask, region-level mask, or global mask described above. and using the current training mask For remote sensing images Perform masking to obtain the mask image. : in Indicates pixel-by-pixel multiplication; Randomly discard semantic vectors: During training, the modified semantic embeddings are processed. The semantic vectors are randomly discarded by setting the semantic embedding vectors corresponding to some spatial locations, some semantic locations, or some categories to zero with a preset probability, resulting in perturbed semantic embeddings. : in, The semantic vector preserves the mask, which is dimensional and Alignment, whose value consists of 0 and 1. When the value is 1, the semantic embedding vector corresponding to the corresponding position or region is retained; when the value is 0, the semantic embedding vector corresponding to the corresponding position or region is set to zero. Based on diffusion denoising training model: the original remote sensing image Encoding into the latent space of Stable Diffusion yields latent variables. ; at random time steps Towards Add Gaussian noise Noisy latent variables are obtained. : in This represents the diffuse noise scheduling parameter, used to control the time step. The mixing ratio of the original latent variable and the noise component; Noisy latent variables Time step Mask image Training mask and the perturbated semantic embedding Input mask to image diffusion model Noise predicted by the model And it is trained using diffusion denoising loss; in, This represents the diffusion denoising loss. Represents the latent variable Time step Gaussian noise Disrupted semantic embedding and training mask Calculate the expectation from the joint sampling; Indicates ControlNet based on The generated multi-scale spatial semantic control features The squared norm (L2) is used to measure true noise. With prediction noise The differences between them; Post-temporal remote sensing image: Image after masking Semantic embedding after changes and pseudo-change mask As a conditional signal, the trained mask is input into the image diffusion model. Generated temporal remote sensing images: 。 8. A remote sensing open vocabulary change detection sample data synthesis system, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to invoke the program instructions in the memory to execute the method as described in any one of claims 1-7.

Images