A remote sensing image general change detection method based on open vocabulary segmentation

Abstract

The application provides a remote sensing image change detection method and system based on open vocabulary semantic segmentation. The method decomposes the open vocabulary change detection task into two subtasks of class-independent potential change region proposal and dual-phase open vocabulary semantic segmentation executed in parallel, and constructs a unified framework which does not need to be retrained for the target change detection dataset in the reasoning stage. The core is that a general processing flow is designed to enable the pre-trained open vocabulary semantic segmentation model to be combined with the pre-trained image feature extraction network; a feature upsampling scheme based on inverse mapping constraint is proposed, which effectively improves the feature quality and the adaptability of the framework to the change of the input resolution through the inverse mapping constraint based on the Fourier neural operator network (FNO network); and a post-processing method including change probability suppression processing based on dual-phase background probability and fusion processing of change region probability graph and semantic category probability graph is designed to realize stable change detection across multi-category features. The application has superior accuracy, adaptability to different category vocabularies and supports different module combinations.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and computer vision technology, specifically relating to a remote sensing image change detection method and system applicable to open vocabulary environments, which does not require retraining on the target change detection dataset during the inference stage. It can be used in scenarios such as remote sensing image interpretation, geographic information updating, disaster monitoring, and urban change analysis. Background Technology

[0002] Remote sensing image change detection aims to identify changes in the land surface of the same geographic area at different points in time, and is widely used in urban planning, disaster assessment, resource monitoring and other fields. Traditional change detection methods are generally divided into two categories: binary change detection, which only determines whether a change has occurred; and semantic change detection, which not only detects changes, but also identifies the semantic categories before and after the change (such as "from vegetation to buildings").

[0003] Existing semantic change detection methods typically rely on predefined, closed, fixed sets of categories for training and prediction. However, in real-world applications, users may focus on dynamic semantic categories, making traditional methods difficult to meet flexible and varied needs. Open-lexical change detection aims to address this issue by allowing users to input semantic concepts (open lexicalities) to guide change detection.

[0004] Currently, an intuitive approach to open-vocabulary change detection is to adopt a "post-classification comparison" paradigm: that is, perform open-vocabulary semantic segmentation on the two-temporal images separately, and then compare the segmentation results. However, directly applying this paradigm to open-vocabulary change detection faces significant challenges: First, how to achieve the extraction of general, category-independent potential change regions? Second, how to accurately identify target category objects in two-temporal images? Furthermore, how to utilize ground cover semantic information to guide the general change region proposal process?

[0005] Furthermore, existing open-vocabulary semantic segmentation models often suffer from oversegmentation or blurred boundaries due to instance proposal steps (such as using SAM) when processing high-resolution remote sensing images, affecting the accuracy of subsequent change analysis. Meanwhile, the differences in image resolution from different data sources and sensors also place higher demands on the stability of change detection models.

[0006] Therefore, there is an urgent need for a remote sensing image change detection method that can simultaneously obtain category-independent change regions, open-vocabulary semantic categories, and cross-resolution stable feature representations without requiring retraining for the target category. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a remote sensing image change detection method and system that does not require training for a specific change detection task, can handle multiple semantic categories, and is adaptive to changes in input resolution.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A remote sensing image change detection method based on open-vocabulary semantic segmentation is characterized by decoupling the open-vocabulary change detection task into two parallel and complementary sub-tasks: category-independent latent change region proposal and bi-temporal open-vocabulary semantic segmentation. This method, through a unified workflow, enables the pre-trained open-vocabulary semantic segmentation model to be combined with an image feature extraction network, thereby forming an open-vocabulary change detection workflow that does not require retraining for the target change detection category. Its core steps include:

[0010] 1. Feature Extraction and Upsampling: Feature maps of the two-phase images are extracted using image feature extraction networks (such as the DINO series models). A feature upsampling function (Inv-FeatUp) based on inverse mapping constraints upsamples the feature maps to the target resolution. This upsampling function is trained based on inverse mapping constraints, enabling it to reconstruct accurate details from low-resolution features, ensuring that the upsampled features are semantically faithful to the original image and adaptable to resolution changes.

[0011] 2. Category-independent latent change region proposal: Perform multi-scale feature similarity comparison on the upsampled bi-temporal feature map to generate a category-independent probability map that identifies latent change regions.

[0012] 3. Dual-temporal Open-vocabulary Semantic Segmentation: A pre-trained open-vocabulary semantic segmentation model is used to process dual-temporal images. The user provides the target category vocabulary (open vocabulary set), and the model outputs a probability map of each pixel belonging to each category (including the background).

[0013] 4. Post-fusion processing: The potential change region probability map obtained in step 2 is fused with the bi-temporal semantic segmentation probability map obtained in step 3. This process includes change probability suppression processing based on bi-temporal background probability and fusion processing of the change region probability map and semantic category probability map, aiming to stabilize multi-class discrimination and improve the overall consistency of the results, and finally outputting a change detection map with open-vocabulary semantic labels.

[0014] Accordingly, the present invention also provides a system for implementing the above method, comprising: an image feature extraction module, a feature upsampling module (including Inv-FeatUp), a change region proposal module, an open vocabulary semantic segmentation module, and a fusion post-processing module.

[0015] The beneficial effects of this invention are:

[0016] 1. No need for retraining on target change detection dataset during inference phase and generality: The framework of this invention does not require retraining on target change detection dataset during the inference phase, achieving plug-and-play functionality; it also supports combinations of various image encoders and open vocabulary models.

[0017] 2. Accuracy and Stability: By introducing the Inv-FeatUp upsampling unit, the problem caused by resolution differences is effectively alleviated, improving the accuracy of feature comparison. Simultaneously, the designed fusion post-processing method effectively integrates change region information and semantic information. Under the same input image and category vocabulary, compared to the open vocabulary change detection process that only uses post-classification comparison, this invention improves the consistency of change region localization and the stability of semantic change category discrimination.

[0018] 3. Open vocabulary capability: Users can define the change categories they are interested in by inputting a set of semantic concepts, breaking through the limitations of fixed categories in traditional methods and improving the flexibility of applications.

[0019] 4. Handling Multi-Category Ground Feature Scenes: This method demonstrates excellent recognition capabilities for multi-category ground features, fragmented targets in high-resolution scenes, and large-scale continuous ground features, with clearer boundary distinctions. The technical solution of this invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0021] Figure 1 The overall flowchart of the Free-CD framework provided in the embodiments of the present invention is shown.

[0022] Figure 2 This is a schematic diagram of the structure and training of the Inv-FeatUp feature upsampling device. Detailed Implementation

[0023] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0024] The overall flow of the method provided in this embodiment of the invention is as follows: Figure 1As shown. The specific implementation steps are as follows:

[0025] Step S1: Feature Extraction

[0026] Input registered dual-temporal remote sensing images and ,in This represents the first phase image. This represents the second-phase image. A pre-trained image feature extraction network is used. (For example, using the DINOv2-ViT-L / 14 model) Image features are extracted to obtain the first phase low-resolution feature map. Second phase low-resolution feature map .

[0027] Step S2: Feature upsampling based on inverse mapping constraints (Inv-FeatUp)

[0028] Input F^(1) and F^(2) into the pre-trained feature upsampling module The first phase upsampled feature map is obtained. Second phase upsampling feature map Inv-FeatUp is one of the key innovations. Its core lies in using an inverse mapping constraint to constrain the upsampling process, ensuring that the upsampled features can accurately reconstruct the original image content and suppress reconstruction artifacts.

[0029] Inv-FeatUp training (see) Figure 2 It includes two stages:

[0030] Phase 1 (FNO Network Training): The goal is to learn the image encoder. (e.g., the inverse mapping of ViT). We use a Fourier neural operator network (FNO network for short). To approximate this inverse mapping. Training is performed on a low-resolution dataset, where... It is a high-resolution image. The downsampled version, These are the corresponding low-resolution features. The inverse mapping training loss is:

[0031]

[0032] in Represents structural similarity loss. These are the weighting coefficients. Due to the inherent grid independence of Fourier neural operators, the learned mappings are subject to resolution consistency constraints.

[0033] Phase 2 (Upsampler Training): Fixing the trained FNO network Training the upsampling network and learnable downsampling networks The goal is to reconstruct the original image using the upsampled features. The training loss for the upsampler is:

[0034]

[0035] in It is to reconstruct the image. It is the image reconstruction loss. It is feature consistency loss. These are the weighting coefficients. After two stages of training, Inv-FeatUp can generate higher resolution features that are richer in detail and adaptable to different resolutions.

[0036] Step S3: Proposal of Category-Independent Potential Variation Regions

[0037] For high-resolution feature maps and Feature similarity (such as cosine similarity) is compared across multiple scales. Specifically, pixel-wise similarity is calculated as follows:

[0038]

[0039] Where x represents the pixel position, and · represents the vector dot product. This represents the L2 norm. The feature difference plot is as follows:

[0040]

[0041] right After normalization and spatial filtering, a probability map of potential change regions independent of class is obtained. Each pixel value represents the probability that the position will change.

[0042] Step S4: Bi-temporal open-vocabulary semantic segmentation

[0043] In parallel, the original two-phase images and Input a pre-trained open-vocabulary semantic segmentation model (e.g., SegEarth-OV). Input target category text set. ,in This represents the background category. The model outputs a semantic category probability map for each image. This indicates that the image at time t belongs to category x. The probability of, where .

[0044] Step S5: Post-fusion processing

[0045] This is a key step in effectively integrating change information with semantic information:

[0046] 1. Probability suppression processing based on changes in dual-temporal background probability: utilizing... and The background category probability is used to correct the change probability map. The correction formula is:

[0047]

[0048] in and These represent the background class probabilities at position x in the dual-temporal image. If a position is predicted as having a high background probability in both temporal phases, its change probability is reduced. The corrected image is shown below. .

[0049] 2. Fusion processing of the change region probability map and the semantic category probability map: Calculate the final semantic change probability map. Two implementation methods are provided:

[0050] Method 1 (Hard Discriminant Fusion): For each position x, if and If a semantic change occurs, the change category is determined to be the corresponding label pair in C; otherwise, no change is considered. Here, τ is a preset threshold.

[0051] Method 2 (Probability Fusion): , where a≠b, represents the probability of changing from category a to category b.

[0052] 3. Threshold segmentation: Apply a threshold to the semantic change probability map (such as using Otsu's automatic thresholding method) or directly take argmax to obtain the mask of the changed region and the semantic label corresponding to each changed region.

[0053] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A remote sensing image change detection method based on open-vocabulary semantic segmentation, characterized in that, Includes the following steps: S1. Obtain the registered first-phase remote sensing image and the second-phase remote sensing image, and extract features through a pre-trained image feature extraction network to obtain the first-phase feature map and the second-phase feature map; S2. Input the first time-phase feature map and the second time-phase feature map into the pre-trained feature upsampling module respectively to obtain the first time-phase upsampling feature map and the second time-phase upsampling feature map. The feature upsampling module is trained through inverse mapping constraints so that the image result reconstructed by the upsampling features through the preset inverse mapping network satisfies the consistency constraint between the image and the corresponding input image. S3. Calculate the feature difference between the first phase upsampled feature map and the second phase upsampled feature map to obtain a feature difference map, and generate a category-independent probability map of potential change regions based on the feature difference map; S4. Input the first temporal remote sensing image and the second temporal remote sensing image into the pre-trained open vocabulary semantic segmentation model, and receive the target category text set to obtain the first temporal semantic category probability map and the second temporal semantic category probability map; S5. Perform fusion processing on the first temporal semantic category probability map, the second temporal semantic category probability map, and the potential change region probability map to obtain a change region mask and semantic change category information corresponding to the change region mask.

2. The method according to claim 1, characterized in that, The inverse mapping constraint mentioned in step S2 includes: Low spatial resolution features of low-resolution training images are extracted using a pre-trained image feature extraction network. The inverse mapping network is trained to reconstruct the low-resolution training image based on the low spatial resolution features; When training the feature upsampling module, the inverse mapping network is fixed, and the parameters of the feature upsampling module are updated according to the reconstruction loss between the reconstruction result of the upsampled features by the inverse mapping network and the target image.

3. A training method for the feature upsampling module in the method of claim 1, characterized in that, Includes the following steps: S31. Acquire training images and perform downsampling processing on the training images to obtain low-resolution training images; S32. Extract low spatial resolution features of the low-resolution training image using a pre-trained image feature extraction network; S33. Train an inverse mapping network based on a Fourier neural operator network (FNO network) to reconstruct the low-resolution training image based on the low spatial resolution features; S34. Fix the trained inverse mapping network and connect the feature upsampling module to be trained in series with the inverse mapping network; S35. Input the low spatial resolution features into the feature upsampling module to obtain upsampled features, and then use the inverse mapping network to obtain a reconstructed image based on the upsampled features; S36. Update the parameters of the feature upsampling module based on the image reconstruction loss between the reconstructed image and the target training image, and the feature consistency loss between the upsampled features and the reference features.

4. The method according to claim 1, characterized in that, Step S3 includes: Pixel-by-pixel similarity is calculated on the first phase upsampled feature map and the second phase upsampled feature map to obtain a similarity map; A feature difference map is generated based on the similarity map; The feature difference map is processed by at least one of normalization, spatial filtering, and multi-scale fusion to obtain the probability map of the potential change region.

5. The method according to claim 1, characterized in that, Step S5 includes: Based on the background category probabilities in the first and second temporal semantic category probabilities, the potential change region probability map is subjected to change probability suppression processing based on the dual-temporal background probabilities to obtain the corrected change region probability map. Based on the corrected probability map of the change region, the probability map of the first phase semantic category, and the probability map of the second phase semantic category, calculate the semantic change probability map; The change region mask and the corresponding semantic change category information are determined based on the semantic change probability map.

6. The method according to claim 5, characterized in that, The semantic change category information includes at least one of the following: the category before the change, the category after the change, and the category pair before and after the change.

7. The method according to claim 1, characterized in that, The image feature extraction network includes a convolutional neural network, a visual transformer network, a visual encoder in a visual-language pre-trained model, or an image encoding network pre-trained based on at least one of supervised learning, self-supervised learning, and weakly supervised learning.

8. The method according to claim 1, characterized in that, The open-vocabulary semantic segmentation model is a segmentation model that can output pixel-level semantic category probabilities based on a set of target category texts.

9. A remote sensing image change detection system based on open-vocabulary semantic segmentation, characterized in that, include: The image feature extraction module is used to extract features from the first time-phase remote sensing image and the second time-phase remote sensing image respectively, to obtain the first time-phase feature map and the second time-phase feature map; The feature upsampling module is used to perform feature upsampling on the first phase feature map and the second phase feature map respectively to obtain the first phase upsampled feature map and the second phase upsampled feature map; The change region proposal module is used to generate a category-independent probability map of potential change regions based on the first phase upsampled feature map and the second phase upsampled feature map. The open-vocabulary semantic segmentation module is used to perform open-vocabulary semantic segmentation on the first-phase remote sensing image and the second-phase remote sensing image according to the target category text set, and obtain the first-phase semantic category probability map and the second-phase semantic category probability map. The fusion post-processing module is used to output the change region mask and the corresponding semantic change category information based on the potential change region probability map, the first phase semantic category probability map, and the second phase semantic category probability map.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images