Remote sensing image semantic change detection method for non-strictly open words

Abstract

The invention discloses a remote sensing image semantic change detection method for non-strict open words. The method comprises the following steps: acquiring a dual-temporal remote sensing image; and inputting the dual-temporal remote sensing image into the trained remote sensing image semantic change detection model, and outputting a corresponding binary change mask image and a semantic change mask image. The semantic change detection model is constructed on the basis of a Semantic-CD framework, and comprises the following four components: a CLIP vision encoder with BCSF Adapter used for extracting double-temporal image features, a Prompt Learner used for generating visual embedding of text semantics, a BCD decoder used for generating a binary detection mask, and an SCD decoder used for marking semantic tags on the binary mask. In addition, the framework completely decouples the change detection process into a BCD task and an SCD task, double-temporal image features are modeled in the BCD task by utilizing an admitted vision encoder, change semantic decoding is realized in the SCD task by utilizing vision-language models, and finally more accurate semantic change detection mask generation and low-cost multi-class generalization are realized.

Description

Technical Field

[0001] The present invention relates to a remote sensing image semantic change detection method for non-strict open words, belonging to the field of machine learning and digital image processing, and has broad application prospects in the fields of electronic information technology, geographic information interpretation and remote sensing. Background Art

[0002] In the field of remote sensing, change detection refers to the use of multi-temporal remote sensing images covering the same surface area to determine and analyze surface changes. This includes identifying the nature of the change, assessing the temporal and spatial distribution patterns of the change, etc. As an important part of remote sensing image change interpretation technology, it has played a key role in many fields such as environmental change, urban development, and disaster monitoring.

[0003] Remote sensing image change detection includes two tasks: Remote Sensing Image Binary Change Detection (RSIBCD) and Remote Sensing Image Semantic Change Detection (RSISCD). Binary remote sensing image change detection usually refers to the process of using remote sensing image data from two different time points to identify whether the surface or objects have changed significantly between the two time points through comparative analysis. This detection method simplifies the change detection results into a binary state, namely "changed" and "unchanged". Semantic remote sensing image change detection is based on binary remote sensing image change detection, and further analyzes and interprets the changed area at the semantic level to identify the specific type of change (such as changes in land use types, the addition or removal of buildings, etc.) or the type of objects before and after the change.

[0004] Although the current RSISCD methods have shown powerful results, the change categories that these methods can detect are a strictly closed set, which means that these algorithms using closed category strategies have to bear high costs when generalizing to new categories. In practical application scenarios, the requirements for detection results are often more refined and the categories are more diverse, that is, semantic change detection in remote sensing scenes requires algorithms to have strong generalization capabilities in semantic categories. Unfortunately, current change detection schemes cannot meet this stringent requirement. Therefore, we hope to explore a method that can achieve multi-category generalization at a lower cost, thereby achieving the goal of rich and fine-grained semantic-level change detection.

[0005] Fortunately, the concept of open words provides new possibilities for SCD. In computer vision tasks such as object detection and semantic segmentation, open word solutions refer to the use of techniques such as image-text pairs, weak supervision signals, or visual language models to solve the problem that traditional closed-set methods cannot generalize to new categories. Conventional open word solutions include Region-Aware Training or Pseudo Labeling using image-text pairs. In recent years, with the development of visual language foundation models (VLFMs), open word solutions based on Knowledge Distillation and Transfer Learning have also emerged. The above methods have been demonstrated in tasks such as object detection and semantic segmentation, providing a new solution for multi-class generalization of SCD. It is worth noting that the open word setting does not require that the unseen categories are completely invisible during training, but cleverly uses weak supervision signals (such as image-caption, image-level label) or advanced vision-language models (such as CLIP) to solve the closed-set problem. Although the categories of the current SCD dataset are insufficient to support a fully open-word setting, the core idea of ​​open words is to use rich category semantic priors to guide the implementation of downstream tasks, which introduces an innovative solution for the multi-category generalization of SCD.

[0006] Therefore, this paper proposes Semantic-CD, a semantic change detection framework for non-strict open vocabulary remote sensing images. This framework completely decouples the change detection process into BCD and SCD tasks, using adapted vision encoder to model bi-temporal image features in the BCD task, and using vision-language models to achieve change semantic decoding in the SCD task, ultimately achieving more accurate semantic change detection mask generation and low-cost multi-class generalization. Summary of the invention

[0007] In view of the above problems, the present invention provides a remote sensing image semantic change detection method for non-strictly open words that solves at least some of the above technical problems. The remote sensing image semantic change detection task can be effectively completed by this method.

[0008] The embodiment of the present invention provides a remote sensing image semantic change detection method for non-strict open words, including:

[0009] Step 1: Dataset preparation

[0010] Among them, in the "dataset preparation" described in step 1, the method is as follows: the SECOND dataset is used. This dataset collects 4662 pairs of aerial images from multiple platforms and sensors. These image pairs are distributed in cities such as Hangzhou, Chengdu and Shanghai. The size of each image is 512x 512 and annotated at the pixel level. This dataset includes 6 major land cover categories, namely non-vegetated surface, trees, low vegetation, water, buildings and playgrounds, covering common natural and man-made geographical changes.

[0011] Step 2: Data preprocessing;

[0012] The principle of the “data preprocessing” described in step 2 is as follows: extract the semantic annotations of the two time phases, and take their intersection to form a binary change mask.

[0013] Step 3: Build the Semantic-CD network framework.

[0014] Among them, the "Semantic-CD network" described in step three has the following structure: Semantic-CD includes four components: CLIP vision encoder with BCSF Adapter, Prompt Learner, BCD decoder and SCD decoder.

[0015] Step 4: Insert BCSF Adapter into CLIP Vision Encoder

[0016] Among them, the "Insert BCSF Adapter into the CLIP visual encoder" described in step 4 is done as follows: First, build the backbone of the image encoder based on CLIP-ViT-L-14. Then, design the BCSF Adapter. Bi-temporalChange Semantic Filter (BCSF) flows and filters the changing semantic features from the dual dimensions of space and channel, providing a solution paradigm for the dual-temporal feature encoding of the basic model encoder. In Semantic-CD, BCSF is inserted into the CLIP image encoder as an adapter to enable the CLIP encoder to have the ability to model dual-temporal change features. Specifically, we inserted four BCSF layers, kept the network backbone frozen during model training, and only fine-tuned the parameters of the BCSF layer.

[0017] Step 5: Design Prompt Learner to generate semantic prior visual embedding

[0018] The structure of the "Prompt Learner" described in step 5 is as follows: Prompt Learner is built on the text encoder of CLIP, and achieves downstream task adaptation through learnable context embedding. It builds instance-adaptive context prompts based on meta tokens to enhance the model's representation ability for new categories. Prompt Learner can effectively mine the semantic priors in the text encoder of CLIP, and adapt to downstream tasks and new semantic categories through prompt learning.

[0019] Step 6: Design a BCD decoder to generate a binary mask

[0020] The "BCD decoding head" described in step 6 is as follows: the BCD decoder uses a simple multi-scale change detection decoding head to decode the bi-temporal visual features and generate a binary change mask. Specifically, we concatenate the multi-temporal image features obtained by the visual encoder along the channel dimension, construct feature maps of multiple scales based on SimpleFPN, and then unify them to the same scale for decoding the change mask.

[0021] Step 7: Design SCD decoding head to generate bi-temporal semantic mask

[0022] The "SCD decoding head" described in step 7 is as follows: the SCD decoder includes two twin decoding networks to decode visual features and a multi-stage fusion network to mine semantic priors from the cost map. First, similar to the BCD decoder, the bi-temporal features enter the two FPN networks of the twin network to construct a multi-scale bi-temporal feature map. At the same time, the cost map is input into the multi-stage fusion network to integrate the semantic prior into the decoding process.

[0023] Step 8: Train the network using a two-stage fully decoupled approach

[0024] The "two-stage fully decoupled network training method" described in step eight is as follows: the loss function of the BCD task is set to the binary cross entropy loss. The loss function of the SCD task is set to the cross entropy loss calculated by removing the background class loss. The training process is set to a two-stage paradigm in which BCD and SCD are completely decoupled, that is: i) when training the BCD branch, freeze the prompt learner and the scd decoder, ii) when training the SCD branch, freeze the vision encoder and the bcd decoder. This training process setting ensures that the BCD and SCD tasks are completely independent during training, and are only connected through the shared vision encoder during inference.

[0025] Through the above steps, the algorithm can obtain a high-precision and high-generalization change detection model. The model can distinguish the range and category of changes in dual-temporal remote sensing images and output the results.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] This paper proposes a semantic change detection framework for non-strict open vocabulary remote sensing images, Semantic-CD. It enhances the generalization of the model by introducing the latent knowledge of the visual-linguistic foundation model, and makes the detection results more accurate through a fully decoupled multi-task learning approach. In addition, we design a training process with fully decoupled BCD and SCD branches, aiming to accurately output BCD masks and fully exploit the latent knowledge of the visual-linguistic foundation in the SCD process. Compared with previous methods, Semantic-CD can produce more accurate mask areas and smaller semantic classification errors, and has extremely low generalization cost and strong generalization ability.

[0028] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0029] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0031] Figure 1 Schematic diagram of the framework designed for this algorithm.

[0032] Figure 2 BCSF Adapter module designed for this algorithm.

[0033] Figure 3 The SCD decoding network designed for this algorithm. DETAILED DESCRIPTION

[0034] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0035] The embodiment of the present invention provides a remote sensing image semantic change detection method for non-strict open words, which specifically includes the following steps:

[0036] S1, obtaining dual-phase remote sensing images;

[0037] S2. The dual-temporal remote sensing image and the semantic category to be identified are put into a trained remote sensing image semantic change detection model, and a semantic change detection mask image corresponding to the dual-temporal remote sensing image is output; the vehicle paint defect detection and segmentation model adopts a Semantic-CD framework.

[0038] In S2 above, the training method of the Semantic-CD model is as follows:

[0039] Step 1: obtaining a dual-temporal remote sensing image with semantic change annotations as a training data set; preprocessing the training data set to extract a binary change mask;

[0040] Step 2: Design the Semantic-CD network, which includes the following four parts: CLIP vision encoder with BCSF Adapter is used to extract bi-temporal image features, Prompt Learner generates visual embedding of text semantics, BCD decoder generates binary detection masks, and SCD decoder adds semantic labels to binary masks. ;

[0041] Step 3: Use the dual-temporal remote sensing image and the corresponding binary change mask to train the BCD branch;

[0042] Step 4: Use the dual-temporal remote sensing images and the corresponding semantic change masks to train the SCD branch;

[0043] See the schematic diagram of Semantic-CD framework for Figure 1 shown.

[0044] Next, the above steps 1 to 4 are described in detail.

[0045] In the above step 1, for the semantic change detection task, the embodiment of the present invention adopts the SECOND dataset. This dataset collects 4662 pairs of aerial images from multiple platforms and sensors. These image pairs are distributed in cities such as Hangzhou, Chengdu and Shanghai. The size of each image is 512x 512 and is annotated at the pixel level. This dataset includes 6 main land cover categories, namely non-vegetated surface, trees, low vegetation, water, buildings and playgrounds, covering common natural and man-made geographical changes. In addition, it is necessary to extract the semantic annotations of the two phases, and take their intersection to form a binary change mask.

[0046] In the above step 2, the Semantic-CD network specifically includes the following four parts: CLIP vision encoder with BCSF Adapter is used to extract bi-temporal image features, Prompt Learner generates visual embedding of text semantics, BCD decoder generates binary detection mask and SCD decoder adds semantic labels to binary mask. The specific definition of the above structure is as follows:

[0047] CLIP vision encoder with BCSF Adapter: Change detection tasks need to focus on effective changes in bi-temporal images and ignore the interference of irrelevant factors. This ability is lacking in existing base model encoders. The bi-temporal change semantic filter (BCSF) flows and filters the semantic features of changes from both spatial and channel dimensions, providing a solution paradigm for the bi-temporal feature encoding of the base model encoder. The BCSF module can be modeled as follows:

[0048]

[0049] Among them, Φ sf and Φ cf-k They are used to represent the spatial dual-time-varying semantic filter and the channel dual-time-varying semantic filter respectively. and are feed-forward networks (FFNs) with different parameter settings. and They represent the image features after spatial filtering and channel filtering respectively. represents the output of the filter with bi-temporal interaction, where k represents different time points. Among them, the spatial and channel filters can be expressed by the following formula:

[0050]

[0051] BCSF uses a direct method to fuse the spatial / channel features of dual-time images. In the spatial filter, x represents the feature to be filtered, and y represents the corresponding feature at another time point, which provides the filtering criteria. cat Indicates feature concatenation along the channel dimension. Through a linear layer Φ proj , converting the number of channels from 2×c to 1. σ represents the sigmoid activation function. represents pixel-by-pixel multiplication. The channel filter mirrors the spatial filtering process, with the main difference being that it operates along the channel axis. The channel filter requires a preliminary step of transposing and flattening the original feature map, i.e., H×W×c→c×(H×W).

[0052] In Semantic-CD, BCSF is inserted into the CLIP image encoder as an adapter to enable the CLIP encoder to have the ability to model bi-temporal variation features. Specifically, we insert BCSF layers after the 5th, 11th, 17th, and 23rd attention layers, keep the network backbone frozen during model training, and only fine-tune the parameters of the BCSF layer.

[0053] Prompt Learner: This paper constructs a method called Prompt Learner, which aims to efficiently apply the CLIP visual language model to the SCD task. Specifically, Prompt Learner is based on the pre-trained CLIP text encoder and instance-adaptive context prompt, and its structure can be expressed as the following formula:

[0054] π i =Φ meta (F i ),i∈{1,2}

[0055]

[0056] v m (F i )=v m +π i

[0057] P y (F i )={v 1 (F i ),v 2 (F i ),…,v m (F i ).c y},c y ∈C

[0058] {t y (F i )}=Φ CLIP-text ({P y (F i )})

[0059]

[0060] Prompt Learner receives dual-time features from the visual encoder, denoted as {F i},i∈{1,2}, and the category set C as input. First, it uses a meta-network Φ meta F i Convert to instance-adaptive contextual hint meta tags (metatoken) π i For each category c in the category set C y , we construct a learnable text prompt where {v 1 ,v 2 ,…,v m} represents a learnable category-specific context. Subsequently, the meta-tag π i is appended to each learnable category-specific contextual tag to form the instance-adaptive hint P y (F i ). The principle behind this instance-adaptive prompt design is to generate a meta-tag that helps create a unique prompt for each sample, so that the context can focus on the specific features or attributes of the sample. This approach makes the model more robust to category changes, thereby improving its representation ability for unseen categories. Finally, this prompt is fed into the CLIP text encoder Φ CLIP-text To obtain the semantic features of all categories, and through the cosine similarity Φ sim Export pixel-level cost map It is worth noting that in the output , the dimension of N-tokens must be expanded to accommodate the subsequent pixel-level semantic decoding process.

[0061] BCD Decoder: A concise multi-scale change detection decoding head is used to decode bi-temporal visual features and generate binary change masks. Specifically, we concatenate the multi-temporal image features obtained by the visual encoder along the channel dimension, and construct feature maps of multiple scales based on SimpleFPN, which are then unified to the same scale for decoding the change mask, as described in the following formula:

[0062] {F′ i}=Φ sampling (Φ cat (F1 ,F 2 ))

[0063] F′ i =Φ resize (Φ conv (F′ i ))

[0064] M BCD =Φ proj (Φ cat ({F′ i}))

[0065] Here, F 1 and F 2 Respectively represent the pre- and post-temporal semantic feature maps from the visual encoder. Φ sampling Represents the multi-scale construction process based on SimpleFPN. Subsequently, the multi-scale feature map It is refined through convolution to enhance its spatial characteristics. Φ resize refers to bilinear interpolation, which is used to unify the spatial resolution, while Φ proj A 1×1 convolutional layer is used to project the channel dimension of the feature map to the number of categories.

[0066] SCD decoder: accepts two types of input, visual features from the vision encoder and cost map from promptlearner. Specifically, SCD decoder includes two twin decoding networks to decode visual features and a Multi-stage fusion network to mine semantic guidance from the cost map, as shown in Figure x. First, similar to the BCD decoder, the dual-phase features enter the two FPN networks of the twin network to construct a multi-scale dual-phase feature map. At the same time, the cost map is input into the Multi-stage fusion network to fuse the semantic prior. The above process can be described by the following formula:

[0067] {F i}=Φ sampling (F 1 / 2 )

[0068]

[0069] Initially, similar to the BCD decoder, the bi-temporal features are independently input into two feature pyramid networks composed of a siamese network. 1 / 2 and cost graph are input into the feature pyramid networks of the two branches of the twin decoding network to generate their corresponding multi-scale feature maps {F i}and

[0070] Subsequently, {F i}and The fusion is performed layer by layer and input into the decoding head network to obtain the final semantic change mask. This process can be described by the following formula:

[0071]

[0072] It is worth mentioning that, unlike some SCD methods that use twin decoders with completely independent parameters, the two twin branches of the SCD decoder in this paper share parameters. The idea behind this design is that we hope that the SCD decoder is only responsible for the task of restoring the semantic segmentation map from image features and semantic features, while leaving the task of determining the semantic category entirely to the prompt learner. This setting makes the model more inclined to use the semantic prior in the prompt learner for category judgment, rather than using the decoding network to search for categories within a closed set.

[0073] In the above steps three and four, the present invention constructs a two-stage training paradigm in which BCD and SCD are completely decoupled. The design idea of ​​this training paradigm is that we hope to separate the work focus of BCD and SCD tasks as much as possible. Therefore, the training process adopts a two-stage paradigm in which BCD and SCD are completely decoupled, that is: i) when training the BCD branch, freeze the prompt learner and the SCD decoder, ii) when training the SCD branch, freeze the vision encoder and the BCD decoder. This setting of the training process ensures the complete independence of the BCD and SCD tasks during training, and they are only connected through the shared vision encoder during reasoning. The loss function of the above process is as follows:

[0074]

[0075] Here, represents a binary change detection mask, and Represent the semantic change masks before and after timing, respectively. and iRepresent the predicted value and labeled value of a single pixel in binary change detection or semantic change detection, respectively. N represents the total number of pixels. Since the BCD task is a binary classification task to distinguish whether there is a change in the image, the loss function of the BCD task is in the form of binary cross entropy. In the loss function of the SCD task, we removed the loss calculation of the background class. The starting point of this setting is that we hope that the SCD task is responsible for the judgment of each category with specific semantics, and the judgment of whether there is a change is completely handed over to the BCD task.

[0076] The present invention uses the AdamW optimizer with an initial learning rate of 1e-4 for training, and uses a cosine annealing scheduler with linear warm-up to reduce the learning rate. The number of epochs in the BCD training phase is 300, and the batch size is 1, and the number of epochs in the SCD training phase is 5, and the batch size is 1.

Claims

1. A remote sensing image semantic change detection method for non-strictly open words, characterized in that: include: S1, obtaining dual-phase remote sensing images; S2. Submit the dual-temporal remote sensing image and the semantic category to be identified to a trained remote sensing image semantic change detection model, and output a semantic change detection mask image corresponding to the dual-temporal remote sensing image; the semantic change detection model adopts a Semantic-CD framework.

2. The remote sensing image semantic change detection method for non-strict open words according to claim 1, characterized in that: In S2, the training process of the remote sensing image semantic change detection model is as follows: Step 1: obtaining a dual-temporal remote sensing image with semantic change annotations as a training data set; preprocessing the training data set to extract a binary change mask; Step 2: Design the Semantic-CD network, which includes the following four parts: CLIP vision encoder with BCSF Adapter is used to extract bi-temporal image features, Prompt Learner generates visual embedding of text semantics, BCD decoder generates binary detection masks, and SCD decoder adds semantic labels to binary masks. ； Step 3: Use the dual-temporal remote sensing image and the corresponding binary change mask to train the BCD branch; Step 4: Use the dual-temporal remote sensing images and the corresponding semantic change masks to train the SCD branch.

3. The remote sensing image semantic change detection method for non-strict open words as claimed in claim 2, characterized in that: In the step 1, the preprocessing includes: based on the bi-temporal semantic change annotation, the intersection of the annotations is taken as a binary change mask annotation.

4. The remote sensing image semantic change detection method for non-strict open words as claimed in claim 2, characterized in that: The step 2 specifically includes: Insert 4 BCSF Adapter layers into the visual encoder based on the CLIP-ViT-L-14 structure. Build a Prompt Learner network and design learnable text embeddings. Build a BCD decoding header based on the Simple FPN structure. The SCD decoding head is constructed based on the Simple FPN structure, and a multi-stage-fusion structure is added to fuse semantic priors and dual-phase image features.

5. The remote sensing image semantic change detection method for non-strict open words as claimed in claim 2, characterized in that: The steps three and four specifically include: The loss function of the BCD task is set to binary cross entropy loss. The loss function of the SCD task is set to the cross entropy loss calculated by removing the background class loss. The training process is set as a two-stage paradigm in which BCD and SCD are completely decoupled, that is: i) when training the BCD branch, the prompt learner and the scd decoder are frozen; ii) when training the SCD branch, the vision encoder and the bcd decoder are frozen. This training process setting ensures that the BCD and SCD tasks are completely independent during training and are only connected through the shared vision encoder during inference. The number of epochs in the BCD training phase is 300 and the batch size is 1, and the number of epochs in the SCD training phase is 5 and the batch size is 1.

Images