A remote sensing image change detection method based on language guidance

Abstract

The present invention relates to the field of remote sensing image change detection, and discloses a remote sensing image change detection method based on language guidance. Efficient SAM is used as a visual basic model to extract semantic feature maps of remote sensing images in two phases, and a multi-layer adapter is constructed to extract multi-scale feature maps in two phases; spatial attention is constructed to extract visual features; the input text description is processed by a text encoder to obtain word embedding and overall expression embedding; a multi-head cross attention method is used to construct a pre-fusion module for the fusion operation of word embedding and spliced ​​multi-layer feature maps; a dual-channel decoder structure is constructed to calculate the similarity between the obtained image features and the semantic embedding, and output the final change detection mask. The present invention designs a multi-layer visual-language feature cross layer to enable the model to fully learn the relevant information between the image and the corresponding type of text, thereby realizing universal detection of changes in multi-category remote sensing images guided by natural language.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image change detection, and in particular to a remote sensing image change detection method based on language guidance. Background Art

[0002] Multi-temporal remote sensing imagery change detection technology aims to discover, identify, and interpret geographic change by automatically analyzing remote sensing imagery acquired at different points in time over the same geographic area. This technology has demonstrated widespread application value in a variety of fields, including land-use planning, urban expansion monitoring, farmland change assessment, geological disaster monitoring, ecological and environmental protection, wetland management, and forest protection. However, while current deep learning-based change detection schemes have made significant progress in bi-temporal remote sensing imagery, enabling simple identification of changes in a single land type or location, they still face limitations in understanding semantic changes and extracting multi-dimensional information. In practical applications, this multi-dimensional information, such as change type, is crucial for decision-making processes such as land planning, urban management, and disaster assessment. Therefore, deep learning technology urgently needs to achieve technological breakthroughs in capturing multi-dimensional change information. Furthermore, existing change detection methods lack a unified paradigm for model interaction and universal multi-class segmentation, which to some extent restricts the further development and application of this technology.

[0003] To address these challenges, some initial work used convolutional neural networks (CNNs) to extract spatial and spectral features, followed by long short-term memory (LSTM) networks to extract temporal information for multi-type change detection. Other researchers have used pre-trained CNNs to extract deep features and then applied deep change vector analysis to identify multi-type changes. However, training deep learning models requires a large number of labeled samples, and both sample quantity and quality are crucial for improving change detection performance. In remote sensing change detection, this is particularly problematic due to the insufficient number and imbalance between changed and unchanged samples. High-quality samples require manual annotation by experts, a tedious, time-consuming, and expensive task. This makes the sample problem a pressing issue in deep learning-based remote sensing change detection. Furthermore, purely convolutional change detection methods for remote sensing images struggle to handle long-range conceptual relationships. To address these challenges, recent efforts have introduced attention mechanisms based on the Transformer architecture for remote sensing change detection. Compared to purely convolutional methods, these models can extract richer semantic information and establish global relationships in the feature space, achieving superior change detection performance. Based on this, some researchers have proposed leveraging the powerful zero-shot semantic recognition and segmentation capabilities of visual foundational models to reduce the model's reliance on sample size. By fine-tuning downstream models, these approaches can be applied to the specific task of remote sensing change detection. However, these approaches require re-tuning or training new network models for specific tasks, hindering effective interaction with the model and the universal detection of changes in multiple categories of objects. Summary of the Invention

[0004] In order to address the shortcomings of the above-mentioned background technologies, the present invention proposes a language-guided remote sensing image change detection method. By designing a multi-layer visual-language feature cross layer, the model can fully learn the relevant information between the image and the corresponding type of text, and realize the universal detection of multi-category remote sensing image changes guided by natural language.

[0005] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0006] A remote sensing image change detection method based on language guidance comprises the following steps:

[0007] Step 1: Collect public remote sensing image change detection datasets and preprocess the datasets;

[0008] Step 2: Select two remote sensing change detection images corresponding to each other from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images of the two temporal phases;

[0009] Step 3: construct a multi-layer adapter to extract a two-phase multi-scale feature map based on the semantic feature map of the remote sensing image of the two temporal phases;

[0010] Step 4: Construct spatial attention, and further mine the visual features of the remote sensing image at each scale after global feature learning based on the two-phase multi-scale feature map, that is, the global spatial feature map. The global spatial feature maps of different scales are spliced ​​into a multi-layer global spatial feature map;

[0011] Step 5: Use the pre-trained text encoder in the image-natural language model CLIP to process the input text prompts corresponding to the training samples to obtain word embeddings and overall expression embeddings;

[0012] Step 6: Using the multi-head cross attention method, a pre-fusion module is constructed to fuse the word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that are integrated with the semantic features.

[0013] Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder; input the activated visual features fused with semantic features, output the total visual features after passing through the visual pathway decoder, and output the semantic embedding after passing through the semantic pathway decoder; calculate the similarity between the total visual features and the semantic embedding, and output the final change detection mask through interpolation and binarization operations.

[0014] Furthermore, the data set is preprocessed in step 1 as follows:

[0015] Step 1.1: Extract the change detection targets of different categories separately through color value detection to obtain the binary segmentation images corresponding to each category;

[0016] Step 1.2: Process the corresponding segmentation maps of the two time phases by using pixel value XOR to obtain a single-category change detection dataset and store it in a classified manner;

[0017] Step 1.3: Divide the dataset of each category into training set, validation set and test set according to the proportion.

[0018] Furthermore, the specific process of step 2 is as follows:

[0019] Step 2.1: Select two temporal phases of remote sensing change detection images I1 and I2, expressed as:

[0020] I1,I2∈R H×W×3

[0021] Where H and W are the height and width of the input image respectively;

[0022] Step 2.2: Use the SAMI pre-trained lightweight encoder as the image encoder and the default mask decoder of SAM as the decoder of EfficientSAM. Fine-tune the EfficientSAM model on the SA-1B dataset to obtain the model pre-training weights.

[0023] Step 2.3: Process the remote sensing change detection images I1 and I2 of the two temporal phases respectively to obtain the semantic feature maps f of the remote sensing images of the two temporal phases. v1 and f v2 , expressed as:

[0024]

[0025] Among them, H v and W v Represents the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

[0026] Furthermore, the multi-layer adapter in step 3 includes four multi-layer convolution and deconvolution layers with different parameters and the same structure, named Adapter4, Adapter8, Adapter16, and Adapter32, respectively, for extracting different fine-grained features of the image; Adapter4 and Adapter8 are set for upsampling, that is, increasing the resolution of the feature map and adjusting the number of channels; Adapter16 adjusts the feature dimension without changing the resolution; Adapter32 increases the number of channels and reduces the spatial resolution through a deconvolution operation with a stride of 2;

[0027] The multi-scale feature learning process is as follows:

[0028] Semantic feature maps of remote sensing images of two temporal phases f v1 and f v2 After four adaptation operations, we can get multi-scale feature maps with downsampling times of 32, 16, 8 and 4 respectively. For a single feature map f vk , k=1,2, the output of the i-th layer is expressed by the following formula:

[0029]

[0030] Among them, Conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; is the feature map of the i-th layer in the k-th phase after downsampling; is the final multi-scale feature map after the adaptation operation.

[0031] Furthermore, the spatial attention layer in step 4 is a multi-layer convolutional layer structure, which is used to learn the overall features after the two phases are spliced ​​together. The specific steps are as follows:

[0032] Step 4.1: The two temporal multi-scale feature maps corresponding to the same scale and Join along the channel dimension to obtain the concatenated feature map

[0033]

[0034] in, Represents channel-level feature concatenation operations;

[0035] Step 4.2: Construct the spatial attention layer and input the concatenated feature map The first convolution layer uses a 1x1 convolution kernel to concatenate the input feature maps. The channel is compressed and then remapped to the original or specified channel dimension using 1x1 convolution to generate an attention weight matrix consistent with the input image size and multiplied by the input image; it is expressed by the following formula:

[0036]

[0037] in, Represents the visual features after global feature learning.

[0038] Furthermore, the step 5 specifically includes:

[0039] Step 5.1: Initialize the CLIP text encoder using existing pre-trained weights.

[0040] Step 5.2: Get the input text hint L∈R l , where l is the length of the input text prompt; during the training phase, the input text prompt is replaced by the label category of the specific template sentence; the input text prompt is input into the text encoder to obtain the word embedding and overall expression embedding where the word embedding f w Learn specific task information and overall expression embedding f all Grasp the overall message expressed in the text.

[0041] Furthermore, the specific operation of fusion feature in step 6 is expressed as the following formula:

[0042]

[0043] in, represents the visual features after global feature learning, fw is word embedding; MCA stands for multi-head cross attention operation; W q 、W k and W v Linear matrices representing query, key, and value respectively, mapping the feature vector to the corresponding feature space, C i represents the number of channels in the fusion feature space, It represents the activated visual features after the semantic features are fused into the i-th layer.

[0044] Furthermore, the specific construction method of the dual-channel decoder in step 7 is as follows:

[0045] Step 7.1: In the visual pathway, in the visual pathway decoder, for each layer of image features after the pre-fusion module, flatten them in the spatial dimension and add sinusoidal position encoding, which is expressed as follows:

[0046]

[0047] Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, Represents activated visual features embedded in sinusoidal positional coding;

[0048] Step 7.2: Embed the activated visual features of the sinusoidal position encoding With word embedding f w After concatenation, the multimodal tokens are subjected to multi-head self-attention operation to further learn the relevant information between multimodal features. The formula is as follows:

[0049]

[0050] Among them, MSA represents multi-head self-attention operation, Represents multi-layer visual features after fusing multimodal information;

[0051] Step 7.3: Use a pyramid-structured decoder to fuse multiple layers of visual features, from the bottom-level details to the top-level abstract information, and integrate the information at each level by upsampling and fusing the low-level features step by step; introduce a cross-attention mechanism between visual features and text features at each pyramid level, with visual features as queries and text features as keys and values; that is, the visual features of each layer are combined into a single layer. and word embedding f w The total visual features after activation are obtained by decoding through the cross attention mechanism, and the public representation is as follows:

[0052]

[0053] Among them, f VRepresents the total visual features after activation, and FPN represents the pyramid structure decoder;

[0054] Step 7.4: In the semantic path, embed f with the overall expression all For query, activate visual features f actv For the key and value, a cross attention operation is performed, followed by a multi-head self-attention operation. The formula is as follows:

[0055] f L =MSA(MCA(f all ,f actv ))

[0056] Among them, f L is the semantic embedding obtained after passing through the semantic pathway decoder;

[0057] Step 7.5: The total visual feature f after activation is obtained V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

[0058] The beneficial effects brought about by adopting the above technical solution are:

[0059] 1) This paper designs a language-guided architecture for remote sensing image change detection. Specifically, based on a multi-head cross-attention mechanism, by designing multiple layers of visual-language feature cross-layers, the model fully learns the relevant information between images and corresponding types of text, achieving universal detection of changes in multi-category remote sensing images guided by natural language.

[0060] 2) This paper uses Efficient SAM (Segment Anything Model) as its foundational visual model. Compared to traditional SAM models, Efficient SAM reduces the number of parameters to 5% of the original by pre-training using SAM mask images (SAMI), while running 20 times faster. This makes it a lightweight and efficient foundational visual model. Its powerful feature extraction and semantic segmentation capabilities improve model performance while reducing its dependence on the quantity and quality of remote sensing image samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flow chart of the method of the present invention.

[0062] Figure 2 Each category in the SECOND dataset is processed separately to obtain the corresponding binary segmentation image.

[0063] Figure 3Schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0065] The present invention designs a language-guided remote sensing image change detection architecture. Through a multi-head cross-attention mechanism and the design of a multi-layer visual-language feature cross-fusion layer, the model fully learns the relevant information between the image and the corresponding type of text, and realizes the universal detection of multi-category remote sensing image changes guided by natural language. In addition, a lightweight Efficient SAM is used as the visual base model to extract features from the image, which reduces the number of parameters while improving the running speed and reducing the model's dependence on the number of training samples. At the same time, a multi-layer adapter is designed to extract multi-scale features of the image, and a pyramid-structured decoder is used to enable the model to learn the range of changes at different granularities.

[0066] A language-guided change detection method for remote sensing images includes training set data classification, constructing an Efficient SAM to learn preliminary features of bitemporal remote sensing images, building a multi-layer adapter to extract multi-scale features of remote sensing images, building a spatial attention module to generate attention weights for the spliced ​​images, initializing a pre-trained text encoder and obtaining word embeddings and overall expression embeddings based on input text prompts, building a pre-fusion module to enable the interaction of text and image information, building a dual-channel decoder to obtain image features and semantic embeddings, and outputting segmentation masks through similarity calculation.

[0067] The specific implementation flow chart of the remote sensing image change detection method based on language guidance of the present invention is as follows: Figure 1 As shown, the network structure diagram is as follows Figure 3 The specific implementation steps are as follows:

[0068] Step 1: Collect public remote sensing image change detection datasets and preprocess the datasets.

[0069] The SECOND dataset, a publicly available dataset for remote sensing change detection, is a semantic change detection dataset collected from 4,662 pairs of aerial images from multiple platforms and sensors. These pairs are distributed over cities such as Hangzhou, Chengdu, and Shanghai. Each image is 512 x 512 pixels and annotated at the pixel level. The SECOND dataset focuses on six major land cover categories: non-vegetated surface, trees, low vegetation, water, buildings, and playgrounds, which are frequently involved in natural and anthropogenic geographic changes.

[0070] like Figure 2 As shown, the specific implementation of the present invention for preprocessing the training set in the data set is as follows:

[0071] 1) Extract the change detection targets of different categories separately through color value detection to obtain the binary segmentation images corresponding to each category;

[0072] 2) The corresponding segmentation maps of the two time phases are processed by the pixel value XOR method to obtain a single-category change detection dataset and store it in a classified manner.

[0073] 3) Divide each category data set into training set, validation set and test set in a ratio of 7:1:2.

[0074] Step 2: Select two remote sensing change detection images corresponding to each other from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images of the two temporal phases.

[0075] Initialize the Efficient SAM model of the visual basic model. The details are as follows:

[0076] 1) SAMI pre-trained lightweight encoders (such as ViT-Tiny and ViT-Small) are used as image encoders, and SAM's default mask decoder is used as the decoder of EfficientSAM. The EfficientSAM model is fine-tuned on the SA-1B dataset to obtain the model pre-training weights.

[0077] 2) Set the training batch size to 4, and select two corresponding remote sensing change detection images I1, I2∈R each time H×W×3 The EfficientSAM model is used to obtain the semantic features of remote sensing images of two phases with 256 channels and 16-fold downsampling of feature dimensions (4, 256, 32, 32). v1 and f v2 , expressed as:

[0078]

[0079] Among them, H v and W v Represents the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

[0080] Step 3: Construct a multi-layer adapter to extract a two-phase multi-scale feature map based on the two-phase remote sensing image semantic feature map.

[0081] The multi-layer adapter is composed of four adapters in total, each of which is composed of multiple layers of convolution and deconvolution layers with the same structure. By adjusting the resolution and number of channels of the feature map, the multi-scale features of the input image are learned to achieve multi-scale feature fusion.

[0082] In order to obtain multi-scale feature maps by downsampling the original image by 32 times, 16 times, 8 times, and 4 times, four adapters are set, named Adapter4, Adapter8, Adapter16, and Adapter32 respectively.

[0083] For a single feature map f vk , k=1,2, the output of the i-th layer is expressed by the following formula:

[0084]

[0085] Among them, conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; is the feature map of the i-th layer in the k-th phase after downsampling, k = 1, 2, i = 1, 2, 3, 4, corresponding to downsampling by 32 times, 16 times, 8 times and 4 times; For the final multi-layer feature map after convolution operation, the adapter includes two steps of downsampling and subsequent convolutional layers.

[0086] Considering that Efficient SAM is based on the ViT architecture and has already downsampled the original image by a factor of 16, Adapter4 and Adapter8 are used for upsampling, increasing the resolution of the feature map while adjusting the number of channels. Adapter16 adjusts the feature dimension without changing the resolution. Adapter32 increases the number of channels and reduces the spatial resolution through deconvolution with a stride of 2. The adjusted number of channels is 64, 128, 256, and 512, respectively, compensating for the image information lost by downsampling.

[0087] The specific implementation of building a multi-layer adapter is as follows:

[0088] 1) Based on the formula for changing the output feature map size of the deconvolution layer: output height (width) = (input height (width) - 1) × stride - 2 × padding + kernel height (width) + output padding, adjust the deconvolution layer parameters to downsample the original image by a factor of 32, 16, 8, or 4, respectively. Then, use batch normalization and the ReLU activation function to stabilize the training process.

[0089] 2) After adjusting the sampling rate, the feature dimensions are adjusted to 64, 128, 256, and 512 respectively using 1x1 convolutional layer batch normalization and ReLU.

[0090] Step 4: Build spatial attention to further mine the global spatial features of remote sensing images at various scales.

[0091] The spatial attention layer is a multi-layer convolutional layer structure that is used to learn the overall features of the two phases after splicing, and enables the network to learn which spatial locations in the input feature map are more important, so that more attention is paid to these locations in subsequent processing. The specific steps are as follows:

[0092] 1) The two temporal multi-scale feature maps corresponding to the same scale and Splicing along the channel dimension, we get the spliced ​​feature maps with dimensions of 128, 256, 512, and 1024 respectively (the original dimensions are 64, 128, 256, and 512. The channel dimension is doubled after splicing the two phase feature maps. It can be understood as a global feature map after simple splicing, and waits for subsequent global feature learning). It is expressed by the following formula;

[0093]

[0094] in Represents channel-level feature concatenation operations;

[0095] 2) Construct a spatial attention layer, where the first convolution layer uses a 1x1 convolution kernel to compress the channels of the input concatenated feature map. This allows the network to learn how to aggregate information in the channel dimension, that is, which channels’ information is more important to the final attention distribution. Setting the convolution kernel size to 1 allows spatial information to be retained, but the information between channels is effectively integrated, reducing the dimension and facilitating efficient calculation. Finally, a 1x1 convolution is used again to remap to the original or specified channel dimension, generating an attention weight matrix that is consistent with the input image size and multiplied by the input image. It is expressed as follows:

[0096]

[0097] in Represents the visual features after global feature learning.

[0098] Step 5: Use the pre-trained image-natural language model CLIP (Contrastive Language-Image Pre-Training) to process the input text prompts through the natural language encoder to obtain word embeddings and overall expression embeddings.

[0099] A Transformer-based multimodal pre-trained model, CLIP, is used to extract information at different scales from text prompts. This is based on the fact that CLIP is pre-trained on a dataset of 400 million image-text pairs collected from the internet, enabling the model to understand image content and generate matching text descriptions, while also being able to retrieve corresponding images based on text queries. Its core advantage lies in the strong semantic consistency between the learned image and text embeddings, which enables CLIP's text encoder to excel in a variety of cross-modal downstream tasks.

[0100] During the training phase, the trainer creates text prompts for different categories using model sentences containing that category. For example, if the training sample is forest category change, the corresponding text prompt is "Remote sensing image change detection for forest category." During the prediction phase after model training, users can create their own text prompts, such as "Give land change for forest category," to enable efficient interaction with the model.

[0101] Leveraging its powerful semantic understanding and generalization capabilities, we extract task-specific information and overall text expression information from the input text prompts for application in downstream cross-modal fusion tasks. The specific steps are as follows:

[0102] 1) Use the official pre-trained weights to initialize the CLIP text encoder and set the upper limit of text length to 77;

[0103] 2) Get the input text prompt L∈R l , where l is the length of the input text. During the training phase, the input text prompt is replaced by the label category of the specific template sentence. The text prompt is input into the text encoder (pre-trained text encoder in CLIP) to obtain the word embedding and overall expression embedding C t is the dimension of the matrix; where word embedding f w Responsible for learning more detailed specific task information, while the overall expression embedding f all Then you can grasp the overall information expressed in the text.

[0104] Step 6: Use the multi-head cross attention method to build a pre-fusion module for the fusion operation of word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that are fused with semantic features.

[0105] The pre-fusion module is a key component based on the multi-head attention mechanism, designed to build a bridge between visual and language features and promote their interaction. Using visual features as queries and word embeddings as keys and values, the multi-head cross-attention operation helps focus on visual areas relevant to language instructions, reducing interference from irrelevant background information, thereby improving the model's performance on cross-modal tasks.

[0106] The specific operation of fusion features can be expressed as the following formula:

[0107]

[0108] Among them, MCA stands for multi-head cross attention operation, W q 、W k and W v Represents the corresponding linear matrix, mapping the eigenvector to the corresponding eigenspace, C i represents the number of channels in the fusion feature space, It represents the activated visual features after the semantic features are fused into the i-th layer.

[0109] Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder. Calculate the similarity between the image features obtained after the visual pathway decoder and the semantic embedding obtained after the semantic pathway decoder, and output the final change detection mask through interpolation and binarization operations.

[0110] The dual-channel decoder design uses a deep cross-modal fusion strategy to effectively integrate visual and language information at different scales and levels, improving the model's performance and generalization capabilities in complex scene understanding tasks. The specific construction method is as follows:

[0111] Step 7.1) In the visual pathway, for each layer of activated visual features, we first flatten them spatially and apply a sinusoidal position encoding. Note that 2D position embedding is applied to visual features, while 1D position embedding is used for text features. Applying position embedding ensures that the model correctly aligns spatial and sequential information during the fusion process, which is crucial for understanding relative positions in the visual scene and temporal relationships in the text description. The specific formula is as follows:

[0112]

[0113] Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, Visual features representing activations embedded in sinusoidal positional encodings.

[0114] Step 7.2) Concatenate the position-encoded visual features with the word embeddings, and then perform a multi-head cross-attention operation on the resulting multimodal tokens to further learn the correlation between the multimodal features. The formula is as follows:

[0115]

[0116] Among them, MSA represents multi-head self-attention operation, Represents multi-layer visual features after fusing multimodal information.

[0117] Step 7.3) A pyramid (FPN) structured decoder is used to fuse features at different scales, from bottom-level details to high-level abstract information. By upsampling and fusing low-level features step by step, the model can integrate information at all levels and enhance the richness and accuracy of feature expression. A cross-attention mechanism between visual features and text features is introduced at each FPN level (visual features are regarded as queries, and text features are used as keys and values). This not only enables visual features to be adjusted according to the semantic guidance of the text, but also enables text features to be integrated into the spatial context of visual features, achieving deeper cross-modal fusion. The visual features of each layer are combined into a single layer. and word embedding f w The total visual features after activation are obtained by decoding through the cross attention mechanism, and the public representation is as follows;

[0118]

[0119] Among them, f V Represents the total visual features after activation, and FPN represents the pyramid structure decoder;

[0120] Step 7.4) In the semantic path, we use the overall expression embedding as the query and the activated visual features as the key and value, and perform a cross-attention operation followed by a multi-head self-attention operation. The formula is as follows:

[0121] f L =MSA(MCA(f all ,f actv ))

[0122] Step 7.5) The total visual feature f after activation is obtained V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

[0123] It is worth noting that during the training process, this embodiment adopts a strategy of auxiliary loss calculation. That is, considering that the last layer of visual features passing through the decoder is generally considered to have more global and abstract information and is more suitable for expressing high-level semantics, this embodiment selects the last layer as the input of the final predicted output mask, and the features of the remaining layers are only added to the calculation of the loss function to assist model adjustment, and are not used as the final output.

Claims

1. A remote sensing image change detection method based on language guidance, characterized in that: The steps include: Step 1: Collect public remote sensing image change detection datasets and preprocess the datasets; Step 2: Select two remote sensing change detection images corresponding to each other from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images of the two temporal phases; Step 3: construct a multi-layer adapter to extract a two-phase multi-scale feature map based on the semantic feature map of the remote sensing image of the two temporal phases; Step 4: Construct spatial attention, and further mine the visual features of the remote sensing image at each scale after global feature learning based on the two-phase multi-scale feature map, that is, the global spatial feature map. The global spatial feature maps of different scales are spliced ​​into a multi-layer global spatial feature map; Step 5: Use the pre-trained text encoder in the image-natural language model CLIP to process the input text prompts corresponding to the training samples to obtain word embeddings and overall expression embeddings; Step 6: Using the multi-head cross attention method, a pre-fusion module is constructed to fuse the word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that are integrated with the semantic features. Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder; Input is the activated visual feature fused with the semantic feature, after passing through the visual pathway decoder, the total visual feature is output, and after passing through the semantic pathway decoder, the semantic embedding is output; The total visual features and semantic embeddings are similarly calculated, and the final change detection mask is output through interpolation and binarization operations; In step 3, the multi-layer adapter includes four multi-layer convolution and deconvolution layers with different parameters but the same structure, named Adapter4, Adapter8, Adapter16, and Adapter32, respectively, for extracting different fine-grained features of the image; Adapter4 and Adapter8 are set for upsampling, that is, increasing the resolution of the feature map and adjusting the number of channels; Adapter16 adjusts the feature dimension without changing the resolution; Adapter32 increases the number of channels and reduces the spatial resolution through a deconvolution operation with a stride of 2; The multi-scale feature learning process is as follows: Semantic feature maps of remote sensing images of two temporal phases f v1 and f v2 After four adaptation operations, we can get multi-scale feature maps with downsampling times of 32, 16, 8 and 4 respectively. For a single feature map f vk , k=1,2, the output of the i-th layer is expressed by the following formula: Among them, Conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; is the feature map of the i-th layer in the k-th phase after downsampling; is the final multi-scale feature map after the adaptation operation.

2. The method for detecting changes in remote sensing images based on language guidance according to claim 1, characterized in that: In step 1, the data set is preprocessed as follows: Step 1.1: Extract the change detection targets of different categories separately through color value detection to obtain the binary segmentation images corresponding to each category; Step 1.2: Process the corresponding segmentation maps of the two time phases by using pixel value XOR to obtain a single-category change detection dataset and store it in a classified manner; Step 1.3: Divide the dataset of each category into training set, validation set and test set according to the proportion.

3. The method for detecting changes in remote sensing images based on language guidance according to claim 1, wherein: The specific process of step 2 is as follows: Step 2.1: Select two temporal phases of remote sensing change detection images I1 and I2, expressed as: I1,I2∈R H×W×3 Where H and W are the height and width of the input image respectively; Step 2.2: Use the SAMI pre-trained lightweight encoder as the image encoder and use the default mask decoder of SAM as the decoder of EfficientSAM, and fine-tune the EfficientSAM model on the SA-1B dataset. Get the model pre-training weights; Step 2.3: Process the remote sensing change detection images I1 and I2 of the two temporal phases respectively to obtain the semantic feature maps f of the remote sensing images of the two temporal phases. v1 and f v2 , expressed as: Among them, H v and W v Represents the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

4. The method for detecting changes in remote sensing images based on language guidance according to claim 1, wherein: The spatial attention layer in step 4 is a multi-layer convolutional layer structure used to learn the overall features of the two phases after splicing. The specific steps are as follows: Step 4.1: The two temporal multi-scale feature maps corresponding to the same scale and Join along the channel dimension to obtain the concatenated feature map in, Represents channel-level feature concatenation operations; Step 4.2: Construct the spatial attention layer and input the concatenated feature map The first convolution layer uses a 1x1 convolution kernel to concatenate the input feature maps. The channel is compressed and then remapped to the original or specified channel dimension using 1x1 convolution to generate an attention weight matrix consistent with the input image size and multiplied by the input image; it is expressed by the following formula: in, Represents the visual features after global feature learning.

5. The method for detecting changes in remote sensing images based on language guidance according to claim 1, wherein: The step 5 specifically includes: Step 5.1: Initialize the CLIP text encoder using existing pre-trained weights. Step 5.2: Get the input text hint L∈R l , where l is the length of the text prompt; during the training phase, the input text prompt is replaced by the label category of the specific template sentence; the text prompt is input into the text encoder to obtain the word embedding and overall expression embedding C t is the dimension of the matrix; where word embedding f w Learn specific task information and overall expression embedding f all Grasp the overall message expressed in the text.

6. The method for detecting changes in remote sensing images based on language guidance according to claim 5, characterized in that: The specific operation of fusion features in step 6 is expressed as the following formula: in, represents the visual features after global feature learning, f w is word embedding; MCA stands for multi-head cross attention operation; W q 、W k and W v Linear matrices representing query, key, and value respectively, mapping the feature vector to the corresponding feature space, C i Represents the number of channels in the fusion feature space, It represents the activated visual features after the i-th layer fuses the semantic features.

7. The method for detecting changes in remote sensing images based on language guidance according to claim 6, characterized in that: The specific construction method of the dual-channel decoder in step 7 is as follows: Step 7.1: In the visual pathway decoder, for each layer of image features after the pre-fusion module, It flattens the spatial dimension and adds sinusoidal position encoding, which is expressed as follows: Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, * represents the activated visual features embedded in the sinusoidal positional encoding; Step 7.2: Embed the activated visual features of the sinusoidal position encoding *With word embedding f w After concatenation, the multimodal tokens are subjected to multi-head self-attention operation to further learn the relevant information between multimodal features. The formula is as follows: Among them, MSA represents multi-head self-attention operation, Represents multi-layer visual features after fusing multimodal information; Step 7.3: Use a pyramid-structured decoder to fuse multiple layers of visual features, from the bottom-level details to the top-level abstract information, and integrate the information at each level by upsampling and fusing the low-level features step by step; introduce a cross-attention mechanism between visual features and text features at each pyramid level, with visual features as queries and text features as keys and values; that is, the visual features of each layer are combined into a single layer. and word embedding f w The total visual features after activation are obtained by decoding through the cross attention mechanism, and the public representation is as follows; Among them, f V Represents the total visual features after activation, and FPN represents the pyramid structure decoder; Step 7.4: In the semantic path decoder, embed f with the overall expression all For query, activate visual features f actv For the key and value, a cross attention operation is performed, followed by a multi-head self-attention operation. The formula is as follows: Among them, f L is the semantic embedding obtained after passing through the semantic pathway decoder; Step 7.5: The total visual feature f after activation is obtained V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

Images