Remote sensing image change detection method based on language guidance

Abstract

The invention relates to the field of remote sensing image change detection, and discloses a remote sensing image change detection method based on language guidance. The method comprises the following steps of: extracting a two-time-phase remote sensing image semantic feature map by adopting an Officient SAM as a visual basic model, constructing a multi-layer adapter, and extracting a two-time-phase multi-scale feature map; constructing spatial attention, and extracting visual features; the input text description is processed through a text encoder, and word embedding and overall expression embedding are obtained; a multi-head cross attention method is adopted, and a pre-fusion module is constructed and used for fusion operation of multiple layers of feature maps after word embedding and splicing; and constructing a two-channel decoder structure, performing similarity calculation on the obtained image features and semantic embedding, and outputting a final change detection mask. According to the method, a multi-layer vision-language feature cross layer is designed, so that the model fully learns related information of images and corresponding types of texts, and universal detection of multi-category remote sensing image changes can be realized by taking natural language as guidance.

Description

Technical Field

[0001] The invention relates to the field of remote sensing image change detection, and in particular to a remote sensing image change detection method based on language guidance. Background Art

[0002] Multi-temporal remote sensing image change detection technology aims to discover, identify and explain geographic changes by automatically analyzing remote sensing images acquired at different time points in the same geographic area. This technology has demonstrated its wide application value in many fields such as land use planning, urban expansion monitoring, farmland change assessment, geological disaster monitoring, ecological and environmental protection, wetland management and forest protection. However, although the current change detection schemes based on deep learning have made significant progress in dual-temporal remote sensing images and achieved simple identification of changes in a single land type or change location, they are still limited in understanding semantic changes and extracting multivariate information. In practical applications, these multivariate information, such as change types, are crucial for decision-making processes such as land planning, urban management and disaster assessment. Therefore, deep learning technology urgently needs to achieve technical breakthroughs in obtaining multivariate change information. In addition, existing change detection methods lack a unified paradigm in terms of model interaction and multi-category general segmentation, which to some extent limits the further development of this technology and the expansion of its application scope.

[0003] In order to address these challenges, some works initially used CNN (Convolutional Neural Network) to extract spatial spectral features, and then extracted time information through Long Short-Term Memory (LSTM) to achieve multi-type change detection; some researchers also used pre-trained CNN to extract deep features, and then used deep change vector analysis to identify multi-type changes. However, the training of deep learning models requires a large number of labeled samples, and the number and quality of samples are the key to improving change detection performance. In remote sensing change detection tasks, there is an issue of insufficient and unbalanced number of samples that have changed and not changed, and high-quality samples require manual annotation by experts, which is tedious, time-consuming and expensive. This makes the sample problem an urgent problem to be solved in the field of remote sensing change detection based on deep learning. In addition, remote sensing image change detection methods based on pure convolution have difficulties in dealing with long-distance conceptual relationships. Considering these problems, in recent years, some remote sensing change detection methods based on Transformer architecture have introduced attention mechanisms. Compared with pure convolution methods, such models can extract richer semantic information and establish global relationships in feature space, achieving better remote sensing image change detection performance. Based on this, some researchers have proposed to use the powerful zero-shot semantic recognition and segmentation capabilities of the visual basic model to reduce the model's dependence on the number of samples, and then fine-tune the downstream model to apply it to the specific task of remote sensing change detection. However, these methods require re-adjusting or training new network models for specific tasks, and cannot achieve effective interaction with the model, nor universal detection of changes in multiple categories of targets. Summary of the invention

[0004] In order to address the shortcomings of the above-mentioned background technologies, the present invention proposes a remote sensing image change detection method based on language guidance. By designing a multi-layer visual-language feature cross-layer, the model can fully learn the relevant information between the image and the corresponding type of text, thereby realizing universal detection of changes in multi-category remote sensing images guided by natural language.

[0005] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0006] A remote sensing image change detection method based on language guidance comprises the following steps:

[0007] Step 1: Collect the public remote sensing image change detection dataset and preprocess the dataset;

[0008] Step 2: Select two remote sensing change detection images corresponding to each other in two time periods from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images in the two time periods;

[0009] Step 3: construct a multi-layer adapter to extract a two-phase multi-scale feature map according to the two-phase remote sensing image semantic feature map;

[0010] Step 4: Construct spatial attention, and further mine the visual features of the remote sensing image at each scale after global feature learning according to the two-phase multi-scale feature map, that is, the global spatial feature map, and splice the global spatial feature maps of different scales into a multi-layer global spatial feature map;

[0011] Step 5: Use the pre-trained text encoder in the image-natural language model CLIP to process the input text prompts corresponding to the training samples to obtain word embeddings and overall expression embeddings;

[0012] Step 6: Use the multi-head cross attention method to build a pre-fusion module for the fusion operation of word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that integrate the semantic features;

[0013] Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder; input the activated visual features fused with semantic features, output the total visual features after passing through the visual pathway decoder, and output the semantic embedding after passing through the semantic pathway decoder; calculate the similarity between the total visual features and the semantic embedding, and output the final change detection mask through interpolation and binarization operations.

[0014] Furthermore, the data set is preprocessed in step 1 as follows:

[0015] Step 1.1: Extract the change detection targets of different categories separately through color value detection to obtain the binary segmentation images corresponding to each category;

[0016] Step 1.2: Process the corresponding segmentation images of the two phases by means of pixel value XOR to obtain a single-category change detection data set and store it by category;

[0017] Step 1.3: Divide the dataset of each category into training set, validation set and test set according to the proportion.

[0018] Furthermore, the specific process of step 2 is as follows:

[0019] Step 2.1: Select two temporal phases of remote sensing change detection images I1 and I2, expressed as:

[0020] I1,I2∈R H×W×3

[0021] Where H and W are the height and width of the input image respectively;

[0022] Step 2.2: Use the SAMI pre-trained lightweight encoder as the image encoder, and use the default mask decoder of SAM as the decoder of EfficientSAM, and fine-tune the EfficientSAM model on the SA-1B dataset to obtain the model pre-training weights;

[0023] Step 2.3: Process the remote sensing change detection images I1 and I2 of the two time phases respectively to obtain the semantic feature maps f of the remote sensing images of the two time phases v1 and f v2 , expressed as:

[0024]

[0025] Among them, H v and W v Represent the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

[0026] Furthermore, the multi-layer adapter in step 3 includes four multi-layer convolution and deconvolution layers with different parameters and the same structure, respectively named Adapter4, Adapter8, Adapter16, and Adapter32, which are used to extract features of different fine-grained images; Adapter4 and Adapter8 are set for upsampling, that is, the resolution of the feature map is improved and the number of channels is adjusted at the same time; Adapter16 adjusts the feature dimension without changing the resolution; Adapter32 increases the number of channels and reduces the spatial resolution through a deconvolution operation with a step size of 2;

[0027] The multi-scale feature learning process is as follows:

[0028] Semantic feature map of remote sensing images in two phases v1 and f v2 After four adaptation operations, we get multi-scale feature maps with downsampling times of 32, 16, 8, and 4. For a single feature map f vk , k = 1, 2, the output of the i-th layer is expressed by the following formula:

[0029]

[0030] Among them, Conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; It is the feature map of the i-th layer in the k-th phase after downsampling; is the final multi-scale feature map after the adaptation operation.

[0031] Furthermore, the spatial attention layer in step 4 is a multi-layer convolutional layer structure, which is used to learn the overall features after the two phases are spliced. The specific steps are as follows:

[0032] Step 4.1: The two-phase multi-scale feature maps corresponding to the same scale and Concatenate along the channel dimension to obtain the concatenated feature map

[0033]

[0034] in, Represents channel-level feature concatenation operations;

[0035] Step 4.2: Construct the spatial attention layer and input the concatenated feature map The first convolution layer uses a 1x1 convolution kernel to concatenate the input feature maps. The channels are compressed, and then remapped to the original or specified channel dimension using 1x1 convolution to generate an attention weight matrix consistent with the input image size and multiply it with the input image; it is expressed by the following formula:

[0036]

[0037] in, Represents the visual features after global feature learning.

[0038] Furthermore, the step 5 specifically includes:

[0039] Step 5.1: Initialize the CLIP text encoder using existing pre-trained weights;

[0040] Step 5.2: Get the input text hint L∈R l , where l is the length of the input text prompt; during the training phase, the input text prompt is replaced by the label category of the specific template sentence; the input text prompt is input into the text encoder to obtain the word embedding and overall expression embedding The word embedding f w Learn specific task information and embed the overall expression f all Grasp the overall message expressed in the text.

[0041] Furthermore, the specific operation of fusion feature in step 6 is expressed as the following formula:

[0042]

[0043] in, represents the visual features after global feature learning, fw is word embedding; MCA stands for multi-head cross attention operation; W q , W k and W v Linear matrices representing query, key, and value, respectively, map feature vectors to corresponding feature spaces, C i represents the number of channels in the fusion feature space, It represents the activated visual features after the i-th layer fuses the semantic features.

[0044] Furthermore, the specific construction method of the dual-channel decoder in step 7 is as follows:

[0045] Step 7.1: In the visual pathway, in the visual pathway decoder, for each layer of image features after the pre-fusion module, flatten it in the spatial dimension and add sinusoidal position encoding, the formula is as follows:

[0046]

[0047] Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, Represents activated visual features embedded in sinusoidal positional coding;

[0048] Step 7.2: Embed the activated visual features of the sinusoidal position encoding With word embedding f w After concatenation, the multimodal tokens are subjected to multi-head self-attention operation to further learn the relevant information between multimodal features. The formula is as follows:

[0049]

[0050] Among them, MSA represents multi-head self-attention operation, Represents the multi-layer visual features after fusing multimodal information;

[0051] Step 7.3: Use a pyramid-structured decoder to fuse multiple layers of visual features, from the bottom-level details to the high-level abstract information, and integrate the information at each level by upsampling and fusing the low-level features step by step; introduce a cross-attention mechanism between visual features and text features at each pyramid level, with visual features as queries and text features as keys and values; that is, combine the visual features of each layer into a single decoder. and word embedding f w The total visual features after activation are obtained by decoding through the cross-attention mechanism, and the public representation is as follows:

[0052]

[0053] Among them, f VRepresents the total visual features after activation, and FPN represents the pyramid structure decoder;

[0054] Step 7.4: In the semantic path, embed f with the overall expression all To query, activate the visual feature f actv For the key and value, a cross attention operation is performed, followed by a multi-head self-attention operation. The formula is as follows:

[0055] f L =MSA(MCA(f all ,f actv ))

[0056] Among them, f L is the semantic embedding obtained after passing through the semantic pathway decoder;

[0057] Step 7.5: Obtain the total activated visual feature f V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

[0058] The beneficial effects brought by adopting the above technical solution are:

[0059] 1) The present invention designs a language-guided remote sensing image change detection architecture. Specifically, based on a multi-head cross-attention mechanism, by designing a multi-layer visual-language feature cross-layer, the model fully learns the relevant information between the image and the corresponding type of text, and realizes universal detection of multi-category remote sensing image changes guided by natural language.

[0060] 2) The present invention adopts Efficient SAM (Segment Anything Model) as the basic visual model. Compared with the traditional SAM model, Efficient SAM reduces the number of parameters to 5% of the original by using the SAM mask image pre-training method (SAMI), and runs 20 times faster. It is a lightweight and efficient basic visual model. Based on its powerful feature extraction and semantic segmentation capabilities, it can improve the performance of the model while reducing the model's dependence on the number and quality of remote sensing image samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 The present invention is a flow chart of the method.

[0062] Figure 2 To process each category in the SECOND dataset separately to obtain the corresponding binary segmentation image.

[0063] Figure 3A schematic diagram of the network structure of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0065] The present invention designs a language-guided remote sensing image change detection architecture. Through a multi-head cross-attention mechanism and a multi-layer visual-language feature cross-fusion layer, the model can fully learn the relevant information between the image and the corresponding type of text, and realize the universal detection of multi-category remote sensing image changes guided by natural language. In addition, a lightweight Efficient SAM is used as the visual basic model to extract features from the image, which improves the running speed while reducing the number of parameters and reduces the model's dependence on the number of training samples. At the same time, a multi-layer adapter is designed to extract multi-scale features of the image, and a pyramid-structured decoder is used to enable the model to learn the range of changes at different granularities.

[0066] A remote sensing image change detection method based on language guidance includes training set data classification, constructing Efficient SAM to learn preliminary features of two-temporal remote sensing images, constructing a multi-layer adapter to extract multi-scale features of remote sensing images, constructing a spatial attention module to generate attention weights of spliced ​​images, initializing a pre-trained text encoder and obtaining word embedding and overall expression embedding according to input text prompts, constructing a pre-fusion module to enable text and image information to interact, constructing a dual-channel decoder to obtain image features and semantic embedding, and outputting segmentation masks through similarity calculation.

[0067] The specific implementation flow chart of the remote sensing image change detection method based on language guidance of the present invention is as follows: Figure 1 As shown, the network structure diagram is as follows Figure 3 The specific implementation steps are as follows:

[0068] Step 1: Collect public remote sensing image change detection datasets and preprocess the datasets.

[0069] Collect the remote sensing change detection public dataset SECOND, which is a semantic change detection dataset that collects 4662 pairs of aerial images from multiple platforms and sensors. These image pairs are distributed in cities such as Hangzhou, Chengdu, and Shanghai. The size of each image is 512x 512 and annotated at the pixel level. The second focus is on 6 major land cover categories, namely non-vegetated surface, trees, low vegetation, water, buildings, and playgrounds, which are often involved in natural and man-made geographic changes.

[0070] like Figure 2 As shown, the specific implementation of the present invention for preprocessing the training set in the data set is as follows:

[0071] 1) Extract the change detection targets of different categories separately by color value detection to obtain the binary segmentation images corresponding to each category;

[0072] 2) The corresponding segmentation maps of the two phases are processed by means of pixel value XOR to obtain a single-category change detection data set and store it in a classified manner.

[0073] 3) Divide each category data set into training set, validation set and test set in the ratio of 7:1:2.

[0074] Step 2: Select two remote sensing change detection images corresponding to each other in two time periods from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images in the two time periods.

[0075] Initialize the Efficient SAM model of the visual basic model. The details are as follows:

[0076] 1) Use SAMI pre-trained lightweight encoders (such as ViT-Tiny and ViT-Small) as image encoders, use SAM’s default mask decoder as the decoder of EfficientSAM, and fine-tune the EfficientSAM model on the SA-1B dataset to obtain the model pre-training weights.

[0077] 2) Set the training batch size to 4, and select two corresponding remote sensing change detection images I1, I2∈R each time H×W×3 Through the EfficientSAM model, we obtain the semantic features of remote sensing images of two phases with 256 channels and 16-fold downsampling of feature dimensions (4, 256, 32, 32). v1 and f v2 , expressed as:

[0078]

[0079] Among them, H v and W v Represent the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

[0080] Step 3: construct a multi-layer adapter to extract a two-phase multi-scale feature map based on the two-phase remote sensing image semantic feature map.

[0081] The multi-layer adapter is composed of four adapters in total, each of which is composed of multiple layers of convolution and deconvolution layers with the same structure. By adjusting the resolution and number of channels of the feature map, the multi-scale features of the input image are learned to achieve multi-scale feature fusion.

[0082] In order to obtain a multi-scale feature map by downsampling the original image by 32 times, 16 times, 8 times, and 4 times, four adapters are set, named Adapter4, Adapter8, Adapter16, and Adapter32 respectively.

[0083] For a single feature map f vk , k = 1, 2, the output of the i-th layer is expressed by the following formula:

[0084]

[0085] Among them, conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; is the feature map of the i-th layer in the k-th phase after downsampling, k = 1, 2, i = 1, 2, 3, 4, corresponding to downsampling by 32 times, 16 times, 8 times and 4 times; For the final multi-layer feature map after the convolution operation, the adapter includes two steps of downsampling and subsequent convolutional layers.

[0086] Considering that Efficient SAM is based on the ViT architecture and has downsampled the original image 16 times, Adapter4 and Adapter8 are set for upsampling, that is, the resolution of the feature map is increased and the number of channels is adjusted at the same time. Adapter16 adjusts the feature dimension without changing the resolution. Adapter32 increases the number of channels and reduces the spatial resolution through a deconvolution operation with a step size of 2. The adjusted number of channels is 64, 128, 256, and 512, respectively, using the number of channels to compensate for the image information lost by downsampling.

[0087] The specific implementation of building a multi-layer adapter is as follows:

[0088] 1) According to the calculation formula of the deconvolution layer to change the output feature map size: output height (width) = (input height (width) - 1) × step size - 2 × padding + convolution kernel height (width) + output padding, adjust the deconvolution layer parameters, and downsample the original image by 32 times, 16 times, 8 times, and 4 times respectively. Then use batch normalization and ReLU activation function to stabilize the training process.

[0089] 2) After adjusting the sampling rate, the feature dimensions are adjusted to 64, 128, 256, and 512 respectively using 1x1 convolutional layer batch normalization and ReLU.

[0090] Step 4: Build spatial attention to further mine the global spatial features of remote sensing images at various scales.

[0091] The spatial attention layer is a multi-layer convolutional layer structure, which is used to learn the overall features of the concatenation of two phases and enable the network to learn which spatial positions in the input feature map are more important, so that more attention can be paid to these positions in subsequent processing. The specific steps are as follows:

[0092] 1) The two-phase multi-scale feature maps corresponding to the same scale and Splicing is performed along the channel dimension to obtain the spliced ​​feature maps with dimensions of 128, 256, 512, and 1024 respectively (the original dimensions are 64, 128, 256, and 512. The channel dimension is doubled after the two-phase feature maps are spliced. It can be understood as a global feature map after simple splicing, and waits for subsequent global feature learning), which is expressed by the following formula;

[0093]

[0094] in Represents channel-level feature concatenation operations;

[0095] 2) Construct a spatial attention layer, where the first layer of convolution uses a 1x1 convolution kernel to compress the channels of the input concatenated feature map. The network learns how to aggregate information in the channel dimension, that is, which channels’ information is more important to the final attention distribution. The convolution kernel size is set to 1 so that the spatial information is retained, but the information between channels is effectively integrated, reducing the dimension for efficient calculation. Finally, use 1x1 convolution again to remap to the original or specified channel dimension, generate an attention weight matrix that is consistent with the size of the input map, and multiply it with the input image, expressed by the following formula:

[0096]

[0097] in Represents the visual features after global feature learning.

[0098] Step 5: Use the pre-trained image-natural language model CLIP (Contrastive Language-Image Pre-Training) to process the input text prompts through the natural language encoder to obtain word embeddings and overall expression embeddings.

[0099] A Transformer-based multimodal pre-trained model, CLIP, is used to extract information of different scales in text prompts. The rationale is that the CLIP model is pre-trained on a dataset of 400 million (image, text) pairs collected from the Internet, allowing the model to understand the image content and generate matching text descriptions, while also being able to retrieve corresponding images based on text queries. Its core advantage is that the learned embedding vectors of images and texts have strong semantic consistency, which makes CLIP's text encoder perform well in a variety of cross-modal downstream tasks.

[0100] During the training phase, for different categories, the trainer gives text prompts by applying model sentences containing the category. For example, if the training sample is forest category change, the corresponding text prompt is "Remote sensing image change detection of forest category". In the prediction phase after the model is trained, the user can give text prompts by himself, such as "Give land changes of forest category", to achieve efficient interaction with the model.

[0101] With its powerful semantic understanding and generalization capabilities, it extracts information about specific tasks and the overall expression of the text from the input text prompts to be applied to downstream cross-modal fusion tasks. The specific steps are as follows:

[0102] 1) Use the official pre-trained weights to initialize the CLIP text encoder and set the upper limit of the text length to 77;

[0103] 2) Get the input text prompt L∈R l , where l is the length of the input text. During the training phase, the input text prompt is replaced by the label category of the specific template sentence. The text prompt is input into the text encoder (the pre-trained text encoder in CLIP) to obtain the word embedding and overall expression embedding C t is the dimension of the matrix; where word embedding f w is responsible for learning more detailed task-specific information, while the overall expression embedding f all Then you have grasped the overall information expressed in the text.

[0104] Step 6: Use the multi-head cross-attention method to build a pre-fusion module for the fusion operation of word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that integrate the semantic features.

[0105] The pre-fusion module is a key component based on the multi-head attention mechanism, which aims to build a bridge between visual features and language features and promote the interaction between the two. Using visual features as queries and word embeddings as keys and values ​​for multi-head cross-attention operations helps focus on visual areas related to language instructions and reduce the interference of irrelevant background information, thereby improving the performance of the model on cross-modal tasks.

[0106] The specific operation of fusion features can be expressed as the following formula:

[0107]

[0108] Among them, MCA stands for multi-head cross attention operation, W q , W k and W v Represents the corresponding linear matrix, mapping the eigenvector to the corresponding eigenspace, C i represents the number of channels in the fusion feature space, Represents the activated visual features after the i-th layer fuses the semantic features.

[0109] Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder. Calculate the similarity between the image features obtained after the visual pathway decoder and the semantic embedding obtained after the semantic pathway decoder, and output the final change detection mask through interpolation and binarization operations.

[0110] The dual-channel decoder design achieves effective integration of visual and language information at different scales and levels through a deep cross-modal fusion strategy, improving the performance and generalization ability of the model in complex scene understanding tasks. The specific construction method is as follows:

[0111] Step 7.1) In the visual pathway, for each layer of activated visual features, first flatten them in the spatial dimension and add sinusoidal position encoding. Note that the visual features here use 2D position embedding, while the text features use 1D position embedding. Applying position embedding ensures that the model can correctly align spatial and sequential information during the fusion process, which is crucial for understanding the relative position in the visual scene and the temporal relationship in the text description. The specific formula is as follows:

[0112]

[0113] Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, Visual features representing activations embedded in sinusoidal positional encodings.

[0114] Step 7.2) Concatenate the position-encoded visual features with the word embeddings, and then perform a multi-head cross-attention operation on the obtained multimodal tokens to further learn the relevant information between the multimodal features. The formula is as follows:

[0115]

[0116] Among them, MSA represents multi-head self-attention operation, Represents multi-layer visual features after fusing multimodal information.

[0117] Step 7.3) A pyramid (FPN) structured decoder is used to fuse features at different scales, from bottom-level details to high-level abstract information. By upsampling and fusing low-level features step by step, the model can integrate information at all levels and enhance the richness and accuracy of feature expression. A cross-attention mechanism between visual features and text features is introduced at each FPN level (visual features are regarded as queries, and text features are used as keys and values). This not only enables visual features to be adjusted according to the semantic guidance of the text, but also enables text features to be integrated into the spatial context of visual features, achieving deeper cross-modal fusion. The visual features of each layer are combined into a single entity. and word embedding f w The total visual features after activation are obtained by decoding through the cross attention mechanism, and the public representation is as follows;

[0118]

[0119] Among them, f V Represents the total visual features after activation, and FPN represents the pyramid structure decoder;

[0120] Step 7.4) In the semantic path, the overall expression embedding is used as the query, the activated visual features are used as the key and value, and a cross-attention operation is performed, followed by a multi-head self-attention operation, which is expressed as follows:

[0121] f L =MSA(MCA(f all ,f actv ))

[0122] Step 7.5) The total visual feature f after activation is obtained V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

[0123] It is worth noting that during the training process, this embodiment adopts a strategy of auxiliary loss calculation, that is, considering that the last layer of visual features passing through the decoder is generally considered to have more global and abstract information and is more suitable for expressing high-level semantics, this embodiment selects the last layer as the input of the final predicted output mask, and the features of the remaining layers are only added to the calculation of the loss function to assist the model adjustment, and are not used as the final output.

Claims

1. A remote sensing image change detection method based on language guidance, characterized in that: The steps include: Step 1: Collect the public remote sensing image change detection dataset and preprocess the dataset; Step 2: Select two remote sensing change detection images corresponding to each other in two time periods from the preprocessed remote sensing image change detection dataset, use Efficient SAM as the visual basic model, and extract the semantic feature maps of the remote sensing images in the two time periods; Step 3: construct a multi-layer adapter to extract a two-phase multi-scale feature map according to the two-phase remote sensing image semantic feature map; Step 4: Construct spatial attention, and further mine the visual features of the remote sensing image at each scale after global feature learning according to the two-phase multi-scale feature map, that is, the global spatial feature map, and splice the global spatial feature maps of different scales into a multi-layer global spatial feature map; Step 5: Use the pre-trained text encoder in the image-natural language model CLIP to process the input text prompts corresponding to the training samples to obtain word embeddings and overall expression embeddings; Step 6: Use the multi-head cross attention method to build a pre-fusion module for the fusion operation of word embedding and the concatenated multi-layer global spatial feature map to obtain the activated visual features that integrate the semantic features; Step 7: Construct a dual-channel decoder structure, including a visual pathway decoder and a semantic pathway decoder; The input is the activated visual features fused with semantic features, and after passing through the visual pathway decoder, the total visual features are output, and after passing through the semantic pathway decoder, the semantic embedding is output; The total visual features and semantic embeddings are similarly calculated, and the final change detection mask is output through interpolation and binarization operations.

2. The method for detecting changes in remote sensing images based on language guidance according to claim 1, characterized in that: In step 1, the data set is preprocessed as follows: Step 1.1: Extract the change detection targets of different categories separately through color value detection to obtain the binary segmentation images corresponding to each category; Step 1.2: Process the corresponding segmentation images of the two phases by means of pixel value XOR to obtain a single-category change detection data set and store it by category; Step 1.3: Divide the dataset of each category into training set, validation set and test set according to the proportion.

3. The method for detecting changes in remote sensing images based on language guidance according to claim 1, characterized in that: The specific process of step 2 is as follows: Step 2.1: Select two temporal phases of remote sensing change detection images I1 and I2, expressed as: I1,I2∈R H×W×3 Where H and W are the height and width of the input image respectively; Step 2.2: Use the SAMI pre-trained lightweight encoder as the image encoder, and use the default mask decoder of SAM as the decoder of EfficientSAM, and fine-tune the EfficientSAM model on the SA-1B dataset to obtain the model pre-training weights; Step 2.3: Process the remote sensing change detection images I1 and I2 of the two time phases respectively to obtain the semantic feature maps f of the remote sensing images of the two time phases v1 and f v2 , expressed as: Among them, H v and W v Represent the height and width of the output feature map, C v represents the number of channels, and EFFSAM represents the pre-trained Efficient SAM model.

4. The method for detecting changes in remote sensing images based on language guidance according to claim 3, characterized in that: The multi-layer adapter in step 3 includes four multi-layer convolution and deconvolution layers with different parameters and the same structure, respectively named Adapter4, Adapter8, Adapter16, and Adapter32, which are used to extract features of different fine-grained images; Adapter4 and Adapter8 are set for upsampling, that is, the resolution of the feature map is improved and the number of channels is adjusted at the same time; Adapter16 adjusts the feature dimension without changing the resolution; Adapter32 increases the number of channels and reduces the spatial resolution through a deconvolution operation with a step size of 2; The multi-scale feature learning process is as follows: Semantic feature map of remote sensing images in two phases v1 and f v2 After four adaptation operations, we get multi-scale feature maps with downsampling times of 32, 16, 8, and 4. For a single feature map f vk , k = 1, 2, the output of the i-th layer is expressed by the following formula: Among them, Conv represents a 1*1 convolutional layer, BN represents a batch normalization operation, γ represents a RELU activation function, and Downsample represents a downsampling operation consisting of a convolutional layer; It is the feature map of the i-th layer in the k-th phase after downsampling; is the final multi-scale feature map after the adaptation operation.

5. The method for detecting changes in remote sensing images based on language guidance according to claim 4, characterized in that: The spatial attention layer in step 4 is a multi-layer convolutional layer structure, which is used to learn the overall features after the two phases are spliced. The specific steps are as follows: Step 4.1: The two-phase multi-scale feature maps corresponding to the same scale and Concatenate along the channel dimension to obtain the concatenated feature map in, Represents channel-level feature concatenation operations; Step 4.2: Construct the spatial attention layer and input the concatenated feature map The first convolution layer uses a 1x1 convolution kernel to concatenate the input feature maps. The channels are compressed, and then remapped to the original or specified channel dimension using 1x1 convolution to generate an attention weight matrix consistent with the input image size and multiply it with the input image; it is expressed by the following formula: in, Represents the visual features after global feature learning.

6. The method for detecting changes in remote sensing images based on language guidance according to claim 1, characterized in that: The step 5 specifically includes: Step 5.1: Initialize the CLIP text encoder using existing pre-trained weights; Step 5.2: Get the input text hint L∈R l , where l is the length of the text prompt; during the training phase, the input text prompt is replaced by the label category of the specific template sentence; the text prompt is input into the text encoder to obtain the word embedding and overall expression embedding C t is the dimension of the matrix; where word embedding f w Learn specific task information and embed the overall expression f all Grasp the overall message expressed in the text.

7. The method for detecting changes in remote sensing images based on language guidance according to claim 6, characterized in that: The specific operation of fusion feature in step 6 is expressed as the following formula: in, represents the visual features after global feature learning, f w is word embedding; MCA stands for multi-head cross attention operation; W q , W k and W v Linear matrices representing query, key, and value, respectively, map feature vectors to corresponding feature spaces, C i represents the number of channels in the fusion feature space, Represents the activated visual features after the i-th layer fuses the semantic features.

8. The method for detecting changes in remote sensing images based on language guidance according to claim 7, characterized in that: The specific construction method of the dual-channel decoder in step 7 is as follows: Step 7.1: In the visual pathway decoder, for each layer of image features after the pre-fusion module, It is flattened in the spatial dimension and adds sinusoidal position encoding, the formula is as follows: Among them, F represents the spatial dimension flattening operation, Pos sin represents the sinusoidal position encoding, Represents activated visual features embedded in sinusoidal positional coding; Step 7.2: Embed the activated visual features of the sinusoidal position encoding With word embedding f w After concatenation, the multimodal tokens are subjected to multi-head self-attention operation to further learn the relevant information between multimodal features. The formula is as follows: Among them, MSA represents multi-head self-attention operation, Represents the multi-layer visual features after fusing multimodal information; Step 7.3: Use a pyramid-structured decoder to fuse multiple layers of visual features, from the bottom-level details to the high-level abstract information, and integrate the information at each level by upsampling and fusing the low-level features step by step; introduce a cross-attention mechanism between visual features and text features at each pyramid level, with visual features as queries and text features as keys and values; that is, combine the visual features of each layer into a single decoder. and word embedding f w The total visual features after activation are obtained by decoding through the cross attention mechanism, and the public representation is as follows; Among them, f V Represents the total visual features after activation, and FPN represents the pyramid structure decoder; Step 7.4: In the semantic pathway decoder, embed f with the overall expression all To query, activate the visual feature f actv For the key and value, a cross attention operation is performed, followed by a multi-head self-attention operation. The formula is as follows: f L =MSA(MCA(f all ,f actv )) Among them, f L is the semantic embedding obtained after passing through the semantic pathway decoder; Step 7.5: Obtain the total activated visual feature f V With semantic embedding f L A matrix multiplication operation is performed to calculate the similarity between the two, and finally a predicted change detection mask is generated through bilinear interpolation and binarization operations.

Images