A Remote Sensing Image Change Detection Method Based on Fine-tuned CLIP

Abstract

This invention discloses a remote sensing image change detection method based on fine-tuned CLIP, comprising: acquiring original dual-time remote sensing images and preprocessing the remote sensing images; constructing an initial remote sensing image change detection model, training the initial remote sensing image change detection model using a training set to obtain a target remote sensing image change detection model; and using the target remote sensing image change detection model to detect changes in the remote sensing images. This invention introduces a large model, CLIP, pre-trained on a large dataset, into the change detection task, aiming to guide the model to focus on the general features of changed ground features, thereby improving its generalization ability. This invention uses visual cues to fine-tune CLIP, ensuring its good transferability in downstream remote sensing tasks. Furthermore, when constructing the loss function, this invention also introduces the Dice coefficient loss to enhance the model's sensitivity to positive examples.

Description

Technical Field

[0001] This invention relates to remote sensing image detection technology, and in particular to a method for detecting changes in remote sensing images based on fine-tuning CLIP. Background Technology

[0002] Remote sensing image change detection refers to identifying targets that have changed at the pixel level by comparing remote sensing images of the same area at different time periods. As a crucial Earth observation tool in remote sensing technology, it plays a key role in numerous application areas such as disaster assessment, land use statistics, and urban and rural planning. Therefore, designing a change detection method with high accuracy and strong generalization is of great significance.

[0003] In recent years, with the continuous advancement of deep learning technology, deep learning methods have been widely used in remote sensing image change detection and have achieved remarkable success. However, the following shortcomings still exist:

[0004] Deep learning methods typically train on a specific training set and then apply the model to a test set to evaluate its performance. This approach can lead to the model learning only specific types of variation from a limited set of labeled data, thus affecting its ability to adapt to new situations and reducing its generalization ability. For example, if the training set mainly contains buildings with white roofs, while the test set includes some buildings with black roofs, the model may fail to recognize this type of variation, resulting in false positives or false negatives.

[0005] Therefore, how to design a method to enhance the generalization ability of the model has become an urgent problem to be solved in the field of remote sensing image change detection. Summary of the Invention

[0006] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a method for detecting changes in remote sensing images based on fine-tuned CLIP.

[0007] Technical solution: The present invention provides a remote sensing image change detection method based on fine-tuned CLIP, comprising:

[0008] The original dual-time remote sensing images were acquired and preprocessed. The preprocessed remote sensing images were then used to construct a dataset, which was divided into a training set and a test set according to the proportions.

[0009] An initial remote sensing image change detection model is constructed, and the initial remote sensing image change detection model is trained using the training set to obtain the target remote sensing image change detection model; the loss function of the initial remote sensing image change detection model consists of optimized binary cross-entropy loss and Dice coefficient loss;

[0010] Target remote sensing image change detection model is used to detect changes in remote sensing images;

[0011] The initial remote sensing image change detection model includes:

[0012] Construct a first residual network, a second residual network, a first visual encoder, and a second visual encoder;

[0013] Any one of the two-time remote sensing images is input into the first residual network to generate the first specific feature, and the remote sensing image is input into the first visual encoder to generate the first general feature;

[0014] Another remote sensing image is input into the second residual network to generate a second specific feature, and the remote sensing image is input into the second visual encoder to generate a second general feature;

[0015] Both the first and second visual encoders are CLIP-based visual encoders with visual cueing fine-tuning.

[0016] Furthermore, constructing the initial remote sensing image change detection model also includes:

[0017] Construct a first feature fusion module and a second feature fusion module;

[0018] The first specific feature and the first general feature are preprocessed separately and then input into the first feature fusion module to obtain the first fused feature;

[0019] The second specific feature and the second general feature are preprocessed separately and then input into the second feature fusion module to obtain the second fused feature;

[0020] The first fusion feature and the second fusion feature are concatenated along the sequence length dimension, and then self-attention mechanism operation and feedforward neural network operation are performed to obtain the first comprehensive feature;

[0021] The first comprehensive feature is separated to obtain the second comprehensive feature and the third comprehensive feature;

[0022] The second comprehensive feature and the first specific feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information specific feature;

[0023] By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second specific feature, the second change information specific feature is obtained;

[0024] The second comprehensive feature and the first general feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information general feature;

[0025] By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second general feature, the second change information general feature is obtained;

[0026] A classification head module is constructed, and the first change information-specific features, the second change information-specific features, the first change information-general features, and the first change information-general features are input into the classification head module to obtain the final detection result.

[0027] Further, the first specific feature and the first general feature are preprocessed and then input into the first feature fusion module to obtain the first fused feature, which includes:

[0028] The first specific feature is passed through a convolutional layer, and then the operation is performed in the spatial dimension. Softmax The operation yields the attention score for each spatial location within the channel. This attention score is then used to perform a weighted summation of the first specific features before dimensionality reduction along the spatial dimension, generating semantic tags. , represented as:

[0029] ,

[0030] In the formula, Indicates the first specific feature, Represents matrix multiplication. This indicates a convolution operation with a kernel size of 3×3. The semantic markup shape is , For sequence length, c Number of channels;

[0031] The first general feature is reduced in dimensionality using convolution and a multilayer perceptron to generate semantic tags. , represented as:

[0032] ,

[0033] In the formula, Indicates the first general feature, MLP Represents a multilayer perceptron. For activation function, This indicates a convolution operation using two 1×1 kernels;

[0034] Using dot product to transfer semantic tags and semantic tags The first fusion feature is obtained by fusion. , represented as:

[0035] ,

[0036] In the formula, This indicates element-wise multiplication of matrices.

[0037] Furthermore, the first and second fusion features are concatenated along the sequence length dimension, followed by self-attention mechanism operations and feedforward neural network operations to obtain the first comprehensive feature, represented as:

[0038] ,

[0039] ,

[0040] In the formula, Indicates the features after splicing. This indicates a splicing operation. Indicates the second fusion feature, Indicates the first comprehensive feature, , and Represents the transformation matrix. This represents a feedforward neural network; The attention mechanism operation is represented by the following formula:

[0041] ,

[0042] In the formula, , , These represent the three input items for the attention mechanism operation, indicated by superscript. T Indicates matrix transpose. For scaling items.

[0043] Furthermore, the first comprehensive feature is separated to obtain the second and third comprehensive features, including:

[0044] The first comprehensive feature The sequence is divided into two parts along the length dimension to obtain the second comprehensive feature. and the third comprehensive feature , represented as:

[0045] .

[0046] Furthermore, the second comprehensive feature and the first specific feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information specific feature, which is represented as:

[0047] ,

[0048] In the formula, This is a special feature representing the first change information.

[0049] Furthermore, the second comprehensive feature and the first general feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information general feature, which is represented as:

[0050] ,

[0051] In the formula, This indicates the general characteristics of the first change information.

[0052] Furthermore, the first change information-specific feature, the second change information-specific feature, the first change information general feature, and the first change information general feature are input into the classification head module to obtain the final detection result, including:

[0053] Subtracting the first change information general feature from the first change information general feature yields the dual-time image general difference feature;

[0054] Subtracting the first and second change-specific features yields the dual-time image-specific difference features. Then, bilinear interpolation upsampling and a 1×1 convolution are performed, followed by addition to the dual-time image-general difference features, resulting in difference features containing change information, general features, and specific features. , is represented as:

[0055] ,

[0056] In the formula, This indicates the general characteristics of the second change information. This indicates a special feature for representing the second change information. H , W , C These represent the height, width, and number of channels of the original remote sensing image, respectively. Upsample This indicates an upsampling operation based on bilinear interpolation;

[0057] Difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , is represented as:

[0058] .

[0059] Furthermore, the loss function of the initial remote sensing image change detection model is expressed as:

[0060] ,

[0061] in, The optimized binary cross-entropy loss is expressed as:

[0062] ,

[0063] The Dice coefficient loss is expressed as:

[0064] ,

[0065] In the formula, y Indicates training labels, This indicates the result of the model's prediction. This represents the L1 norm.

[0066] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0067] 1. By introducing the large model CLIP, this invention enables the remote sensing image change detection model to obtain general features in the image data, thereby improving the accuracy and generalization of the model. This invention verifies the generalization and accuracy of the invention from the perspectives of qualitative and quantitative analysis.

[0068] 2. In terms of accuracy, through quantitative analysis, the accuracy indicators of this invention on the public dataset WHU-CD are significantly improved compared with existing change detection methods; through qualitative analysis, the detection results generated by this invention have a significant improvement in overall detection performance compared with other methods.

[0069] 3. From the perspective of generalization, the ablation experiment verified that after removing the CLIP model, the detection index of the model decreased significantly, and the detection effect on the same land cover type but different manifestations was significantly worse, indicating that the use of CLIP plays an important role in the generalization of the model. Attached Figure Description

[0070] Figure 1 This is a flowchart of the remote sensing image change detection method based on fine-tuning CLIP of the present invention;

[0071] Figure 2 This is a structural diagram of the initial remote sensing image change detection model of the present invention;

[0072] Figure 3 This is a flowchart illustrating the fusion of general and specific features of this invention;

[0073] Figure 4 This is a flowchart of the visual cues for the fine-tuning method used in this invention;

[0074] Figure 5 This is a flowchart of the self-attention and feedforward neural network used in this invention;

[0075] Figure 6 This is a flowchart of the cross-attention and feedforward neural network used in this invention;

[0076] Figure 7 This is a comparison chart of the detection results of this invention and other models;

[0077] Figure 8 This is a diagram showing the ablation results of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0079] The flowchart of the remote sensing image change detection method based on fine-tuning CLIP described in this embodiment is as follows: Figure 1 As shown, the method includes at least steps 1 to 3 as follows.

[0080] Step 1: Obtain the original dual-time remote sensing images and preprocess them. Use the preprocessed remote sensing images to construct a dataset, which is divided into training and test sets according to the proportions.

[0081] Step 2: Construct an initial remote sensing image change detection model. Train the initial remote sensing image change detection model using the training set to obtain the target remote sensing image change detection model.

[0082] The loss function of the initial remote sensing image change detection model consists of optimized binary cross-entropy loss and Dice coefficient loss;

[0083] Step 3: Use the target remote sensing image change detection model to detect changes in the remote sensing image;

[0084] like Figure 2 As shown, the initial remote sensing image change detection model includes:

[0085] Construct a first residual network, a second residual network, a first visual encoder, and a second visual encoder;

[0086] Any one of the two-time remote sensing images is input into the first residual network to generate the first specific feature, and the remote sensing image is input into the first visual encoder to generate the first general feature;

[0087] Another remote sensing image is input into the second residual network to generate a second specific feature, and the remote sensing image is input into the second visual encoder to generate a second general feature;

[0088] Both the first and second visual encoders are CLIP-based visual encoders with visual cueing fine-tuning.

[0089] A bitemporal remote sensing image consists of two images of the same object taken at different times: one before the change and the other after the change. Due to the large size of the remote sensing images, and to reduce computational complexity and resource consumption, the original bitemporal remote sensing images need to be cropped to obtain the bitemporal input features. and ,in, Represents remote sensing images before the change. This represents the changed remote sensing image. Represents the height of the image. The width represents the image, and 3 indicates that the remote sensing image is an RGB three-channel image. A large number of high-resolution dual-time remote sensing images were acquired, cropped into blocks, and constructed into a dataset, which was then proportionally divided into training and testing sets.

[0090] The first and second residual networks have identical structures, and they share weights to ensure consistency in feature extraction. To reduce the number of parameters and computational complexity, the two residual networks employ a lightweight ResNet18 network model.

[0091] The first and second visual encoders have identical structures, and weights are shared between them to ensure consistency in feature extraction. The first visual encoder is the CLIP visual encoder with visual cue fine-tuning, and it can use the lightweight ViT-B / 16. CLIP is a multimodal pre-trained model, trained using 400 million pairs of images and text crawled from the internet, including not only natural scene images but also specialized data such as remote sensing and medical images. Pre-training on large-scale datasets allows the CLIP model to achieve zero-shot performance comparable to state-of-the-art supervised learning on many datasets.

[0092] Because ground features in change detection tasks vary significantly in shape, color, and other aspects, traditional models that rely solely on specialized knowledge extraction may only learn limited knowledge with a small amount of labeled data, resulting in insufficient generalization performance. To address this issue, this invention uses a residual network to extract specialized features from images and combines it with a CLIP visual encoder to extract general features. By combining general and specialized features, the model's ability to adapt to new changes is improved, thereby enhancing its accuracy and generalization.

[0093] However, the CLIP model only used limited remote sensing data during the pre-training phase. To improve its performance in the specific downstream task of remote sensing image change detection, the parameters of the CLIP model need to be adjusted. However, due to limitations in GPU memory and computing resources, directly adjusting the CLIP model parameters is not feasible. Therefore, this invention uses a fine-tuning method to solve this problem. Fine-tuning is a technique for adjusting the parameters of a pre-trained model to adapt to a specific downstream task. It can optimize the model's performance on new tasks without significantly changing the model structure. Specifically, this invention uses visual cues for fine-tuning. Visual Prompt effectively adjusts the input of the visual model by introducing cue information before the input data and each Transformer Block, thereby guiding the model to extract features relevant to specific contexts or tasks more efficiently. Specifically, in Figure 4 In this model, the embedding layer first vectorizes the original image features through convolution operations, generating a series of feature representations. Then, learnable parameters are concatenated with these feature representations and fed into the Transformer layer for learning. During this process, the output of each Transformer layer is again concatenated with the learnable parameters, further optimizing the model's feature representation capabilities and facilitating more efficient fine-tuning.

[0094] Furthermore, constructing the initial remote sensing image change detection model also includes:

[0095] Construct a first feature fusion module and a second feature fusion module;

[0096] The first specific feature and the first general feature are preprocessed separately and then input into the first feature fusion module to obtain the first fused feature;

[0097] The second specific feature and the second general feature are preprocessed separately and then input into the second feature fusion module to obtain the second fused feature;

[0098] The first fusion feature and the second fusion feature are concatenated along the sequence length dimension, and then self-attention mechanism operation and feedforward neural network operation are performed to obtain the first comprehensive feature;

[0099] The first comprehensive feature is separated to obtain the second comprehensive feature and the third comprehensive feature;

[0100] The second comprehensive feature and the first specific feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information specific feature;

[0101] By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second specific feature, the second change information specific feature is obtained;

[0102] The second comprehensive feature and the first general feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information general feature;

[0103] By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second general feature, the second change information general feature is obtained;

[0104] A classification head module is constructed, and the first change information-specific features, the second change information-specific features, the first change information-general features, and the first change information-general features are input into the classification head module to obtain the final detection result.

[0105] To effectively fuse specialized and general features, a feature fusion module is constructed, comprising a first feature fusion module and a second feature fusion module. This module fuses general features with specialized features through dimensionality reduction and gating mechanisms. The flowchart for this step is shown below. Figure 3 As shown, since specialized features and general features differ significantly in tensor shape and semantics, this invention proposes a feature fusion module based on a gating mechanism to more effectively fuse these two types of features.

[0106] Further, the first specific feature and the first general feature are preprocessed and then input into the first feature fusion module to obtain the first fused feature, which includes:

[0107] The first specific feature is passed through a convolutional layer to reduce its dimensionality in the channel dimension, and then processed in the spatial dimension, i.e., the height dimension. Softmax The operation normalizes the spatial information within each channel, obtaining an attention score for each spatial location within the channel. This attention score is then used to perform a weighted summation of the first dedicated features before dimensionality reduction, generating low-dimensional semantic tags rich in semantic information. , represented as:

[0108] ,

[0109] In the formula, Indicates the first specific feature, Represents matrix multiplication. This indicates a convolution operation with a kernel size of 3×3. The semantic markup shape is , For sequence length, c Number of channels;

[0110] The first general feature is subjected to convolution and multilayer perceptron to reduce its dimensionality in terms of sequence length and channel dimension, respectively, thereby aligning the general feature with the specific feature in terms of shape and generating semantic tags. , represented as:

[0111] ,

[0112] In the formula, This represents the first general feature; MLP stands for Multilayer Perceptron. For activation functions; This indicates a convolution operation using two 1×1 kernels. The first 1×1 convolution performs dimensionality reduction on general features, and the second 1×1 convolution... The operations together form a gating mechanism to ensure that general features can adaptively flow into the semantic tags after modeling specific features.

[0113] Using dot product to transfer semantic tags and semantic tags The first fusion feature is obtained by fusion. , represented as:

[0114] ,

[0115] In the formula, This indicates element-wise multiplication of matrices.

[0116] Given that general features and special features may have significant semantic differences, the above fusion process uses a gating mechanism to allow general features to adaptively fuse with the semantic labels of special features to obtain the first fused feature.

[0117] The second fusion feature expression is obtained by implementing the same process as the first fusion feature described above:

[0118] ,

[0119] In the formula, Indicates semantic tags, This indicates a semantic tag.

[0120] To obtain comprehensive features with advanced semantics and rich variation information, the first and second fused features are further concatenated along the sequence length dimension, followed by self-attention mechanism operations and feedforward neural network operations, such as... Figure 5 As shown, the first comprehensive feature is obtained, represented as:

[0121] ,

[0122] ,

[0123] In the formula, Indicates the features after splicing. This indicates a splicing operation. Indicates the second fusion feature, Indicates the first comprehensive feature, , and Represents the transformation matrix. This represents a feedforward neural network, which in its implementation employs a multilayer perceptron with a residual mechanism. The attention mechanism operation is represented by the following formula:

[0124] ,

[0125] In the formula, , , These represent the three input items for the attention mechanism operation, indicated by superscript. T Indicates matrix transpose. For scaling items.

[0126] For the obtained first fusion feature Second fusion features The current approach only incorporates semantic information from the corresponding single-time image, failing to effectively fuse information from both time-based images. Furthermore, simple subtraction and other differential operations cannot effectively represent the features of changing information and lack awareness of contextual information. Attention mechanisms, on the other hand, can acquire richer global and contextual information. Therefore, this invention first concatenates the two-time fused features along the sequence length dimension, and then employs a self-attention mechanism and a feedforward neural network to enhance the perception of changing information, resulting in the first comprehensive feature.

[0127] Furthermore, the first comprehensive feature is separated to obtain the second and third comprehensive features, including:

[0128] The first comprehensive feature The sequence is divided into two parts along the length dimension to obtain the second comprehensive feature. and the third comprehensive feature , is represented as:

[0129] ,

[0130] Furthermore, such as Figure 6 As shown, the second comprehensive feature and the first specific feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information specific feature, which is represented as:

[0131] ,

[0132] In the formula, This is a special feature representing the first change information.

[0133] Although the first comprehensive feature contains high-level semantics and rich variation information, directly using the first comprehensive feature for upsampling and convolution to obtain the final result will cause problems. Since the upsampling kernels are mostly preset and static, such as bilinear interpolation or nearest neighbor interpolation, this leads to poor image segmentation results. Therefore, this invention integrates the first comprehensive feature back into a large-size dedicated feature space and a general feature space through a cross-attention mechanism, thereby obtaining a large-size pixel-level feature space that incorporates variation information. , and special features , .

[0134] Furthermore, the second comprehensive feature and the first general feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information general feature, which is represented as:

[0135] ,

[0136] In the formula, This indicates the general characteristics of the first change information.

[0137] To obtain the final difference between general features and special features, this invention performs subtraction operations on the dual-time special features and general features respectively, thereby completing the modeling of the difference between dual-time general features and special features; then, for the smaller differential general features, simple upsampling combined with 1×1 convolution is used to make the size of the differential general features the same as the size of the differential special features, and then a simple addition operation is used to complete feature aggregation.

[0138] Furthermore, the first change information-specific feature, the second change information-specific feature, the first change information general feature, and the first change information general feature are input into the classification head module to obtain the final detection result, including:

[0139] Subtracting the first change information general feature from the first change information general feature yields the dual-time image general difference feature;

[0140] Subtracting the first and second change-specific features yields the dual-time image-specific difference features. Then, bilinear interpolation upsampling and a 1×1 convolution are performed, followed by addition to the dual-time image-general difference features, resulting in difference features containing change information, general features, and specific features. , represented as:

[0141] ,

[0142] In the formula, This indicates the general characteristics of the second change information. This indicates a special feature for representing the second change information. H , W , C These represent the height, width, and number of channels of the original remote sensing image, respectively. Upsample This indicates an upsampling operation based on bilinear interpolation;

[0143] Difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , represented as:

[0144] ,

[0145] In the formula, Upsample This represents upsampling. In the example, differential features can be used... Upsampled by 4 times, then passed through two 3×3 convolutional layers, where the first convolutional layer has 4 input and 3 output channels. The second convolutional layer input channel is With the output channel set to 1, the final result is obtained. If it's the testing phase, features also need to be considered. conduct The operation involves setting a threshold of 0.5 for each pixel; pixels greater than the threshold are considered variable, and pixels less than 0.5 are considered invariant. During the training phase, the final result... It is fed into the loss function.

[0146] Furthermore, the loss function of the initial remote sensing image change detection model is expressed as:

[0147] ,

[0148] in, The optimized binary cross-entropy loss is expressed as:

[0149] ,

[0150] The Dice coefficient loss is expressed as:

[0151] ,

[0152] In the formula, y Indicates training labels, This indicates the result of the model's prediction. This represents the L1 norm.

[0153] It is a binary classification cross-entropy loss. An optimized version of the algorithm combines the Sigmoid activation function and the binary cross-entropy loss function, alleviating the problem of numerical instability during training. This is used to solve the problem of imbalance between positive and negative classes in change detection tasks.

[0154] To verify the effectiveness of this invention, it was trained and tested on the publicly available dataset WHU-CD. The detection results using this invention were compared with current state-of-the-art change detection networks, including the FC-Diff network based on fully convolutional and differencing, the FC-Unet network based on fully convolutional and UNet, the STANet spatiotemporal attention network based on feature pyramids, and the DASNet dual-temporal attention fully convolutional network. Simultaneously, both comparative and ablation experiments were conducted with quantitative and qualitative analyses.

[0155] The WHU-CD dataset is a publicly available dataset for detecting building changes. It contains pairs of high-resolution aerial images, each 32507×15354 pixels in size. Due to computational resource constraints, these images were first cropped into non-overlapping patches of 256×256 pixels and then randomly divided into two parts in a 7:3 ratio: 5335 pairs were used for training and 2285 pairs were used for testing.

[0156] To verify the effectiveness of the method of this invention from the perspective of accuracy, this invention is compared with the FC-Diff network based on fully convolution and difference, the FC-Unet network based on fully convolution and UNet, the STANet spatiotemporal attention network based on feature pyramid, and the DASNet dual-temporal attention fully convolutional network on the WHU-CD dataset.

[0157] From a qualitative analysis perspective, the test results are visualized, and the results are as follows: Figure 7 As shown in the figure, white represents correctly classified pixels, red represents misclassified pixels, and green represents missed pixels. Figure (a) represents the image before the change, Figure (b) represents the image after the change, Figure (c) represents the label, Figure (d) represents the FC-Unet detection result, Figure (e) represents the FC-diff detection result, Figure (f) represents the DASNet detection result, Figure (g) represents the STANet detection result, and Figure (h) represents the detection result based on the method proposed in this invention. Figure 7 As can be seen, the segmentation effect of the present invention is significantly better than that of other models.

[0158] From a quantitative analysis perspective, the recall, F1 score, precision, crossover ratio, and accuracy of the detection results on the test set are calculated to evaluate the detection performance. The calculation formulas are as follows:

[0159] ,

[0160] ,

[0161] ,

[0162] ,

[0163] ,

[0164] in, Indicates accuracy. Indicates recall rate, Indicates the F1 score. Indicates accuracy rate. The crossover ratio (CRO) is represented by TP; TP indicates that the model correctly detected regions of change, TN indicates that the model correctly detected regions of no change, FP indicates that the model incorrectly detected regions of no change as regions of change, and FN indicates that the model failed to detect regions of actual change. The evaluation results are shown in Table 1.

[0165] Table 1

[0166]

[0167] As shown in Table 1, the method proposed in this invention significantly outperforms the other four methods in all metrics. In summary, compared to the FC-Diff network based on fully convolution and difference, the FC-Unet network based on fully convolution and UNet, the STANet spatiotemporal attention network based on feature pyramids, and the DASNet dual-temporal attention fully convolutional network, this invention has significant advantages in segmentation performance and accuracy.

[0168] To verify that the introduction of CLIP enhances the accuracy and generalization of the present invention, the present invention conducted ablation experiments on the CLIP branch and verified the positive effect of the introduction of the CLIP model on the present invention from the perspectives of quantitative and qualitative analysis.

[0169] From a quantitative analysis perspective, ablation experiments were conducted on the CLIP branch, and the recall, F1 score, precision, crossover ratio, and accuracy of the detection results were calculated on the test set to evaluate the detection performance. The detection results are shown in Table 2.

[0170] Table 2

[0171]

[0172] As can be seen from the data in Table 2, not using CLIP will lead to a significant decrease in the indicators.

[0173] From a qualitative analysis perspective, the visualization results are as follows: Figure 8As shown in the figures, Figure (a) represents the image before the change, Figure (b) represents the image after the change, Figure (c) represents the label, Figure (d) represents the detection result without using the CLIP model, and Figure (e) represents the detection result with the CLIP model. The changed targets in the two sets of images are very similar. From the detection results, both methods can identify the changes in the first set of images quite perfectly. However, for the second set of images, without using CLIP, the model cannot perceive the changes because the feature representation extracted from general knowledge is insufficient to detect them. Although the detection results of the model after using CLIP are not perfect, its ability to perceive changes is significantly enhanced. In summary, adding CLIP as a general feature can effectively improve the model's generalization ability.

[0174] In summary, quantitative and qualitative analysis shows that the present invention has achieved good results in terms of accuracy and generalization.

Claims

1. A method for detecting changes in remote sensing images based on fine-tuned CLIP, characterized in that, include: The original dual-time remote sensing images were acquired and preprocessed. The preprocessed remote sensing images were then used to construct a dataset, which was divided into a training set and a test set according to the proportions. An initial remote sensing image change detection model is constructed, and the initial remote sensing image change detection model is trained using the training set to obtain the target remote sensing image change detection model; the loss function of the initial remote sensing image change detection model consists of optimized binary cross-entropy loss and Dice coefficient loss; Target remote sensing image change detection model is used to detect changes in remote sensing images; The initial remote sensing image change detection model includes: Construct a first residual network, a second residual network, a first visual encoder, a second visual encoder, a first feature fusion module, a second feature fusion module, and a classification head module; both the first and second visual encoders are CLIP visual encoders with visual cues for fine-tuning. Any one of the two-time remote sensing images is input into the first residual network to generate the first specific feature, and the remote sensing image is input into the first visual encoder to generate the first general feature; The first specific feature and the first general feature are preprocessed separately and then input into the first feature fusion module to obtain the first fused feature; Another remote sensing image is input into the second residual network to generate a second specific feature, and the remote sensing image is input into the second visual encoder to generate a second general feature; The second specific feature and the second general feature are preprocessed separately and then input into the second feature fusion module to obtain the second fused feature; The first fusion feature and the second fusion feature are concatenated along the sequence length dimension, and then self-attention mechanism operation and feedforward neural network operation are performed to obtain the first comprehensive feature; The first comprehensive feature is separated to obtain the second comprehensive feature and the third comprehensive feature; The second comprehensive feature and the first specific feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information specific feature; By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second specific feature, the second change information specific feature is obtained; The second comprehensive feature and the first general feature are subjected to cross-attention operation and feedforward neural network operation to obtain the first change information general feature; By performing cross-attention operations and feedforward neural network operations on the third comprehensive feature and the second general feature, the second change information general feature is obtained; The first change information specific feature, the second change information specific feature, the first change information general feature, and the first change information general feature are input into the classification head module to obtain the final detection result.

2. The remote sensing image change detection method based on fine-tuning CLIP according to claim 1, characterized in that, After preprocessing the first specific feature and the first general feature respectively, they are input into the first feature fusion module to obtain the first fused feature, which includes: The first specific feature is passed through a convolutional layer, and then a softmax operation is performed in the spatial dimension to obtain the attention score for each spatial location within the channel. The attention scores are then used to perform a weighted summation of the first specific feature before dimensionality reduction in the spatial dimension to generate semantic tags. , represented as: ， In the formula, Indicates the first specific feature, Represents matrix multiplication. This indicates a convolution operation with a kernel size of 3×3. The semantic markup shape is , c is the sequence length, and c is the number of channels; The first general feature is reduced in dimensionality using convolution and a multilayer perceptron to generate semantic tags. , represented as: ， In the formula, This represents the first general feature; MLP stands for Multilayer Perceptron. For activation function, This indicates a convolution operation using two 1×1 kernels; Using dot product to transfer semantic tags and semantic tags The first fusion feature is obtained by fusion. , represented as: ， In the formula, This indicates element-wise multiplication of matrices.

3. The remote sensing image change detection method based on fine-tuning CLIP according to claim 2, characterized in that, The first and second fusion features are concatenated along the sequence length dimension, followed by self-attention mechanism operations and feedforward neural network operations to obtain the first comprehensive feature, represented as: ， ， In the formula, Indicates the features after splicing. This indicates a splicing operation. Indicates the second fusion feature, Indicates the first comprehensive feature, , and Represents the transformation matrix. This represents a feedforward neural network; The attention mechanism operation is represented by the following formula: ， In the formula, , , These represent the three input terms of the attention mechanism operation, with the superscript T indicating matrix transpose. For scaling items.

4. The remote sensing image change detection method based on fine-tuning CLIP according to claim 3, characterized in that, Separating the first comprehensive feature yields the second and third comprehensive features, including: The first comprehensive feature The sequence is divided into two parts along the length dimension to obtain the second comprehensive feature. and the third comprehensive feature , represented as: 。 5. The remote sensing image change detection method based on fine-tuning CLIP according to claim 4, characterized in that, By performing cross-attention operations and feedforward neural network operations on the second comprehensive feature and the first specific feature, the first change information specific feature is obtained, represented as follows: ， In the formula, This is a special feature representing the first change information.

6. The remote sensing image change detection method based on fine-tuning CLIP according to claim 5, characterized in that, By performing cross-attention operations and feedforward neural network operations on the second comprehensive feature and the first general feature, the first change information general feature is obtained, represented as: ， In the formula, This indicates the general characteristics of the first change information.

7. The remote sensing image change detection method based on fine-tuning CLIP according to claim 6, characterized in that, The first change information-specific feature, the second change information-specific feature, the first change information general feature, and the first change information general feature are input into the classification head module to obtain the final detection result, including: Subtracting the general features of the first change information from the general features of the second change information yields the general difference features of the two-time images. Subtracting the first and second change-specific features yields the dual-time image-specific difference features. Then, bilinear interpolation upsampling and a 1×1 convolution are performed, followed by addition to the dual-time image-general difference features, resulting in difference features containing change information, general features, and specific features. , represented as: ， In the formula, This indicates the general characteristics of the second change information. The second feature represents the specific characteristics of the change information. H, W, and C represent the height, width, and number of channels of the original remote sensing image, respectively. This indicates an upsampling operation based on bilinear interpolation; Difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , represented as: 。 8. The remote sensing image change detection method based on fine-tuning CLIP according to claim 1, characterized in that, The loss function of the initial remote sensing image change detection model is expressed as: ， in, The optimized binary cross-entropy loss is expressed as: ， The Dice coefficient loss is expressed as: ， In the formula, y represents the training label. This indicates the result of the model's prediction. This represents the L1 norm.

Images