Remote sensing image change detection method based on fine tuning CLIP

Abstract

The invention discloses a remote sensing image change detection method based on fine tuning CLIP, and the method comprises the steps: obtaining an original dual-time remote sensing image, and carrying out the preprocessing of the remote sensing image; constructing an initial remote sensing image change detection model, and training the initial remote sensing image change detection model by using the training set to obtain a target remote sensing image change detection model; and performing change detection on the remote sensing image by using the target remote sensing image change detection model. According to the method, a large model CLIP pre-trained in a large-scale data set is introduced into a change detection task, and the method aims at guiding the model to pay attention to universal features of changing ground features, so that the generalization ability of the model is improved; according to the method, the CLIP is finely adjusted by adopting a visual prompt method, so that the CLIP can be well migrated in a remote sensing downstream task; in addition, when a loss function is constructed, Dice coefficient loss is introduced to enhance the sensitivity of the model to a positive example.

Description

Technical Field

[0001] The invention relates to remote sensing image detection technology, and in particular to a remote sensing image change detection method based on fine-tuning CLIP. Background Art

[0002] Remote sensing image change detection refers to identifying changed targets at the pixel level by comparing remote sensing images of the same area at different time periods. As one of the important earth observation methods in remote sensing technology, it plays a key role in many application fields such as disaster assessment, surface utilization statistics, urban and rural planning, etc. Therefore, it is of great significance to design a change detection method with high accuracy and strong generalization.

[0003] In recent years, with the continuous advancement of deep learning technology, deep learning methods have been widely used in remote sensing image change detection and have achieved remarkable success, but there are still the following shortcomings: Deep learning methods are usually trained on a training set of a specific data set, and after training, the model is applied to the test set to evaluate its performance. This method causes the model to only learn specific types of changes from limited labeled data, which affects its ability to adapt to new changes and reduces the model's generalization ability. For example, if the buildings that change in the training set are mostly white roofs, but the test set contains some corresponding buildings with black roofs, the model may not be able to recognize this type of change, resulting in false detection or missed detection.

[0004] Therefore, how to design a method to enhance the generalization ability of the model has become an urgent problem to be solved in the field of remote sensing image change detection. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a remote sensing image change detection method based on fine-tuning CLIP.

[0006] Technical solution: A remote sensing image change detection method based on fine-tuning CLIP of the present invention comprises: Obtain the original dual-time remote sensing images, preprocess the remote sensing images, use the preprocessed remote sensing images to construct a data set, and divide it into a training set and a test set in proportion; An initial remote sensing image change detection model is constructed, and the initial remote sensing image change detection model is trained using the training set to obtain a target remote sensing image change detection model; the loss function of the initial remote sensing image change detection model is composed of an optimized binary cross entropy loss and a Dice coefficient loss; Using the target remote sensing image change detection model to perform change detection on remote sensing images; Among them, building the initial remote sensing image change detection model includes: Constructing a first residual network, a second residual network, a first visual encoder, and a second visual encoder; Input any one of the dual-time remote sensing images into a first residual network to generate a first dedicated feature, and input the remote sensing image into a first visual encoder to generate a first general feature; Inputting another remote sensing image into a second residual network to generate a second dedicated feature, and inputting the remote sensing image into a second visual encoder to generate a second universal feature; Both the first visual encoder and the second visual encoder are visual encoders of CLIP with visual cue fine-tuning.

[0007] Furthermore, constructing the initial remote sensing image change detection model also includes: Construct a first feature fusion module and a second feature fusion module; Preprocessing the first special feature and the first general feature respectively and inputting them into a first feature fusion module to obtain a first fusion feature; Preprocessing the second special feature and the second general feature respectively and inputting them into the second feature fusion module to obtain the second fusion feature; The first fusion feature and the second fusion feature are concatenated in the sequence length dimension, and then the self-attention mechanism operation and the feedforward neural network operation are performed to obtain the first comprehensive feature; Separating the first comprehensive feature to obtain a second comprehensive feature and a third comprehensive feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first special feature to obtain a first change information special feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second special feature to obtain a second change information special feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first general feature to obtain a first change information general feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second general feature to obtain a second change information general feature; A classification head module is constructed, and the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature are input into the classification head module to obtain a final detection result.

[0008] Furthermore, the first special feature and the first general feature are pre-processed and then input into the first feature fusion module, and the first fusion feature obtained includes: The first specialized feature is passed through a convolutional layer and then performed on the spatial dimension SoftmaxOperation, get the attention score of each spatial position in the channel, use the attention score to perform weighted summation of the first dedicated feature before dimensionality reduction on the spatial dimension, and generate semantic tags , expressed as: , In the formula, represents the first special feature, represents matrix multiplication, represents a convolution operation with a convolution kernel size of 3×3. The semantic mark shape is , is the sequence length, c is the number of channels; The first common feature is reduced in dimension using convolution and multi-layer perceptron to generate semantic tags , expressed as: , In the formula, represents the first common feature, MLP represents a multi-layer perceptron, is the activation function, Indicates the use of two convolution operations with a kernel size of 1×1; Use dot product operation to mark semantic and semantic markup Fusion is performed to obtain the first fusion feature , expressed as: , In the formula, Represents the multiplication of corresponding matrix elements.

[0009] Furthermore, the first fusion feature and the second fusion feature are concatenated in the sequence length dimension, and then the self-attention mechanism operation and the feedforward neural network operation are performed to obtain the first comprehensive feature, which is expressed as: , , In the formula, Represents the features after splicing, Represents a splicing operation, represents the second fusion feature, represents the first comprehensive feature, , and represents the transformation matrix, represents a feed-forward neural network; Represents the attention mechanism operation, and the calculation formula is: , In the formula, , , Represent the three input items of the attention mechanism operation, with superscripts T represents the matrix transpose, A scaling item.

[0010] Furthermore, the first comprehensive feature is separated to obtain the second comprehensive feature and the third comprehensive feature, which include: The first comprehensive feature The sequence length dimension is divided into two parts, and the second comprehensive feature is obtained. and the third comprehensive characteristic , expressed as: .

[0011] Furthermore, the second comprehensive feature and the first special feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information special feature, which is expressed as: , In the formula, Indicates the first change information-specific feature.

[0012] Furthermore, the second comprehensive feature and the first general feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information general feature, which is expressed as: , In the formula, Indicates the general characteristics of the first change information.

[0013] Further, the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature are input into the classification head module to obtain the final detection result, including: Subtracting the first change information universal feature from the first change information universal feature to obtain a dual-time image universal difference feature; The first change information-specific feature and the second change information-specific feature are subtracted to obtain the dual-time image-specific difference feature, and then bilinear interpolation upsampling and 1×1 convolution are performed, and then added to the dual-time image universal difference feature to obtain a difference feature with change information, universal features and special features. , expressed as: , In the formula, Indicates the second change information general characteristics, Indicates the second change information dedicated feature, H , W , CRespectively represent the height, width and number of channels of the original remote sensing image, Upsample Represents an upsampling operation based on bilinear interpolation; The difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , expressed as: .

[0014] Furthermore, the loss function of the initial remote sensing image change detection model is expressed as: , in, Represents the optimized binary cross entropy loss, expressed as: , represents the Dice coefficient loss, expressed as: , In the formula, y represents the training label, Represents the result predicted by the model, represents the L1 norm.

[0015] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: 1. The present invention introduces the large model CLIP, so that the remote sensing image change detection model can obtain the common features in the image data, thereby improving the accuracy and generalization of the model. The present invention verifies the generalization and accuracy of the invention from the perspectives of qualitative and quantitative analysis; 2. From the perspective of accuracy, through quantitative analysis, the accuracy indicators of the present invention on the public data set WHU-CD are significantly improved compared with the existing change detection methods; through qualitative analysis, the detection results produced by the present invention are greatly improved in overall detection effect compared with other methods; 3. From the perspective of generalization, the ablation experiment verified that after removing the CLIP model, the detection index of the model decreased significantly, and the detection effect of the same type of land object but different forms of expression became significantly worse, indicating that the use of CLIP plays an important role in the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of a remote sensing image change detection method based on fine-tuning CLIP of the present invention; Figure 2 is a structural diagram of the initial remote sensing image change detection model of the present invention; Figure 3 It is a flow chart of integrating the general features and the special features of the present invention; Figure 4 is a flow chart of visual prompts of the fine-tuning method used in the present invention; Figure 5 is a flow chart of the self-attention and feed-forward neural network used in the present invention; Figure 6 is a flow chart of the cross-attention and feed-forward neural network used in the present invention; Figure 7 It is a comparison diagram of the detection results of the present invention and other models; Figure 8 This is a diagram of the ablation results of the present invention. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0018] The present embodiment describes a remote sensing image change detection method based on fine-tuning CLIP, the flow chart of which is as follows: Figure 1 As shown, the method at least includes the following steps 1 to 3.

[0019] Step 1, obtain the original dual-time remote sensing image, preprocess the remote sensing image, use the preprocessed remote sensing image to construct a data set, and divide it into a training set and a test set in proportion; Step 2, constructing an initial remote sensing image change detection model, using the training set to train the initial remote sensing image change detection model, and obtaining a target remote sensing image change detection model; The loss function of this initial remote sensing image change detection model consists of the optimized binary cross entropy loss and Dice coefficient loss; Step 3, using the target remote sensing image change detection model to perform change detection on the remote sensing image; like Figure 2 As shown, constructing the initial remote sensing image change detection model includes: Constructing a first residual network, a second residual network, a first visual encoder, and a second visual encoder; Input any one of the dual-time remote sensing images into a first residual network to generate a first dedicated feature, and input the remote sensing image into a first visual encoder to generate a first general feature; Inputting another remote sensing image into a second residual network to generate a second dedicated feature, and inputting the remote sensing image into a second visual encoder to generate a second universal feature; Both the first visual encoder and the second visual encoder are visual encoders of CLIP with visual cue fine-tuning.

[0020] Dual-time remote sensing images include two remote sensing images of the same object taken at different times, one is the remote sensing image before the change, and the other is the remote sensing image after the change. Due to the large size of the remote sensing image itself, for the sake of computational complexity and computing resources, it is necessary to crop the original dual-time remote sensing image to obtain the dual-time input feature after cropping. and ,in, represents the remote sensing image before the change, Represents the remote sensing image after the change. Represents the height of the image, Represents the width of the image, and 3 represents that the remote sensing image is an RGB three-channel image. A large number of high-resolution dual-temporal remote sensing images are obtained, cropped into blocks, constructed into a data set, and divided into a training set and a test set in proportion.

[0021] The first residual network and the second residual network have the same structure. To ensure the consistency of feature extraction, the two residual networks share weights. To reduce the number of model parameters and computational complexity, the two residual networks use a lightweight ResNet18 network model.

[0022] The first visual encoder and the second visual encoder have the same structure. To ensure the consistency of feature extraction, the weights are shared between the two visual encoders. The first visual encoder is the CLIP visual encoder with visual cue fine-tuning. The CLIP visual encoder can use the lightweight ViT-B / 16. CLIP is a multimodal pre-trained model. The CLIP model uses 400 million pairs of images and texts crawled from the Internet for multimodal training, including not only natural scene images, but also professional data such as remote sensing images and medical images. Pre-training on large-scale datasets makes the zero-shot performance of the CLIP model comparable to the state-of-the-art level of supervised learning on many datasets.

[0023] Since there are large differences in shape, color, etc. of objects in change detection tasks, traditional models that only rely on specialized knowledge extraction may only be able to learn limited knowledge under a small amount of labeled data, resulting in insufficient generalization performance. In order to solve this problem, the present invention uses a residual network to extract specialized features of the image, and combines the CLIP visual encoder to extract the general features of the image. By combining general features with specialized features, the model's ability to adapt to new changes is improved, thereby improving the accuracy and generalization of the model.

[0024] However, the CLIP model only used limited remote sensing data in the pre-training stage. In order to improve its performance in the specific downstream task of remote sensing image change detection, the parameters of the CLIP model are adjusted. However, due to the limitations of video memory and computing resources, it is not feasible to directly adjust the parameters of the CLIP model. Therefore, the present invention uses the fine-tuning method to solve this problem. Fine-tuning is a technique for adjusting the parameters of a pre-trained model to adapt to specific downstream tasks. It can optimize the performance of the model on new tasks without significantly changing the model structure. In terms of specific methods, the present invention uses a visual prompt method for fine-tuning. The visual prompt Visual Prompt effectively adjusts the input of the visual model by introducing prompt information before the input data and each Transformer Block, thereby guiding the model to more efficiently extract features related to specific contexts or tasks. Specifically, in Figure 4 In the example, the embedding layer first vectorizes the original image features through convolution operations to generate a series of feature representations. Subsequently, the learnable parameters are concatenated with these feature representations and input into the Transformer layer for learning. In this process, the output of each Transformer layer is concatenated with the learnable parameters again, thereby further optimizing the model's feature expression capabilities and promoting more efficient fine-tuning.

[0025] Furthermore, constructing the initial remote sensing image change detection model also includes: Construct a first feature fusion module and a second feature fusion module; Preprocessing the first special feature and the first general feature respectively and inputting them into a first feature fusion module to obtain a first fusion feature; Preprocessing the second special feature and the second general feature respectively and inputting them into the second feature fusion module to obtain the second fusion feature; The first fusion feature and the second fusion feature are concatenated in the sequence length dimension, and then the self-attention mechanism operation and the feedforward neural network operation are performed to obtain the first comprehensive feature; Separating the first comprehensive feature to obtain a second comprehensive feature and a third comprehensive feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first special feature to obtain a first change information special feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second special feature to obtain a second change information special feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first general feature to obtain a first change information general feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second general feature to obtain a second change information general feature; A classification head module is constructed, and the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature are input into the classification head module to obtain a final detection result.

[0026] In order to effectively fuse the special features and the general features, a feature fusion module is constructed, including a first feature fusion module and a second feature fusion module, and the general features are fused with the special features through dimensionality reduction and gating mechanisms. The flowchart of this step is as follows: Figure 3 As shown, since there are large differences in tensor shape and semantics between dedicated features and general features, in order to more effectively fuse these two types of features, the present invention proposes a feature fusion module of dedicated features and general features based on a gating mechanism.

[0027] Furthermore, the first special feature and the first general feature are pre-processed and then input into the first feature fusion module, and the first fusion feature obtained includes: The first special feature is passed through a convolutional layer to reduce the dimension of the special feature in the channel dimension, and then it is performed in the spatial dimension, that is, the height dimension. Softmax The operation is performed to normalize the spatial information in each channel, obtain the attention score of each spatial position in the channel, and use the attention score to perform weighted summation on the spatial dimension of the first dedicated feature before dimensionality reduction to generate a low-dimensional semantic tag with rich semantic information. , expressed as: , In the formula, represents the first special feature, represents matrix multiplication, represents a convolution operation with a convolution kernel size of 3×3. The semantic mark shape is , is the sequence length, c is the number of channels; The first universal feature is used for convolution and multi-layer perceptron to reduce the sequence length and channel dimension respectively, so as to align the universal feature with the special feature in shape and generate semantic tags. , expressed as: , In the formula, represents the first universal feature, MLP stands for multi-layer perceptron, is the activation function; Indicates the use of two convolution operations with a kernel size of 1×1. The first 1×1 convolution completes the dimensionality reduction operation for common features, and the second 1×1 convolution and The operations together form a gating mechanism that ensures that general features can adaptively flow into the semantic tags after modeling specialized features.

[0028] Use dot product operation to mark semantic and semantic markup Fusion is performed to obtain the first fusion feature , expressed as: , In the formula, Represents the multiplication of corresponding matrix elements.

[0029] In view of the fact that there may be great differences between general features and special features at the semantic level, in the above fusion process, a gating mechanism is used to enable the general features to be adaptively fused with the semantic tags of the special features to obtain the first fused features.

[0030] The implementation process of the first fusion feature is the same as the above, and the expression of the second fusion feature is obtained as follows: , In the formula, Represents semantic markup, Represents a semantic tag.

[0031] In order to obtain comprehensive features with high-level semantics and rich variation information, the first fusion feature and the second fusion feature are further concatenated in the sequence length dimension, and then the self-attention mechanism operation and feedforward neural network operation are performed, such as Figure 5 As shown, the first comprehensive feature is obtained, which is expressed as: , , In the formula, Represents the features after splicing, Represents a splicing operation, represents the second fusion feature, represents the first comprehensive feature, , and represents the transformation matrix, represents a feedforward neural network, and a multilayer perceptron with a residual mechanism is used in its specific implementation; Represents the attention mechanism operation, and the calculation formula is: , In the formula, , , Represent the three input items of the attention mechanism operation, with superscripts T represents the matrix transpose, A scaling item.

[0032] For the first fusion feature obtained and the second fusion feature It only contains the semantic information of the corresponding single-time image, while the information of the dual-time image is not effectively fused. In addition, differentiation operations performed only through simple operations such as subtraction cannot effectively represent the characteristics of the change information, and also lack the perception of contextual information. The attention mechanism can obtain richer global information and contextual information. Therefore, the present invention first splices the dual-time fusion features in the sequence length dimension, and then uses the self-attention mechanism and feedforward neural network to enhance the perception of change information to obtain the first comprehensive feature.

[0033] Furthermore, the first comprehensive feature is separated to obtain the second comprehensive feature and the third comprehensive feature, which include: The first comprehensive feature The sequence length dimension is divided into two parts, and the second comprehensive feature is obtained. and the third comprehensive characteristic , expressed as: , Furthermore, if Figure 6 As shown, the second comprehensive feature and the first special feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information special feature, which is expressed as: , In the formula, Indicates the first change information-specific feature.

[0034] Although the first comprehensive feature contains high-level semantics and rich change information, there will be problems if the first comprehensive feature is directly used for upsampling and convolution to obtain the final result. Since the upsampling kernels are mostly preset and static, such as bilinear interpolation or nearest neighbor interpolation, this will lead to poor image segmentation effect. Therefore, the present invention integrates the first comprehensive feature back into a large-scale dedicated feature space and a universal feature space through a cross-attention mechanism, thereby obtaining a large-scale pixel-level feature space that incorporates change information. , and special features , .

[0035] Furthermore, the second comprehensive feature and the first general feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information general feature, which is expressed as: , In the formula, Indicates the general characteristics of the first change information.

[0036] In order to obtain the final difference between the general features and the special features respectively, the present invention performs subtraction operations on the dual-time special features and the general features respectively, thereby completing the modeling of the difference between the dual-time general features and the special features; then, a simple upsampling combined with a 1×1 convolution is used for the smaller-sized differentiated general features, so that the size of the differentiated general features is the same as that of the differentiated special features, and then a simple addition operation is used to complete the feature aggregation.

[0037] Further, the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature are input into the classification head module to obtain the final detection result, including: Subtracting the first change information universal feature from the first change information universal feature to obtain a dual-time image universal difference feature; The first change information-specific feature and the second change information-specific feature are subtracted to obtain the dual-time image-specific difference feature, and then bilinear interpolation upsampling and 1×1 convolution are performed, and then added to the dual-time image universal difference feature to obtain a difference feature with change information, universal features and special features. , expressed as: , In the formula, Indicates the second change information general characteristics, Indicates the second change information dedicated feature, H , W , C Respectively represent the height, width and number of channels of the original remote sensing image; Upsample Represents an upsampling operation based on bilinear interpolation; The difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , expressed as: , In the formula, Upsample represents upsampling. In this example, the difference feature can be Upsampled 4 times, and then passed through two 3×3 convolutional layers, where the first convolutional layer has input and output channels of , the second convolutional layer input channel is , the output channel is 1, and the final result is If it is the testing phase, you also need to conduct Operation, each pixel is set as the threshold of 0.5, and pixels greater than the threshold are considered to be changed, and pixels less than 0.5 are considered to be unchanged. If it is the training stage, the final result is fed into the loss function.

[0038] Furthermore, the loss function of the initial remote sensing image change detection model is expressed as: , in, Represents the optimized binary cross entropy loss, expressed as: , represents the Dice coefficient loss, expressed as: , In the formula, y represents the training label, Represents the result predicted by the model, represents the L1 norm.

[0039] is the binary cross entropy loss An optimized version of , which combines the Sigmoid activation function and the binary cross entropy loss function together, alleviating the problem of numerical instability during training. Used to solve the problem of imbalance between positive and negative classes in change detection tasks.

[0040] To verify the effectiveness of the present invention, the present invention was trained and tested on the public dataset WHU-CD. The detection results of the present invention were compared with the current more advanced change detection networks, including the FC-Diff network based on full convolution and difference, the FC-Unet network based on full convolution and UNet, the spatiotemporal attention network STANet based on feature pyramid, and the dual-temporal attention full convolution network DASNet. At the same time, both the comparative experiments and ablation experiments were quantitatively and qualitatively analyzed.

[0041] The WHU-CD dataset is a publicly available building change detection dataset containing a pair of high-resolution aerial images with a size of 32507 × 15354 pixels. Due to the limitation of computing resources, these images are first cropped into non-overlapping patches with a size of 256 × 256 pixels and randomly divided into two parts in a ratio of 7:3: 5335 pairs for training and 2285 pairs for testing.

[0042] In order to verify the effectiveness of the method of the present invention from the perspective of accuracy, the present invention is compared with the FC-Diff network based on full convolution and difference, the FC-Unet network based on full convolution and UNet, the spatiotemporal attention network STANet based on feature pyramid, and the dual-temporal attention fully convolutional network DASNet on the WHU-CD dataset.

[0043] From the perspective of qualitative analysis, the test results are visualized. Figure 7 As shown, white represents correct pixel classification, red represents wrong classification, and green represents missed classification. Figure (a) represents the image before the change, Figure (b) represents the image after the change, Figure (c) represents the label, Figure (d) represents the FC-Unet detection result, Figure (e) represents the FC-diff detection result, Figure (f) represents the DASNet detection result, Figure (g) represents the STANet detection result, and Figure (h) represents the detection result based on the method proposed in the present invention. Figure 7 It can be seen from the figure that the segmentation effect of the present invention is significantly better than that of other models.

[0044] From the perspective of quantitative analysis, the recall rate, F1 score, precision, intersection-over-union ratio, and accuracy are calculated on the test set to evaluate the detection performance. The calculation formulas are as follows: , , , , , in, Indicates the accuracy, represents the recall rate, represents the F1 score, Indicates the accuracy, represents the intersection-over-union ratio; TP represents the area where the model correctly detects changes, TN represents the area where the model correctly detects no changes, FP represents the area where the model incorrectly detects no changes as changes, and FN represents the area where the model fails to detect real changes. The evaluation results are shown in Table 1.

[0045] Table 1

[0046] As can be seen from Table 1, the method proposed in this invention is significantly higher than the other four methods in all indicators. In summary, compared with the FC-Diff network based on full convolution and difference, the FC-Unet network based on full convolution and UNet, the spatiotemporal attention network STANet based on feature pyramid, and the dual-temporal attention full convolution network DASNet, this invention has great advantages in segmentation performance and accuracy. In order to verify that the introduction of CLIP enhances the accuracy and generalization of the present invention, the present invention conducts an ablation experiment on the CLIP branch, and verifies the positive effect of the introduction of the CLIP model on the present invention from the perspectives of quantitative and qualitative analysis.

[0047] From the perspective of quantitative analysis, an ablation experiment is conducted on the CLIP branch, and the recall rate, F1 score, precision, intersection-over-union ratio, and accuracy are calculated on the test set to evaluate the detection performance. The detection results are shown in Table 2.

[0048] Table 2

[0049] From the data in Table 2, it can be seen that not using CLIP will lead to a significant decrease in indicators.

[0050] From the perspective of qualitative analysis, the visualization results are as follows: Figure 8 As shown in the figure, (a) represents the image before the change, (b) represents the image after the change, (c) represents the label, (d) represents the detection result without using the CLIP model, and (e) represents the detection result with the CLIP model. The change targets in these two groups of pictures are very similar. From the detection results, both methods can perfectly identify the changes in the first group of pictures. However, for the second group of pictures, if CLIP is not used, the model will not be able to perceive the changes, because the feature representation extracted by general knowledge is not enough to detect such changes. Although the detection results of the model after using CLIP are not perfect, its ability to perceive changes is significantly enhanced. In summary, adding CLIP as a general feature can effectively improve the generalization ability of the model.

[0051] In summary, through quantitative and qualitative analysis, it can be seen that the present invention has achieved good results in both accuracy and generalization.

Claims

1. A remote sensing image change detection method based on fine-tuning CLIP, characterized in that: include: Obtain the original dual-time remote sensing images, preprocess the remote sensing images, use the preprocessed remote sensing images to construct a data set, and divide it into a training set and a test set in proportion; An initial remote sensing image change detection model is constructed, and the initial remote sensing image change detection model is trained using the training set to obtain a target remote sensing image change detection model; the loss function of the initial remote sensing image change detection model is composed of an optimized binary cross entropy loss and a Dice coefficient loss; Using the target remote sensing image change detection model to perform change detection on remote sensing images; Among them, building the initial remote sensing image change detection model includes: Constructing a first residual network, a second residual network, a first visual encoder, and a second visual encoder; Input any one of the dual-time remote sensing images into a first residual network to generate a first dedicated feature, and input the remote sensing image into a first visual encoder to generate a first general feature; Inputting another remote sensing image into a second residual network to generate a second dedicated feature, and inputting the remote sensing image into a second visual encoder to generate a second universal feature; Both the first visual encoder and the second visual encoder are visual encoders of CLIP with visual cue fine-tuning.

2. The remote sensing image change detection method based on fine-tuning CLIP according to claim 1, characterized in that: Building the initial remote sensing image change detection model also includes: Construct a first feature fusion module and a second feature fusion module; Preprocessing the first special feature and the first general feature respectively and inputting them into a first feature fusion module to obtain a first fusion feature; Preprocessing the second special feature and the second general feature respectively and inputting them into the second feature fusion module to obtain the second fusion feature; The first fusion feature and the second fusion feature are concatenated in the sequence length dimension, and then the self-attention mechanism operation and the feedforward neural network operation are performed to obtain the first comprehensive feature; Separating the first comprehensive feature to obtain a second comprehensive feature and a third comprehensive feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first special feature to obtain a first change information special feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second special feature to obtain a second change information special feature; Performing a cross attention operation and a feedforward neural network operation on the second comprehensive feature and the first general feature to obtain a first change information general feature; Performing a cross attention operation and a feedforward neural network operation on the third comprehensive feature and the second general feature to obtain a second change information general feature; A classification head module is constructed, and the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature are input into the classification head module to obtain a final detection result.

3. The remote sensing image change detection method based on fine-tuning CLIP according to claim 2, characterized in that: The first special feature and the first general feature are pre-processed and then input into the first feature fusion module, and the first fusion feature obtained includes: The first specialized feature is passed through a convolutional layer and then performed on the spatial dimension Softmax Operation, get the attention score of each spatial position in the channel, use the attention score to perform weighted summation of the first dedicated feature before dimensionality reduction on the spatial dimension, and generate semantic tags , expressed as: ， In the formula, represents the first special feature, represents matrix multiplication, represents a convolution operation with a convolution kernel size of 3×3. The semantic mark shape is , is the sequence length, c is the number of channels; The first common feature is reduced in dimension using convolution and multi-layer perceptron to generate semantic tags , expressed as: ， In the formula, represents the first common feature, MLP represents a multi-layer perceptron, is the activation function, Indicates the use of two convolution operations with a kernel size of 1×1; Use dot product operation to mark semantic and semantic markup Fusion is performed to obtain the first fusion feature , expressed as: ， In the formula, Represents the multiplication of corresponding matrix elements.

4. The remote sensing image change detection method based on fine-tuning CLIP according to claim 3, characterized in that: The first fusion feature and the second fusion feature are concatenated in the sequence length dimension, and then the self-attention mechanism operation and the feedforward neural network operation are performed to obtain the first comprehensive feature, which is expressed as: ， ， In the formula, represents the features after splicing, Represents a splicing operation, represents the second fusion feature, represents the first comprehensive feature, , and represents the transformation matrix, represents a feed-forward neural network; Represents the attention mechanism operation, and the calculation formula is: ， In the formula, , , Represent the three input items of the attention mechanism operation, with superscripts T represents the matrix transpose, A scaling item.

5. The remote sensing image change detection method based on fine-tuning CLIP according to claim 4, characterized in that: Separating the first comprehensive feature to obtain the second comprehensive feature and the third comprehensive feature includes: The first comprehensive feature The sequence length dimension is divided into two parts, and the second comprehensive feature is obtained. And the third comprehensive characteristic , expressed as: 。 6. The remote sensing image change detection method based on fine-tuning CLIP according to claim 5, characterized in that: The second comprehensive feature and the first special feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information special feature, which is expressed as: , In the formula, Indicates the first change information-specific feature.

7. The remote sensing image change detection method based on fine-tuning CLIP according to claim 6, characterized in that: The second comprehensive feature and the first general feature are subjected to cross attention operation and feedforward neural network operation to obtain the first change information general feature, which is expressed as: ， In the formula, Indicates the general characteristics of the first change information.

8. The remote sensing image change detection method based on fine-tuning CLIP according to claim 7, characterized in that: Inputting the first change information-specific feature, the second change information-specific feature, the first change information-general feature, and the first change information-general feature into the classification head module to obtain the final detection result, including: Subtracting the first change information universal feature from the first change information universal feature to obtain a dual-time image universal difference feature; The first change information-specific feature and the second change information-specific feature are subtracted to obtain the dual-time image-specific difference feature, and then bilinear interpolation upsampling and 1×1 convolution are performed, and then added to the dual-time image universal difference feature to obtain a difference feature with change information, universal features and special features. , expressed as: ， In the formula, Indicates the second change information general characteristics, Indicates the second change information dedicated feature, H , W , C Respectively represent the height, width and number of channels of the original remote sensing image, Upsample Represents an upsampling operation based on bilinear interpolation; The difference characteristics The final output features are obtained through upsampling based on bilinear interpolation and two 3×3 convolutions. , expressed as: 。 9. The remote sensing image change detection method based on fine-tuning CLIP according to claim 1, characterized in that: The loss function of the initial remote sensing image change detection model is expressed as: ， in, Represents the optimized binary cross entropy loss, expressed as: ， represents the Dice coefficient loss, expressed as: ， In the formula, y represents the training label, Represents the result predicted by the model, represents the L1 norm.

Images