An adaptive fusion remote sensing image change detection method based on semantic compensation

Abstract

The present invention discloses a method for adaptively fusion-based change detection of remote sensing images based on semantic compensation. The method comprises the following steps: using a CNN-Transformer hybrid hierarchical backbone network to extract features from remote sensing images, combining the local detail sensitivity of the CNN with the global context perception of the Transformer to capture multi-scale features ranging from subtle changes to broad patterns; ensuring semantic consistency by compensating and aligning local features from the CNN and global features from the Transformer through a semantic compensation module; and using a CNN-Trans feature adaptive fusion module to adaptively balance and fuse features from the CNN and Transformer branches, thereby enhancing the model's generalization ability in different scenarios. The present invention uses a change feature recognition module to identify real changes while suppressing environmental interference, effectively improving recognition accuracy.

Description

Technical Field

[0001] The invention relates to an adaptive fusion remote sensing image change detection method based on semantic compensation, and belongs to the technical field of remote sensing image detection. Background Art

[0002] Change detection in remote sensing images is a key application area of ​​remote sensing technology, playing a vital role in diverse fields, including natural resource management, disaster monitoring, urban planning, and environmental monitoring. With technological advancements, methods for remote sensing image change detection have also been continuously improving, from early post-classification comparison methods and transformation vector analysis to direct classification methods. While these methods each have their own unique characteristics, most rely on manually extracted features. In recent years, with the widespread use of high-resolution and multispectral imagery, the features and spectral information contained in images have become richer and more complex. However, these traditional methods have limitations in describing real-world scenarios, particularly in complex environments, where their detection performance is often unsatisfactory. To improve the accuracy and efficiency of change detection, more advanced techniques are needed to adapt to the growing complexity of data.

[0003] In recent years, the rapid development of deep learning technology has revolutionized the field of change detection in remote sensing images. Applying deep learning to remote sensing image change detection has become an unstoppable trend, and a large number of research results have emerged in this field. These studies leverage the powerful capabilities of deep learning to more accurately and efficiently detect changes in remote sensing images. Based on the semantic segmentation network UNet, Daudt et al. proposed three end-to-end fully convolutional architectures (FC-EF, FC-Siam-conc, and FC-Siam-diff) for remote sensing image change detection. Zhang et al. fused feature maps of remote sensing images at different time points and utilized channel attention and spatial attention techniques to address the inconsistency between image features at different time points. Fang et al. improved upon the Unet++ network and designed the SNUNet-CD model to promote the natural fusion of high-level and low-level features. By introducing a channel attention mechanism into deep supervision, they effectively alleviated the semantic information loss that can be caused by deep supervision. Zheng et al. proposed an innovative Siamese CNN architecture. By integrating high-frequency attention blocks (HFABs), this architecture enhances the model's ability to identify changed buildings in remote sensing imagery, improving the accuracy of change detection. However, change information in remote sensing images often involves complex spatial relationships. Traditional CNNs rely on fixed-size convolutional kernels with limited receptive fields, making it difficult to capture large-scale spatial information and potentially failing to adequately capture global changes. Compared to convolutional neural networks (CNNs), Transformers are better able to model global image features. This is because Transformers can process interactions between all pixels, not just local neighborhoods. With the application of Transformers in remote sensing imagery, some researchers have found that combining them for feature extraction can achieve better performance than using either CNN or Transformer alone. Feng et al. innovatively integrated convolutional neural networks (CNNs) and Transformers through intra-scale cross-interaction and inter-scale feature fusion strategies, effectively capturing both local and global features and improving change detection accuracy. Cui et al. combined the local feature extraction capabilities of CNNs with the global context modeling capabilities of ViTs, improving multi-scale object perception through a U-shaped architecture and Siamese network design. A learnable mask-guided module was designed to focus on true changes in multi-scale heterogeneous representations while gradually suppressing spurious changes, improving the accuracy of change detection. However, these CNN-Transformer backbones often overlook the semantic gap between the two architectures when extracting features from remote sensing images. CNNs typically use convolutional kernels to extract local features and perform dimensionality reduction through pooling layers, while Transformers use a self-attention mechanism to extract global features.The two architectures have significantly different feature representations, making direct fusion difficult. Since CNNs and Transformers excel at extracting different types of features, failing to effectively fuse these two features can affect the model's overall understanding of the image, leading to a decrease in change detection accuracy. Summary of the Invention

[0004] In view of the problems existing in the existing methods, the present invention aims to provide a remote sensing image change detection method with semantic compensation, comprising the following steps:

[0005] The original change detection dataset images are cropped into non-overlapping sub-images of a specified size and divided into training, validation, and test sets. Data augmentation is also performed before inputting into the change detection network.

[0006] Input the bi-temporal data into the CNN-Transformer twin encoder backbone network;

[0007] The features extracted by the CNN and Transformer branches are passed into the semantic compensation module for semantic compensation and then passed into the CNN and Transformer branches of the backbone network respectively;

[0008] The features input into the CNN branch and the features extracted by the Transformer in the semantic compensation module are passed into the CNN-Transformer feature adaptive fusion module to adaptively fuse the two features;

[0009] The features of different scales and different times that have been fused are passed into different change feature identification modules to extract the difference information of multiple scales;

[0010] The bottom-level difference features are upsampled to make their scale the same as the previous level; the upsampled features are spliced ​​with the previous level features in the channel dimension; the spliced ​​features are deconvolved to change their scale and channel; the previous process is repeated to the first level; the features fused with the first-level features are deconvolved and convolved with a kernel size of 3; the features are upsampled and convolved twice with a kernel size of 3 to generate a change map; the model parameters with the best performance are trained on the training set; the dual-phase remote sensing images of the test set are input into the remote sensing image change detection model to obtain predictions of changing objects.

[0011] Furthermore, the original datasets include WHU-CD and CDD. The above datasets are cropped into non-overlapping images of size 256x256, and then these images are divided into training set, validation set, and test set according to the corresponding proportions of each dataset; the training set is randomly flipped and cropped, and all these images are normalized before being input into the network.

[0012] Furthermore, the CNN-Transformer twin encoder backbone network combines traditional convolution with a parallel hierarchical network of multi-head attention mechanisms. More specifically, the input image with a resolution of 256x256 is subjected to an initial downsampling and dimensionality-raising operation using a CNN-Transformer dual-branch. The CNN branch uses two consecutive convolutions with a stride of 2, a padding of 1, and a convolution kernel size of 3 to reduce the input image size to 64x64, and the number of channels increases from 3 to 128. The Transformer branch uses a convolution operation with a convolution kernel size of 7, a padding of 3, and a stride of 4 to increase the dimension of the feature map, where the dimension size is 64. The feature map is layer-normalized and then multi-head self-attention operation is performed; the feature map obtained after the operation is added to the feature map without multi-head self-attention, and then layer normalization and multi-layer perceptron are performed in sequence; the previous operation is iterated N times to complete the feature extraction of the layer, N varies according to the level, the backbone network has a total of four levels, and N of each layer is set to 3, 4, 6, and 3. The features after the extraction of the layer are downsampled by 2 using a convolution kernel of 3 and the number of channels is doubled before being passed to the next level of the backbone network until the end.

[0013] Furthermore, the semantic compensation module has the following features: Figure 1 and features Figure 2 They are from CNN and Transformer branches respectively. First, extract the corresponding Q, K, V vectors from the two feature maps respectively, and transform the features Figure 2 The Q vector and characteristics Figure 1 The K, V of the horizontal and vertical directions are calculated, and the features Figure 1 The Q vector and characteristics Figure 2 The K and V of the two branches are used to calculate the horizontal and vertical correlation. The feature maps after the horizontal and vertical correlation calculations are added to the feature maps input into the two branches and global maximum and global average pooling are performed respectively. The pooled features are fully connected, and then the features of the two branches are added and activated using the Sigmoid function. The activated features are then combined with the feature maps. Figure 1 and features Figure 2 After multiplication, the sum is added. Finally, some channels are exchanged with the feature map to obtain the fused features.

[0014] Furthermore, the CNN-Transformer feature adaptive fusion module is specifically: feature Figure 1 and features Figure 2Represents the features extracted by the CNN and Transformer branches at different levels. The feature maps are input into convolutions with different kernel sizes of 3 and activated with the ReLU function before layer normalization to obtain the vectors Q, K, and V. The two pairs of Q, K, and V are cross-correlated and then convolved with learnable weights, and then added to the input feature map with learnable weights. The resulting feature map is layer normalized and convolved with a kernel size of 1. The convolved features are sliced ​​and input into the two branches respectively. The features of the first branch are depth-wise separable convolutions and multiplied with the features of the second branch. The multiplied features are convolved and assigned learnable weights, and then added to the weighted input feature map. These features are concatenated along the channel dimension and convolved to obtain the fused features.

[0015] Furthermore, the fused features at different levels are input into the change feature identification module. Specifically, the change feature identification module includes the following features: Figure 1 and features Figure 2 Represents the different time feature maps fused by the CNN-Transformer feature adaptive fusion module. The two features are channel-spliced. The spliced ​​feature maps are subjected to global maximum pooling, 1x1 convolution, ReLU activation, 1x1 convolution and global average pooling, 1x1 convolution, ReLU activation, and 1x1 convolution, and then added. The added features are activated using the Sigmoid function and then activated. Figure 1 and features Figure 2 Multiply to get new features Figure 1 and features Figure 2 , the two feature maps are convolved with a kernel of 3, batch normalized and ReLU activated on different branches, and then multiplied with the feature Figure 1 and features Figure 2 Perform channel splicing, and use the convolution kernel size of 3 to convolve the spliced ​​features with the previous multiplied feature map and feature Figure 1 and features Figure 2 After splicing, the convolution kernel is 1 and the Sigmoid function is activated, and the features obtained after activation are combined with the features Figure 1 and features Figure 2 Multiply, add the features after multiplication, and use the GELU activation function to get the difference features.

[0016] Furthermore, when the network is trained under supervision, we use the cross entropy loss function for this type of problem.

[0017] Compared with the existing technology, the present invention has the following beneficial effects:

[0018] (1) This paper proposes a CNN-Transformer dual-branch encoder for extracting features from remote sensing images. The encoder combines the sensitivity of CNN to local details and the broad perception of Transformer to global context, and can simultaneously capture multi-scale features from small changes to large-scale patterns.

[0019] (2) This paper proposes a semantic compensation module and inserts it between the CNN-Transformer dual-branch encoder to reduce the semantic gap between the features in the dual branches. By using the semantic compensation module, the fine local features extracted by the CNN encoder and the global semantic information captured by the Transformer encoder can be effectively supplemented and adjusted to ensure semantic consistency between the two.

[0020] (3) This paper proposes a CNN-Transformer adaptive fusion module for fusing features between CNN branches and Transformer branches. The CNN-Transformer adaptive fusion module uses learnable factors to adaptively balance the local sensitivity of CNN and the global generalization ability of Transformer, optimizing the combination of local details and global context. By adaptively fusing features from different sources, the CNN-Transformer adaptive fusion module enhances the generalization ability for various complex scenarios and improves the applicability of the model to diverse remote sensing image data.

[0021] (4) The effectiveness of our method is verified on two well-known remote sensing image change detection datasets, CDD and WHU-CD. To verify the effectiveness of our method, we conducted a detailed comparison with existing methods on multiple evaluation metrics, including precision, recall, overall accuracy, intersection-over-union (IoU), and F1 score. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present invention will become more apparent from the following description of examples of the present invention with reference to the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a method for detecting changes in remote sensing images using adaptive fusion of semantic compensation according to an embodiment of the present invention is shown;

[0024] Figure 2 The figure shows an overall schematic diagram of a method for adaptive fusion remote sensing image change detection based on semantic compensation according to an embodiment of the present invention;

[0025] Figure 3 A diagram of a semantic compensation module according to an embodiment of the present invention is shown;

[0026] Figure 4A diagram of a CNN-Transformer feature adaptive fusion module according to an embodiment of the present invention is shown;

[0027] Figure 5 A diagram of a change feature identification module according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, advantages and technical methods of the present invention more clear, the following is described in conjunction with specific examples. The specific examples described here are only used to explain the present invention and are not used to limit the present invention.

[0029] like Figure 1 FIG. 1 is a flowchart of a remote sensing image change detection method using adaptive fusion of multi-level difference features according to an embodiment of the present invention, including steps S1 to S7:

[0030] S1: Crop the original change detection dataset images into non-overlapping sub-images of a specified size and divide them into training, validation, and test sets. Perform data augmentation before inputting into the network.

[0031] In this example, preprocessing was performed on two publicly available datasets to obtain the bi-temporal data required for model training and testing. These datasets include the WHU-CD dataset and the CDD dataset. These two datasets are described below.

[0032] (1) WHU-CD dataset:

[0033] The WHU-CD dataset covers the area affected by the 6.3 magnitude earthquake in February 2011 and the subsequent reconstruction efforts in the years following it. The dataset consists of aerial imagery acquired in April 2012 and contains 12,796 buildings within a 20.5 square kilometer area (the 2016 dataset contained 16,077 buildings in the same area). The original WHU-CD image size is 32,507 by 15,354 pixels, with a resolution of 0.2 meters per pixel.

[0034] (2) CDD dataset:

[0035] The CDD dataset consists of seven pairs of seasonally varying images (with a resolution of 4725 x 2700 pixels) used to manually create ground truth, and four pairs of seasonally varying images (with minimal variation, at a resolution of 1900 x 1000 pixels) used to manually add additional objects. The spatial resolution of the acquired images ranges from 3 cm / pixel to 100 cm / pixel. The CDD dataset includes objects of varying sizes (e.g., from cars to large structures) and seasonal variations of natural objects (e.g., from individual trees to vast forest areas).

[0036] In the example of the present invention, step S1 specifically includes steps S11-S13.

[0037] S11, crop the original dataset image of change detection into 256x256 non-overlapping sub-images according to the specified size;

[0038] S12: Divide the cropped images into training, validation, and test sets according to different ratios. For the CDD dataset, the training, validation, and test sets are divided according to a ratio of 10000 / 3000 / 3000; for the WHU-CD dataset, the training, validation, and test sets are divided according to a ratio of 5947 / 743 / 744.

[0039] S13, these data sets are randomly flipped and cropped 8 times before being input into the network, and then normalized.

[0040] S2, inputs the dual-phase data into the twin encoder backbone network, and extracts features of the dual-phase data in turn through different layers of the backbone network to obtain multi-level features.

[0041] like Figure 2 As shown, an overall framework diagram of an adaptive fusion remote sensing image change detection method using semantic compensation is provided for an embodiment of the present invention. In the example of the present invention, the CNN-Transformer hybrid twin encoder backbone network is divided into four parts, namely, feature extraction at different levels.

[0042] (1) CNN branch

[0043] The first layer performs two consecutive convolutions with a kernel size of 3, a padding of 1, and a stride of 2 on the input remote sensing image with a channel number of 3, reducing the resolution from 256x256 to 64x64 and increasing the number of channels from 3 to 64; the second layer uses two convolutions with a kernel size of 3 to reduce the dimension of the feature map from 64x64 to 32x32 and increase the number of channels from 64 to 128; the third layer uses two convolutions with a kernel size of 3 to reduce the dimension of the feature map from 32x32 to 16x16 and increase the number of channels from 128 to 256; the fourth layer uses two convolutions with a kernel size of 3 to reduce the dimension of the feature map from 16x16 to 8x8 and increase the number of channels from 256 to 512.

[0044] (2) Transformer branch

[0045] The first layer uses a convolution operation with a kernel size of 7, a padding of 3, and a stride of 4 to increase the dimension of the feature map, where the dimension size is 64 and the feature map scale is 64x64. The feature map is layer-normalized and then multi-head self-attention operation is performed; the feature map obtained after the operation is added to the feature map without multi-head self-attention, and then layer-normalization and multi-layer perceptron are performed in sequence; the second layer uses a convolution operation with a kernel size of 3 and a padding of 1 to convert the feature map from 64x64 to 32x32, and the channel The number of channels increases from 64 to 128, and then the feature map is subjected to multi-head self-attention operation; the third layer uses convolution with a kernel size of 3 and padding of 1 to change the scale of the feature map from 32x32 to 16x16, and the number of channels increases from 128 to 256, and then the feature map is subjected to multi-head self-attention operation; the fourth layer uses convolution with a kernel size of 3 and padding of 1 to change the scale of the feature map from 16x16 to 8x8, and the number of channels increases from 256 to 512, and then the feature map is subjected to multi-head self-attention operation.

[0046] S3: The dual-phase features extracted by the backbone network at different levels and branches are input into the semantic compensation module and then into the corresponding branches at different levels for corresponding feature calculation.

[0047] like Figure 3 As shown, a structure diagram of a semantic compensation module according to an embodiment of the present invention is provided. Step S3 specifically includes steps S31-S34:

[0048] S31, feature map F 1 and F 2 They come from the CNN and Transformer branches respectively, and are transformed into three vectors Q, K, and V through convolution with a convolution kernel size of 1x1.

[0049] S32, each vector is subjected to different reshape operations to generate vectors for calculating horizontal and vertical attention. In order to avoid unreasonable attention distribution caused by comparing the feature vector at each position with itself, the vertical Q and K vectors are multiplied and then a diagonal matrix with negative infinity as the main diagonal element is added. After that, it is spliced ​​with the horizontal attention and activated using Sigmoid. Finally, the activated feature distribution is split into horizontal and vertical directions, multiplied with the corresponding V vector and then added to obtain the horizontal and vertical attention distribution. This process is used Cross_Att w_h ( ) Represents. The feature map is fused with the feature map of the original input at the element level. F fus1 , F fus2 , the process is expressed as:

[0050] ,

[0051] S33, in calculation F fus1 , F fus2 The corresponding channel weight is multiplied by the input feature map F 1 ，F 2 and Cross_Att w_h (F 1, F 2 ) , Cross_Att w_h (F 2, F 1 ) And add the two together to get F CW1 , F CW2 This process is expressed as:

[0052] ,

[0053] in S ig is the Sigmoid function, FC is the fully connected layer, GAP is the global average pooling, GMP is the global maximum pooling, is element-wise addition, This is element-wise multiplication.

[0054] S34, will F CW1 and F CW2 With the input feature map F 1 , F 2 After adding, add it to the input feature map F 1 , F 2 Perform channel-level information exchange, that is, exchange elements of the input feature map and the fused feature map at even positions in the channel dimension:

[0055] ,

[0056] In the above formula, CE represents channel-level element exchange. The semantic compensation module adds the fused features while retaining the original features, compensating the semantic information between the CNN branch and the Transformer branch.

[0057] In S4, the modules of different branches after semantic compensation are subjected to dual-branch feature fusion through the CNN-Transformer feature adaptive fusion module.

[0058] like Figure 4 As shown, a network structure diagram of a CNN-Transformer feature adaptive fusion module according to an embodiment of the present invention is provided. The specific steps of step S4 include S41-S43:

[0059] S41, feature map F 1 and F 2 They come from the CNN and Transformer branches respectively, and generate Q, K, and V vectors through convolution with a kernel size of 3x3 and layer normalization. They can be expressed as follows:

[0060] ,

[0061] S42, the two groups Q, K, V are fused using cross multi-head attention and multiplied by the learnable factor β to obtain and Then multiply the input feature map by the learnable factor α and Add them together to get . It can be expressed as follows:

[0062] ,

[0063] ,

[0064] S43, will After layer normalization, the channel direction dimension is increased by 1x1 convolution and then segmented in the channel direction. and ;Will After depth-wise separable convolution and Multiply the features to enhance the feature extraction capability. Apply activation function and convolution to it to get F e1 and F e2 . It can be expressed as follows:

[0065] ,

[0066] Where split means dividing the input feature into half of the original one along the channel dimension; Conv d represents depthwise convolution; Convp Represents point convolution. Then the learning factor is introduced and F e1 , F e2 Multiply and combine with learnable factors Input features F 1 , F 2 Add.

[0067] In S44, the two branches are connected and convolved to complete adaptive fusion. The formula is as follows:

[0068] ,

[0069] Among them, BN stands for batch normalization and Concat stands for channel dimension concatenation.

[0070] S5, inputting the fused feature maps of different time phases into the change feature identification module to extract the change features.

[0071] like Figure 5 FIG. 1 is a structural diagram of a change feature identification module according to an embodiment of the present invention. Step S5 includes steps S51 to S54:

[0072] S51, feature map F t1 and F t2 These features are extracted from remote sensing images at different time phases using the twin network. First, the two feature maps are concatenated in the channel dimension. Global average pooling and global maximum pooling are then performed on each feature. Two convolutions with a kernel size of 1x1 are then performed to halve their dimensions. After element-wise addition of these two branches, a Sigmoid activation is performed, and the result is then element-wise multiplied with the original input features. This can be expressed as follows:

[0073] ,

[0074] S52, will F t1_w , F t2_w Use convolution with a kernel size of 3x3 and multiply them to get F m , the multiplied features are connected with these two features in the channel dimension to obtain F mc1 , F mc2 :

[0075] ,

[0076] S53, yes F mc1 , F mc2 Use convolution with kernel size of 3x3 and F m , Fc3 1 , Fc3 2 Connect the channel dimensions to get F .right F Use convolution with kernel size of 1x1 to change features F The channel dimension is activated by the Sigmoid function to obtain the feature F s . F s Differences and characteristics Fc3 1 , Fc3 2 After multiplication, the sum is added, and finally the GELU activation function is used for activation to obtain the change feature output by the change feature identification module. The formula is:

[0077] ,

[0078] ,

[0079] S6, inputs the fused difference features into the detection head to perform pixel-level remote sensing image change detection and generate a change prediction map after the detection is completed, and iteratively trains the model parameters that save the best results.

[0080] S7, input the dual-temporal remote sensing images of the test set into the remote sensing image change detection model to obtain the prediction of the changed ground objects.

[0081] In this embodiment, the cross entropy loss function is used to train the network model. The cross entropy loss function is defined as follows:

[0082] ,

[0083] in H, W Refers to the length and width of the true value label, label(i) represents the i-th pixel in the label image, p(i) represents the probability of the model predicting the i-th pixel, H(.) This embodiment uses AdamW as the optimizer, sets the weight decay rate to 0.01, the initial learning rate to 0.0001, and the epoch to 200.

[0084] Table 1 Comparison of the proposed method with well-known change detection methods on the WHU-CD dataset

[0085]

[0086] Table 2 Comparison of the proposed method with well-known change detection methods on the CDD dataset

[0087]

[0088] Tables 1 and 2 compare the accuracy, intersection-over-union (IoU), precision, recall, and F1 score of the proposed method with those of currently known change detection methods on the WHU-CD and CDD datasets to further illustrate the effectiveness of the proposed method. As can be seen from these tables, the proposed method significantly improves upon these well-known change detection methods, effectively validating the effectiveness of the proposed method.

[0089] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A method for adaptive fusion remote sensing image change detection based on semantic compensation, characterized in that: The steps include: S1: Crop the original change detection dataset images into non-overlapping sub-images of a specified size and divide them into training, validation, and test sets. Perform data augmentation on the training set images before inputting them into the network. The details are as follows: S1.1, crop the original images of the dataset into images of size 256x256; S1.2, divide the cropped images into training set, validation set, and test set according to different ratios; S1.3, randomly flip and crop the training set; S1.4, normalize the enhanced training set, validation set, and test set images; S2 performs an initial downsampling operation on the input image, converting the 256x256 resolution image into a 64x64 feature map, and the number of image channels increases from 3 to 128. The feature maps are respectively passed into the convolution and Transformer blocks with a convolution kernel size of 3. The semantic compensation module is used to compensate the information of the two semantically different feature maps. The scale of the feature map is reduced and the number of channels of the feature map is increased before passing it into the convolution and Transformer blocks of the next level. After four layers of CNN-Transformer feature extraction, it reaches the end of the backbone network. S3, the features output by the semantic compensation module in the backbone network are fused using the CNN-Transformer feature adaptive fusion module; The feature maps F1 and F2 come from the CNN and Transformer branches respectively, and are generated by convolution with a kernel size of 3 and layer normalization to generate Q, K, V vectors, which can be expressed as follows: Q1,K1,V1=LN(ReLU(Conv 3×3 (F1)))), <h2 style=";text-align:left;direction:ltr">Q2,K2,V2 = LN(ReLU(Conv<h2 style=";text-align:left;direction:ltr"> 3×3 <h2 style=";text-align:left;direction:ltr"> (F2)) The two groups Q, K, V are fused using cross multi-head attention and multiplied by the learnable factor β to obtain F β1 and F β2 Then multiply the input feature map F by the learnable factor α α1 and F α2 Add them together to get F α1+β1 , F α2+β2 , which is expressed as follows: F CA1 =MHA(Q2,K1,V1), <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> CA2 <h2 style=";text-align:left;direction:ltr"> (MHA(Q1,K2,V2)) F β1 =reshape(Conv 3×3 (F CA1 ))×β1, F β2 =reshape(Conv 3×3 (F CA2 ))×β2, F α1+β1 , F α2+β2 After layer normalization, the channel dimension is increased by 1x1 convolution and then segmented in the channel direction to obtain F a1 , F a2 and F b1 , F b2 ; After passing the depth-separable convolution, it is combined with F a2 , F b2 Multiply the features to enhance the feature extraction capability; use activation function and convolution to get F e1 and F e2 , which is expressed as follows: F a =LN(Conv 1×1 (F α1+β1 )), F b =LN(Conv 1×1 (F α2+β2 )), F a1 ,F a2 =split(F a ), F b1 ,F b2 =split(F b ), Split means dividing the input feature into half of the original one along the channel dimension; Conv d Denotes depth convolution; Conv p Represents point convolution; then introduces learning factors δ and F e1 , F e2 Multiply and add the input features F1 and F2 multiplied by the learnable factor γ; finally, connect the two branches and perform convolution to complete the adaptive fusion; the formula is as follows: CTFAF out =RELU(BN(Conv 1×1 (Concat(Fus1,Fus2))))), Among them, BN means batch normalization, and Concat means channel dimension concatenation; S4, input the fused CNN-Transformer features at different levels into the change feature identification module to extract difference features; S5, upsample the bottom-level difference features to make them the same scale as the previous level; concatenate the upsampled features with the previous level features in the channel dimension; deconvolve the concatenated features to change their scale and channel; repeat the previous process to the first level; deconvolve the features fused with the first level features and then use a convolution kernel size of 3; The features are upsampled and convolved twice with a kernel size of 3 to generate a change map; 200 rounds of iterative training are performed on the training set and the model parameters with the best effect are saved; S6, input the dual-temporal remote sensing images of the test set into the remote sensing image change detection model to obtain the change map predicted by the model.

2. The adaptive fusion remote sensing image change detection method based on semantic compensation according to claim 1, characterized in that: The semantic compensation module performs semantic compensation on the features extracted by CNN and Transformer, and S2 includes: The feature maps F1 and F2 come from the CNN and Transformer branches respectively, and are transformed into three vectors Q, K, and V through convolution with a convolution kernel size of 1. Furthermore, each vector is subjected to different reshape operations to generate vectors for calculating horizontal and vertical attention. In order to avoid unreasonable attention distribution caused by calculating the feature vector at each position with itself, the vertical Q and K vectors are multiplied and then added to a diagonal matrix with negative infinity as the main diagonal element. After that, they are spliced ​​with the horizontal attention and activated using Sigmoid. Finally, the activated feature distribution is split into horizontal and vertical directions, multiplied with the corresponding V vector, and then added to obtain the horizontal and vertical attention distribution. This process is called Cross_Att w_h () indicates that the feature map is fused with the feature map of the original input at the element level to obtain F fus1 and F fus2 , expressed as: In calculating F fus1 , F fus2 The corresponding channel weights are then multiplied by the input feature maps F1, F2 and Cross_Att w_h (F1,F2), Cross_Att w_h (F2, F1) and add the two to get F CW1 ,F CW2 This process is expressed as: Among them, Sig is the Sigmoid function, FC is the fully connected layer, GAP is the global average pooling, GMP is the global maximum pooling, is element-wise addition, is element-wise multiplication; F CW1 and F CW2 After adding the input feature maps F1 and F2, the channel-level information is exchanged with the input feature maps F1 and F2, that is, the elements of the input feature map and the fused feature map are exchanged at the even positions of the channel dimension: In the above formula, CE represents channel-level element exchange. The semantic compensation module adds the fused features while retaining the original features, compensating the semantic information between the CNN branch and the Transformer branch.

3. The adaptive fusion remote sensing image change detection method based on semantic compensation according to claim 1, characterized in that: The change feature identification module is good at identifying the construction or demolition of buildings and vegetation changes. The change feature identification module can handle small-scale structural changes from macro-city expansion to micro-scale expansion by capturing edge, texture and shape features. In addition, it can also reduce the misjudgment rate by distinguishing between real changes and pseudo changes caused by changes in imaging conditions; S4 is specifically: Feature map F t1 and F t2 They come from the corresponding features extracted from remote sensing images of different phases in the twin network. First, the two feature maps are spliced ​​in the channel dimension. Then, global average pooling and global maximum pooling are performed on the two features respectively, and two convolutions with a kernel size of 1x1 are performed to reduce their dimensions by half. After element-by-element addition of the two branches, Sigmoid activation is used and then element-by-element multiplication with the original input features is performed. The formula is as follows: F C =Concat(F t1 ,F t2 ), F w1 =Conv 1×1 (RELU(Conv 1×1 (GAP(F c )))), F w2 =Conv 1×1 (RELU(Conv 1×1 (GMP(F c )))), Then, F t1_w , F t2_w Use convolution with a kernel size of 3x3 and multiply to get F m , connect the multiplied features with these two features in the channel dimension to get F mc1 , F mc2 : Fc31=RELU(BN(Conv 3×3 (F t1_w )))), Fc32=RELU(BN(Conv 3×3 (F t2_w )))), F mc1 =Concat(F m Fc31), F mc2 =Concat(F m Fc32), F mc1 , F mc2 After using convolution kernel size 3 and F m , Fc31, Fc32 are connected in the channel dimension to obtain F; the convolution kernel size of F is used to change the channel dimension of feature F, and the Sigmoid function is used for activation to obtain feature F s ; The feature F s After multiplying with features Fc31 and Fc32 respectively, the results are added together, and finally the GELU activation function is used to activate the change features output by the change feature identification module; the formula is: F=Concat(Conv 3×3 (F mc1 ),Conv 3×3 (F mc2 ),F m ), F s =Sig(Conv 1×1 (F)),

Images