Semantic compensation-based adaptive fusion remote sensing image change detection method

Abstract

The invention discloses an adaptive fusion remote sensing image change detection method based on semantic compensation, and the method comprises the following steps: carrying out the feature extraction of a remote sensing image through employing a CNN-Transform mixed hierarchical backbone network, combining the local detail sensitivity of a CNN with the global context perception of Transform, and carrying out the feature extraction of the remote sensing image, capturing multi-scale features from a small change to a wide mode; through a semantic compensation module, local features from the CNN and global features from Transform are compensated and aligned to ensure semantic consistency; a CNN-Trans feature adaptive fusion module is used to adaptively balance and fuse features from CNN and Transform branches, thereby enhancing the generalization ability of the model in different scenes. According to the invention, through the change feature identification module, real changes can be identified while environmental interference is suppressed, and the identification accuracy is effectively improved.

Description

Technical Field

[0001] The invention relates to an adaptive fusion remote sensing image change detection method based on semantic compensation, and belongs to the technical field of remote sensing image detection. Background Art

[0002] Remote sensing image change detection is an important application field of remote sensing technology. It plays an important role in many fields, including natural resource management, disaster monitoring, urban planning, environmental monitoring, etc. With the development of technology, the methods of remote sensing image change detection are also constantly improving, from the early post-classification comparison method, transformation vector analysis method to the direct classification method, etc. These methods have their own characteristics, but most of them rely on manually extracted features. In recent years, with the widespread use of high-resolution and multispectral images, the features and spectral information contained in the images have become richer and more complex. However, these traditional methods have certain limitations in describing actual scenes, especially in the face of complex environments, their detection effects are often unsatisfactory. In order to improve the accuracy and efficiency of change detection, it is necessary to explore more advanced technical means to adapt to the growing data complexity.

[0003] In recent years, the rapid development of deep learning technology has completely changed the field of remote sensing image change detection. At present, the application of deep learning technology to remote sensing image change detection has become an unstoppable trend, and a large number of research results have emerged in this field. These studies use the powerful ability of deep learning to detect changes in remote sensing images more accurately and efficiently. Daudt et al. proposed three end-to-end fully convolutional architectures (FC-EF, FC-Siam-conc, FC-Siam-diff) for remote sensing image change detection based on the semantic segmentation network UNet. Zhang et al. used channel attention and spatial attention techniques to solve the inconsistency problem between image features at different time points by fusing remote sensing image feature maps at different time points. Fang et al. made improvements on the basis of the Unet++ network and designed the SNUNet-CD model to promote the natural fusion of high-level features and low-level features. They introduced the channel attention mechanism in deep supervision to effectively alleviate the problem of semantic information loss that may be caused by deep supervision. Zheng et al. proposed an innovative Siamese CNN structure, which enhances the model's ability to identify changed buildings in remote sensing images by integrating high-frequency attention blocks (HFAB) and improves the accuracy of change detection. However, the change information in remote sensing images often involves complex spatial relationships. Traditional CNNs rely on fixed-size convolution kernels with limited receptive fields, making it difficult to capture large-scale spatial information, which may lead to insufficient capture of global changes. Compared with convolutional neural networks (CNNs), Transformers are better able to model the global features of images. This is because Transformers can handle interactions between all pixels, not just local neighborhoods. With the application of Transformers in the field of remote sensing images, some researchers have found that combining them for feature extraction can achieve better performance than using CNN or Transformer alone. Feng et al. innovatively integrated convolutional neural networks (CNNs) and Transformers through internal scale cross-interaction and inter-scale feature fusion strategies to effectively capture local and global features and improve the accuracy of change detection. Cui et al. combined the local feature extraction capabilities of CNNs with the global context modeling capabilities of ViT, and improved the perception capabilities of multi-scale targets through U-shaped structures and Siamese network design. A learnable mask-guided module is designed to focus on the real changes in multi-scale heterogeneous representations, while gradually suppressing pseudo changes, improving the accuracy of change detection. However, these CNN and Transformer-based backbones often ignore the semantic gap brought by these two architectures when extracting features from remote sensing images. CNN usually uses convolution kernels to extract local features and performs dimensionality reduction through pooling layers, while Transformer uses a self-attention mechanism to extract global features.The feature representations of the two architectures are quite different, making it difficult to fuse them directly. Since CNN and Transformer are good at extracting different types of features, if the two features cannot be effectively fused, it will affect the model's overall understanding of the image, resulting in a decrease in the accuracy of change detection. Summary of the invention

[0004] In view of the problems existing in the existing methods, the object of the present invention is to provide a remote sensing image change detection method with semantic compensation, comprising the following steps: The original dataset images of change detection are cut into non-overlapping sub-images according to the specified size, and divided into training set, validation set, and test set. In addition, data enhancement is performed before inputting into the change detection network; Input the bi-phase data into the CNN-Transformer twin encoder backbone network; The features extracted by the CNN and Transformer branches are respectively passed into the semantic compensation module for semantic compensation and then passed into the CNN and Transformer branches of the backbone network respectively; The features input into the CNN branch and the features extracted by the Transformer in the semantic compensation module are passed into the CNN-Transformer feature adaptive fusion module to adaptively fuse the two features; The features of different scales and different times that have been fused are passed into different change feature identification modules to extract the difference information of multiple scales; The bottom-level difference features are upsampled to make their scale the same as that of the previous level; the upsampled features are concatenated with the previous level features in the channel dimension; the concatenated features are deconvolved to change their scale and channel; the previous process is repeated to the first level; the features fused with the first-level features are deconvolved and then convolved with a kernel size of 3; the features are upsampled and convolved twice with a kernel size of 3 to generate a change map; the model parameters with the best preservation effect are trained on the training set; the dual-phase remote sensing images of the test set are input into the remote sensing image change detection model to obtain the prediction of the changing objects.

[0005] Furthermore, the original data sets include WHU-CD and CDD. The above data sets are cut into non-overlapping images of size 256x256 and then these images are divided into training set, validation set, and test set according to the corresponding proportions of each data set; the training set is randomly flipped and cropped, and all these images are normalized and then input into the network.

[0006] Furthermore, the CNN-Transformer twin encoder backbone network combines traditional convolution with a parallel hierarchical network of multi-head attention mechanism; more specifically, the image with an input resolution of 256x256 is subjected to a CNN-Transformer dual-branch initial downsampling and dimensionality-raising operation, and the CNN branch uses two consecutive convolutions with a step size of 2, a padding of 1, and a convolution kernel size of 3 to change the input image size to 64x64, and the number of channels is changed from 3 to 128; The Transformer branch uses a convolution operation with a convolution kernel size of 7, a padding of 3, and a stride of 4 to increase the dimension of the feature map, where the dimension size is 64. The feature map is layer-normalized and then multi-head self-attention is performed; the feature map obtained after the operation is added to the feature map without multi-head self-attention, and then layer-normalization and multi-layer perceptron are performed in sequence; the previous operation is iterated N times to complete the feature extraction of this layer, and N varies according to the level. The backbone network has four levels, and N of each layer is set to 3, 4, 6, and 3. The features after the extraction of this level are downsampled by 2 using a convolution with a convolution kernel of 3 and the number of channels is doubled before being passed to the next level of the backbone network until the end.

[0007] Furthermore, the semantic compensation module has the following features: Figure 1 and Features Figure 2 They come from the CNN and Transformer branches respectively. First, extract the corresponding Q, K, and V vectors from the two feature maps respectively, and transform the features Figure 2 The Q vector and characteristics Figure 1 The K and V of the feature vector are used to calculate the horizontal and vertical correlation. Figure 1 The Q vector and characteristics Figure 2 The K and V of the two branches are used to calculate the horizontal and vertical correlation. The feature maps after the horizontal and vertical correlation calculations are added to the feature maps input into the two branches and global maximum and global average pooling are performed respectively. The pooled features are fully connected, and then the features of the two branches are added and activated using the Sigmoid function. The activated features are then combined with the feature maps. Figure 1 and Features Figure 2 After multiplication, add them together. Finally, exchange some channels with the feature map to get the fused features.

[0008] Furthermore, the CNN-Transformer feature adaptive fusion module is specifically: feature Figure 1 and Features Figure 2Represents the features extracted by CNN and Transformer branches at different levels. The feature map is input into different convolutions with a kernel size of 3, and the vectors of Q, K, and V are obtained by layer normalization after activation with the ReLU function. The two pairs of Q, K, and V are cross-calculated and convolved with learnable weights and added to the input feature map with learnable weights. The feature map obtained after addition is layer normalized, and the normalized features are convolved with a kernel size of 1. The convolved features are sliced ​​and input into the two branches respectively. The features of the first branch are subjected to depthwise separable convolution and multiplied with the features of the second branch. The multiplied features are convolved and given learnable weights and added to the input feature map with weights. These features are connected in the channel dimension and convolved to obtain the fused features.

[0009] Furthermore, the fused features at different levels are input into a change feature identification module. Specifically, the change feature identification module includes: Figure 1 and Features Figure 2 The different time feature maps fused by the CNN-Transformer feature adaptive fusion module are shown in Figure 1. The two features are spliced ​​in channels. The spliced ​​feature maps are subjected to global maximum pooling, 1x1 convolution, ReLU activation, 1x1 convolution and global average pooling, 1x1 convolution, ReLU activation, and 1x1 convolution, and then added. The added features are activated using the Sigmoid function and then activated. Figure 1 and Features Figure 2 Multiply to get new features Figure 1 and Features Figure 2 , the two feature maps are convolved with a kernel of 3, batch normalized, and ReLU activated on different branches, and then multiplied with the feature Figure 1 and Features Figure 2 Perform channel splicing, and use the convolution kernel size of 3 to convolve the spliced ​​features with the previous multiplication feature map and feature Figure 1 and Features Figure 2 After splicing, convolution with a convolution kernel of 1 and Sigmoid function are performed to activate it, and the features obtained after activation are combined with the features Figure 1 and Features Figure 2 Multiply, add the multiplied features, and use the GELU activation function to get the difference features.

[0010] Furthermore, when the network is trained in a supervised manner, we use the cross entropy loss function for this type of problem.

[0011] Compared with the existing technology, the present invention has the following beneficial effects: (1) This paper proposes a CNN-Transformer dual-branch encoder for extracting features from remote sensing images. The encoder combines the sensitivity of CNN to local details and the broad perception of Transformer to global context, and can simultaneously capture multi-scale features ranging from small changes to large-scale patterns.

[0012] (2) The present invention proposes a semantic compensation module and inserts it between the CNN-Transformer dual-branch encoder to reduce the semantic gap between the features under the dual branches. By using the semantic compensation module, the fine local features extracted by the CNN encoder and the global semantic information captured by the Transformer encoder can be effectively supplemented and adjusted to ensure the semantic consistency between the two.

[0013] (3) The present invention proposes a CNN-Transformer adaptive fusion module for fusing features between CNN branches and Transformer branches. The CNN-Transformer adaptive fusion module uses learnable factors to adaptively balance the local sensitivity of CNN and the global generalization ability of Transformer, optimizing the combination of local details and global context. The CNN-Transformer adaptive fusion module enhances the generalization ability of various complex scenes by adaptively fusing features from different sources, and improves the applicability of the model to diverse remote sensing image data.

[0014] (4) The effectiveness of the method of the present invention is verified on two well-known remote sensing image change detection datasets CDD and WHU-CD. In order to verify the effectiveness of the present invention, a detailed comparison is made with the existing methods on multiple evaluation indicators, including precision, recall, overall accuracy, intersection over union, and F1 score. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other objects, features and advantages of the present invention will become more apparent through the following description of examples of the present invention with reference to the accompanying drawings, in which: Figure 1 A flowchart of a method for detecting changes in remote sensing images using adaptive fusion with semantic compensation according to an embodiment of the present invention is shown; Figure 2 An overall schematic diagram of a method for adaptive fusion remote sensing image change detection based on semantic compensation according to an embodiment of the present invention is shown; Figure 3 A diagram of a semantic compensation module according to an embodiment of the present invention is shown; Figure 4 A CNN-Transformer feature adaptive fusion module diagram according to an embodiment of the present invention is shown; Figure 5 A diagram of a change feature identification module according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0016] In order to make the purpose, advantages and technical methods of the present invention clearer, the following is a description of the specific examples. The specific examples described here are only used to explain the present invention and are not used to limit the present invention.

[0017] like Figure 1 As shown, it is a flowchart of the steps of a remote sensing image change detection method of multi-level difference feature adaptive fusion provided by an embodiment of the present invention, including steps S1-S7: S1, cut the original dataset images of change detection into non-overlapping sub-images according to the specified size, and divide them into training set, validation set, and test set. Perform data enhancement before inputting into the network.

[0018] In the present invention, preprocessing is performed on two publicly available datasets to obtain the bi-phase data required for model training and testing. These publicly available datasets include the WHU-CD dataset and the CDD dataset. The following will introduce these two datasets respectively.

[0019] (1) WHU-CD dataset: The WHU-CD dataset covers the area that was hit by a 6.3 magnitude earthquake in February 2011 and rebuilt in the following years. The dataset consists of aerial images acquired in April 2012, which contain 12,796 buildings within 20.5 square kilometers (16,077 buildings in the same area in the 2016 dataset). The original image size of WHU-CD is 32,507*15,354. Its resolution is 0.2 meters per pixel.

[0020] (2) CDD dataset: The CDD dataset consists of 7 pairs of seasonal change images (resolution 4725*2700 pixels) for manual creation of ground truth and 4 pairs of seasonal change images (very small changes, resolution 1900*1000 pixels) for manual addition of additional objects. The spatial resolution of the obtained images is 3 to 100 cm / pixel. CDD includes objects of different sizes (such as from cars to large building structures) and seasonal changes of natural objects (such as from single trees to vast forest areas).

[0021] In the example of the present invention, step S1 specifically includes steps S11-S13.

[0022] S11, cutting the original dataset image of change detection into 256x256 non-overlapping sub-images according to the specified size; S12, divide the cropped images into training set, validation set, and test set according to different ratios. For the CDD dataset, the training set, validation set, and test set are divided into a ratio of 10000 / 3000 / 3000; for the WHU-CD dataset, the training set, validation set, and test set are divided into a ratio of 5947 / 743 / 744; S13, these data sets are randomly flipped 8 times and cropped before being input into the network, and then normalized.

[0023] S2, input the dual-phase data into the twin encoder backbone network, and extract features of the dual-phase data in turn through different levels of the backbone network to obtain multi-level features.

[0024] like Figure 2 As shown, an overall framework diagram of an adaptive fusion remote sensing image change detection method using semantic compensation is provided for an embodiment of the present invention. In the example of the present invention, the CNN-Transformer hybrid twin encoder backbone network is divided into four parts, namely, feature extraction at different levels.

[0025] (1) CNN branch The first layer performs two consecutive convolutions with a kernel size of 3, a padding of 1, and a stride of 2 on the input remote sensing image with a channel number of 3, reducing the resolution from 256x256 to 64x64 and increasing the number of channels from 3 to 64; the second layer uses two convolutions with a kernel size of 3 to reduce the feature map from 64x64 to 32x32 and increase the number of channels from 64 to 128; the third layer uses two convolutions with a kernel size of 3 to reduce the feature map from 32x32 to 16x16 and increase the number of channels from 128 to 256; the fourth layer uses two convolutions with a kernel size of 3 to reduce the feature map from 16x16 to 8x8 and increase the number of channels from 256 to 512.

[0026] (2) Transformer branch The first layer uses a convolution operation with a kernel size of 7, a padding of 3, and a stride of 4 to increase the dimension of the feature map, where the dimension size is 64 and the feature map scale is 64x64. The feature map is layer-normalized and then multi-head self-attention is performed. The feature map obtained after the operation is added to the feature map without multi-head self-attention, and then layer-normalization and multi-layer perceptron are performed in turn. The second layer uses a convolution operation with a kernel size of 3 and a padding of 1 to convert the feature map from 64x64 to 32x32, and the channel The number of channels increases from 64 to 128, and then the feature map is subjected to multi-head self-attention operation; the third layer uses convolution with a kernel size of 3 and padding of 1 to change the scale of the feature map from 32x32 to 16x16, and the number of channels increases from 128 to 256, and then the feature map is subjected to multi-head self-attention operation; the fourth layer uses convolution with a kernel size of 3 and padding of 1 to change the scale of the feature map from 16x16 to 8x8, and the number of channels increases from 256 to 512, and then the feature map is subjected to multi-head self-attention operation.

[0027] S3, the dual-phase features extracted by the backbone network at different levels and branches are respectively input into the semantic compensation module and then input into the corresponding branches at different levels to perform corresponding feature calculations.

[0028] like Figure 3 As shown, a structure diagram of a semantic compensation module of an embodiment of the present invention is provided, and the specific steps of step S3 include S31-S34: S31, feature map F 1 and F 2 They come from the CNN and Transformer branches respectively, and are transformed into three vectors Q, K, and V through convolution with a convolution kernel size of 1x1.

[0029] S32, for each vector, different reshape operations are performed to generate vectors for calculating horizontal and vertical attention. In order to avoid unreasonable attention distribution caused by comparing the feature vector at each position with itself, the Q and K vectors in the vertical direction are multiplied and then added to a diagonal matrix with negative infinity as the main diagonal element. After that, it is concatenated with the horizontal attention and activated using Sigmoid. Finally, the activated feature distribution is split into horizontal and vertical directions, multiplied with the corresponding V vector, and then added to obtain the horizontal and vertical attention distribution. This process is used Cross_Att w_h ( ) The feature map is fused with the original input feature map by element-wise addition. F fus1 , F fus2 , the process is expressed as: , S33, in calculation F fus1 , F fus2 The corresponding channel weight is then multiplied by the input feature map F 1 ，F 2 and Cross_Att w_h (F 1, F 2 ) , Cross_Att w_h (F 2, F 1 ) And add the two together to get F CW1 , F CW2 This process is expressed as: , in S ig is the Sigmoid function, FC is the fully connected layer, GAP is the global average pooling, GMP is the global maximum pooling, is element-wise addition, This is element-wise multiplication.

[0030] S34, will F CW1 and F CW2 With the input feature map F 1 , F 2 After adding, add it to the input feature map F 1 , F 2 Perform channel-level information exchange, that is, exchange the elements of the input feature map and the fused feature map at the even positions of the channel dimension: , In the above formula, CE represents channel-level element exchange. The semantic compensation module adds the fused features on the basis of retaining the original features, compensating the semantic information between the CNN branch and the Transformer branch.

[0031] S4, the modules of different branches after semantic compensation are subjected to dual-branch feature fusion through the CNN-Transformer feature adaptive fusion module.

[0032] like Figure 4 As shown, a network structure diagram of a CNN-Transformer feature adaptive fusion module according to an embodiment of the present invention is provided, and the specific steps of step S4 include S41-S43: S41, feature map F 1 and F 2 They come from the CNN and Transformer branches respectively, and generate Q, K, V vectors through convolution with a kernel size of 3x3 and layer normalization. The formula is as follows: , S42, the two groups Q, K, V are fused using cross multi-head attention and multiplied by the learnable factor β to obtain and Then multiply the input feature map by the learnable factor α and Add together to get . It can be expressed as follows: , , S43, will After layer normalization, the channel dimension is increased by 1x1 convolution and then segmented in the channel direction. and ;Will After depth-wise separable convolution, The features are multiplied to enhance the feature extraction capability. The activation function and convolution are used to obtain F e1 and F e2 . It can be expressed as follows: , Among them, split means to cut the input feature into half of the original along the channel dimension; Conv d represents depthwise convolution; Conv p represents point convolution. Then the learning factor is introduced and F e1 , F e2 Multiply and multiply by the learnable factor Input features F 1 , F 2 Add.

[0033] S44, connects the two branches and performs convolution to complete adaptive fusion. It can be expressed as follows: , Among them, BN means batch normalization, and Concat means channel dimension concatenation.

[0034] S5, inputting the fused feature maps of different time phases into the change feature identification module to extract the change features.

[0035] like Figure 5 The figure shows a structural diagram of a change feature identification module according to an embodiment of the present invention. The specific steps of step S5 include S51-S54: S51, feature map F t1 and F t2 They come from the corresponding features extracted from remote sensing images of different phases in the twin network. First, the two feature maps are spliced ​​in the channel dimension, and then the two features are globally averaged and globally max pooled, and two convolutions with a kernel size of 1x1 are performed to halve their dimensions. After adding the two branches element by element, Sigmoid is used for activation and then element by element multiplication with the original input features. It can be expressed as follows: , S52, will F t1_w , F t2_w Use convolution with a kernel size of 3x3 and multiply them to get F m , the multiplied features are connected with these two features in the channel dimension to obtain F mc1 , F mc2 : , S53, yes F mc1 , F mc2 Use convolution with a kernel size of 3x3 and F m , Fc3 1 , Fc3 2 Connect the channel dimensions to get F .right F Use convolution with kernel size 1x1 to change features F The channel dimension is activated by the Sigmoid function to obtain the featureF s . F s Differences and characteristics Fc3 1 , Fc3 2 After multiplication, the sum is added, and finally the GELU activation function is used for activation to obtain the change feature output by the change feature identification module. It can be expressed as: , , S6, inputs the fused difference features into the detection head to perform pixel-level remote sensing image change detection and generate a change prediction map after the detection is completed, and iteratively trains the model parameters that save the best results.

[0036] S7, input the dual-temporal remote sensing images of the test set into the remote sensing image change detection model to obtain the prediction of the changed objects.

[0037] In this embodiment, the cross entropy loss function is used to train the network model. The cross entropy loss function is defined as follows: , in H, W Refers to the length and width of the true value label, label(i) represents the i-th pixel in the label image, p(i) represents the probability of the model predicting the i-th pixel, H(.) is the cross entropy function. This embodiment uses AdamW as the optimizer, sets the weight decay rate to 0.01, the initial learning rate to 0.0001, and the epoch to 200.

[0038] Table 1 Comparison of the proposed method with well-known change detection methods on the WHU-CD dataset

[0039] Table 2 Comparison of the proposed method with well-known change detection methods on CDD dataset

[0040] Tables 1 and 2 show the comparison of the accuracy, intersection-over-union ratio, precision, recall rate and F1 score of the method of the present invention with the currently well-known change detection methods on the WHU-CD and CDD datasets to better illustrate the effectiveness of the method. From the above tables, it can be seen that the current method has a significant improvement compared with the currently well-known change detection methods, which effectively verifies the effectiveness of the present invention.

[0041] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A method for adaptive fusion remote sensing image change detection based on semantic compensation, characterized in that: The steps include: S1, cut the original dataset images of change detection into non-overlapping sub-images according to the specified size, and divide them into training set, validation set, and test set; perform data enhancement on the training set images before inputting them into the network; the specific process is as follows: S1.1, crop the original images of the dataset into images of size 256x256; S1.2, divide the cropped images into training set, validation set, and test set according to different proportions; S1.3, randomly flip and crop the training set; S1.4, normalize the enhanced training set, validation set, and test set images; S2, the input image is initially downsampled to convert the image with a resolution of 256x256 into a feature map of 64x64, and the number of channels of the image is changed from 3 to 128; the feature maps are respectively passed into the convolution and Transformer blocks with a convolution kernel size of 3; the semantic compensation module is used to compensate the information of the above two semantically different feature maps; the scale of the feature map is reduced and the number of channels of the feature map is increased before passing it into the convolution and Transformer blocks of the next level; after four layers of CNN-Transformer feature extraction, it reaches the end of the backbone network; S3, the features output by the semantic compensation module in the backbone network are fused using the CNN-Transformer feature adaptive fusion module; S4, input the fused CNN-Transformer features at different levels into the change feature identification module to extract difference features; S5, upsample the bottom layer difference features to make them the same scale as the previous layer; concatenate the upsampled features with the previous layer features in the channel dimension; deconvolve the concatenated features to change their scale and channel; repeat the previous process to the first layer; deconvolve the features fused with the first layer features and then use a convolution kernel size of 3; The features are upsampled and convolution is performed twice with a kernel size of 3 to generate a change map; 200 rounds of iterative training are performed on the training set and the model parameters with the best effect are saved; S6, inputting the dual-temporal remote sensing images of the test set into the remote sensing image change detection model to obtain the change map predicted by the model.

2. The adaptive fusion remote sensing image change detection method based on semantic compensation according to claim 1, characterized in that: The semantic compensation module performs semantic compensation on the features extracted by CNN and Transformer, and S2 includes: Feature Map F 1 and F 2 They come from the CNN and Transformer branches respectively, and are transformed into three vectors, Q, K, and V, through convolution with a convolution kernel size of 1. Furthermore, different reshape operations are performed on each vector to generate vectors for calculating horizontal and vertical attention. In order to avoid unreasonable attention distribution caused by calculating the feature vector at each position with itself, the vertical Q and K vectors are multiplied and then added to a diagonal matrix with negative infinity as the main diagonal elements. After that, it is concatenated with the horizontal attention and activated using Sigmoid. Finally, the activated feature distribution is split into horizontal and vertical directions, multiplied with the corresponding V vector, and then added to obtain the horizontal and vertical attention distribution. This process is explained with Cross_Att w_h ( ) The feature map is fused with the original input feature map by element-wise addition. F fus1 and F fus2 , the process is expressed as: ， In calculation F fus1 , F fus2 The corresponding channel weight is then multiplied by the input feature map F 1 ，F 2 and Cross_Att w_h (F 1, F 2 ) , Cross_Att w_h (F 2, F 1 ) And add the two together to get F CW1 , F CW2 This process is expressed as: ， ， in S ig is the Sigmoid function, FC is the fully connected layer, GAP is the global average pooling, GMP is the global maximum pooling, is element-wise addition, is element-wise multiplication; Will F CW1 and F CW2 With the input feature map F 1 , F 2 After adding, add it to the input feature map F 1 , F 2 Perform channel-level information exchange, that is, exchange the elements of the input feature map and the fused feature map at the even positions of the channel dimension: ， In the above formula, CE represents channel-level element exchange. The semantic compensation module adds the fused features while retaining the original features, compensating the semantic information between the CNN branch and the Transformer branch.

3. The adaptive fusion remote sensing image change detection method based on semantic compensation according to claim 1, characterized in that: The feature adaptive fusion module uses a learnable factor to perform adaptive fusion between features; S3 includes: Feature Map F 1 and F 2 They come from the CNN and Transformer branches respectively, and the Q, K, and V vectors are generated by convolution with a kernel size of 3 and layer normalization. The formula is as follows: ， The two groups Q, K, and V are fused using cross multi-head attention and multiplied by the learnable factor β to obtain and Then multiply the input feature map by the learnable factor α and Add together to get , . It can be expressed as follows: ， Will After layer normalization, the channel dimension is increased by 1x1 convolution and then segmented in the channel direction. and ; After passing the depth-wise separable convolution, The features are multiplied to enhance the feature extraction capability. The activation function and convolution are used to obtain F e1 and F e2 . It can be expressed as follows: ， Among them, split means to cut the input feature into half of the original along the channel dimension; Conv d represents depthwise convolution; Conv p Represents point convolution; learning factors are then introduced and F e1 , F e2 Multiply and multiply by the learnable factor Input features F 1 , F 2 Add; finally, connect the two branches and perform convolution to complete adaptive fusion; the formula is as follows: ， Among them, BN means batch normalization, and Concat means channel dimension concatenation.

4. The adaptive fusion remote sensing image change detection method based on semantic compensation according to claim 1, characterized in that: The change feature identification module is good at identifying different types of changes such as building construction or demolition, vegetation changes, etc. The change feature identification module can handle small-scale structural changes from macro-city to micro-scale by capturing key features such as edges, textures and shapes; In addition, it can also reduce the misjudgment rate by distinguishing between real changes and pseudo changes caused by changes in imaging conditions; S4 is specifically: Feature Map F t1 and F t2 They come from the corresponding features extracted from remote sensing images of different phases in the twin network. First, the two feature maps are spliced ​​in the channel dimension, and then the two features are globally averaged and globally max-pooled respectively, and two convolutions with a kernel size of 1x1 are performed to halve their dimensions. After adding the two branches element by element, Sigmoid is used for activation and then multiplied element by element with the original input features. The formula is as follows: ， ， Secondly, F t1_w , F t2_w Use convolution with a kernel size of 3x3 and multiply them to get F m , the multiplied features are connected with these two features in the channel dimension to obtain F mc1 , F mc2 : ， right F mc1 , F mc2 After using convolution with a convolution kernel size of 3, F m , Fc3 1 , Fc3 2 Connect the channel dimensions to get F ;right F Use convolution with kernel size 1 to change features F The channel dimension is activated by the Sigmoid function to obtain the feature F s ; The feature F s Differences and characteristics Fc3 1 , Fc3 2 After multiplication, the addition is performed, and finally the GELU activation function is used for activation to obtain the change feature output by the change feature identification module; it can be expressed as: 。

Images