Multi-level fusion building change detection method based on CNN and Transformer

Abstract

A multi-level fusion building change detection method based on CNN and Transformer includes the following steps: Step 1: Preprocessing two remote sensing images; Step 2: Inputting the hierarchical features extracted from three sets of channels into an improved spatiotemporal attention cross-enhancement module; Step 3: Inputting the enhanced two-channel multi-level features obtained in Step 2 into an improved channel interaction multi-level fusion module; Step 4: Extracting building boundaries using the Soble operator; Step 5: Inputting the fused features obtained in Step 3 into a multi-layer perceptron (MLP) and a data-dependent upsampling module, training the model through backpropagation; and finally, using the trained model for change detection. This method effectively integrates the advantages of CNN and Transformer, extracting features from two image phases more efficiently, and achieving superior change detection results.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing monitoring technology, and in particular to a multi-level fusion building change detection method based on CNN and Transformer. Background Art

[0002] Building change detection using remote sensing is the process of extracting building change information by analyzing remote sensing images acquired at different times at the same geographic location. This process is crucial for urban planning, cadastral surveying, map updates, and the rational use of land resources. With the rapid development of deep learning theory and technology, building change detection based on deep learning has become a key research area in remote sensing.

[0003] Convolutional neural networks (CNNs) are one of the most commonly used deep learning models, widely used in fields such as natural language processing, image classification, object detection, and semantic segmentation. CNNs possess powerful feature representation capabilities, capable of extracting rich and abstract local contextual information from imagery, and have become the mainstream network for remote sensing change detection. However, CNNs employ a local connection mechanism, which limits their receptive field size and hinders their ability to effectively exploit global image dependencies.

[0004] In recent years, the Transformer model has gained popularity in the field of deep learning. Using a multi-head global attention mechanism, the Transformer excels at capturing global dependencies and effectively overcomes the limitations of CNN receptive field size. The Transformer model has also been successfully applied to remote sensing change detection. However, the Transformer's uniform use of global representations at every level of the model can easily lead to redundancy between shallow and deep layers. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-level fusion building change detection method based on CNN and Transformer, which can effectively improve the accuracy of building change detection.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The multi-level fusion building change detection method based on CNN and Transformer has the following steps: Step 1, preprocess the two-phase remote sensing images, and then extract the multi-level features of the two-phase images through three channels: Channel 1 cascades the two-phase images and extracts the multi-level features of the cascaded images through the residual CNN network; Channel 2 performs difference calculation on the two-phase images and extracts the multi-level features of the difference image through the residual CNN network; Channel 3 cascades the two-phase images and extracts the multi-level features of the cascaded image through the Transformer network;

[0008] Step 2: Input the features of each level extracted from the three groups of channels into the spatiotemporal attention cross enhancement module, use the spatiotemporal difference features output by channel 2 to perform spatiotemporal attention cross enhancement on the features of channels 1 and 3, and output the enhanced multi-level features of channels 1 and 3;

[0009] Step 3: Input the enhanced two-channel multi-level features obtained in Step 2 into the channel interaction multi-level fusion module, and perform two multi-level fusion operations on the features of the two channels, namely channel priority and level priority;

[0010] Step 4: Use the Soble operator to extract the building boundary and expand the extracted building boundary area through Gaussian blur operation;

[0011] Step 5: First, the fusion features obtained in Step 3 are input into the multi-layer perceptron MLP and the data-related upsampling module in sequence, and the change probability map is solved through the SoftMax layer; then, based on the change probability map, the change reference map and the boundary area obtained in Step 4, the loss is calculated using the boundary weighted cross entropy loss function, and the model is trained through back propagation; finally, the trained model is used for change detection.

[0012] The specific process of Step 1 above is:

[0013] Step 1.1: Perform cascade operation and difference operation on the two pre-processed remote sensing images X1 and X2 to obtain the cascade image CI and the difference image DI respectively:

[0014] CI=Concat(X1,X2) (1)

[0015] DI=X1-X2 (2)

[0016] Where Concat represents the cascade operation;

[0017] Step 1.2, channel 1 inputs the cascade image CI into the residual CNN network ResNet50 to obtain four levels of features F c ={c i|i=1,2,3,4}; Channel 2 inputs the difference image DI into the residual CNN network ResNet18 to obtain four levels of features F d ={d i |i=1,2,3,4}; Channel 3 inputs the cascade image CI into the Swin Transformer network to obtain four levels of features F t ={t i |i=1,2,3,4}.

[0018] The specific steps of Step 2 above are as follows:

[0019] The spatiotemporal attention cross enhancement module consists of two submodules: the temporal-spatial attention-map module (TSAM) and the cross enhancement module (CEM).

[0020] Step 2.1, the multi-level features F extracted from channel 2 in Step 1.2 d ={d i |i=1,2,3,4} is input to the TSAM module to generate a multi-level attention map S={s i |i=1,2,3,4}, where the attention map s of the i-th level i Based on the hierarchical feature d i Generate, the specific process is as follows:

[0021] F d The size of the i-th level is C i ×H i ×W i Features of d i Perform average and maximum double pooling operations to obtain two 1×H i ×W i The feature map is then concatenated, and the concatenated features are passed through 7×7 convolution and Sigmoid function to generate the attention map s. i (1×H i ×W i ); generate s i The formula is as follows:

[0022] s i =σ(Conv(Concat(AvgPool(d i ),MaxPool(d i ))))(i=1,2,3,4) (3)

[0023] In the formula, σ represents the Sigmoid function, Conv represents the convolution operation, Concat represents the channel cascade operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively;

[0024] Step 2.2, the multi-level features F extracted from channel 1 in Step 1.2 c The multi-level attention map S generated by Step 2.1 is input into the cross enhancement module CEM, and the attention map S is used to analyze the feature F. c Perform weighted enhancement to obtain enhanced multi-level features F c ′={c′ i |i=1,2,3,4}; Cross enhancement is performed between the same layers. c The i-th level feature c i (C i ×H i ×W i ), through the i-th level attention map s i Make enhancements;

[0025] Step 2.3, the multi-level features F extracted from channel 3 in Step 1.2 t The multi-level attention map S generated in Step 2.1 is input into the cross enhancement module CEM, and the attention map S is used to analyze the feature F. t Perform weighted enhancement to obtain enhanced multi-level features F t ′={t i ′|i=1,2,3,4}; Cross enhancement is performed between the same layers. t The i-th level feature t i (C i ×H i ×W i ), through the i-th level attention map s i Make enhancements.

[0026] In Step 3 above, the channel interaction multi-level fusion module includes two fusion branches: channel priority and level priority. The specific fusion steps are as follows:

[0027] Step 3.1: Multi-level features F obtained in Step 2 c ′ and F t 'Carry out hierarchical priority multi-level fusion;

[0028] Step 3.1.1, multi-level features F c ′={c i ′|i=1,2,3,4} and F t ′={t i′|i=1,2,3,4} performs fusion between the same levels to generate four-level fusion features F f ={f i |i=1,2,3,4}: In the i-th level, the hierarchical features c of the two channels are i ′ and t i ′ are added together, and then the 3×3 convolution, batch normalization, and ReLU function are performed in sequence to obtain the i-th level fusion feature f i , the formula is as follows:

[0029] f i =ReLU(BN(Conv(c i ′+t i ′)))(i=1,2,3,4) (4)

[0030] Where BN stands for batch normalization, and Conv stands for convolution operation;

[0031] Step 3.1.2, F f The four-level features are fused at different levels, and the following operations are performed in sequence: i (i=1,2,3,4) are upsampled to 1 / 4 of the original image size and cascaded together. The cascaded features are input into the MLP to obtain the output F1 of the layer-priority multi-level fusion branch;

[0032] Step 3.2: Multi-level features F obtained in Step 2 c ′ and F t 'Perform channel priority multi-level fusion; Step 3.2.1, enhance the features F of channels 1 and 3 c ′ and F t 'Perform intra-channel fusion: The enhanced feature F of channel 1 c Each level feature c in ′ i ′Upsample to the original Figure 1 / 4 size and cascade, and perform convolution operation on the cascaded features to obtain the multi-level fusion feature F in the channel. f1 At the same time, the enhanced feature F of channel 3 t Each level feature t in ′ i ′Upsample to the original Figure 1 / 4 size and cascade, and perform convolution operation on the cascaded features to obtain the multi-level fusion feature F in the channel. f2 ;

[0033] Step 3.2.2, F f1 and F f2 Perform inter-channel mask fusion: First, perform mask fusion on F f1 and F f2Generate masks M1 and M2 through 1×1 convolution; then cross-multiply the in-channel fusion features with the mask, that is, let F f1 × M2, get F f1 ′, let F f2 ×M1 multiplication, we get F f2 ′. Finally, F f1 ′ and F f2 'The cascade input is fed into the MLP to obtain the output F2 of the channel-priority multi-level fusion branch;

[0034] Step 3.2.3. Concatenate the fusion features F1 and F2 output by the two branches and input them into the channel attention module CAM to obtain the final output of the channel interaction multi-level fusion module, denoted as F3.

[0035] The specific steps of Step 5 above are:

[0036] Step 5.1. Input F3 into the multi-layer perceptron (MLP) and the data-related upsampling module in sequence to obtain the features of the original image size; and generate a change probability map through the SoftMax layer, and record the probability map as p.

[0037] Step 5.2: Use the boundary area obtained in Step 4 to obtain the weight map w in the loss function, which is used to enhance the boundary. The formula is as follows:

[0038]

[0039] Where α is a hyperparameter obtained from the experiment, e represents the boundary area obtained in Step 4, and i∈e indicates that pixel i belongs to the boundary area;

[0040] Step 5.3. Calculate the weighted cross entropy loss function Loss based on the change reference graph y, the change probability graph p, and the weights solved in Step 5.2:

[0041]

[0042] Where Ω represents the image, H and W are the height and width of the image, and p i represents the change probability of pixel i, y i represents the category label of pixel i, y i =1 means pixel i belongs to the change class, y i = 0 means pixel i belongs to the unchanged class, w i represents the weight of pixel i.

[0043] The present invention provides a multi-level fusion building change detection method based on CNN and Transformer, which adopts a parallel architecture of CNN and Transformer as a whole: first, multi-level features of two-phase image cascades and differences are extracted through three channels; secondly, a spatiotemporal attention cross-enhancement module is designed to realize the enhancement of channel 2 features on channel 1 and 3 features; then, a channel interaction multi-level fusion module is designed to perform multi-level interactive fusion of channel 1 and 3 enhanced features through two multi-level fusion operations, channel priority and level priority; finally, the building boundary area is enhanced based on the Soble operator, Gaussian blur operation and boundary weighted cross entropy function.

[0044] Compared with the existing technology, the technical solution proposed in the present invention can more effectively integrate the advantages of CNN and Transformer, more effectively extract the features of the two-phase images, and obtain better change detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described below with reference to the accompanying drawings and examples:

[0046] Figure 1 It is the overall framework diagram of the building change detection network proposed in the present invention;

[0047] Figure 2 Schematic diagram of the spatiotemporal attention cross-enhancement module of the network proposed in this invention;

[0048] Figure 3 Schematic diagram of the hierarchical priority fusion branch of the channel interaction multi-level fusion module in the network proposed by the present invention;

[0049] Figure 4 Schematic diagram of the channel priority fusion branch of the channel interactive multi-level fusion module in the network proposed by the present invention;

[0050] Figure 5 These are three typical examples of change detection results of different networks on the LEVIR-CD dataset;

[0051] Figure 6 These are three typical examples of change detection results of different networks on the WHU-CD dataset;

[0052] Figure 7 These are three typical examples of change detection results of different networks on the Google Data dataset. DETAILED DESCRIPTION

[0053] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0054] This example uses three sets of commonly used public building change detection datasets for experiments:

[0055] LEVIR-CD: Proposed by H. Chen and Z. Shi in the paper "Aspatial-temporal attention-based method and a new dataset for remote sensing image change detection" (RemoteSens, 2020), it contains 637 pairs of remote sensing images of size 1024×1024 pixels. The images are cropped into image blocks of size 256×256 pixels without overlapping, and 7120, 1024, and 2048 groups of image blocks are obtained as training sets, validation sets, and test sets, respectively.

[0056] WHU-CD: Proposed by S. Ji et al. in the paper "Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set" (IEEE Trans. Geosci. Remote Sens., 2019). The experimental data size is 32507 × 15354 pixels. It is cropped to 256 × 256 pixels without overlap, resulting in 7620 pairs of image blocks. They are randomly divided into training, validation, and test sets in a ratio of 8:1:1.

[0057] Google Data: Proposed by D. Peng et al. in the paper "SemiCDNet: Asemisupervised convolutional neural network for change detection in high-resolution remote-sensing images" (IEEE Trans. Geosci. Remote Sens., 2021), this dataset contains 19 pairs of seasonally varying images, ranging in size from 1006 × 1168 pixels to 4936 × 5224 pixels. The images were uniformly cropped to 256 × 256 pixels without overlap, resulting in 3130 sets. These sets were randomly divided into training, validation, and test sets in an 8:1:1 ratio.

[0058] The network framework diagram of the method of the present invention is as follows Figure 1 As shown, the specific steps include:

[0059] Step 1: Preprocess the two phases of remote sensing images, and then extract the multi-level features of the two phases of images through three channels: Channel 1 cascades the two phases of images and extracts the multi-level features of the cascaded images through the residual CNN network; Channel 2 performs difference calculation on the two phases of images and extracts the multi-level features of the difference image through the residual CNN network; Channel 3 cascades the two phases of images and extracts the multi-level features of the cascaded images through the Transformer network. Figure 1 , the specific steps are as follows:

[0060] 1.1) Preprocess the dataset: Numerically, the RGB channels of the image are first normalized, and then standardized to a variance of 0.5 and a mean of 0.5. At the same time, conventional data augmentation is performed on the image, including random rotation, random flipping, and random scaling.

[0061] 1.2) Perform cascade operation and difference operation on the two pre-processed remote sensing images X1 and X2 to obtain the cascade image CI and the difference image DI respectively:

[0062] CI=Concat(X1,X2) (1)

[0063] DI=X1-X2 (2)

[0064] Concat represents a cascade operation.

[0065] 1.3) In channel 1, the cascade image CI is input into the ResNet50 network, the output of each layer is extracted, and then processed by 1×1 convolution to obtain four levels of convolution features, denoted as F c ={c i |i=1,2,3,4}, where c i represents the output of the i-th layer.

[0066] 1.4) In channel 2, the difference image DI is input into the ResNet18 network, the output of each layer is extracted, and then processed by 1×1 convolution to obtain four levels of spatiotemporal difference features, denoted as F d ={d i |i=1,2,3,4}, where d i represents the output of the i-th layer.

[0067] 1.5) In channel 3, the cascaded image CI is input into the Swin Transformer network, the output of each layer is extracted, and then processed by 1×1 convolution to obtain four levels of Transformer features, denoted as F t ={t i |i=1,2,3,4}, where t i represents the output of the i-th layer;

[0068] In the embodiment of the present invention, the last layer of the ResNet50, ResNet18 and Swin Transformer networks is improved without performing downsampling operations, so the sizes of the four groups of hierarchical features output by the three groups of channels are 1 / 4, 1 / 8, 1 / 16 and 1 / 16 of the original images respectively.

[0069] This method uses three parallel channels. Channels 1 and 3 employ CNN and Transformer networks, respectively, integrating the strengths of CNN and Transformer to more effectively capture local and global information from two image phases. Channel 2 is an auxiliary channel that enhances the features extracted by channels 1 and 3 by extracting the spatiotemporal differences between the two image phases.

[0070] Step 2: Input the features of each level extracted by the three groups of channels into the spatiotemporal attention cross enhancement module, use the spatiotemporal difference features output by channel 2 to perform spatiotemporal attention cross enhancement on the features of channels 1 and 3, and output the enhanced multi-level features of channels 1 and 3. Figure 1 and 2 The spatiotemporal attention cross-enhancement module consists of two submodules: the temporal-spatial attention-map module (TSAM) and the cross-enhancement module (CEM). The specific steps are as follows:

[0071] 2.1) The multi-level features F extracted from channel 2 d ={d i |i=1,2,3,4} is input to the TSAM module to generate a multi-level attention map S={s i |i=1,2,3,4}, where the attention map s of the i-th level i Based on the hierarchical feature d i Generate, the specific process is as follows:

[0072] F d The size of the i-th level is C i ×H i ×W i Features of d i Enter TSAM i The module performs the following operations in sequence: First, two 1×H i ×W i The feature map is then cascaded, and the cascaded features are passed through 7×7 convolution and Sigmoid function to generate the attention map s. i (1×H i ×W i ). Generate s iThe formula is as follows:

[0073] s i =σ(Conv(Concat(AvgPool(d i ),MaxPool(d i ))))(i=1,2,3,4) (3)

[0074] Where σ represents the Sigmoid function, Conv represents the convolution operation, Concat represents the channel cascade operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively.

[0075] 2.2) The multi-level features F extracted from channel 1 c The multi-level attention map S generated in step 2.1 is input into the cross enhancement module CEM1, and the attention map S is used to analyze the feature F c Perform weighted enhancement to obtain enhanced multi-level features F c ′={c i ′|i=1,2,3,4}. Cross enhancement is performed between the same levels, and F c The i-th level feature c i (C i ×H i ×W i ) Input CEM1 i Module, through the i-th level attention map s i Make enhancements.

[0076] 2.3) The multi-level features F extracted from channel 3 t The multi-level attention map S generated in step 2.1 is input into the cross enhancement module CEM2, and the attention map S is used to analyze the feature F t Perform weighted enhancement to obtain enhanced multi-level features F t ′={t i ′|i=1,2,3,4}. Cross enhancement is performed between the same levels, and F t The i-th level feature t i (C i ×H i ×W i ) Input CEM2 i Module, through the i-th level attention map s i Make enhancements.

[0077] In step 3, the enhanced two-channel multi-level features obtained in step 2 are input into the channel interaction multi-level fusion module, and the features of the two channels are subjected to channel priority and level priority multi-level fusion operations respectively. The channel interaction multi-level fusion module includes two fusion branches: channel priority and level priority. CNN features and Transformer features are complementary fused from the perspective of channels and levels. The level priority fusion branch is to fuse the two channel features within the level first and then between the levels, while the channel priority fusion branch is to fuse within the channel first and then between the channels. The following steps are specifically adopted:

[0078] 3.1) In order to retain valuable information at each level during the fusion process, the multi-level features F obtained in step 2 are c ′ and F t 'Carry out hierarchical priority multi-level fusion, refer to Figure 3 .

[0079] ① For multi-level features F c ′={c i ′|i=1,2,3,4} and F t ′={t i ′|i=1,2,3,4} performs fusion between the same levels to generate four-level fusion features F f ={f i |i=1,2,3,4}: In the i-th level, the hierarchical features c of the two channels are i ′ and t i ′Intra-Layer Fusion Module (ILFM). In ILFM, c i ′ and t i ′ is added and then goes through 3×3 convolution, batch normalization, and ReLU function to obtain the i-th level fusion feature f i , the formula is as follows:

[0080] f i =ReLU(BN(Conv(c i ′+t i ′)))(i=1,2,3,4) (4)

[0081] Where BN stands for batch normalization and Conv stands for convolution operation.

[0082] ② For F f The four-level features are fused at different levels, and the following operations are performed in sequence: i (i=1,2,3,4) are upsampled to 1 / 4 of the original image size and cascaded together. The cascaded features are input into the MLP to obtain the output F1 of the level-priority multi-level fusion branch.

[0083] 3.2) In order to enhance the two channels and filter redundant information, the multi-level features F obtained in step 2 are c ′ and F t 'Perform channel priority multi-level fusion, refer to Figure 4 .

[0084] ① Enhanced feature F for channels 1 and 3 c ′ and F t 'Perform intra-channel fusion: The enhanced feature F of channel 1 c Each level feature c in ′ i ′Upsample to the original Figure 1 / 4 size and cascade, and perform convolution operation on the cascaded features to obtain the multi-level fusion feature F in the channel. f1 At the same time, the enhanced feature F of channel 3 t Each level feature t in ′ i ′Upsample to the original Figure 1 / 4 size and cascade, and perform convolution operation on the cascaded features to obtain the multi-level fusion feature F in the channel. f2 .

[0085] ② For F f1 and F f2 Perform inter-channel mask fusion: First, perform mask fusion on F f1 and F f2 Generate masks M1 and M2 through 1×1 convolution; then cross-multiply the in-channel fusion features with the mask, that is, let F f1 × M2, get F f1 ′, let F f2 ×M1 multiplication, we get F f2 ′. Finally, F f1 ′ and F f2 ’ is cascaded and input into MLP to obtain the output F2 of the channel-priority multi-level fusion branch.

[0086] 3.3) The outputs F1 and F2 of the two fusion branches are cascaded and input into the channel attention module (CAM) to obtain the final output of the channel interaction multi-level fusion module, which is recorded as F3.

[0087] Step 4: Use the Soble operator to extract the building boundary and expand the extracted building boundary area through Gaussian blur operation. The specific steps are as follows:

[0088] 4.1) For the dataset label map y, use the Soble operator to perform boundary detection and obtain the preliminary building boundary area, which is recorded as y′.

[0089] 4.2) Expand the building boundary in y′ through Gaussian blur operation to obtain the final building boundary area, denoted as e.

[0090] In step 5, the fused features obtained in step 3 are first input into the multi-layer perceptron (MLP) and the data-related upsampling module in sequence, and the change probability map is solved through the SoftMax layer. Then, based on the change probability map, the change reference map, and the boundary area obtained in step 4, the loss is calculated using the boundary weighted cross entropy loss function, and the model is trained through backpropagation. Finally, the trained model is used for change detection. The specific steps are as follows:

[0091] 5.1) Input F3 into the multi-layer perceptron (MLP) and the data-dependent upsampling module in sequence to obtain the features of the original image size; and generate a change probability map through the SoftMax layer, which is denoted as p.

[0092] 5.2) Using the boundary region obtained in step 4, we obtain the weight map w in the loss function, which is used to enhance the boundary. The formula is as follows:

[0093]

[0094] Where α is a hyperparameter obtained experimentally, e represents the boundary region obtained in step 4, and i∈e indicates that pixel i belongs to the boundary region. In this embodiment, α is set to 1.

[0095] 5.3) Calculate the weighted cross entropy loss function Loss based on the change reference graph y, the change probability graph p, and the weights solved in step 5.2:

[0096]

[0097] Where Ω represents the image, H and W are the height and width of the image, and p i represents the change probability of pixel i, y i represents the category label of pixel i, y i =1 means pixel i belongs to the change class, y i = 0 means pixel i belongs to the unchanged class, w i Indicates the weight of pixel i. In this embodiment, both H and W are 256.

[0098] To verify the effectiveness of the network model proposed in this invention, the present invention is compared with the following change detection network:

[0099] (1) FC-EF, FC-Siam-diff, and FC-Siam-conc networks [change detection methods proposed by R. Caye Daudt et al. in the paper “Fully convolutional Siamese networks for change detection” (ICIP, 2018)]

[0100] (2) IFN network [a change detection network proposed by C. Zhang et al. in the paper “A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images” (ISPRS, 2020)]

[0101] (3) SNUNet network [a change detection network proposed by S. Fang et al. in the paper “SNUNet-CD: A densely connected Siamese network for change detection of VHR images” (IEEE Geosci. Remote Sens. Lett., 2022)]

[0102] (4) BIT network [a change detection network proposed by H. Chen et al. in the paper “Remote sensing image change detection with transformers” (IEEE Trans. Geosci. Remote Sens., 2022)]

[0103] (5) MSCANet network [a change detection network proposed by Mengxi Liu et al. in the paper “A CNN-transformer network with multiscale context aggregation for fine-grained cropland change detection” (IEEE J.Sel. Topics Appl. Earth Observ. Remote Sens., 2022)]

[0104] In order to demonstrate the actual effect of change detection of this method and compare it with other methods, precision, F1 score, recall, intersection over union (IoU), and overall accuracy (OA) are used as evaluation indicators. The specific formula is as follows:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] In the above formula, TP represents the number of pixels that are actually buildings but predicted to be non-buildings; TN represents the number of pixels that are actually non-buildings but predicted to be non-buildings; FP represents the number of pixels that are actually non-buildings but predicted to be buildings; and FN represents the number of pixels that are actually buildings but predicted to be non-buildings.

[0111] Figure 5 、 Figure 6 and Figure 7 The change detection result diagrams of the above-mentioned different change detection networks and typical examples of the present invention on three sets of data sets are respectively given.

[0112] Table 1 shows the evaluation indicators of the above-mentioned different change detection networks and the present invention on three datasets.

[0113]

[0114]

[0115] Comparing the change detection result graph and the evaluation index, the change detection effect of the present invention is significantly better than the change detection results of several other advanced level deep learning change detection technologies. Figure 5 、 Figure 6 and Figure 7 The detection result diagram shows that the present invention achieves more accurate results in both the overall change area and the detailed building boundaries.

[0116] As shown in Table 1, the change detection results of our proposed method significantly outperform those of the other comparison networks, achieving the highest F1, IoU, and OA on all three datasets. For example, on the WHU-CD dataset, the F1 score is 91.19%, which is 14.5%, 12.79%, 15.56%, 3.29%, 3.95%, 4.75%, and 4.2% higher than those of the other networks, respectively.

[0117] The experimental results verify the effectiveness of the multi-level fusion building change detection network model based on CNN and Transformer proposed in the present invention. In order to effectively utilize the respective advantages of CNN and Transformer, the network as a whole adopts a parallel architecture of CNN and Transformer: first, multi-level features of two-phase image cascades and differences are extracted through three channels; secondly, a spatiotemporal attention cross-enhancement module is designed to enhance the features of channels 1 and 3 by channel 2 features, making full use of spatiotemporal priors to better perform change detection tasks; then a channel interaction multi-level fusion module is designed to perform multi-level interactive fusion of channel 1 and 3 enhanced features through two multi-level fusion operations of channel priority and level priority, solving the problem of inter-level misalignment when CNN and Transformer features are fused, highlighting task-related information and suppressing redundant information; finally, the building boundary area is enhanced based on the Soble operator, Gaussian blur operation and boundary weighted cross entropy function, making the building boundary prediction more accurate. In summary, the change detection technology proposed in the present invention can obtain better change detection results.

[0118] The above is only one embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-level fusion building change detection method based on CNN and Transformer is characterized by the following steps: for: Step 1: Preprocess the two remote sensing images, and then extract the multi-level features of the two images through three channels: Channel 1 cascades the two images and extracts the multi-level features of the cascaded image through the residual CNN network; Channel 2 performs difference calculation on the two images and extracts the multi-level features of the difference image through the residual CNN network; Channel 3 cascades the two images and extracts the multi-level features of the cascaded image through the Transformer network; Step 2: Input the features of each level extracted from the three groups of channels into the spatiotemporal attention cross enhancement module, use the spatiotemporal difference features output by channel 2 to perform spatiotemporal attention cross enhancement on the features of channels 1 and 3, and output the enhanced multi-level features of channels 1 and 3; Step 3: Input the enhanced two-channel multi-level features obtained in Step 2 into the channel interaction multi-level fusion module, and perform two multi-level fusion operations on the features of the two channels, namely channel priority and level priority; Step 4: Use the Soble operator to extract the building boundary and expand the extracted building boundary area through Gaussian blur operation; Step 5: First, the fusion features obtained in Step 3 are sequentially input into the multi-layer perceptron (MLP) and the data-related upsampling module, and the change probability map is solved through the SoftMax layer. Then, based on the change probability map, the change reference map, and the boundary area obtained in Step 4, the loss is calculated using the boundary weighted cross entropy loss function, and the model is trained through backpropagation. Finally, the trained model is used for change detection. The specific steps of Step 2 are as follows: The spatiotemporal attention cross enhancement module consists of two submodules: the spatiotemporal attention map generation module and the cross enhancement module; Step 2.1, the multi-level features extracted from channel 2 in Step 1.2 Input to the TSAM module to generate a multi-level attention map S = { s i | i = 1, 2, 3, 4}, where i Attention map of each level s i Based on hierarchical features d i Generate, the specific process is as follows: right F d Middle i The level size is C i × H i × W i Features d i Perform average and maximum double pooling operations to obtain two 1× H i × W i The feature map is then concatenated, and the concatenated features are passed through 7×7 convolution and Sigmoid function to generate the attention map. s i (1× H i × W i );generate s i The formula is as follows: (3) Where, Represents the Sigmoid function, Conv represents the convolution operation, Concat represents the channel cascade operation, AvgPool and MaxPool represent the average pooling and maximum pooling operations respectively; Step 2.2, the multi-level features extracted from channel 1 in Step 1.2 F c And the multi-level attention map generated by Step 2.1 S Input into the cross enhancement module CEM, using the attention map S Features F c Perform weighted enhancement to obtain enhanced multi-level features ; Cross enhancement is performed between the same layers. F c Middle i Level features c i ( C i × H i × W i ), through the i Level-level attention map s i Make enhancements; Step 2.3, the multi-level features extracted from channel 3 in Step 1.2 F t and the multi-level attention map generated in Step 2.1 S Input into the cross enhancement module CEM, using the attention map S Features F t Perform weighted enhancement to obtain enhanced multi-level features ; Cross enhancement is performed between the same layers. F t Middle i Level features t i ( C i × H i × W i ), through the i Level-level attention map s i Make enhancements.

2. The multi-level fusion building change detection method based on CNN and Transformer according to claim 1 is characterized in that: The specific process of Step 1 is as follows: Step 1.1, the two phases of remote sensing images after preprocessing X 1 and X 2 Perform cascade operation and difference operation to obtain cascade images CI Sum difference image DI : (1) (2) Where Concat represents the cascade operation; Step 1.2, channel 1 will cascade the image CI Input to the residual CNN network ResNet50 to obtain four levels of features ; Channel 2 will be the difference image DI Input to the residual CNN network ResNet18 to obtain four levels of features ; Channel 3 will cascade the image CI Input into the Swin Transformer network to obtain four levels of features .

3. The multi-level fusion building change detection method based on CNN and Transformer according to claim 2 is characterized in that: In Step 3, the channel interaction multi-level fusion module includes two fusion branches: channel priority and level priority. The specific fusion steps are as follows: Step 3.1: Multi-level features obtained in Step 2 and Carry out hierarchical priority and multi-level integration; Step 3.1.1, multi-level features and Perform fusion between the same levels to generate four levels of fusion features : In the i In each level, the hierarchical features of the two channels are and Add them together and then go through 3×3 convolution, batch normalization, and ReLU function to get the i Level fusion features f i , the formula is as follows: (4) Where BN stands for batch normalization, and Conv stands for convolution operation; Step 3.1.2, The four-level features in the middle are fused at different levels, and the following operations are performed in sequence: f i ( i =1, 2, 3, 4) upsample to 1 / 4 of the original image size and concatenate them together. The concatenated features are input into the MLP to obtain the output of the layer-priority multi-level fusion branch F 1; Step 3.2: Multi-level features obtained in Step 2 and Perform channel-priority multi-level fusion; Step 3.2.

1. Enhanced features for channels 1 and 3 and Perform intra-channel fusion: enhance the features of channel 1 Features at each level Upsample to 1 / 4 of the original image size and concatenate, and perform convolution on the concatenated features to obtain multi-level fusion features within the channel. F f1， At the same time, the enhanced features of channel 3 Features at each level Upsample to 1 / 4 of the original image size and concatenate, and perform convolution on the concatenated features to obtain multi-level fusion features within the channel. F f2 ; Step 3.2.2, F f1 and F f2 Perform inter-channel mask fusion: First, F f1 and F f2 Generate mask through 1×1 convolution M 1 and M 2; Then cross-multiply the channel fusion feature with the mask, that is, F f1 × M 2. Get ,make F f2 × M Multiply by 1 and we get , and finally and Cascade input to MLP to obtain the output of channel-priority multi-level fusion branch F 2; Step 3.2.3, fusion features of the two branches output F 1 and F After cascading, the channel attention module CAM is input to obtain the final output of the channel interaction multi-level fusion module, which is recorded as F 3.

4. The multi-level fusion building change detection method based on CNN and Transformer according to claim 3 is characterized in that: The specific steps of Step 5 are: Step 5.

1. F 3. Input the multi-layer perceptron MLP and the data-related upsampling module in sequence to obtain the features of the original image size; And generate the change probability map through the SoftMax layer, and record the probability map as p ; Step 5.2: Use the boundary area obtained in Step 4 to obtain the weight map in the loss function w , used to enhance the boundary, the formula is as follows: (5) in is an experimentally derived hyperparameter, represents the boundary area obtained in Step 4, Represents pixels i Belongs to the border area; Step 5.3, based on the reference image of the change y , change probability map p Calculate the weighted cross entropy loss function with the weights solved in Step 5.2 Loss : (6) In the formula Indicates image, H and W is the height and width of the image, p i Represents pixels i The probability of change, y i Represents pixels i The category label, y i = 1 for pixels i Belongs to the change category, y i = 0 for pixels i Belongs to the unchanged category, w i Represents pixels i The weight of .

Images