Strong supervision change detection method based on convolutional neural network and visual attention model

Abstract

The invention relates to a strong supervision change detection method based on a convolutional neural network and a visual attention model, and belongs to the field of remote sensing and computer vision. A pair of remote sensing paste images at different moments is read into the model, the model comprises a CNNTTransConv module, a mixed double-attention module and a multi-scale strong supervision module, the model is optimized through a loss function, the remote sensing images at different moments pass through the optimized model, a change graph is obtained, and a change detection task is achieved. The system has the advantages that the feature extraction module improves the expression ability of global features; the hybrid double attention module enhances the ability of the network to obtain more obvious change information, and the multi-scale strong supervision module supervises output features of different scales without introducing additional parameters, thereby further enhancing the ability of the network to distinguish change features. The method realizes the most advanced performance, and can be widely applied to remote sensing change detection tasks of various scenes.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing and computer vision, and in particular refers to a strongly supervised change detection method based on a convolutional neural network (CNN) and a visual attention model (Transformer). Background Art

[0002] Change detection can obtain semantic change maps by locating and identifying pixel-level changes in dual-temporal remote sensing images. In recent years, using change detection technology to analyze building and land cover changes has become a new trend and plays an important role in damage assessment, environmental protection, and land planning. The implementation process of change detection can be expressed by formula (1).

[0003] P(x,y)=G[F(I1(x,y),I2(x,y))] (1)

[0004] Among them, (x, y) represents the position index in the pixel space, and is the input bi-phase image. C, H and W represent the channel, height and width of the image respectively. F(·) is the change feature extractor, and G(·) is the classifier that maps the change feature to the final change result. By thresholding the detected change amplitude map P, the final binary change detection map can be obtained.

[0005] In the early stages of change detection, techniques such as image difference, image ratio, and regression analysis are often used to extract change information. These methods mainly rely on the spectral differences of individual pixels, usually ignoring specific categories and only identifying the locations of changed or unchanged areas. In order to utilize more image information, some methods based on change vector analysis, principal component analysis, and slow feature analysis have been proposed. Although these methods are simple in principle and computationally efficient, they do not consider the semantic relevance of surrounding pixels, making them very sensitive to noise. With the development of machine learning, change detection algorithms based on support vector machines, random forests, and artificial neural networks have significantly improved their accuracy. However, the limitation of these methods is that they manually design features, which restricts their application in practical scenarios.

[0006] With the significant progress of deep learning technology in other fields such as image processing and natural language processing, more and more researchers have devoted themselves to the study of change detection based on convolutional neural networks. Compared with traditional algorithms, these change detection methods can generate high-quality change maps and show superior noise resistance. Change detection methods based on fully convolutional networks have obtained satisfactory change detection maps because they can independently and efficiently predict the label of each pixel. Among them, Long et al. proposed a fully convolutional network in 2015. This network performs well in semantic segmentation tasks because it can independently and efficiently predict the label of each pixel and has been widely used in the field of change detection. Subsequently, Varghese et al. proposed a change detection network that combines a twin network with a fully convolutional network, which can effectively detect changes under different lighting conditions, different viewing angles, and seasonal changes. Daudt et al. used three end-to-end fully convolutional networks to handle change detection tasks, and designed an encoder-decoder network with a symmetrical structure, while using a skip connection mechanism to improve accuracy. In addition, Daudt et al. constructed a change detection dataset for training a newly proposed network that utilizes two different convolutional neural networks to detect changes in image pairs and performs well in a fully supervised manner.

[0007] Inspired by generative adversarial networks, researchers applied generative adversarial networks to change detection tasks, which regarded the connection process between dual-time images and change images as a generative learning process, thus achieving significant performance improvements. Among them, Gong et al. proposed an unsupervised change detection method using generative adversarial networks, which first used generative adversarial networks to generate simulated remote sensing images and enhance training samples. Then, the mapping function between the training samples and the corresponding image regions was constructed through continuous adversarial learning to improve the change map. Hou et al. proposed a dual-branch change detection model based on GAN network. This dual-branch structure can distinguish images in the feature domain, thereby alleviating the loss of important information. In addition, the model uses generative adversarial networks to adaptively model the target distribution and improve performance through image generation. Niu et al. used conditional generative adversarial networks to model the conversion process between synthetic aperture radar images and optical images, and designed an approximate network to minimize the pixel differences in synthetic aperture radar images. The network promotes the update of alternating feature information, making direct comparison of the final change map possible. Although the above methods can produce better change detection results, their global feature extraction capabilities are limited. In addition, they do not adequately consider spatial context information and internal relationships between high-level and low-level features.

[0008] With the emergence of UNet networks, researchers have increasingly applied them to change detection tasks. Among them, Chen et al. introduced attention mechanisms and jump connection mechanisms in UNet-based networks, which not only solved the problem of sample data imbalance to a certain extent, but also enhanced the network's anti-interference ability and overall robustness to noise, and improved the performance of change detection. Peng et al. proposed an enhanced U-Net++ network based on an encoder-decoder architecture. The network extracts global and fine-grained information from the input image to generate high-precision feature maps. Subsequently, it uses a fusion strategy to integrate change maps from different semantic layers, and finally achieves excellent performance in change detection tasks. To further improve network performance, Ghaffarian et al. also proposed an extended U-Net network that combines a deep residual network and uses conditional random fields to update high-resolution post-disaster building images. In order to obtain more accurate change maps, researchers have also introduced various strategies. For example, the network based on the feature fusion strategy improves the accuracy and robustness of the change detection task through weighted fusion, feature cascading, and feature stacking. The method using the attention mechanism focuses on the key information of features (changed and unchanged) to detect changes more accurately.

[0009] Since convolutional neural networks can extract powerful change features, the above algorithms have achieved some success in change detection research. However, there are still some potential problems: (1) Obtaining accurate change maps requires mining relevant contextual information, but the limited receptive field mainly focuses on small change areas, which limits the network performance; (2) Most existing methods adopt various strategies (e.g., skip connections, step-by-step fusion) to enhance network performance, but ignore the mining and coordination of spatiotemporal relationships between multi-scale features, which hinders the extraction of accurate change areas; (3) Some scale reduction operations will lead to the loss of fine-grained information, thus limiting network performance.

[0010] Therefore, in order to improve the context modeling ability and expand the receptive field, the algorithm based on the visual attention model Transformer was applied to the change detection task and achieved satisfactory results. Chen et al. introduced Transformer into the change detection task to refine shallow features, enhance original features, and generate more accurate predictions. Zhang et al. proposed SwinSUNet to extract long-term global information in time and space, thereby improving the extraction of change information. Wu et al. applied CrossSwin Transformer to enhance image communication and improve the accuracy of change maps. Although these methods effectively present multi-scale long-distance details, they still face limitations in modeling capabilities. Therefore, how to improve the limitations of limited network receptive field and weak modeling capabilities, improve feature expression capabilities, compensate for potential information loss and obtain significant change information are issues that need to be seriously considered by the present invention, and are also challenges faced by change detection tasks. Summary of the invention

[0011] The present invention provides a strongly supervised change detection method based on convolutional neural network and visual attention model to solve the problems of limited receptive field, weak feature extraction ability and loss of change information of most current networks, aiming to obtain a more refined change map.

[0012] The technical solution adopted by the present invention comprises the following steps:

[0013] (1) Read a pair of remote sensing images I1 and I2 at different times into the model;

[0014] (2) Use 3×3 convolution to obtain the shallow features x1 and x2 of input images I1 and I2 respectively;

[0015] (3) In the network coding stage, the CNN_TransConv module is first used to process the shallow features x1 and x2 in the twin upper branch coding stage and the twin lower branch coding stage respectively to obtain rich information at scales of 1 / 2, 1 / 4, 1 / 8 and 1 / 16; then, the hybrid dual attention module is used to further refine the different scale features of the upper and lower branches to help the model obtain more delicate change features;

[0016] (4) In the network decoding stage, firstly, the features of different scales from the encoder and decoder stages are concatenated, and the CNN_TransConv module is used to further improve the expression of the features. Finally, the multi-scale strong supervision module is used to supervise the output features of different scales in the decoding stage.

[0017] (5) Use 3×3 convolution to obtain the final output change map;

[0018] (6) Optimize the model through the network total loss function;

[0019] Change detection based on remote sensing images is a pixel-level classification task. In the process of change map generation, binary cross entropy loss is used to optimize the network. The loss function of change detection is defined as follows:

[0020]

[0021] Among them, U represents the total number of training samples, Y i represents the true value of the i-th sample, and EICTNet(.) represents the proposed network model;

[0022] The total network loss can be defined as the following formula;

[0023] l total = l CD +ll MSM (3)

[0024]

[0025] Among them, λ is the weight parameter of the multi-scale strong supervision module, which is set to 0.05;

[0026] (7) For the input image pair I1 and I2, a more refined change map is obtained through the model optimized by step (6), thereby achieving a refined change detection task.

[0027] The CNN_TransConv module in step (3) of the present invention is a feature extractor CNN_TransConv Block (CTCB) based on CNN and Transformer, and CTCB is divided into CNN_Branch branch, Transformer_Branch branch and 3D-Relations_Branch branch;

[0028] 1) Input features of size W×H×C, where C is the number of feature channels, and then use 1×1 convolution to change its number of channels to obtain a feature f of W×H×C / 2, which will be used as the input of the CNN_Branch and Transformer_Branch branches respectively;

[0029] 2) In the CNN_Branch branch, a cyclic multi-scale convolution is used to capture rich information from different receptive fields (1, 3, 6). This method uses different expansion rates to process the convolution kernel in the same layer, which can effectively capture multi-scale features without introducing additional parameters or computational costs. Subsequently, the output feature f of the CNN_Branch branch containing rich local information can be obtained through the splicing operation. c ;

[0030] 3) In the Transformer_Branch branch, the entire image is first segmented, and the semantic tokenizer is used to obtain the token T∈R L×C , the analyzer aggregates the feature map space into a set of tags by learning a set of spatial attention maps, where L represents the word length. Specifically, for each pixel on the feature map f, point-wise convolution is used to obtain L word tuples, each word tuple represents a semantic concept, and then the spatial attention map A∈R is calculated by the softmax function. HW×L , H and W represent the dimensions of each word tuple. Finally, the attention map A is used to calculate the weighted average sum of the pixels in f to obtain a semantic word tuple T of size L. This process can be expressed by the following formula:

[0031] T=A T f=(Softmax(f·W)) T f (5)

[0032] Where, · represents the input feature f and the learnable convolution kernel W∈R C×L The Softmax(·) function can standardize each word tuple to obtain the attention map A. The word tuple T is calculated by multiplying A and f.

[0033] Then, the word is input into the Transformer encoder to model the spatiotemporal global semantic relationship and enhance the word representation. The Transformer encoder consists of Multihead Self-Attention (MSA) and MultiLayer Perceptron (MLP). In this module, the word is first embedded into the layer normalization layer LN and MSA. Subsequently, it also uses residual connection to minimize information loss. The above process is modeled by formula (6):

[0034] T'=MSA(LN(T))+T (6)

[0035] Among them, MSA contains 8 parallel non-shared weight heads, in each head, the input word T will be converted into a triplet based on the attention mechanism (query Q en , key K en ,value V en ), can be expressed as follows:

[0036]

[0037] in, and is a learnable parameter in the encoder, and d represents Qen , K en and V en Then, an attention head is applied to model the relationship between triplets:

[0038]

[0039] Then, by concatenating the results of all attention heads, the output of the multi-head self-attention MSA is obtained. In order to further encode the word embedding, a multi-layer perceptron MLP is also applied, which includes two linear transformation layers and a Gaussian error linear unit GeLU;

[0040] In order to enhance the global information, a Transformer decoder containing multiple cross attentions (CA) is used to learn the correlation between the input feature f and the word unit T'. Different from the MSA in the encoder, its query comes from the image feature f, and the key and value are linearly transformed to the word unit T' obtained by the encoder, which can be expressed as follows:

[0041]

[0042] Finally, the final output f of the Transformer_Branch branch is obtained by using the same operations as the encoder, including attention mechanism, element-wise addition, and multi-layer perceptron. t ;

[0043] 4) In the 3D-Relations_Branch branch, in order to obtain richer feature representation, the spatial-spectral dependency is modeled through the 3D attention strategy. First, the output features of the two branches are processed separately using pseudo 3D convolution, that is, 1×3×3 and 3×1×1 convolution kernels are used to capture the correlation between space and channels respectively. Then, the outputs of the two branches are fused by element-by-element addition to obtain the output f p , and use ReLU and Softmax operations to obtain the attention weight of the 3D-Relations_Branch branch. Finally, it is multiplied with the fusion output of the two branches by element-wise multiplication to obtain rich information f that jointly considers channel and spatial information. r , this process can be expressed by the following formula:

[0044] f r =f p +α·β(f p )⊙f p (10)

[0045] Among them, α(·) represents the Sigmoid function, β(·) represents the ReLU activation function, and ⊙ represents the element-wise product. By fusing the original input f with the output f of the 3D-Relation_Branch branch, r , and the final output f of the CTCB module is obtained out , which contains rich global and local information at the same time, and also uses the attention mechanism to fully model the relationship between the two, thereby improving the feature representation of the model.

[0046] The hybrid dual attention module in step (3) of the present invention: In order to better perceive the contextual information of the dual-time remote sensing image, this module exchanges space and channels between the dual-branch features. It first generates an exchange mask, and then retains and exchanges space and channel features based on the mask. The process can be expressed as the following formula:

[0047]

[0048] Among them, Mask refers to the exchange mask composed of the change mask 1 and the non-change mask 0;

[0049] Through the spatial-spectral exchange between the dual time domain features, the feature distributions of the two branches will be more similar, which helps to achieve adaptation between the dual time domains. Subsequently, in order to improve the ability of single-branch features to perceive contextual information, this module also introduces a lightweight attention mechanism CS Attention for single-branch feature space and channel exchange. In this dual attention branch, a deep convolution with a kernel size of 1×1 is used to change the number of channels and reduce the amount of calculation. Finally, by element-by-element addition, it can obtain fine pixel information from the channel and spatial attention branches.

[0050] In step (4) of the present invention, the multi-scale strong supervision module adds additional supervision on the change graphs of different scales, thereby improving network performance. The multi-scale strong supervision module first implements two parts of different scale image information collection:

[0051] 1) In the decoder stage, three convolution kernels of size 3×3 are used to obtain the output change map of the intermediate layer, and the decoded change maps with scales of 1 / 8, 1 / 4, and 1 / 2 are obtained respectively;

[0052] 2) Use two convolutional layers and ReLU operations to process the real image to obtain different scale change maps corresponding to the decoder level;

[0053] For a decoder stage with a total of m intermediate layers, the following change diagram can be obtained:

[0054]

[0055] Among them, conv(·) represents the convolution operation, Represents the output features at the mth decoding stage. For the real image y, the change graph can be obtained as follows:

[0056] I m '=F m (y) (13)

[0057] Where F(·) is the function used to process the real image, which first applies a convolutional layer for downsampling, then uses a ReLU operation, and finally uses another convolutional layer to obtain the changed image;

[0058] By designing a multi-scale strong supervision module, the network can improve the feature learning ability of the intermediate layer and obtain more discriminative change features. Therefore, the auxiliary objective function of the intermediate layer can be defined as the following formula:

[0059]

[0060] Where N represents the size of the image, i,j∈[0,N), and They represent the pixel values ​​of the change map at the mth decoding stage and the real image at point (i, j), respectively. The loss of the multi-scale supervision strong module measures their alignment in the pixel space by calculating the mean square error (MSE) between the decoder stage and the real image.

[0061] This paper constructs a strongly supervised change detection network based on CNN and Transformer. First, a plug-and-play feature extraction module CNN_TransConv is designed, which uses lightweight convolution to capture local features and uses the global feature representation capability of Transformer to extract detailed information; secondly, a hybrid dual attention module containing exchange operation and CS attention mechanism is designed, which enhances the network's ability to capture significant change information through spatiotemporal information exchange; finally, a multi-scale strong supervision module is proposed to strongly supervise the output features of various scales in the decoding stage, further improving the network's ability to distinguish change features. A large number of experimental results show that this method can obtain more refined change maps and outperform existing representative advanced change detection methods.

[0062] The beneficial effects of the present invention are as follows: (1) The present invention is an accurate change detection method for remote sensing images; (2) The present invention designs a CNN_TransConv module (CTCB) to enhance the feature extraction capability. The module utilizes the powerful local feature extraction capability of CNN on the one hand, and the self-attention mechanism of Transformer on the other hand. Therefore, each pixel can access global information, enabling the network to obtain more detailed information in all directions; a hybrid dual attention module is proposed to promote the exchange of dual-temporal features in spatial and channel dimensions, which not only enhances the expressive power of features, but also improves network performance without the need for additional parameters or computational operations; a multi-scale strong supervision module is designed to further improve network performance and enhance constraints on deep features by introducing additional supervision on output features at different scales. The present invention conducts a large number of experimental comparisons and analyses on three public datasets, LEVIR-CD, CDD and DSIFN-CD, and evaluates the effectiveness and superiority of the proposed method from both qualitative and quantitative perspectives. A large number of comparative experimental results show that the method proposed in the present invention can obtain the highest F1 score and a more refined change map, achieving a more significant change detection task; (3) The present invention can be widely used in remote sensing image change detection tasks in various scenarios, which helps to promote the development of change detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a pair of remote sensing images at different times on the LEVIR-CD dataset;

[0064] Figure 2 It is the overall framework diagram of the network of the present invention;

[0065] Figure 3 This is the structure diagram of the CNN_TransConv module;

[0066] Figure 4 This is the structure diagram of the hybrid dual attention module;

[0067] Figure 5 This is a visual comparison of the change maps obtained with other methods on the LEVIR-CD dataset;

[0068] Figure 6 It is a visual comparison of the change graphs obtained with other methods on the CDD dataset;

[0069] Figure 7 It is a visual comparison of the change graphs obtained with other methods on the DSIFN-CD dataset;

[0070] Figure 8 is a visual comparison of different ablation settings on three datasets;

[0071] Fig. 9 It is a visual comparison of different ablation settings in the CTCB module on three datasets;

[0072] Fig.10 Figure 2 is a visual comparison of different ablation settings in the HDAB module on three datasets. DETAILED DESCRIPTION

[0073] The following steps are involved:

[0074] (1) Read a pair of remote sensing images I1 and I2 at different times into the model, see Figure 1 ;

[0075] (2) Use 3×3 convolution to obtain the shallow features x1 and x2 of input images I1 and I2 respectively,

[0076] (3) In the network coding stage, the CNN_TransConv module is first used to process the shallow features x1 and x2 in the twin upper branch coding stage and the twin lower branch coding stage respectively to obtain rich information at scales of 1 / 2, 1 / 4, 1 / 8 and 1 / 16; then, the hybrid dual attention module is used to further refine the different scale features of the upper and lower branches to help the model obtain more delicate change features;

[0077] The encoding stage includes three stages in total: the twin upper branch encoding stage, the twin lower branch encoding stage and the encoding change feature stage, wherein the twin upper branch encoding stage uses the CNN_TransConv module to obtain rich information of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 scales of the I1 image. This CNN_TransConv module is a new plug-and-play feature extraction module, which combines lightweight convolution with Transformer, not only uses CNN to improve the expression ability of local features, but also applies the self-attention mechanism of Transformer, so that each pixel can obtain global information, thereby improving the expression ability of global features; the twin lower branch encoding stage uses the same operation as the twin upper branch encoding stage to obtain rich information of 1 / 2, 1 / 4, 1 / 8 and 1 / 16 scales of the I2 image; the encoding change feature stage uses a hybrid dual attention module to help the model obtain more obvious change information, that is, firstly use the spatiotemporal conversion operation to realize information exchange within the scale, and then apply the parallel attention mechanism to obtain more detailed change features;

[0078] Among them, the CNN_TransConv module designs a parallel structure of Transformer and deep convolution, including Transformer_Branch branch, CNN_Branch branch and 3D-Relation_Branch branch. Among them, the Transformer_Branch branch represents the features as a small number of tokens, which uses the encoder to model the context, then feeds it back to the pixel space, and uses the decoder to refine the original features. The CNN_Branch branch expands different expansion rates to the corresponding convolution kernels of the same convolution layer through cyclic multi-scale convolution, so it can flexibly capture multi-scale change features. The 3D-Relation_Branch branch further enriches the feature representation by using the spatial-spectral dependency. This multi-branch feature extraction module based on CNN and Transformer can not only produce deeper feature representations, but also model cross-window connections to expand the receptive field. The hybrid dual attention module is designed to compensate for potential information loss and obtain significant change information. In the encoding stage, the module first uses the spatiotemporal conversion operation to realize the information exchange of spatial features and channel features without adding additional parameters and calculations, and adaptively performs domain adaptation between dual-phase domains. Subsequently, it adopts a parallel attention mechanism to obtain more refined output features;

[0079] (4) In the network decoding stage, firstly, the different scale change features from the encoder and decoder stages are concatenated, and the CNN_TransConv module is used to further improve the feature expression. Finally, the multi-scale strong supervision module is used to supervise the different scale output features of the decoding stage. This multi-scale strong supervision module not only learns the mapping of the change map through forward learning, but also learns additional dual regression maps through reverse supervision in the decoding stage, thereby supervising the output features of different scales, forming a closed loop for additional strong supervision.

[0080] In the decoding change feature stage, in order to compensate for the possible loss of information, the model also splices the different scale change features from the encoder and decoder stages, and uses the CNN_TransConv module to further improve the expression of features; in order to further improve the network performance and enhance the recovery of information in the decoding stage, the model uses a multi-scale strong supervision module to supervise the different scale output features of the decoding stage. It is worth noting that this module does not introduce additional parameters and can further improve the network's ability to distinguish change features;

[0081] (5) Use 3×3 convolution to get the final output change map, see Figure 2 ;

[0082] (6) Optimize the model through the network total loss function;

[0083] Change detection based on remote sensing images is a pixel-level classification task. In the process of change map generation, binary cross entropy loss is used to optimize the network. The loss function of change detection is defined as follows:

[0084]

[0085] Among them, U represents the total number of training samples, Y i represents the true value of the i-th sample, and EICTNet(.) represents the proposed network model;

[0086] The total network loss can be defined as the following formula;

[0087] l total = l CD +ll MSM (3)

[0088]

[0089] Among them, λ is the weight parameter of the multi-scale strong supervision module, which is set to 0.05;

[0090] (7) For the input image pair I1 and I2, a more refined change map is obtained through the model optimized by step (6), thereby achieving a refined change detection task.

[0091] In the step (3), the CNN_TransConv module: convolution can extract local detail information, and the Transformer can obtain global semantic information. In order to improve the feature expression in the representation learning process, the present invention proposes a feature extractor CNN_TransConv Block (CTCB) based on CNN and Transformer. The network structure is as follows: Figure 3 As shown in the figure, CTCB is divided into CNN_Branch branch, Transformer_Branch branch and 3D-Relations_Branch branch;

[0092] 1) Input features of size W×H×C, where C is the number of feature channels, and then use 1×1 convolution to change its number of channels to obtain a feature f of W×H×C / 2, which will be used as the input of the CNN_Branch and Transformer_Branch branches respectively;

[0093] 2) In the CNN_Branch branch, a cyclic multi-scale convolution is used to capture rich information from different receptive fields (1, 3, 6). This method uses different expansion rates to process the convolution kernel in the same layer, which can effectively capture multi-scale features without introducing additional parameters or computational costs. Subsequently, the output feature f of the CNN_Branch branch containing rich local information can be obtained through the splicing operation. c ;

[0094] 3) In the Transformer_Branch branch, the entire image is first segmented, and the semantic tokenizer is used to obtain the token T∈R L×C , the analyzer aggregates the feature map space into a set of tags by learning a set of spatial attention maps, where L represents the word length. Specifically, for each pixel on the feature map f, point-wise convolution is used to obtain L word tuples, each word tuple represents a semantic concept, and then the spatial attention map A∈R is calculated by the softmax function. HW×L , H and W represent the dimensions of each word tuple. Finally, the attention map A is used to calculate the weighted average sum of the pixels in f to obtain a semantic word tuple T of size L. This process can be expressed by the following formula:

[0095] T=A T f=(Softmax(f·W)) T f (5)

[0096] Where, · represents the input feature f and the learnable convolution kernel W∈R C×L The Softmax(·) function can standardize each word tuple to obtain the attention map A. The word tuple T is calculated by multiplying A and f.

[0097] Then, the word is input into the Transformer encoder to model the spatiotemporal global semantic relationship and enhance the word representation. The Transformer encoder consists of Multihead Self-Attention (MSA) and MultiLayer Perceptron (MLP). In this module, the word is first embedded into the layer normalization layer LN and MSA. Subsequently, it also uses residual connection to minimize information loss. The above process is modeled by formula (6):

[0098] T'=MSA(LN(T))+T (6)

[0099] Among them, MSA contains 8 parallel non-shared weight heads. In each head, the input word T will be converted into a triple (query Q en , key K en ,value V en ), can be expressed as follows:

[0100]

[0101] in, and is a learnable parameter in the encoder, and d represents Q en , K en and V en The dimension of is set to 64. Then, an attention head is applied to model the relationship between triples:

[0102]

[0103] Then, the present invention obtains the output of multi-head self-attention (MSA) by concatenating the results of all attention heads. In order to further encode word embedding, a multi-layer perceptron (MLP) is also applied, which includes two linear transformation layers and a Gaussian error linear unit (GeLU);

[0104] In order to enhance the global information, the present invention uses a Transformer decoder containing multiple cross attentions (CA) to learn the correlation between the input feature f and the word unit T'. Different from the MSA in the encoder, its query comes from the image feature f, and the key and value perform a linear transformation on the token T' obtained by the encoder, which can be expressed as follows:

[0105]

[0106] Finally, by using the same operations as the encoder (attention mechanism, element-by-element addition, and multi-layer perceptron), the present invention can obtain the final output f of the Transformer_Branch branch t ;

[0107] 4) In the 3D-Relations_Branch branch, in order to obtain richer feature representation, the present invention models the spatial-spectral dependency through a 3D attention strategy. First, it uses pseudo-3D convolution to process the output features of the two branches respectively, that is, it uses 1×3×3 and 3×1×1 convolution kernels to capture the correlation between space and channels respectively. Then, the outputs of the two branches are fused by element-by-element addition to obtain the output f p, and use ReLU and Softmax operations to obtain the attention weight of the 3D-Relations_Branch branch. Finally, it is multiplied with the fusion output of the two branches by element-wise multiplication to obtain rich information f that jointly considers channel and spatial information. r . This process can be expressed by the following formula:

[0108] f r =f p +α·β(f p )⊙f p (10)

[0109] Among them, α(·) represents the Sigmoid function, β(·) represents the ReLU activation function, and ⊙ represents the element-wise product. By fusing the original input f with the output f of the 3D-Relation_Branch branch r The present invention obtains the final output f of the CTCB module out , which contains rich global and local information at the same time, and also uses the attention mechanism to fully model the relationship between the two, thereby improving the feature representation of the model.

[0110] The hybrid dual attention module in step (3): Similar to most works, the present invention also obtains the variation features of different scales in the encoder stage, and its module structure is as follows Figure 4 As shown in Figure 2, in order to better perceive the contextual information of dual-time remote sensing images, first, this module exchanges space and channels between dual-branch features. Specifically, it first generates an exchange mask, and then retains and exchanges space and channel features based on the mask. The process can be expressed as the following formula:

[0111]

[0112] Among them, Mask refers to the exchange mask composed of the change mask 1 and the non-change mask 0;

[0113] Through the spatial-spectral exchange between the dual-time domain features, the feature distributions of the two branches will be more similar, which helps to achieve adaptation between the dual time domains. Subsequently, in order to improve the ability of single-branch features to perceive contextual information, this module also introduces a lightweight attention mechanism (C-SAttention) for single-branch feature space and channel exchange. In this dual attention branch, a deep convolution with a kernel size of 1×1 is used to change the number of channels and reduce the amount of calculation. Finally, by element-by-element addition, it can obtain fine pixel information from the channel and spatial attention branches.

[0114] The multi-scale strong supervision module in step (4): Most current methods only supervise the final output of the network and lack constraints on intermediate features, which results in the model being unable to learn more effective change information, thereby affecting network performance. Unlike these methods, the present invention introduces a multi-scale strong supervision module in the decoder, which can add additional supervision to the change graphs of different scales, thereby improving network performance. Specifically, the multi-scale strong supervision module first implements two parts of image information acquisition at different scales:

[0115] 1) In the decoder stage, three convolution kernels of size 3×3 are used to obtain the output change map of the intermediate layer, and the decoded change maps with scales of 1 / 8, 1 / 4, and 1 / 2 are obtained respectively;

[0116] 2) Use two convolutional layers and ReLU operations to process the real image to obtain the change maps of different scales corresponding to the decoder stage. It is worth noting that this module only uses simple convolution operations to process the real image compared to the operation of generating the change map. This is because the present invention only uses the change map obtained from the real image to constrain the decoder stage. Therefore, there is no need to design a complex structure, which is much lower than the computational cost of the original model.

[0117] More specifically, for a decoder stage with a total of m intermediate layers (set to 3), the following change graph can be obtained:

[0118]

[0119] Among them, conv(·) represents the convolution operation, Represents the output features at the mth decoding stage. For the real image y, the change graph can be obtained as follows:

[0120] I m '=F m (y) (13)

[0121] Among them, F(.) is the function used to process the real image. It first applies a convolutional layer for downsampling, then uses a ReLU operation, and finally uses another convolutional layer to obtain the transformed image.

[0122] By designing a multi-scale strong supervision module, the network can improve the feature learning ability of the intermediate layer and obtain more discriminative change features. Therefore, the auxiliary objective function of the intermediate layer can be defined as the following formula:

[0123]

[0124] Where N represents the size of the image, i,j∈[0,N), and They represent the pixel values ​​of the change map at the mth decoding stage and the real image at point (i, j), respectively. The loss of the multi-scale supervision strong module measures their alignment in the pixel space by calculating the mean square error (MSE) between the decoder stage and the real image.

[0125] The beneficial effects of the present invention are further illustrated below through analysis and comparison of specific experimental results.

[0126] Experimental examples In order to effectively and systematically evaluate the proposed method, experimental comparisons and analyses were conducted on three public datasets: LEVIR-CD, CDD, and DSIFN-CD. The LEVIR-CD dataset contains 637 pairs of remote sensing images, each with a resolution of 0.5 meters and a resolution of 1024×1024 pixels, capturing a series of complex change features. In order to optimize GPU memory usage and prevent overfitting, the images are cropped into 13072 blocks, each with a size of 256×256 pixels. Among them, 10,000 blocks are used for training, 1024 blocks are used for validation, and 2048 blocks are used for testing. The CDD dataset contains 11 pairs of multispectral images from Google Earth, which capture seasonal changes in the same area with resolutions ranging from 0.03 to 1 meter. Through preprocessing, 16,000 blocks of 256×256 pixels can be extracted from the original image. Finally, 10,000 blocks are used for training, and 3,000 blocks are used for validation and testing, respectively. The DSIFN-CD dataset contains six high-resolution images collected from multiple cities in China. These images are cropped into 512×512 patches, of which 48 are used for testing, 3600 and 340 patches are used for training and validation, respectively.

[0127] In the specific implementation process, the present invention is implemented using PyTorch and trained on an NVIDIA GeForce RTXA6000 GPU. The Adam optimizer is used, and the decay rate is set to 0.99. The model is trained for 1000 cycles with a batch size of 32. The initial learning rate is set to 1e-4 and the weight decay is 0.0005. In order to evaluate the performance of the method, the present invention uses F1 score, precision, recall, and Iou for quantitative evaluation. The quantitative results and visual comparison are shown in Tables 1 and 2, respectively. Figure 5 middle.

[0128] Table 1 Performance comparison with other advanced methods on the LEVIR-CD dataset

[0129]

[0130]

[0131] As can be seen from Table 1, first, most CNN-based methods can obtain the change area of ​​dual-phase images through UNet and twin structures, which promotes the development of change detection tasks. Among them, the classic method FC-EF can effectively process change features by designing a simple network structure and skip connection mechanism. Other methods, such as DASNet, DDCNN, SNUNet, TSNet, CANet and D-TNet, respectively design different attention modules to capture long-range dependencies, obtain more diverse feature representations, and enhance contextual information. In order to improve the ability to distinguish change features, methods such as DMMSTNet, IFN and FDFF-Net also propose different feature fusion methods. In addition, some works also introduce detail semantic enhancement modules to improve the feature learning ability of the network and the quality of change features. Although the above methods improve network performance by introducing different mechanisms, they still have the limitation of not being able to fully capture the rich semantic features of the input image; secondly, Transformer-based methods, including SiamixFormer, Bit, SFGT-CD, LSAT, EATDeer, and MSSCTAS, use powerful global information extraction capabilities to model contextual information, refine original features, and enhance the saliency of change regions, thereby achieving a more accurate representation of change features through improvements in the Transformer architecture. Although these methods perform well in capturing detailed global information, local information is still crucial for identifying detailed change regions. Finally, recent studies combining CNN and Transformer methods, such as MCTNet and MM-Trans, can enrich the details of change regions by integrating the global features of Transformer and the local features of CNN, further promoting the development of change detection tasks. Despite the success of these methods, the method proposed in the present invention is still superior to other state-of-the-art methods. The superiority of the present invention can be attributed to the following reasons.

[0132] (1) The present invention designs a CNN_TransConv module, which has a dual-branch structure that can extract global and local information simultaneously. Unlike previous methods that only rely on CNN to obtain local information or Transformer to obtain global information, this module uniquely integrates the two in one framework. It is worth noting that the CNN_TransConv module is essentially different from the CNN-Transformer based method because it captures global and local information simultaneously in one module. This enhanced feature extraction capability significantly improves the overall performance of the present invention.

[0133] (2) The above methods usually ignore the interaction of dual-temporal features in spatial and temporal dimensions. In contrast, the present invention introduces a hybrid dual attention module to facilitate information exchange between these dimensions and adopts a parallel dual attention mechanism to enhance feature performance. Therefore, the method improves network performance without adding additional parameters and computational operations.

[0134] (3) The multi-scale strong supervision module provides additional supervision for output features of different scales, which is crucial for capturing subtle changes in features. In summary, the design of these modules enables the present invention to surpass existing methods.

[0135] Visual comparison on the LEVR-CD dataset is shown in Figure 5 As shown. For a more intuitive comparison and analysis, the present invention uses white, black, red and green to represent true positive examples, true negative examples, false positive examples and false negative examples, respectively. The following points can be observed from the figure: First, the present invention uses the feature extraction modules of CNN and Transformer to capture the local and global information of the input image, effectively avoiding false negative examples; secondly, the hybrid dual attention module exchanges spatial and channel information to enhance the variation features, thereby reducing false positive examples; finally, due to the introduction of a new supervision method, the present invention can present more complete results.

[0136] Table 2 Performance comparison with other advanced methods on the CDD dataset

[0137]

[0138]

[0139] From the experimental results in Table 2 and Figure 6 The advantages of the present invention can be clearly seen in the visual comparison results. That is, due to the introduction of the CNN_TransConv module, the hybrid dual attention module and the multi-scale strong supervision module for output features of different scales, the present invention can capture more detailed change information from the dual-phase images and obtain a better F1 score than other advanced methods.

[0140] Table 3 Performance comparison with other advanced methods on the DSIFN-CD dataset

[0141]

[0142] From the experimental results in Table 3 and Figure 7The advantages of the present invention can be further found in the visual comparison results. It is worth noting that, similar to the experimental results on the LEVIR-CD and CDD datasets, the present invention achieves 88.49%, 91.21% and 89.83% in precision, recall and F1 score, respectively, surpassing other advanced methods. Visual comparison also shows that the change map generated by the present invention is more accurate, showing the lowest false negative and false positive probabilities. These findings demonstrate its superior performance in change detection tasks.

[0143] To verify the effectiveness of each module in the model proposed by the present invention, firstly, the CTCB module is directly replaced with the traditional convolution operation in the network, which means that the present invention only captures local information instead of extracting rich local-global information. Secondly, the present invention removes the HDAB module from the network to ignore the information interaction from different spatial and temporal dimensions. Finally, the present invention abandons the multi-scale strong supervision module MSM in the model, which means that the present invention does not supervise the changing features in the intermediate layer.

[0144] Through the above settings, the present invention can obtain three new network structures: a model without CTCB, a model without HDAB, and a model without MSM. In order to ensure the fairness of performance comparison, the present invention uses the same parameter settings to train the above networks. Table 4 shows the results of ablation experiments on three data sets. The experimental results show that the CTCB, HDAB, and MSM modules are all crucial to the present invention. First, without the CTCB module, it is impossible to obtain delicate information about the change area. On the one hand, the rich information of the change area will be reflected by global and local information. Therefore, if the present invention ignores global information and only applies CNN to focus on local change information, the network will perform poorly due to the lack of visual semantic expression of change features in pixel space; on the other hand, if the expression of multi-scale local information is ignored, the ability of the network to model contextual semantic information will be limited, which is very important for obtaining accurate change maps. Secondly, removing the HDAB module will also affect the improvement of model performance. This is because the HDAB module can establish connections between different spatial and temporal features, thereby achieving better information fusion and interaction. Finally, we can see that the lack of constraints on intermediate features by the MSM module also has an adverse effect on the output results of the model. This demonstrates the importance of constraining multi-scale variation features in the intermediate layers to improve the performance of change detection tasks. Visual comparison results of different ablation settings are shown in Figure 8 .

[0145] Table 4 Comparison of different ablation experiments on three datasets

[0146]

[0147] In addition, the present invention evaluates the effectiveness of the Multiscale Atrous CNN in the CTCB module. Table 5 and Fig. 9 This proves the advantages of multi-scale hole convolutional neural networks. From Table 5 and Fig. 9 The results clearly show that using a traditional convolutional neural network instead of a multi-scale hole convolutional neural network will reduce the F1 accuracy on the three datasets by 1.44%, 1.13%, and 2.71%, respectively. This shows that in the CNN branch, the use of a multi-scale hole CNN can obtain rich information from different receptive fields and effectively capture multi-scale features, which is particularly important for extracting subtle changes in features.

[0148] Table 5 Comparison of different ablations of CTCB module on three datasets

[0149]

[0150] Finally, the present invention also tests the rationality of each component in the HDAB module. First, the present invention does not directly perform spatial and channel exchange on the dual-time features in the network, which means that it no longer implements dual-time domain adaptation, which will result in the model being unable to perceive rich change information. Second, the present invention will no longer use the CS attention component, which means that it will not perform any processing on the dual-time features, which will result in the network being unable to obtain more detailed change information. The results of the three data sets are shown in Table 6, and the visual comparison is shown in Fig.10 Through the above results, the effectiveness of each component can also be verified, thus facilitating the extraction of subtle change features.

[0151] Table 6 Comparison of different ablations of HDAB module on three datasets

[0152]

[0153] In view of this, the present invention proposes a strongly supervised change detection network based on CNN and Transformer. First, in order to make full use of local and global feature information, the present invention proposes a plug-and-play feature extraction module, called the CNN_TransConv Block module, which is used to refine dual-phase features. It uses a lightweight CNN to capture local features and uses the global feature representation capability of the Transformer to extract detailed information. Secondly, the present invention designs a hybrid dual attention block, which contains an exchange operation and a CS attention mechanism, and enhances the model's ability to capture significant change information through the exchange of spatiotemporal information. Finally, the present invention proposes a multi-scale strong supervision module, which further improves the model's ability to distinguish change features by supervising the output features of various scales in the decoding stage. Experiments on three datasets show that the present invention performs well in change detection tasks.

[0154] The above is only a preferred embodiment of the present invention. The protection scope of the present invention is not limited to the above embodiment. All technical solutions belonging to the principle of the present invention belong to the protection scope of this aspect. For those skilled in the art, several improvements and modifications made without departing from the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A strongly supervised change detection method based on convolutional neural network and visual attention model, characterized in that: The following steps are involved: (1) Read a pair of remote sensing images I1 and I2 at different times into the model; (2) Use 3×3 convolution to obtain the shallow features x1 and x2 of input images I1 and I2 respectively; (3) In the network coding stage, the shallow features x1 and x2 are first processed using the CNN_TransConv module in the twin upper branch coding stage and the twin lower branch coding stage, respectively, to obtain rich information at scales of 1 / 2, 1 / 4, 1 / 8, and 1 / 16; Then, the hybrid dual attention module is used to further refine the different scale features of the upper and lower branches to help the model obtain more delicate change features; (4) In the network decoding stage, firstly, the features of different scales from the encoder and decoder stages are concatenated, and the CNN_TransConv module is used to further improve the expression of the features. Finally, the multi-scale strong supervision module is used to supervise the output features of different scales in the decoding stage. (5) Use 3×3 convolution to obtain the final output change map; (6) Optimize the model through the network total loss function; Change detection based on remote sensing images is a pixel-level classification task. In the process of change map generation, binary cross entropy loss is used to optimize the network. The loss function of change detection is defined as follows: Among them, U represents the total number of training samples, Y i represents the true value of the i-th sample, and EICTNet(.) represents the proposed network model; The total network loss can be defined as the following formula; Among them, λ is the weight parameter of the multi-scale strong supervision module, which is set to 0.05; (7) For the input image pair I1 and I2, a more refined change map is obtained through the model optimized by step (6), thereby achieving a refined change detection task.

2. According to claim 1, a strongly supervised change detection method based on convolutional neural network and visual attention model is characterized in that: In the step (3), the CNN_TransConv module is a feature extractor CNN_TransConv Block (CTCB) based on CNN and Transformer, and CTCB is divided into CNN_Branch branch, Transformer_Branch branch and 3D-Relations_Branch branch; 1) Input features of size W×H×C, where C is the number of feature channels, and then use 1×1 convolution to change its number of channels to obtain a feature f of W×H×C / 2, which will be used as the input of the CNN_Branch and Transformer_Branch branches respectively; 2) In the CNN_Branch branch, a cyclic multi-scale convolution is used to capture rich information from different receptive fields (1, 3, 6). This method uses different expansion rates to process the convolution kernel in the same layer, which can effectively capture multi-scale features without introducing additional parameters or computational costs. Subsequently, the output feature f of the CNN_Branch branch containing rich local information can be obtained through the splicing operation. c ; 3) In the Transformer_Branch branch, the entire image is first segmented, and the semantic tokenizer is used to obtain the token T∈R L×C , the analyzer aggregates the feature map space into a set of tags by learning a set of spatial attention maps, where L represents the word length. Specifically, for each pixel on the feature map f, point-wise convolution is used to obtain L word tuples, each word tuple represents a semantic concept, and then the spatial attention map A∈R is calculated by the softmax function. HW×L , H and W represent the dimensions of each word tuple. Finally, the attention map A is used to calculate the weighted average sum of the pixels in f to obtain a semantic word tuple T of size L. This process can be expressed by the following formula: T=A T f=(Softmax(f·W)) T f (5) Where, · represents the input feature f and the learnable convolution kernel W∈R C×L The Softmax(·) function can standardize each word tuple to obtain the attention map A. The word tuple T is calculated by multiplying A and f. Then, the word is input into the Transformer encoder to model the spatiotemporal global semantic relationship and enhance the word representation. The Transformer encoder consists of Multihead Self-Attention (MSA) and MultiLayer Perceptron (MLP). In this module, the word is first embedded into the layer normalization layer LN and MSA. Subsequently, it also uses residual connection to minimize information loss. The above process is modeled by formula (6): T'=MSA(LN(T))+T (6) Among them, MSA contains 8 parallel non-shared weight heads, in each head, the input word T will be converted into a triplet (query Q en , key K en ,value V en ), which is expressed as follows: in, and is a learnable parameter in the encoder, and d represents Q en , K en and V en Then, an attention head is applied to model the relationship between triplets: Then, by concatenating the results of all attention heads, the output of the multi-head self-attention MSA is obtained. In order to further encode the word embedding, a multi-layer perceptron MLP is also applied, which includes two linear transformation layers and a Gaussian error linear unit GeLU; In order to enhance the global information, a Transformer decoder containing multiple cross attentions (CA) is used to learn the correlation between the input feature f and the word unit T'. Different from the MSA in the encoder, its query comes from the image feature f, and the key and value are linearly transformed to the word unit T' obtained by the encoder, which can be expressed as follows: Finally, the final output f of the Transformer_Branch branch is obtained by using the same operations as the encoder, including attention mechanism, element-wise addition, and multi-layer perceptron. t ; 4) In the 3D-Relations_Branch branch, in order to obtain richer feature representation, the spatial-spectral dependency is modeled through the 3D attention strategy. First, the output features of the two branches are processed separately using pseudo-3D convolution, that is, 1×3×3 and 3×1×1 convolution kernels are used to capture the correlation between space and channels respectively. Then, the outputs of the two branches are fused by element-by-element addition to obtain the output f p , and use ReLU and Softmax operations to obtain the attention weight of the 3D-Relations_Branch branch. Finally, it is multiplied with the fusion output of the two branches by element-wise multiplication to obtain rich information f that jointly considers channel and spatial information. r , this process can be expressed by the following formula: f r =f p +α·β(f p )⊙f p (10) Among them, α(·) represents the Sigmoid function, β(·) represents the ReLU activation function, and ⊙ represents the element-wise product. By fusing the original input f with the output f of the 3D-Relation_Branch branch, r , and the final output f of the CTCB module is obtained out , which contains rich global and local information at the same time, and also uses the attention mechanism to fully model the relationship between the two, thereby improving the feature representation of the model.

3. The method for strongly supervised change detection based on convolutional neural network and visual attention model according to claim 1, characterized in that: The hybrid dual attention module in step (3): In order to better perceive the contextual information of the dual-time remote sensing image, this module exchanges space and channels between the dual-branch features. It first generates an exchange mask, and then retains and exchanges space and channel features based on the mask. The process can be expressed as the following formula: Wherein, Mask refers to the swap mask consisting of a change mask 1 and a non-change mask 0; Through the spatial-spectral exchange between the dual time domain features, the feature distributions of the two branches will be more similar, which helps to achieve adaptation between the dual time domains. Subsequently, in order to improve the ability of single-branch features to perceive contextual information, this module also introduces a lightweight attention mechanism CS Attention for single-branch feature space and channel exchange. In this dual attention branch, a deep convolution with a kernel size of 1×1 is used to change the number of channels and reduce the amount of calculation. Finally, by element-by-element addition, it can obtain fine pixel information from the channel and spatial attention branches.

4. The method for strongly supervised change detection based on convolutional neural network and visual attention model according to claim 1, characterized in that: In step (4), the multi-scale strong supervision module adds additional supervision on the change graphs of different scales, thereby improving network performance. The multi-scale strong supervision module first implements two parts of different scale image information collection: 1) In the decoder stage, three convolution kernels of size 3×3 are used to obtain the output change map of the intermediate layer, and the decoded change maps with scales of 1 / 8, 1 / 4, and 1 / 2 are obtained respectively; 2) Use two convolutional layers and ReLU operations to process the real image to obtain different scale change maps corresponding to the decoder level; For the decoder stage with a total of m intermediate layers, the following change diagram is obtained: Among them, conv(·) represents the convolution operation, Represents the output features of the mth decoding stage. For the real image y, the change graph is as follows: I m '=F m (y) (13) Where F(·) is the function used to process the real image, which first applies a convolutional layer for downsampling, then uses a ReLU operation, and finally uses another convolutional layer to obtain the changed image; By designing a multi-scale strong supervision module, the network can improve the feature learning ability of the intermediate layer and obtain more discriminative change features. Therefore, the auxiliary objective function of the intermediate layer is defined as the following formula: Where N represents the size of the image, i,j∈[0,N), and They represent the pixel values ​​of the change map at the mth decoding stage and the real image at point (i, j), respectively. The loss of the multi-scale supervision strong module measures their alignment in the pixel space by calculating the mean square error (MSE) between the decoder stage and the real image.

Images