A Semantic Change Detection Method for Remote Sensing Images Based on Siamese Transformers

Abstract

The present invention discloses a remote sensing image semantic change detection method based on twin Transformers. Two weight-shared Swin Transformers are used as the encoder of the model to directly model the global semantic relationship of the image. No additional convolutional layer is required to extract features, and the method has better accuracy and robustness. In the decoding stage, a 3D convolution module and a deconvolution module with axial attention are used to generate a multi-category change map from the cascaded dual-phase features, thereby improving the detection accuracy and computational efficiency of the model.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image change detection, and specifically relates to a remote sensing image semantic change detection method based on twin Transformers. Background Art

[0002] Change detection is one of the important research topics in the field of remote sensing. Its purpose is to detect the change targets of interest in the dual-phase remote sensing images of the same area and generate change maps. In semantic change maps, multi-labels are used to represent the changes of different objects, providing detailed "from-to" change information. Semantic change detection technology can reduce a lot of labor costs and time consumption, and is widely used in urban planning, environmental monitoring, and natural disaster assessment (Ji et al., 2019; Isaienkov et al., 2020; Ye et al., 2021).

[0003] With the explosive growth of high-resolution satellite and aerial remote sensing data, related problems in the field of remote sensing have also been effectively solved. However, in semantic change detection, high resolution increases the complexity of remote sensing images, and the model has difficulty distinguishing between real changes and irrelevant changes in the scene, and the recognition of different change categories is not accurate enough. Therefore, semantic change detection based on high-resolution remote sensing images is still a challenging task.

[0004] At present, many scholars have applied convolutional neural networks (CNNs) to change detection in dual-temporal remote sensing images by taking advantage of their effectiveness in feature extraction and recognition. Existing methods focus on three aspects: 1) Using twin CNNs to extract features of dual-temporal images separately (Zhang et al., 2020) to obtain accurate change detection results. 2) Adding an attention mechanism to the network to enhance features of interest (Shi et al., 2021) can accelerate the convergence of the model. 3) Deep supervision strategies and improved loss functions are used to enhance the detection effect of subtle change areas (Peng et al., 2019).

[0005] Although the current CNN-based change detection methods have achieved good performance, due to the limitations of the receptive field in the convolution operation, CNN cannot model the long-range contextual relationship of dual-phase images well, and the high computational complexity caused by deep convolution leads to reduced computational efficiency of the model. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a remote sensing image semantic change detection method based on twin Transformers, which can accurately detect the semantic changes of interest in dual-phase images and reduce the calculation complexity of the model.

[0007] The present invention has the following beneficial effects:

[0008] The present invention proposes an efficient change detection method based on Twin Transformers, which uses two weight-shared Swin Transformers as the encoder of the model to directly model the global semantic relationship of the image without the need for additional convolutional layers to extract features, and has better accuracy and robustness.

[0009] In the decoding stage, a 3D convolution module and a deconvolution module with axial attention are used to generate a multi-category change map from the cascaded dual-phase features, which improves the detection accuracy and computational efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of the method of the present invention;

[0011] Figure 2 This is a structural diagram of the Transformers encoder used in the present invention;

[0012] Figure 3 The comparison results of the present invention with other methods on the SECOND semantic change detection dataset;

[0013] Figure 4 A comparison chart of the parameters of the semantic change detection model. DETAILED DESCRIPTION

[0014] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0015] The present invention provides a semantic change detection method based on twin Transformers, which improves the efficiency and robustness of change detection by using an encoder composed of pure Transformers and a decoder composed of CNN. Figure 1 As shown, it mainly includes three parts:

[0016] (1) Linear mapping of image blocks

[0017] Before the Transformers encoder performs global context modeling, the image must first be converted into a sequence, that is, the entire image is split into several visual semantic sequences as the encoder input. Different from the existing method of first extracting features from the image through CNN, this method directly converts the input images T1 and T2 into semantic sequences tokens1 and tokens2 with learnable position encodings in order to save model capacity and reduce unnecessary computational costs. The process can be defined as follows:

[0018] Token i =F(E(x i ))+E pos ,(i=1,2) (1)

[0019] Among them, x i Indicates T i Remote sensing image at a certain moment, E() represents a linear operation, which converts the input image (H×W×3) into like Figure 1 As shown in Figure 1, C is the dimension of the semantic sequence and L is the total number of semantic sequences. The F() function represents the matrix flattening and transposition operation, which transforms the semantic sequence after linear operation into a (L×C) shape. Finally, the learnable position encoding E is added to the semantic sequence. pos ∈R L×C to preserve location information.

[0020] In short, the sequence obtained after the input image passes through the linear mapping module can be expressed as {E1+P1,E2+P2,…,E L +P L}, E i and P i They are the semantic sequences after image block embedding and position embedding respectively.

[0021] (2) Transformers Encoder

[0022] After obtaining two bi-temporal semantic sequences, an encoder composed of two twin cross Swin Transformers is used to model their contexts respectively to obtain rich semantic change information. Cross Swin Transformers consists of window-based multi-head attention (W-MSA), moving window-based multi-head attention (SW-MSA), two multi-layer perceptrons (MLP) with Gaussian error linear unit (GELU) activation and two layer normalizations (LN), and a residual connection is used after each block. The key and value in SW-MSA come from the semantic sequence output by W-MSA, and the query shares this information with W-MSA to fuse the features between the image and the sequence, and better promote the establishment of global contextual relationships. Both W-MSA and SW-MSA modules are composed of basic multi-head self-attention (MSA). Multi-head attention refers to executing multiple separate self-attention modules in parallel in Transformers, jointly processing information from different representation subspaces at different locations, and connecting the outputs together to linearly project the final value. The expression of multi-head attention is as follows:

[0023] q=T raw W Q ,k=T raw W K ,v=T raw W V (2)

[0024]

[0025] MSA(T raw )=Concat[SA(q1,k1,v1),…,SA(q m ,k m ,v m )]Wo (4)

[0026] Among them, the concat function represents cascade; W Q ,W K ,W V ∈R C×d are the learnable parameters of the three linear projection layers, d is the dimension of the triple (query, key, value), and the result after the self-attention (SA) module is calculated by formula (3). The process is as shown in formula (4), m is the number of attention heads, W o ∈R md×C Represents the linear projection matrix of MSA.

[0027] The whole process of the encoder is as follows Figure 2 As shown in Figure 2, the original Tokens are passed through the Transformers encoder to obtain a new semantic sequence with long-range context information for the next step of generating a semantic change graph.

[0028] (3) Feature Fusion Module

[0029] Feature fusion means concatenating the two-phase tokens in chronological order to obtain tokens 12 and Tokens 21 ,Right now:

[0030] Tokens 12 =concat[T new1 ,T new2 ],Tokens 21 =concat[T new2 ,T new1 ] (5)

[0031] (4) CNN decoder based on axial attention

[0032] In the decoder, a 3D convolutional block is used to process the 4D tensor after the shape change: Tokens 12 ,Tokens 21 This helps to maintain the corresponding semantic relationship of the dual-phase features and can better process long-time series image information. Finally, the dual-phase features through 3D convolution are multiplied element by element. Compared with the two fusion methods of element-by-element subtraction and channel merging, the element-by-element multiplication of the merged features can not only avoid information loss, but also the obtained fusion features have time symmetry, that is, T1·T2=T2·T1.

[0033] Then, the fused features are upsampled through three deconvolution layers. After the first two deconvolution layers, three layers of axial attention modules are added in the height, width, and channel dimensions, respectively, and connected to the deconvolution features through residual connections, which enhances the receptive field of the deconvolution module and decodes the rich semantic information in the original features into the change map. The upsampling process based on axial attention can be expressed as follows:

[0034] Z l =Ax C (Ax W (Ax H (TrConv(x l-1 ))))+TrConv(x l-1 ) (11)

[0035] Where Ax H , Ax W , Ax C denotes the axial attention in height, width and channel dimensions respectively, x l-1 is the original feature of layer l-1. TrConv represents the deconvolution operation, including BatchNorm and RelU activation functions, Z l For x l-1 The l-th layer feature after upsampling. The addition of the two parts in formula (11) represents the residual connection. The feature Z output after the residual connection of the first deconvolution layer and the first three-layer axial attention module l After being processed again by the second set of deconvolution layers and the three-layer axial attention module, the residual connection is output to the third deconvolution layer. The axial attention-based CNN decoder maps the semantic sequence vector back to the pixel space and generates the final semantic change map through upsampling.

[0036] In order to verify the effectiveness of the semantic change detection method based on twin Transformers proposed in this paper, experiments were conducted on the semantic change detection dataset SECOND, and compared with several other commonly used methods.

[0037] Figure 3The figure shows some detection results of the present invention and three popular semantic change detection methods on the preprocessed SECOND dataset, with different colors representing different types of land cover change. By comparing the visualization results, the present invention can find that the present method can capture richer semantic features due to the strong modeling ability of Transformers for long-distance context information, so the change map generated in semantic change detection is the clearest and most complete.

[0038] In order to more comprehensively evaluate the accuracy of the proposed method, the present invention shows the evaluation indicators of the proposed method and other semantic change detection methods on the SECOND dataset in Table 1. The bold data represents the optimal value of the category in different methods. It can be seen that except for the "unchanged" category, the detection accuracy of the proposed method for each change category exceeds that of other methods, reaching 76.34% and 67.43% in the mF1 and mIoU indicators respectively. This shows that the present method can well model the semantic information of the distant context and realize the detection of the change category of interest by extracting and identifying complex features. Compared with the CNN-based method, the Transformers-based method proposed by us has more advantages for multi-category change detection of remote sensing images.

[0039] Table 1 Evaluation indicators of the present invention and other methods on the semantic change detection dataset SECOND

[0040]

[0041]

[0042] In some specific tasks, not only the accuracy and recall rate of change detection are required, but also the number of model parameters and the real-time performance of change detection are also very important. Figure 4 The parameter amounts of all semantic change detection models in the experiment are shown. By comparison, it can be seen that the model parameter amount proposed by the present invention is 7.33MB, which is the lowest compared with other existing algorithms. The present invention takes into account two important indicators, model capacity and detection accuracy, and has high efficiency, that is, it achieves competitive change detection performance with fewer model parameters.

[0043] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A remote sensing image semantic change detection method based on twin Transformers, including an image block linear mapping process, an encoding process, a feature fusion process and a decoding process, characterized in that: The encoding process is implemented using an encoder consisting of two weight-shared twin crossover Swin Transformers; Among them, the two SwinTransformers process the semantic sequences obtained by linear mapping of two dual-phase images through image blocks respectively; the query in the moving window-based multi-head attention SW-MSA in each SwinTransformer is shared with the window-based multi-head attention W-MSA; In the decoding process, a 3D convolution module and a deconvolution module with axial attention are used to generate a multi-category change map from the cascaded dual-phase features.

2. The remote sensing image semantic change detection method based on twin Transformers according to claim 1, characterized in that: The feature fusion process is implemented by a feature fusion module, specifically: Assume that after the two SwinTransformers of the encoder encode the two bi-temporal images respectively, the resulting semantic sequences are Tnew1 and Tnew2 respectively; Concatenate Tnew1 and Tnew2 in chronological order to obtain Tokens 12 and Tokens 21 ,Right now: Tokens 12 =concat[T new1 ,T new2 ],Tokens 21 =concat[Tnew2,Tnew1] Among them, the concat function represents cascade; The decoding process is implemented using a CNN decoder, specifically: First, two 3D convolution blocks are used to process Tokens separately 12 and Tokens 21 , and then multiply the two processing results element by element, and send the result to the upsampling module; wherein, the upsampling module includes three deconvolution layers connected in sequence, and a three-layer axial attention module with height, width and channel latitude is set after each of the first two deconvolution layers; wherein, each three-layer axial attention module is residually connected to a deconvolution layer in front of it.

3. A remote sensing image semantic change detection method based on twin Transformers as described in claim 1 or 2, characterized in that: The image block linear mapping process includes: Convert the input bi-temporal images T1 and T2 into semantic sequences tokens1 and tokens2 with learnable position encodings: Tokeni=F(E(xi))+Epos,i=1,2 (1) Among them, x i Indicates T i Remote sensing images at a certain moment, E() function represents linear operation; F() function represents flattening and transposition operation; Epos∈R L × C Represents learnable positional encoding information.

Images