A method for heterogeneous remote sensing building change detection based on linear cross-temporal interaction

Abstract

The present invention discloses a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction, including: obtaining a heterogeneous remote sensing image pair and dividing it into a plurality of sub-images; using a pre-trained CNN or Transformer model to extract features from each sub-image to obtain a corresponding first feature map; performing a linear cross-attention operation on the first feature map obtained from the corresponding sub-image in the image pair to obtain a corresponding second feature map; performing a collaborative feature alignment operation on the second feature map to obtain a corresponding third feature map; performing difference, upsampling and convolution transformation on the third feature map obtained from the corresponding sub-image in the image pair in sequence to obtain a change detection map. This method targets the heterogeneous differences in data in the heterogeneous remote sensing image change detection task, performs image coupling interaction through cross-attention, and constructs collaborative feature alignment to ensure feature alignment under heterogeneous characteristics, which helps to improve the accuracy and generalization performance of heterogeneous remote sensing image change detection.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction. Background Art

[0002] Remote sensing image change detection refers to the process of extracting the changed areas between remote sensing images of the same area at different times. Change detection is an important research direction in the field of remote sensing. As a key technology for monitoring surface conditions, it plays a huge role in many fields such as urban expansion mapping, environmental monitoring, and disaster loss assessment. Since satellite operation requires a certain period of time, the only remote sensing images may be obtained by different satellite sensors on the target date. In this case, a fast, flexible, and accurate method is needed to process heterogeneous remote sensing images obtained by different sensors.

[0003] Traditional change detection algorithms are often based on manually extracted features, and the representation effect is not ideal. Deep neural networks can extract deeper nonlinear features of images and have better representation. For example, convolutional neural networks (CNNs) capture local inductive biases from data through a sliding convolution kernel mechanism, and Transformers capture long-distance dependencies by dynamically calculating weights through similarity or affinity in a self-attention mechanism. However, CNNs lack the ability to model long-distance dependencies due to their limited receptive field. Although Transformers capture long-distance dependencies with an approximately global receptive field from the shallow network, this causes the representations obtained by Transformers in the shallow and deep layers to have more similarities, resulting in excessive redundancy. Therefore, a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction is proposed. Summary of the invention

[0004] The purpose of the present invention is to propose a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction to address the above problems. Aiming at the heterogeneous differences in data in the heterogeneous remote sensing image change detection task, cross-attention is used to perform image coupling interaction, and collaborative feature alignment is constructed to ensure feature alignment under heterogeneous characteristics, which helps to improve the accuracy and generalization performance of heterogeneous remote sensing image change detection.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction, which comprises the following steps:

[0007] S1, obtaining a remote sensing image pair from a different source, and dividing two images in the image pair into a number of sub-images of the same size;

[0008] S2. Use the pre-trained CNN model or Transformer model to extract features from each sub-image to obtain the corresponding first feature map, the size of which is W×H×d, where W is the width of the first feature map, H is the height of the first feature map, and d is the channel depth of the first feature map;

[0009] S3, the first feature maps obtained from the corresponding sub-images in the image pair are recorded as F1 and F2 respectively, and a linear cross-attention operation is performed to obtain the corresponding second feature map F att-1 and F att-2 , the linear cross attention operation is as follows:

[0010] S31, linearly expand F1 and F2 to obtain the corresponding vector group representations (Q1, K1, V1) and (Q2, K2, V2), where Q1 is the query matrix of F1, K1 is the key matrix of F1, V1 is the value matrix of F1, Q2 is the query matrix of F2, K2 is the key matrix of F2, and V2 is the value matrix of F2, and the size is N×d, N=W×H;

[0011] S32, calculate the linearized self-attention output F of F1 based on the exchanged query matrix corss-1 and the linearized self-attention output F of F2 cross-2 , the dimensions are W×H×d;

[0012] S33, perform depth-separable convolution operations on the value matrices V1 and V2 respectively, and then perform Hadamard products with the exchanged query matrices respectively, and obtain the convolution relative position code P with a size of W×H×d corss-1 and P cross-2 ;

[0013] S34, sum the linearized self-attention output and convolution relative position encoding obtained by F1 and F2 respectively, and obtain the second feature map F with a size of W×H×d att-1 and F att-2 ;

[0014] S4, F att-1 and F att-2 Perform collaborative feature alignment to obtain the corresponding third feature map F align-1 and F align-2 , the collaborative feature alignment operation is as follows:

[0015] S41, F att-1 and F att-2 Input the spatial attention module to obtain the spatial attention A with a size of W×H×2. The spatial attention module includes a Concat layer, a first activation function, a convolution layer, a normalization layer, and a second activation function connected in sequence.

[0016] S42, split the spatial attention A into A1 and A2 of size W×H×1 along the channel direction;

[0017] S43, F att-1 Multiply by A1, F att-2 Multiply it with A2 to get the third feature map F align-1 and F align-2 ;

[0018] S5. Obtain the third feature map F by centering the corresponding sub-image in the image. align-1 and F align-2 The difference, upsampling and convolution transformations are performed in sequence to obtain a change detection map with a size of nW×nH×2, where n is the multiple of upsampling.

[0019] Preferably, in step S1, the size of the sub-image is 256×256.

[0020] Preferably, in step S32, the linearized self-attention output F of F1 corss-1 and the linearized self-attention output F of F2 cross-2 , the calculation formula is as follows:

[0021]

[0022]

[0023] Among them, softmax(·) is a nonlinear mapping function and T is a transposition operation.

[0024] Preferably, in step S41, the first activation function is a ReLU function, and the second activation function is a Sigmoid function.

[0025] Preferably, in step S5, upsampling is 4 times interpolation upsampling.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This method aims at the heterogeneous differences of data in the heterogeneous remote sensing image change detection task. It uses the cross-attention mechanism with linear computational complexity and convolutional relative position encoding capability to reduce the heterogeneous differences between heterogeneous remote sensing data and enhance the coupling interaction between features. It also constructs a collaborative feature alignment operation to further aggregate multiple feature maps to ensure alignment between features under heterogeneous characteristics. While learning each spatial attention distribution to highlight task-related features, it suppresses irrelevant features, thereby achieving accurate heterogeneous remote sensing image change detection with good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the heterogeneous remote sensing building change detection method of the present invention;

[0029] Figure 2 A flowchart of the linear cross attention operation of the present invention;

[0030] Figure 3 A flowchart of the collaborative feature alignment operation of the present invention;

[0031] Figure 4 2 is a diagram showing the detection results of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0034] This application aims at the data heterogeneity differences in heterogeneous remote sensing image change detection tasks, and realizes accurate heterogeneous remote sensing image change detection by constructing a cross-attention mechanism with linear complexity and convolutional relative position encoding capabilities and a collaborative feature alignment operation that specifically promotes effective feature mining and irrelevant feature suppression capabilities.

[0035] like Figure 1-4 As shown, a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction includes the following steps:

[0036] S1. Obtain a pair of remote sensing images from different sources, and divide the two images in the pair into a number of sub-images of the same size.

[0037] In one embodiment, in step S1, the size of the sub-image is 256×256, or can be adjusted according to actual needs.

[0038] S2. Use the pre-trained CNN model or Transformer model to extract features from each sub-image to obtain the corresponding first feature map, the size of which is W×H×d, where W is the width of the first feature map, H is the height of the first feature map, and d is the channel depth of the first feature map;

[0039] S3, the first feature maps obtained from the corresponding sub-images in the image pair are recorded as F1 and F2 respectively, and a linear cross-attention operation is performed to obtain the corresponding second feature map F att-1 and Fatt-2 , the linear cross attention operation is as follows:

[0040] S31. Linearly expand F1 and F2 to obtain the corresponding vector group representations (Q1, K1, V1) and (Q2, K2, V2), where Q1 is the query matrix of F1, K1 is the key matrix of F1, V1 is the value matrix of F1, Q2 is the query matrix of F2, K2 is the key matrix of F2, and V2 is the value matrix of F2, all with dimensions of N×d, N=W×H.

[0041] S32, calculate the linearized self-attention output F of F1 based on the exchanged query matrix corss-1 and the linearized self-attention output F of F2 cross-2 , the dimensions are W×H×d.

[0042] In one embodiment, in step S32, the linearized self-attention output F of F1 corss-1 and the linearized self-attention output F of F2 cross-2 , the calculation formula is as follows:

[0043]

[0044]

[0045] Among them, softmax(·) is a nonlinear mapping function and T is a transposition operation.

[0046] S33, perform depth-separable convolution operations on the value matrices V1 and V2 respectively, and then perform Hadamard products with the exchanged query matrices respectively, and obtain the convolution relative position code P with a size of W×H×d corss-1 and P cross-2 .

[0047] Specifically, the convolutional relative position code P corss-1 and P cross-2 The calculation formula is as follows:

[0048] P cross-1 =Q2□DepthwiseConv(V1)

[0049] P cross-2 =Q1□DepthwiseConv(V2)

[0050] Among them, □ is the Hadamard product, and DepthwiseConv(·) is the depthwise separable convolution operation.

[0051] S34, sum the linearized self-attention output and convolution relative position encoding obtained by F1 and F2 respectively, and obtain the second feature map F with a size of W×H×d att-1 and Fatt-2 .

[0052] Specifically, F att-1 The second feature map obtained by F1, F att-2 The second feature map obtained for F2 is the second feature map, F att-1 and F att-2 The calculation formula is as follows:

[0053]

[0054]

[0055] Among them, the linear cross-attention operation eliminates the differences between heterogeneous data through information interaction, achieving high alignment and mutual coupling between features. The cross-attention mechanism with linear computational complexity and convolutional relative position encoding capability is used to reduce the heterogeneous differences between heterogeneous remote sensing data and enhance the coupling interaction between features.

[0056] S4, F att-1 and F att-2 Perform collaborative feature alignment to obtain the corresponding third feature map F align-1 and F align-2 , the collaborative feature alignment operation is as follows:

[0057] S41, F att-1 and F att-2 The input spatial attention module obtains the spatial attention A of size W×H×2. The spatial attention module includes a Concat layer, a first activation function, a convolution layer, a normalization layer, and a second activation function connected in sequence.

[0058] In one embodiment, in step S41, the first activation function is a ReLU function, and the second activation function is a Sigmoid function.

[0059] Specifically, the calculation formula of spatial attention A is as follows:

[0060] A=Sigmoid(BN(Conv(ReLU(Concat(F att-1 ,F att-2 )))))

[0061] Among them, Sigmoid(·) is the Sigmoid nonlinear mapping function, Conv(·) is the convolution operation, ReLU(·) is the Relu nonlinear mapping function, BN(·) is the normalization operation, and Concat(·) is the connection function.

[0062] S42. Split the spatial attention A into A1 and A2 of size W×H×1 along the channel direction.

[0063] S43, F att-1 Multiply by A1, F att-2 Multiply it with A2 to get the third feature map F align-1 and F align-2 .

[0064] Among them, performing collaborative feature alignment operations can highlight the relevant features of the change detection task while suppressing irrelevant features to achieve accurate detection.

[0065] S5. Obtain the third feature map F by centering the corresponding sub-image in the image. align-1 and F align-2 The difference, upsampling and convolution transformations are performed in sequence to obtain a change detection map with a size of nW×nH×2, where n is the multiple of upsampling.

[0066] In one embodiment, in step S5, upsampling is 4 times interpolation upsampling, which can be adjusted according to actual needs.

[0067] As a preferred embodiment, remote sensing images of Christchurch, New Zealand before and after the reconstruction of the 6.3 magnitude earthquake in 2011 are taken as an example, and a heterogeneous remote sensing building change detection method based on linear cross-temporal interaction provided by this application is used for experiment. The images of the detection data set contain three bands of R, G, and B, with a resolution of 0.2m. The specific steps are as follows:

[0068] Step 1: Load remote sensing images of the Christchurch area in New Zealand before and after the 2011 6.3 magnitude earthquake reconstruction, such as Figure 4 As shown in the figure, X1 is the image before earthquake reconstruction, X2 is the image after earthquake reconstruction, (a)-(f) are six groups of images, and the three bands shown (default R, G, B) are selected to perform a 256×256 cutting operation on the heterogeneous remote sensing image pairs and their corresponding detection label maps, and the data are divided into three parts: training set, validation set, and test set.

[0069] Step 2: Pre-train a CNN model (specifically using the Resnet-18 network) on the ImageNet classification dataset. The CNN model can also be replaced by a Transformer model.

[0070] Step 3: Load the training set and validation set data cut in step 1, and send them to the pre-trained CNN model to capture the first feature maps F1 and F2 with a size of 64×64×64. F1 and F2 are the first feature maps of the corresponding sub-images (heterologous images) in the image pair.

[0071] Step 4: Perform a linear cross attention operation on the first feature maps F1 and F2 to obtain the second feature map F after heterogeneous interaction with a size of 64×64×64 att-1 and Fatt-2 .

[0072] Step 5: Perform collaborative feature alignment to highlight task-related information and suppress task-irrelevant features, and obtain the third feature map F of each heterogeneous data with a size of 64×64×64 align-1 and F align-2 .

[0073] Step 6: Output the corresponding sub-images in the image pair to the third feature map F with a size of 64×64×64 align-1 and F align-2 The difference, 4-fold interpolation upsampling and convolution operations are performed in sequence to obtain a change detection map with a size of 256×256×2. The cross entropy loss function and the stochastic gradient descent algorithm can be used to minimize the loss between the detection prediction map and the label map to complete the model training. The technology is well known to those skilled in the art and will not be described here.

[0074] Step 7: Load the trained parameters of this method, obtain the predicted output on the test set, and compare the change detection results with the test set labels to calculate the precision, recall, F1 score, intersection-over-union ratio, and overall accuracy, as shown in Figure 4 As shown, there are six groups of image pairs (a) to (f), which are the comparison results of the change detection diagrams of the proposed method (the method of the present application) and the reference method (dual time converter). The detection evaluation results are shown in Table 1.

[0075] Table 1

[0076]

[0077] Among them, the dual time transformer is a method proposed in the prior art, reference: H.Chen, Z.Qi and Z.Shi,"Remote Sensing Image Change Detection With Transformers,"in IEEE Transactions on Geoscience and Remote Sensing,vol.60,pp.1-14,2022,Art no.5607514,doi:10.1109 / TGRS.2021.3095166. The detection results of the proposed method in Table 1 are numerically superior to the results of the dual time transformer method in the prior art in all aspects. Figure 4 It has fewer missed detections (light gray pixels) and false detections (dark gray pixels), as well as higher accuracy (white pixels), which further verifies the effectiveness of our method in the task of heterogeneous remote sensing building change detection.

[0078] This method aims at the heterogeneous differences of data in the heterogeneous remote sensing image change detection task. It uses the cross-attention mechanism with linear computational complexity and convolutional relative position encoding capability to reduce the heterogeneous differences between heterogeneous remote sensing data and enhance the coupling interaction between features. It also constructs a collaborative feature alignment operation to further aggregate multiple feature maps to ensure alignment between features under heterogeneous characteristics. While learning each spatial attention distribution to highlight task-related features, it suppresses irrelevant features, thereby achieving accurate heterogeneous remote sensing image change detection with good generalization performance.

[0079] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.

Claims

1. A heterogeneous remote sensing building change detection method based on linear cross-temporal interaction, characterized by: The heterogeneous remote sensing building change detection method based on linear cross-temporal interaction comprises the following steps: S1, obtaining a remote sensing image pair from a different source, and dividing two images in the image pair into a number of sub-images of the same size; S2. Use the pre-trained CNN model or Transformer model to extract features from each sub-image to obtain the corresponding first feature map, the size of which is W×H×d, where W is the width of the first feature map, H is the height of the first feature map, and d is the channel depth of the first feature map; S3, the first feature maps obtained from the corresponding sub-images in the image pair are recorded as F1 and F2 respectively, and a linear cross-attention operation is performed to obtain the corresponding second feature map F att-1 and F att-2 , the linear cross attention operation is as follows: S31, linearly expand F1 and F2 to obtain the corresponding vector group representations (Q1, K1, V1) and (Q2, K2, V2), where Q1 is the query matrix of F1, K1 is the key matrix of F1, V1 is the value matrix of F1, Q2 is the query matrix of F2, K2 is the key matrix of F2, and V2 is the value matrix of F2, and the size is N×d, N=W×H; S32, calculate the linearized self-attention output F of F1 based on the exchanged query matrix corss-1 and the linearized self-attention output F of F2 cross-2 , the dimensions are W×H×d; S33, perform depth-separable convolution operations on the value matrices V1 and V2 respectively, and then perform Hadamard products with the exchanged query matrices respectively, and obtain the convolution relative position code P with a size of W×H×d corss-1 and P cross-2 ; S34, sum the linearized self-attention output and convolution relative position encoding obtained by F1 and F2 respectively, and obtain the second feature map F with a size of W×H×d att-1 and F att-2 ; S4, F att-1 and F att-2 Perform collaborative feature alignment to obtain the corresponding third feature map F align-1 and F align-2 , the collaborative feature alignment operation is specifically as follows: S41, F att-1 and F att-2 Input a spatial attention module to obtain a spatial attention A of size W×H×2, wherein the spatial attention module includes a Concat layer, a first activation function, a convolution layer, a normalization layer, and a second activation function connected in sequence; S42, split the spatial attention A into A1 and A2 of size W×H×1 along the channel direction; S43, F att-1 Multiply by A1, F att-2 Multiply it with A2 to get the third feature map F align-1 and F align-2 ; S5. Obtain the third feature map F by centering the corresponding sub-image in the image. align-1 and F align-2 The difference, upsampling and convolution transformations are performed in sequence to obtain a change detection map with a size of nW×nH×2, where n is the multiple of upsampling.

2. The heterogeneous remote sensing building change detection method based on linear cross-temporal interaction as claimed in claim 1, characterized in that: In step S1, the size of the sub-image is 256×256.

3. The heterogeneous remote sensing building change detection method based on linear cross-temporal interaction as claimed in claim 1, characterized in that: In step S32, the linearized self-attention output F of F1 corss-1 and the linearized self-attention output F of F2 cross-2 , the calculation formula is as follows: Among them, softmax(·) is a nonlinear mapping function and T is a transposition operation.

4. The heterogeneous remote sensing building change detection method based on linear cross-temporal interaction as claimed in claim 1, characterized in that: In step S41, the first activation function is a ReLU function, and the second activation function is a Sigmoid function.

5. The heterogeneous remote sensing building change detection method based on linear cross-temporal interaction as claimed in claim 1, characterized in that: In step S5, the upsampling is 4 times interpolation upsampling.

Images