A remote sensing image change detection method based on a dual-time-phase deviation Transformer

Abstract

The application discloses a remote sensing image change detection method based on a dual-time phase deviation Transformer, which comprises the following steps: firstly, pre-processing two remote sensing images at different times based on color difference equalization and stretching; then, using a deep deviation feature extraction network to extract input feature maps X1 and X2 from the pre-processed remote sensing images; obtaining an attention map from the feature maps and performing weighted average and calculation to obtain a semantic label set; then, connecting the semantic label set to obtain a joint feature map, inputting the joint feature map into an encoder of an enhancement module, performing segmentation, respectively decoding, and obtaining refined feature maps; subtracting the refined feature maps to obtain a feature difference map, inputting the feature difference map into a shallow CNN network to perform pixel-level prediction and obtain a change map. The application can efficiently extract and represent deep features, complete a complex feature mapping task, and solve some problems existing in traditional methods and the limitations of existing methods.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a method for detecting changes in remote sensing images. Background Technology

[0002] Remote sensing image change detection plays a crucial role in many fields, including natural disaster early warning and monitoring, resource surveys and applications, environmental monitoring and assessment, military reconnaissance, and strike effect evaluation. To date, scholars both domestically and internationally have proposed numerous models and methods. The change detection process of these methods generally involves preprocessing two remote sensing images first, and then using different methods to extract the changed areas. However, traditional remote sensing image change detection methods generally suffer from drawbacks such as stringent requirements for image preprocessing, the need for manual intervention in some steps, difficulty in automation, and difficulty in handling multi-source image data.

[0003] Deep learning is an algorithm for learning representations of data. It possesses extremely strong learning capabilities, can fit complex mapping relationships, and solve complex task scenarios. Currently, this technology has been rapidly adopted across various fields. Deep learning excels in natural image processing and understanding, thus offering a new approach to the problem of change detection in remote sensing images. However, most existing deep learning-based change detection methods are insufficient in handling image noise, resulting in low accuracy and certain limitations.

[0004] Existing technical solutions for remote sensing image change detection:

[0005] (1) Image data preprocessing;

[0006] (2) Construct a neural network;

[0007] (3) Input the image data from the training set into the neural network for training;

[0008] (4) Save the training result model and make predictions for the changing regions.

[0009] Although the above steps can yield a correct prediction model, the accuracy of the model's results fluctuates each time due to deviations in the number of training iterations and the setting of preset parameters. Therefore, the detection accuracy of existing deep learning change detection methods still needs to be improved.

[0010] Disadvantages of existing technology:

[0011] Most existing methods improve upon some aspects of traditional change detection methods. They are insufficient in noise handling of images during detection, have low accuracy, and are relatively cumbersome to operate. Furthermore, they can only identify certain types of image data and cannot cover all types of image data.

[0012] In summary:

[0013] (1) The remote sensing images are severely noisy;

[0014] (2) It has low applicability to various types of image data. Summary of the Invention

[0015] To overcome the shortcomings of existing technologies, this invention provides a remote sensing image change detection method based on a dual-temporal bias Transformer. First, two remote sensing images from different times are preprocessed using color difference equalization and stretching. Then, a depth bias feature extraction network is used to extract input feature maps X1 and X2 from the preprocessed images. An attention map is obtained from the feature maps, and a weighted average is calculated to obtain a semantic tag set. The semantic tag sets are then concatenated to obtain a joint feature map, which is input into the encoder of the enhancement module for segmentation, followed by individual decoding to obtain refined feature maps. The refined feature maps are subtracted to obtain a feature difference map, which is then input into a shallow CNN network for pixel-level prediction to obtain a change map. This invention can efficiently extract and represent deep features, complete complex feature mapping tasks, and solve some of the problems and limitations of traditional methods.

[0016] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0017] Step 1: Perform preprocessing on two remote sensing images from different times based on color difference equalization and stretching;

[0018] Step 2: Input the preprocessed remote sensing image into the depth aberration feature extraction module to obtain feature maps X1 and X2; the depth aberration feature extraction module includes feature extraction, feature selection, branch output, and feature fusion operations;

[0019] Step 3: Transfer feature map X i The inputs i=1,2 are fed into the dual-temporal feature joint enhancement module; the dual-temporal feature joint enhancement module calculates the spatial attention map using the softmax function, and performs a weighted average to calculate the semantic tag sets T1 and T2; the semantic tag sets T1 and T2 are concatenated to obtain the joint feature map T, which is then input into the encoder of the enhancement module for segmentation, and then decoded separately to obtain the refined feature map X. 1new X 2new ;

[0020] Step 4: Place X 1new X 2new Subtracting the features yields a feature difference map, which is then input into a shallow CNN network for pixel-level prediction to obtain a change map.

[0021] Preferably, step 1 specifically comprises:

[0022] Step 1-1: Color difference equalization;

[0023] Calculate the histogram of the original remote sensing image, then equalize the histogram to obtain an equalized histogram; construct a mapping function to map the original remote sensing image to the range of the equalized histogram, as shown in the following formula:

[0024] f′(x,y)=T[h(f(x,y))]

[0025] Where f′(x,y) is the equalized image, T is the mapping function, and h(f(x,y)) is the histogram of the original image f(x,y);

[0026] Steps 1-2: Stretching;

[0027] Calculate the mean and standard deviation of the pixel values ​​in the original remote sensing image, and then perform a linear transformation on the pixel values ​​of the original remote sensing image, as shown in the following formula:

[0028]

[0029] Where f″(x,y) is the stretched image, g(x,y) is the original image, and Min g and Max g These are the minimum and maximum pixel values ​​of the original image. f and Max f These are the minimum and maximum values ​​of the stretched pixel value range.

[0030] Preferably, step 2 specifically comprises:

[0031] Step 2-1: Divide the input image into several overlapping regions of the same size, and flatten them into a matrix form; the formula for the feature extraction module is expressed as:

[0032] X = MLP out (V out ⊙Norm(V in +SAB(V in )))

[0033] Among them, V in and V out The input and output are sequences of Transformer feature vectors, SAB represents self-attention blocks, and MLP is used. out represents a fully connected layer mapping, ⊙ represents an element-wise vector product, and Norm represents a normalized layer;

[0034] Step 2-2: Calculate the entropy of each feature using the entropy formula:

[0035]

[0036] Where X i Let represent the i-th dimension of the feature vector, k represent the number of values ​​for that dimension, and P(.) represent the probability.

[0037] Then, the feature selection module calculates the usefulness of each feature for the image processing task using the information gain formula:

[0038] G i =H(Y)-H(Y|X) i )

[0039] Where Y represents the task, H(Y) represents the entropy of the task, and H(Y|Xi) represents the conditional entropy of the task given the i-th feature;

[0040] Steps 2-3: The output of the branch network includes classification labels, bounding boxes, and segmentation masks, which are used to guide subsequent task processing.

[0041] Steps 2-4: Fuse the features obtained from different branches to generate the final feature vector to support the processing of various tasks. The feature fusion formula is as follows:

[0042]

[0043] Preferably, step 3 specifically comprises:

[0044] Step 3-1: For feature map X i Each pixel X i Pointwise convolution is used to obtain L semantic groups, each group representing a semantic concept; then the softmax function is applied to the Height and Width dimensions of each semantic group to calculate the spatial attention map and use the attention map to calculate the weighted average sum of pixels to obtain semantic labels T1 and T2.

[0045] Step 3-2: Perform a concat operation on the two semantic tags to obtain a joint feature map T. Input this T into the encoder to encode the context information into the semantic tag set T, and generate the semantic tag set T. new ;

[0046] Step 3-3: Place T new Divide into T 1new T 2new Afterwards, regarding T 1new and T 2new Decoding operations are performed separately to obtain the refined feature map X. 1new and X 2new .

[0047] Preferably, step 4 specifically comprises:

[0048] Will The feature representations of the two images are subtracted pixel by pixel to obtain the feature difference value at each corresponding pixel location. This feature difference value is then applied to the original image to obtain the transformation image. The specific formula is as follows:

[0049] V(x,y)=[T(x±d,x±d)-S(x,y)]^2

[0050] Where V(x,y) represents the value of each pixel in the transformation image, T(x±d,x±d) represents the pixel in the second image at the same position as the first image, S(x,y) represents the pixel at position (x,y) in the first image, and d is the radius of the sliding window.

[0051] Preferably, the encoding operation is as follows:

[0052] The joint feature map T is normalized using pre-normalized residual units, and then input into multi-head self-attention to calculate the query Q, key K, and value V. The attention weights Att(Q, K, V) are then calculated using the following formula:

[0053] Q = T (l-1) W q

[0054] K = T (l-1) W k

[0055] V = T (l-1) W v

[0056]

[0057] Where T (l-1) is the output of the previous layer network, and W is the learnable parameter in the layer.

[0058] The beneficial effects of this invention are as follows:

[0059] 1. This invention utilizes the efficient feature learning capability of the dual-temporal bias Transformer to solve the complex problem of remote sensing image change detection. By iteratively updating network parameters through a large number of training samples, it efficiently extracts and represents deep features, completes complex feature mapping tasks, and can solve some of the problems and limitations of traditional methods.

[0060] 2. This invention can better extract the original features of images and model the context within dual-temporal images, thereby helping to identify changes in the target and eliminate irrelevant changes, enabling remote sensing image change detection. Simultaneously, this invention performs denoising processing on the target image, partially eliminating image noise compared to traditional methods without requiring manual intervention.

[0061] 3. This invention can better capture global contextual information in images and has higher parameter efficiency. Furthermore, because the model uses a self-attention mechanism for feature representation and relation modeling, rather than relying on fixed-size convolutional kernels, this invention can flexibly handle input images of different sizes and has better universality for various image types.

[0062] 4. This invention also enhances the feature map after feature extraction, thereby generating a more refined feature map, providing a more expressive and discriminative feature representation for subsequent change detection tasks. Attached Figure Description

[0063] Figure 1 This is a structural diagram of the method of the present invention. Detailed Implementation

[0064] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0065] The purpose of this invention is to propose a novel remote sensing image change detection network based on dual-temporal-bias Transformer, which combines the advantages of traditional and novel networks to achieve higher accuracy, higher sensitivity, and higher universality in change detection.

[0066] like Figure 1 As shown, a remote sensing image change detection method based on dual-temporal-bias Transformer includes the following steps:

[0067] Step 1: Perform remote sensing image preprocessing on two images taken at different times, based on color difference equalization and stretching.

[0068] Step 2: Use a depth deviation feature extraction network on the preprocessed image as input feature maps X1 and X2 for the dual-temporal feature joint enhancement module.

[0069] Step 3: Transfer the feature map X i The attention maps are obtained and weighted averaged to calculate semantic tag sets T1 and T2. The two semantic tag sets are concatenated to obtain a joint feature map T, which is then input into the encoder of the enhancement module for segmentation. Each segment is then decoded separately to obtain a refined feature map X containing rich connections. 1new X 2new .

[0070] Step 4: Place X 1new X 2new The feature difference map (FDI) is obtained by performing a subtraction operation, and then input into a shallow CNN network for pixel-level prediction to obtain a change map.

[0071] The third step is further detailed as follows:

[0072] 3-1. For feature map X i Each pixel X i For (i = 1, 2), pointwise convolution is used to obtain L semantic groups, each representing a semantic concept. Then, a softmax function operation is performed on the H and W dimensions of each semantic group to calculate a spatial attention map. This attention map is then used to calculate the weighted average sum of pixels to obtain semantic labels T1 and T2. (H and W are height and width, respectively)

[0073] 3-2. Concatenate the two semantic tags to obtain a joint feature map T. Input this joint feature map into the encoder to encode the context information into the semantic tag set T, and generate a context-rich semantic tag set T. new .

[0074] 3-3, T new After splitting, it becomes T 1new T 2new Then, decoding operations are performed on both, ultimately yielding a refined feature map X containing rich connections. 1new X 2new . Specific implementation examples:

[0076] The method of the present invention will be illustrated by taking the application of the method of the present invention on the (LEVIR)-CD dataset as an example:

[0077] Step 1: Perform color difference equalization and grayscale stretching on the original images of the dataset.

[0078] 1-1. Calculate the histogram of the original image, equalize the histogram of the original image to obtain an equalized histogram, and construct a mapping function to map the original image to the range of the equalized histogram. The formula is as follows:

[0079] f(x,y)=T[h(f(x,y))]

[0080] Where f(x,y) is the equalized image, T is the mapping function, and h(f(x,y)) is the histogram of the original image.

[0081] 1-2. Calculate the mean and standard deviation of the pixel values ​​in the original image, and then perform a linear transformation on the pixel values ​​to adjust the range and distribution of the pixel values. The formula is as follows:

[0082]

[0083] Where f(x,y) is the stretched image, g(x,y) is the original image, and Min g and Max g These are the minimum and maximum pixel values ​​of the original image. f and Max f These are the minimum and maximum values ​​of the stretched pixel value range.

[0084] Step 2: Input the processed image into the feature extraction module. The detailed structure is as follows:

[0085] 2-1. Divide the input image into several overlapping regions of equal size and flatten them into a matrix form. The Transformer module can further abstract and integrate the image features extracted by the convolutional layers through self-attention mechanisms and feedforward networks. The formula for the entire feature extraction module is expressed as:

[0086] X = MLP out (V out ⊙Norm(V in +SAB(V in )))

[0087] Among them, V in and V out The input and output are sequences of Transformer feature vectors, SAB represents self-attention blocks, and MLP is used. out ☐ represents a fully connected layer mapping, ⊙ represents the element-wise product of vectors (Hadamard product), and Norm represents a normalized layer.

[0088] 2-2. Calculate the entropy of each feature using the entropy formula:

[0089]

[0090] Where, X i Let represent the i-th dimension of the feature vector, and k represent the number of values ​​that dimension can take.

[0091] Then, the feature selection module calculates the usefulness of each feature to the task using the information gain formula:

[0092] G i =H(Y)-H(Y|X) i )

[0093] Where Y represents the task (e.g., image classification), H(Y) represents the entropy of the task, and H(Y|Xi) represents the conditional entropy of the task given the i-th feature.

[0094] 2-3: The output of the branch network includes classification labels, bounding boxes, and segmentation masks, which are used to guide subsequent task processing.

[0095] 2-4: The features obtained from different branches are fused to generate the final feature vector to support the processing of various tasks. The feature fusion formula is as follows:

[0096]

[0097] Step 3

[0098] 3-1 For pixels X on features X1 and X2 i _p, using dotted convolution to obtain L semantic groups, and applying the softmax function to calculate the spatial attention map. Finally, the attention map is used to perform a weighted average summation of the pixels in Xi, thus obtaining a compact vocabulary of size L, i.e., semantic tags Ti. The specific formula is as follows:

[0099]

[0100] in This indicates convolution using kernel W, where σ is the softmax function, and T... i It is obtained by weighted calculation of attention map and feature map.

[0101] 3-2. Concatenate the two semantic tags to obtain a joint feature map T. Input T into the Encoder for encoding. The specific encoding operation is as follows: apply a pre-normalized residual unit (PreNorm) to T for layer normalization, and then input it into a multi-head self-attention (MSA) function to calculate self-attention (query Q, key K, value V). Then, calculate the attention weights Att(Q, K, V) using the following formula:

[0102] Q = T (l-1) W q

[0103] K = T (l-1) W k

[0104] V = T (l-1) W v

[0105]

[0106] Where T(l-1) is the output of the previous layer, and W is the learnable parameter in the layer.

[0107] The above is the formula for calculating a single attention head. Therefore, MSA can be expressed by the following formula:

[0108] MSA(T (l-1)= Concat(head1,...,head) n W o

[0109] head j =Att(T) (l-1) W j q ,T (l-1) W j k ,T (l-1) W j v )

[0110] The MLP is a multilayer perceptron. An MLP block consists of two linear transformation layers, with a Gaussian error linear unit (GELU) activation function used between them. The MLP is used to perform a non-linear transformation on the output of the MSA, increasing the model's expressive and fitting capabilities. The formula is as follows:

[0111] MLP(T (l-1) )=ELU(T (l-1) W1)W2

[0112] Therefore, the refined feature map T is output. new After dividing it, we can get T. 1new T 2new

[0113] 3-3 For the T output in the Encoder 1new T 2new Decoding operations are performed separately. The decoder consists of an ND layer multi-head cross-attention (MA) layer and an MLP block, utilizing the input feature maps X1 and X2 extracted in step two as Q (query), and using the context-rich semantic tag set Tinew as K (key) and V (value). Attention weights Att are obtained by calculating the similarity between the query and the key, and then the values ​​are weighted and summed using these attention weights. This can be expressed by the formula:

[0114] MSA(X i,(l-1) ,T new = Concat(head1,...,head) n W o

[0115]

[0116] Step 4: Obtained by the Decoder The feature representations of the two images are subtracted pixel-by-pixel (i.e., point-to-point subtraction or element-level difference operation) to obtain the feature difference value at each corresponding pixel location. This feature difference value is then applied to the original image to obtain the transformation image. The specific formula is as follows:

[0117] V(x,y)=[T(x±d,x±d)-S(x,y)]^2

[0118] Where V(x,y) represents the value of each pixel in the transformation image, T(x±d,x±d) represents the pixel in the second image at the same position as the first image, S(x,y) represents the pixel at position (x,y) in the first image, and d is the radius of the sliding window.

Claims

1. A dual-phase bias Transformer-based remote sensing image change detection method, characterized in that, Comprising the following steps: Step 1: color difference equalization and stretching-based preprocessing is performed on two remote sensing images at different times; Step 2: the preprocessed remote sensing images are input into a deep bias feature extraction module to obtain feature maps X1 and X2; the deep bias feature extraction module comprises feature extraction, feature selection, branch output and feature fusion operations; The step 2 is specifically: Step 2-1: the input image is divided into several overlapping regions of the same size, and they are flattened into a matrix form; the formula of the feature extraction module is: wherein, and denote input and output sequences of Transformer feature vectors, SABdenotes a self-attention block, denotes a fully connected layer mapping, denotes an element-wise product of vectors, and Norm denotes a normalization layer. Step 2-2: the entropy of each feature is calculated by the entropy formula: wherein represents the i-th dimension of the feature vector, k represents the number of values of the dimension, represents the probability; Then, the feature selection module calculates the usefulness of each feature to the image processing task by the information gain formula: Where Y represents the task, H(Y) represents the entropy of the task, and H(Y|Xi) represents the conditional entropy of the task given the i-th feature; Step 2-3: the output of the branch network includes classification labels, positioning boxes and segmentation masks, which are used to guide subsequent task processing; Step 2-4: the features obtained by different branches are fused to generate the final feature vector to support the processing of various tasks, and the feature fusion formula is as follows: ； Step 3: input the feature map X i , i = 1, 2 into a dual-time feature joint enhancement module; the dual-time feature joint enhancement module obtains a spatial attention map through a softmax function, and performs weighted averaging and calculation to obtain semantic label sets T1 and T2; the semantic label sets T1 and T2 are connected to obtain a joint feature map T, which is input into an encoder of the enhancement module, segmented, and then decoded respectively to obtain refined feature maps X 1new , X 2new ; Step 4: X 1new , X 2new Subtracting obtains a feature difference map, and the feature difference map is input into a shallow CNN network to obtain a change map at a pixel level.

2. The dual temporal bias Transformer-based remote sensing image change detection method according to claim 1, wherein, The step 1 is specifically: Step 1-1: color difference equalization; Calculate the histogram of the original remote sensing image, equalize the histogram to obtain the equalized histogram, and construct a mapping function to map the original remote sensing image to the range of the equalized histogram, as follows: wherein is the equalized image, T is a mapping function, h(f(x,y)) is a histogram of the original image f(x,y); Step 1-2: stretching; Calculate the mean and standard deviation of the pixel values of the original remote sensing image, and perform linear transformation on the pixel values of the original remote sensing image, as follows: wherein, (x,y) is the stretched image, g(x,y) is the original image, and are the minimum and maximum of the original image pixel values, and are the minimum and maximum of the stretched pixel value range.

3. The dual temporal bias Transformer-based remote sensing image change detection method according to claim 1, wherein, The step 3 is specifically: Step 3-1: for each pixel X on the feature map X i , use point-wise convolution to obtain L semantic groups, each group representing a semantic concept; then perform softmax function operation on the Height and Width dimensions of each semantic group to calculate the spatial attention map and use the attention map to calculate the weighted average and of the pixels to obtain semantic labels T1, T2; i Step 3-1: for each pixel X on the feature map X i , use point-wise convolution to obtain L semantic groups, each group representing a semantic concept; then perform softmax function operation on the Height and Width dimensions of each semantic group to calculate the spatial attention map and use the attention map to calculate the weighted average and of the pixels to obtain semantic labels T1, T2; i Step 3-1: for each pixel X on the feature map X i , use point-wise convolution to obtain L semantic groups, Step 3-2: Concatenate the two semantic labels to obtain a joint feature map T, which is input into the encoder to encode the context information into the semantic label set T and generate the semantic label set T new ; Step 3-3: T new is split into T 1new , T 2new , T 1new and T 2new . After that, T 1new and T 2new are decoded respectively, and finally the refined feature maps X 1new and X 2new are obtained.

4. The dual temporal bias Transformer-based remote sensing image change detection method of claim 1, wherein, The step 4 is specifically: Will , The feature representations of the two images are subtracted pixel by pixel to obtain the feature difference value at each corresponding pixel location. This feature difference value is then applied to the original image to obtain the transformation image. The specific formula is as follows: wherein denotes the value of each pixel on the variation map, denotes the pixel of the second image at the same position as the pixel of the first image, S(x,y) denotes the pixel of the first image at position (x,y), and d is the radius of the sliding window.

5. The dual-phase offset Transformer-based remote sensing image change detection method of claim 3, wherein, The specific operation of the encoding is: The joint feature map T is input into the layer normalization after using the front normalization residual unit, and then it is input into the multi-head self-attention for self-attention calculation to obtain query Q, key K and value V, and then the attention weight Att(Q, K, V) is calculated, and the calculation formula is as follows: where T (l-1) is the output of the previous layer, and W is the learnable parameter in the layer.

Images