Remote sensing image change detection method and system based on frequency-driven Transform

Abstract

The invention discloses a remote sensing image change detection method and system based on frequency-driven Transform in the technical field of image processing. On an overall framework, the network firstly performs feature extraction to obtain primary features and deep features, and then acquires context marks with spatio-temporal information from the primary features through a Transform coding and decoding network to guide the deep features to perform subsequent processing. In a Transform coding part, a module named as frequency-driven attention is provided, and the module is mainly composed of a low-frequency attention branch, a high-frequency attention branch and a local window self-attention part. Wherein the low-frequency attention is used for capturing the global dependency relationship of the features, the high-frequency attention is used for processing important high-frequency information, and the local window self-attention is used for supplementing detail loss of local information. In the Transform decoding part, the context information obtained from the encoder is integrated into the deep features by using an interactive attention module. In addition, the invention further provides an edge enhancement module and a gating channel exchange operation, the edge enhancement module utilizes a Sobel operator to enhance boundary features, and the gating channel exchange operation obtains richer view angle information through exchange channels. According to the method, the performance of the network can be improved, and the change detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image change detection method and system based on frequency-driven Transformer. Background Art

[0002] Change detection in remote sensing images is to quantitatively analyze and determine the characteristics and processes of surface changes from remote sensing data of different periods. Due to different imaging conditions and the complex and diverse characteristics of objects in the scene, high-resolution remote sensing images may show different spectral characteristics with the same semantic concept at different time and spatial positions, which still poses great challenges to the change detection task. In order to accurately identify the changed areas in complex scenes, the change detection model needs to identify the high-level semantic information of the changed areas in the scene and distinguish the real changes from the complex unrelated changed areas.

[0003] By using self-attention to model the semantic relationship between pixel pairs in space and time, better detection results can be achieved. In 2020, Hao Chen et al. from Beihang University proposed STANet, a new twin network based on metric and spatiotemporal attention. It divides the network into different scales and embeds self-attention in each small area to model long-distance dependencies between pixels. It uses batch balanced contrast loss function to alleviate the problem of sample imbalance and help the network extract features rich in contextual information. However, the computational efficiency of the model is very low, the computational complexity is very high, and it grows quadratically with the number of pixels.

[0004] At present, many change detection methods based on deep convolutional neural networks (CNN) cannot effectively capture the long-distance dependencies between pixels. In this context, Hao Chen et al. from the Beijing University of Aeronautics and Astronautics proposed a bitemporal image transformer (BIT) detection algorithm, which achieved good results in change detection tasks. However, since each head of the multi-head attention in the transformer is only responsible for a subset of the input token sequence, when the channel dimension of the subset is small, the dot product of the query and key cannot form an information function, which will lead to reduced network performance. At the same time, BIT uses a direct subtraction feature fusion method that not only easily destroys the feature structure, but also may cause noise interference. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a remote sensing image change detection method and system based on frequency-driven Transformer, so that the final change detection result of dual-phase multispectral remote sensing image is more accurate.

[0006] To achieve the above object, the technical solution adopted by the present invention is: a remote sensing image change detection method and system based on frequency-driven Transformer, comprising:

[0007] a. Satellite imagery pre and I post First, feature extraction is performed using the modified ResNet-34 network and edge enhancement module to obtain their corresponding deep features (F d1 and F d2 ) and primary features (F p1 and F p2 ), and the deep feature F d1 and F d2 Connect In addition, in the last stage of ResNet-34, we adopt a gated channel swap operation to obtain richer perspective information.

[0008] b. Use a pair of twin Transformer encoders to and Processing is performed to complete the modeling of the dual-time information in order to obtain the sequence S 1 and S 2 , and concatenate the two to get the sequence S.

[0009] c. Use a Transformer decoder to integrate the context information contained in the sequence S into the deep features In order to obtain the processed features

[0010] d. Use a lightweight classifier based on digital sampling to classify features Upsampling is performed to restore the image resolution to its original size, thereby obtaining the final semantic binary change map M.

[0011] e. Throughout the training process, we used a hybrid loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. By updating the loss function with gradients, we can obtain the optimal change detection model after multiple rounds of training.

[0012] The ResNet-34 in step a is a modified ResNet-34 without fully connected layers for efficient feature extraction. It is mainly composed of 4 residual stages, each with 3, 4, 6 and 3 residual blocks. After each residual stage, we use a 1×1 convolution layer and concatenate the features of these four stages to obtain the required deep features F. d .

[0013] Recent studies have shown that swapping channels is feasible for change detection tasks and can effectively handle the problem of building height displacement caused by perspective shift. In addition, in the deep layers of the network, there are lower spatial resolutions and higher channel dimensions, which are more suitable for channel interaction. Therefore, we adopt a new type of gated channel swapping operation in the last residual stage of ResNet-34 to obtain richer perspective information.

[0014] First, we input the two-stream feature X 1 and X 2 Perform channel swapping in the channel dimension at an equal interval of 1:1 to obtain the swapped feature X 12 and X 21 Then, the swapped features X 12 and X 21 Respectively with the original input feature X 1 and X 2 The output is obtained by adding a gated operation and performing nonlinear processing through a 1×1 convolution layer. and The specific operations are as follows:

[0015] X 12 ,X 21 =CE(X 1 , X 2 ) (1)

[0016]

[0017] Among them, δ represents a learnable gating coefficient, which ranges from 0 to 1 and is used to control the optimal ratio of original features to exchange features.

[0018] We believe that by exchanging channels, the model can obtain richer perspective information and extract more discriminative features. This feature representation can more accurately capture the details and features of the changed area, thereby improving the accuracy and robustness of remote sensing image change detection tasks.

[0019] The edge enhancement module in step a can effectively utilize boundary information, help improve the accuracy of change detection, and reduce the problem of false detection caused by blurred boundaries. The overall steps are as follows:

[0020] First, we perform a Sobel filter operation on the initial feature F to enhance the edge features, that is, send the feature into an isotropic Sobel operator with a convolution kernel size of 3×3, extract the high-frequency information of the feature map from two different directions, calculate its gradient and normalize the gradient to obtain the Sobel coefficient η. The calculation formula of this coefficient is as follows:

[0021]

[0022] Among them, Conv 1 and Conv 2 Represents two convolutions using the Sobel operator.

[0023] After multiplying η with the original feature, we pass the multiplied feature through a BN layer and a ReLU activation function for nonlinear processing. Next, the processed tensor passes through a 3x3 convolution layer and is added to the initial feature F to achieve a residual connection, thereby improving the stability of the network during backpropagation training. Finally, a 1×1 convolution is performed to compress the information of multiple channels into a single channel to obtain the primary feature F. p The specific operations are as follows:

[0024] F p =Conv 1×1 (F+Conv 3×3 (ReLU(BN(ηF)))) (4)

[0025] The core of the twin Transformer encoder in step b is a frequency-driven attention module proposed in the present invention, which improves the standard self-attention. The module mainly consists of three parts: low-frequency attention (LFA) branch, high-frequency attention (HFA) branch and local window self-attention. Among them, LFA is used to capture the global dependency of features, HFA is used to process high-frequency important information, and local window self-attention is used to supplement the detail loss of local information.

[0026] In order to let the network focus on low-frequency information and high-frequency information respectively, we cleverly adopted two methods: average pooling and Laplacian filtering. Among them, the former is used for LFA and the latter is used for HFA. In average pooling, the pixel values ​​within each pooling window are averaged and then used as the output value. This process can be regarded as applying a smoothing filter to the image, which reduces the high-frequency information in the image by taking the average of the surrounding pixels, making the image smoother. Therefore, we can regard the average pooling operation as a low-pass filter. Laplacian filtering is an image processing technique used to extract high-frequency information.

[0027] In LFA, we first use a 2×2 average pooling layer to pool the input to achieve low-pass filtering, and then project the pooled features onto K according to equation (5). a and V a , project the feature map of the original input to Q. Finally, apply the standard MSA operation to capture the low-frequency information in the feature map. The specific calculation process of LFA is as follows:

[0028]

[0029] In HFA, we first use Laplace filtering to perform high-frequency filtering on the input to extract high-frequency information while reducing complexity, and then project the filtered features onto K according to equation (5). l and V l , project the feature map of the original input to Q, and also take the standard MSA operation to capture high-frequency features. The specific calculation process of HFA is as follows:

[0030]

[0031] Furthermore, to complement the local detail loss, we also capture fine-grained local information by using local window self-attention with a window size of 2 × 2. Therefore, after adding the outputs of LFA and HFA, we concatenate it with the output of local window self-attention to obtain the output of FCAM.

[0032] In addition, we also adopt a head split operation for FDAM. If we simply assign these attentions to the same number of heads as the standard multi-head attention (MSA) layer, the number of heads will be doubled, resulting in higher computational cost. Therefore, in order to obtain better computational efficiency, we divide the heads into three groups in MSA with a split ratio of μ, where μN heads are used for HFA and LFA, and the other (1-μ)N heads are used for local window self-attention. Moreover, both the average pooling operation adopted in HFA and the local window self-attention have reduced computational complexity compared with the standard MSA operation. Therefore, low complexity is ensured throughout the FDAM process and high throughput on the GPU is guaranteed.

[0033] In summary, the calculation process of the entire FDAM process is as follows:

[0034]

[0035] In this process, we ensure that attention has the ability to model global information while also ensuring that high-frequency information and local details are not ignored, and balance performance and efficiency, thereby solving the problem of insensitivity to local information that often occurs in Transformer networks.

[0036] The core of the Transformer decoder in step c is an interactive attention proposed by the present invention, which is used to integrate the context information in the sequence S into the deep feature Specifically, Q i It is a feature The linear transformation is generated, K i and V i It is generated by linear transformation of sequence S and standard MSA calculation. The specific operation of linear transformation is as follows:

[0037]

[0038] in, and Respectively represent the functions used to obtain Q i , K i and V i The weights of the linear layer.

[0039] Through this operation, we have cleverly realized the information interaction between CNN and Transformer. Compared with simply connecting CNN and Transformer in series, we believe that the cross-temporal and spatial information interaction mode adopted by the present invention is more conducive to giving full play to the respective advantages of CNN and Transformer, thereby processing more complex CD tasks.

[0040] The classifier used in step d, specifically, needs to Decode to original image I pre , I post ∈R 3×H×W Pixel space (H and h represent their respective heights, W and w represent their respective widths). Assuming H = W and h = w, we will As the upsampling scale, the final change map can be obtained. Therefore, the classifier first uses a 1×1 convolution layer with an output channel size of 2×k×k to obtain the reshaped feature Then, through two-dimensional transformation, we can Convert to I pre and I post The predicted image M has the same size.

[0041] The hybrid loss function (Dice loss function and cross entropy loss function) used in step e is as follows:

[0042] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (9)

[0043] Among them, pred represents the predicted value and gt represents the true value.

[0044] A remote sensing image change detection method and system based on frequency-driven Transformer, comprising a processor and a storage device, wherein the storage device stores a plurality of instructions for the processor to load and execute the steps of any one of the above methods.

[0045] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the remote sensing image change detection method and system based on frequency-driven Transformer described in the present invention, in the overall framework, the network first performs feature extraction to obtain primary features and deep features, and then obtains context tags with spatiotemporal information from these primary features through the Transformer encoding and decoding network to guide the subsequent processing of deep features. In the Transformer encoding part, we propose a module called frequency-driven attention, which is mainly composed of three parts: low-frequency attention branch, high-frequency attention branch and local window self-attention. Among them, low-frequency attention is used to capture the global dependency of features, high-frequency attention is used to process high-frequency important information, and local window self-attention is used to supplement the detail loss of local information. In the Transformer decoding part, we integrate the context information obtained from the encoder into the deep features by using an interactive attention module. In addition, we also propose an edge enhancement module and a gated channel exchange operation, the former uses the Sobel operator to strengthen the boundary features, and the latter obtains richer perspective information by exchanging channels. This method makes the final change detection results of dual-phase multispectral remote sensing images more reliable and robust, and the detection accuracy is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the implementation process of a remote sensing image change detection method based on a frequency-driven Transformer according to an embodiment of the present invention;

[0047] Figure 2 is a gate-controlled channel exchange (GCCE) of a remote sensing image change detection method based on a frequency-driven Transformer in an embodiment of the present invention, wherein Figure 4 (a) and (b) represent gated channel exchange operation and channel exchange operation, respectively;

[0048] Figure 3 It is an edge enhancement module (EEM) of a remote sensing image change detection method based on a frequency-driven Transformer in an embodiment of the present invention;

[0049] Figure 4 is a Transformer encoding and decoding module of a remote sensing image change detection method based on frequency-driven Transformer in an embodiment of the present invention, wherein Figure 4(a), (b), and (c) represent the frequency-driven Transformer encoder, the frequency-driven attention module (FDAM), and the Transformer decoder, respectively;

[0050] Figure 5 is the LEVER optical remote sensing image change data set and detection results used in the embodiment of the present invention, wherein Figure 4 (a), (b), (c), (d), and (e) respectively represent the image before the change, the image after the change, the change reference image, the STANet detection result, the BIT detection result, and the detection result of the present invention. DETAILED DESCRIPTION

[0051] A remote sensing image change detection method based on frequency-driven Transformer includes: an overall framework, frequency-driven attention, interactive attention, an edge enhancement module and gated channel exchange. In the overall framework, the network first extracts features to obtain primary features and deep features, and then obtains contextual tags with spatiotemporal information from these primary features through the Transformer encoder-decoder network to guide the subsequent processing of deep features. In the Transformer encoding part, we propose a module called frequency-driven attention, which is mainly composed of three parts: low-frequency attention branch, high-frequency attention branch and local window self-attention. Among them, low-frequency attention is used to capture the global dependency of features, high-frequency attention is used to process high-frequency important information, and local window self-attention is used to supplement the detail loss of local information. In the Transformer decoding part, we integrate the context information obtained from the encoder into the deep features by using an interactive attention module. Finally, through the classifier, the final semantic binary change map can be obtained. In addition, we also propose an edge enhancement module and a gated channel exchange operation. The former uses the Sobel operator to strengthen the boundary features, and the latter exchanges channels to obtain richer perspective information.

[0052] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0053] like Figure 1 FIG. 1 is a schematic diagram of the implementation process of remote sensing image change detection based on frequency-driven Transformer according to the present invention, which includes the following steps:

[0054] Step 1: Satellite imagery pre and I postFirst, feature extraction is performed using the modified ResNet-34 network and edge enhancement module to obtain their corresponding deep features (F d1 and F d2 ) and primary features (F p1 and F p2 ), and the deep feature F d1 and F d2 Connect In addition, in the last stage of ResNet-34, we adopt a gated channel swap operation to obtain richer perspective information.

[0055] The ResNet-34 here is a modified ResNet-34 without fully connected layers for efficient feature extraction. It is mainly composed of 4 residual stages, each with 3, 4, 6, and 3 residual blocks. After each residual stage, we use a 1×1 convolutional layer and concatenate the features of these four stages to get the required deep features F. d In addition, in the last residual stage of ResNet-34, we adopt a new type of gated channel exchange operation, the specific structure is as follows Figure 2 shown.

[0056] First, we input the two-stream feature X 1 and X 2 Perform channel swapping in the channel dimension at an equal interval of 1:1 to obtain the swapped feature X 12 and X 21 Then, the swapped features X 12 and X 21 Respectively with the original input feature X 1 and X 2 The output is obtained by adding a gated operation and performing nonlinear processing through a 1×1 convolution layer. and The specific operations are as follows:

[0057] X 12 , X 21 =CE(X 1 , X 2 ) (1)

[0058]

[0059] Among them, δ represents a learnable gating coefficient, which ranges from 0 to 1 and is used to control the optimal ratio of original features to exchange features.

[0060] We believe that by exchanging channels, the model can obtain richer perspective information and extract more discriminative features. This feature representation can more accurately capture the details and features of the changed area, thereby improving the accuracy and robustness of remote sensing image change detection tasks.

[0061] In addition, the edge enhancement module mentioned in step 1 has a specific structure as follows Figure 3 As shown, its overall steps are as follows:

[0062] First, we perform a Sobel filter operation on the initial feature F to enhance the edge features, that is, send the feature into an isotropic Sobel operator with a convolution kernel size of 3×3, extract the high-frequency information of the feature map from two different directions, calculate its gradient and normalize the gradient to obtain the Sobel coefficient η. The calculation formula of this coefficient is as follows:

[0063]

[0064] Among them, Conv 1 and Conv 2 Represents two convolutions using the Sobel operator.

[0065] After multiplying η with the original feature, we pass the multiplied feature through a BN layer and a ReLU activation function for nonlinear processing. Next, the processed tensor passes through a 3x3 convolution layer and is added to the initial feature F to achieve a residual connection, thereby improving the stability of the network during backpropagation training. Finally, a 1×1 convolution is performed to compress the information of multiple channels into a single channel to obtain the primary feature F. p The specific operations are as follows:

[0066] F p =Conv 1×1 (F+Conv 3×3 (ReLU(BN(ηF)))) (4)

[0067] Step 2: After obtaining the primary feature F p1 and F p2 After that, we start to encode and decode the dual-stream features. The frequency-driven Transformer encoding and decoding module proposed in this paper is as follows Figure 4 As shown. Figure 4As shown in (a), the frequency-driven Transformer encoder we proposed is mainly composed of a frequency-driven attention module (Frequency-driven Attention Module, FDAM), a feedforward network (Feedforward Network, FFN) and a normalization layer (Norm). After passing through the two-stream encoder, we can get the sequence S 1 and S 2 , and concatenate the two to obtain sequence S, so that At the same time, we obtain the spatiotemporal embedding for context decoding.

[0068] Among them, the frequency-driven attention module is the core of this step, and its specific structure is as follows: Figure 4 (b) shows an improvement on the standard self-attention. The standard self-attention is shown below:

[0069]

[0070] Where x∈R C×H×W Represents input; and Denote the weights of the linear layers used to obtain Q, K, and V respectively; d h Indicates the number of hidden dimensions of the header.

[0071] For N h The final output of the multi-head attention (MSA) layer is calculated by linearly projecting the output of each self-attention head. The specific calculation formula is as follows:

[0072]

[0073] Among them, w o Represents the linear transformation matrix used for projection.

[0074] However, existing research has shown that standard self-attention is not optimal in processing global and local information. Therefore, this paper proposes a frequency-driven attention module (FDAM) to replace the standard self-attention operation. The module mainly consists of three parts: low-frequency attention (LFA) branch, high-frequency attention (HFA) branch and local window self-attention. Among them, LFA is used to capture the global dependency of features, HFA is used to process high-frequency important information, and local window self-attention is used to supplement the detail loss of local information.

[0075] In order to let the network focus on low-frequency information and high-frequency information respectively, we cleverly adopted two methods: average pooling and Laplacian filtering. Among them, the former is used for LFA and the latter is used for HFA. In average pooling, the pixel values ​​within each pooling window are averaged and then used as the output value. This process can be regarded as applying a smoothing filter to the image, which reduces the high-frequency information in the image by taking the average of the surrounding pixels, making the image smoother. Therefore, we can regard the average pooling operation as a low-pass filter. Laplacian filtering is an image processing technique used to extract high-frequency information.

[0076] In LFA, we first use a 2×2 average pooling layer to pool the input to achieve low-pass filtering, and then project the pooled features onto K according to equation (5). a and V a , project the feature map of the original input to Q. Finally, apply the standard MSA operation to capture the low-frequency information in the feature map. The specific calculation process of LFA is as follows:

[0077]

[0078] In HFA, we first use Laplace filtering to perform high-frequency filtering on the input to extract high-frequency information while reducing complexity, and then project the filtered features onto K l and V l , project the feature map of the original input to Q, and also take the standard MSA operation to capture high-frequency features. The specific calculation process of HFA is as follows:

[0079]

[0080] Furthermore, to complement the local detail loss, we also capture fine-grained local information by using local window self-attention with a window size of 2 × 2. Therefore, after adding the outputs of LFA and HFA, we concatenate it with the output of local window self-attention to obtain the output of FCAM.

[0081] In addition, we also adopt a head split operation for FDAM. If we simply assign these attentions to the same number of heads as the standard multi-head attention (MSA) layer, the number of heads will be doubled, resulting in higher computational cost. Therefore, in order to obtain better computational efficiency, we divide the heads into three groups in MSA with a split ratio of μ, where μN heads are used for HFA and LFA, and the other (1-μ)N heads are used for local window self-attention. Moreover, both the average pooling operation adopted in HFA and the local window self-attention have reduced computational complexity compared with the standard MSA operation. Therefore, low complexity is ensured throughout the FDAM process and high throughput on the GPU is guaranteed.

[0082] In summary, the calculation process of the entire FDAM process is as follows:

[0083]

[0084] In this process, we ensure that attention has the ability to model global information while also ensuring that high-frequency information and local details are not ignored, and balance performance and efficiency, thereby solving the problem of insensitivity to local information that often occurs in Transformer networks.

[0085] Step 3: Perform a decoding operation to integrate the context information contained in the sequence S into the deep features In order to obtain the processed features like Figure 4 As shown in (c), the Transformer decoder we proposed is mainly composed of an interactive attention module (IAM), a feedforward network and a normalization layer.

[0086] The main block in the Transformer decoder is the IAM, which is used to integrate the contextual information in the sequence S into the deep features Specifically, Q i It is a feature The linear transformation is generated, K i and V i It is generated by linear transformation of sequence S and standard MSA calculation. The specific operation of linear transformation is as follows:

[0087]

[0088] in, and Respectively represent the functions used to obtain Q i , K i and V iThe weights of the linear layer.

[0089] Step 4: Use a classifier to classify the features Upsample to the original size to obtain the change map M.

[0090] The classifier needs to Decode to original image I pre , I post ∈R 3×H×W Pixel space (H and h represent their respective heights, W and w represent their respective widths). Assuming H = W and h = w, we will As the upsampling scale, the final change map can be obtained. Therefore, the classifier first uses a 1×1 convolution layer with an output channel size of 2×k×k to obtain the reshaped feature Then, through two-dimensional transformation, we can Convert to I pre and I post The predicted image M has the same size.

[0091] Step 5: During the entire training process, we used a mixed loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. The specific formula is as follows:

[0092] L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (12)

[0093] Among them, pred represents the predicted value and gt represents the true value.

[0094] The Dice loss function is calculated as follows:

[0095]

[0096] Among them, t n represents the true label category of pixel n. When n is a pixel of the same class, t n The value is 0. When n is a change pixel, t n The value is 1; n represents the predicted probability that pixel n belongs to the change class.

[0097] The cross entropy loss function is calculated as follows:

[0098]

[0099] Among them, t m represents the true label category of pixel m, y m represents the predicted probability that pixel m belongs to the change class.

[0100] The present invention is further described in detail below in conjunction with specific experimental data. The experimental data used in this experiment is LEVER change detection data, and the image size is cropped to 256×256. The experimental platform is Win11 system, the graphics card is NVIDIAGeForce RTX 3060, the deep learning framework used is PyTorch framework, and the cuDNN library and CUDA computing architecture are used for training acceleration. During the training process, data enhancement techniques were applied, including Gaussian blur, random scaling, random flipping, random cropping, and random color transformation, and a weighted cross entropy loss function was used. In the experiment, the batch size was set to 4, the initial learning rate was set to 0.0001, the epoch was set to 300, the head segmentation coefficient μ was set to 0.5, and a linear decay operation was used to reduce the learning rate until it dropped to 0 in the last period.

[0101] In order to verify the effectiveness of the present invention, the change detection method of the present invention is compared with the following change detection methods:

[0102] (1) STANet detection method proposed by Hao Chen et al. [“A Spatial-Temporal Attention-Based Method and a New Dataset for Remote Sensing Image Change Detection.” (Remote Sensing, 2020, 12(10)]

[0103] (2) BIT detection method (GAN) proposed by Hao Chen et al. [Hao Chen et al. Detection method proposed in the article "Remotesensing image change detection with transformers. (IEEE Transactions on Geoscience and Remote Sensing, 2022, 60]

[0104] (3) The method of the present invention.

[0105] The detection performance is measured using precision, recall, and F1 score indicators. F1 is a comprehensive evaluation indicator for change detection. The closer its value is to 1, the better the performance of the change detection method. The detection results are shown in Table 1.

[0106] Table 1. Change detection results performance on LEVIR dataset (%)

[0107]

[0108] As can be seen from Table 1, the F1 score of the detection method proposed in the present invention is the largest, which is closer to 1 than the other three detection algorithms. In addition, the Precision and Recall of the present invention are the largest among the comparison algorithms and are closer to 1. In summary, the performance of the change detection algorithm of the present invention is better than the other three detection methods, which shows that the change detection method proposed in the present invention is effective.

[0109] Figure 5 (a) and (b) are the before and after phase images in the LEVER dataset. Figure 4 (c) is the reference image for change detection. Figure 5 (d), (e), and (f) are the change detection results of the STANet algorithm, the BIT algorithm, and the present invention, respectively. Figure 5 Compared with the reference image, the detection effect of the algorithm proposed in the present invention is the best in terms of visual effect.

[0110] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0112] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A remote sensing image change detection method based on frequency-driven Transformer, characterized in that: include: a. Satellite imagery pre and I post First, feature extraction is performed using the modified ResNet-34 network and edge enhancement module to obtain their corresponding deep features (F d1 and F d2 ) and primary features (F p1 and F p2 ), and the deep feature F d1 and F d2 Connect In addition, in the last stage of ResNet-34, we adopt a gated channel swap operation to obtain richer perspective information. b. Use a pair of twin Transformer encoders to and Processing is performed to complete the modeling of the dual-time information to obtain sequences S1 and S2, and the two are concatenated to obtain sequence S. The encoder is mainly composed of a frequency-driven attention module (Frequency - It consists of a driven Attention Module (FDAM), a feedforward network (FFN) and a normalization layer (Norm). c. Use a Transformer decoder to obtain the processed features The core of the decoder is an interactive attention proposed in this paper, which is used to integrate the contextual information in the sequence S into the deep feature Go in. d. Use a classifier to classify features Upsample to the original size to obtain the change map M. e. Throughout the training process, we used a hybrid loss function (Dice loss function and cross entropy loss function) to weaken the impact of unbalanced samples. By updating the loss function with gradients, we can obtain the optimal change detection model after multiple rounds of training.

2. According to claim 1, the modified ResNet-34 in step a has no fully connected layers for efficient feature extraction. It is mainly composed of 4 residual stages, each of which has 3, 4, 6 and 3 residual blocks respectively. After each residual stage, we use a 1×1 convolution layer respectively, and concatenate the features of these four stages to obtain the required deep features F d .

3. According to claim 1, the gated channel exchange operation in step a can be used to obtain richer viewing angle information. First, we perform channel swapping on the input dual-stream features X1 and X2 in the channel dimension at an evenly spaced 1:1 ratio to obtain the swapped features X 12 and X 21 Then, the swapped features X 12 and X 21 They are added to the original input features X1 and X2 through a gating operation, and then processed nonlinearly through a 1×1 convolution layer to obtain the output and The specific operations are as follows: X 12 ,X 21 =CE(X1,X2) (1) in, δ represents a learnable gating coefficient, which ranges from 0 to 1 and is used to control the optimal ratio of original features to exchange features. We believe that by exchanging channels, the model can obtain richer perspective information and extract more discriminative features. This feature representation can more accurately capture the details and features of the changed area, thereby improving the accuracy and robustness of remote sensing image change detection tasks.

4. According to claim 1, the edge enhancement module in step a can effectively utilize boundary information, help improve the accuracy of change detection, and reduce the problem of false detection caused by blurred boundaries. The overall steps are as follows: First, we perform a Sobel filter operation on the initial feature F to enhance the edge features, that is, send the feature into an isotropic Sobel operator with a convolution kernel size of 3×3, extract the high-frequency information of the feature map from two different directions, calculate its gradient and normalize the gradient to obtain the Sobel coefficient η. The calculation formula of this coefficient is as follows: in, Conv1 and Conv2 represent two convolutions using the Sobel operator. After multiplying η with the original feature, we pass the multiplied feature through a BN layer and a ReLU activation function for nonlinear processing. Next, the processed tensor passes through a 3x3 convolution layer and is added to the initial feature F to achieve a residual connection, thereby improving the stability of the network during backpropagation training. Finally, a 1×1 convolution is performed to compress the information of multiple channels into a single channel to obtain the primary feature F. p The specific operations are as follows: F p =Conv 1×1 (F+Conv 3×3 (ReLU(BN(ηF)))) (4).

5. According to claim 1, the frequency-driven attention module (FDAM) in step b improves self-attention. The module mainly consists of three parts: low-frequency attention (LFA) branch, high-frequency attention (HFA) branch and local window self-attention. Among them, LFA is used to capture the global dependency of features, HFA is used to process high-frequency important information, and local window self-attention is used to supplement the detail loss of local information. In order to let the network focus on low-frequency information and high-frequency information respectively, we cleverly adopted two methods: average pooling and Laplace filtering. The former is used for LFA and the latter is used for HFA. In average pooling, the pixel values ​​within each pooling window are averaged and then used as the output value. This process can be regarded as applying a smoothing filter to the image, which reduces the high-frequency information in the image by taking the average of the surrounding pixels, making the image smoother. Therefore, we can regard the average pooling operation as a low-pass filter. Laplace filtering is an image processing technique used to extract high-frequency information. In LFA, we first use a 2×2 average pooling layer to pool the input to achieve low-pass filtering and project the pooled features onto K a and V a , project the feature map of the original input to Q. Finally, apply the standard MSA operation to capture the low-frequency information in the feature map. The specific calculation process of LFA is as follows: In HFA, we first use Laplace filtering to perform high-frequency filtering on the input to extract high-frequency information while reducing complexity, and then project the filtered features onto K l and V l , project the feature map of the original input to Q, and also take the standard MSA operation to capture high-frequency features. The specific calculation process of HFA is as follows: Furthermore, to complement the local detail loss, we also capture fine-grained local information by using local window self-attention with a window size of 2 × 2. Therefore, after adding the outputs of LFA and HFA, we concatenate it with the output of local window self-attention to obtain the output of FCAM. In addition, we also adopt a head split operation for FDAM. If we simply assign these attentions to the same number of heads as the standard multi-head attention (MSA) layer, the number of heads will be doubled, resulting in higher computational cost. Therefore, in order to obtain better computational efficiency, we divide the heads into three groups in MSA with a split ratio of μ, where μN heads are used for HFA and LFA, and the other (1-μ)N heads are used for local window self-attention. Moreover, both the average pooling operation adopted in HFA and the local window self-attention have reduced computational complexity compared with the standard MSA operation. Therefore, low complexity is ensured throughout the FDAM process and high throughput on the GPU is guaranteed. In summary, the calculation process of the entire FDAM process is as follows: In this process, we ensure that attention has the ability to model global information while also ensuring that high-frequency information and local details are not ignored, and balance performance and efficiency, thereby solving the problem of insensitivity to local information that often occurs in Transformer networks.

6. According to claim 1, the interactive attention module in step c is used to integrate the context information in the sequence S into the deep feature Specifically, Q i It is a feature The linear transformation is generated, K i and V i It is generated by linear transformation of sequence S and standard MSA calculation. The specific operation of linear transformation is as follows: in, and Respectively represent the functions used to obtain Q i , K i and V i The weights of the linear layer.

7. According to claim 1, the classifier used in step d, specifically, it needs to Decode to original image I pre , I post ∈R 3×H×W Pixel space (H and h represent their respective heights, W and w represent their respective widths). Assuming H = W and h = w, we will As the upsampling scale, the final change map can be obtained. Therefore, the classifier first uses a 1×1 convolution layer with an output channel size of 2×k×k to obtain the reshaped feature Then, through two-dimensional transformation, we can Convert to I pre and I post The predicted image M has the same size.

8. According to claim 1, the hybrid loss function (Dice loss function and cross entropy loss function) used in step e is used for gradient updating, and the specific formula is as follows: L(pred,gt)=L dice (pred,gt)+L wce (pred,gt) (12) in, Pred represents the predicted value, and gt represents the true value.

9. A remote sensing image change detection system based on frequency-driven Transformer, characterized in that: It comprises a processor and a storage device, wherein the storage device stores a plurality of instructions for the processor to load and execute the steps of the method according to any one of claims 1 to 9.

Images