Remote sensing image change detection network and detection method based on twin branches

Abstract

The present invention discloses a remote sensing image change detection network and detection method based on dual twin branches, which mainly solves the problem of insufficient detection ability of existing technologies in remote sensing images of multiple resolutions. The detection network consists of a twin local information encoding branch, a twin global information encoding branch, a difference enhancement module, a feature fusion module and a classifier. The twin local and global information encoding branches extract multi-level dual-time local fine-grained and global context features; the difference enhancement module generates and enhances difference features; the feature fusion module fuses local, global and cross-level information and mines channel clues; the classifier classifies the final fused features and generates a change prediction map. The present invention enhances the generalization ability of the network by extracting multi-level dual-time local and global features through dual twin branches; and improves the detection accuracy of the network through the difference enhancement module and the feature fusion module, which can be used for land cover identification, urban expansion estimation and agricultural development monitoring.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a remote sensing image change detection technology, which can be used for land cover identification, urban expansion estimation, disaster assessment and forestry and agriculture monitoring. Background Art

[0002] Change detection in remote sensing images is a key area of ​​remote sensing image processing. Its goal is to detect changes of interest between two remote sensing images captured at different times over the same area. Change detection techniques can be used in a variety of remote sensing applications, including land cover identification, urban expansion estimation, disaster assessment, and forestry and agricultural monitoring. With the rapid development of aerospace sensors, the availability of remote sensing images has increased significantly. While this has advanced the field of change detection, it has also brought some significant challenges. First, due to varying environmental conditions, unavoidable non-relevant changes can negatively impact the results of change detection. These non-relevant changes typically refer to changes that are not of interest in a specific application scenario, as well as artifacts caused by imaging and seasonal factors. Second, the diversity of sensor types leads to a wide range of resolutions in the generated remote sensing images, placing higher demands on the generalization capabilities of change detection techniques. In other words, existing change detection models struggle to perform well across remote sensing images of varying scales. Therefore, the development of change detection techniques for remote sensing images is necessary and urgent.

[0003] With the rapid development of deep learning, deep learning-based methods have dominated the field of change detection. Convolutional neural networks (CNNs), with their powerful feature learning and nonlinear fitting capabilities, have achieved impressive results in change detection. However, researchers subsequently discovered that CNNs alone are insufficient for modeling contextual relationships and emphasizing important regions. Consequently, most studies have enhanced CNN-based change detection techniques by stacking more convolutional layers and embedding visual attention mechanisms. However, these methods are still not adept at modeling the global, long-range contextual dependencies hidden in remote sensing images. Consequently, recent change detection techniques have introduced transformers based on self-attention mechanisms to enhance the ability to capture global, long-range contextual dependencies in remote sensing images. Transformers excel at extracting global information and hold great potential for change detection. However, Transformer-based techniques still have room for improvement. First, Transformers are less able to focus on local, fine-grained information than CNNs, which can negatively impact the detection of fine-scale objects. Furthermore, most Transformer-based models divide the input remote sensing image into regular patches based on a rigid grid. These regular patches disrupt the continuity of complex geographical environments and the geometry of changing objects. This runs counter to the goal of change detection. Based on the above discussion, CNN and Transformer are actually two complementary methods.

[0004] Hao Chen et al. proposed a Transformer-based change detection method for remote sensing imagery in IEEE, 2021, 60:1-14. They first extract features from two time phases using a CNN backbone based on a residual network. These features are then converted into semantic tags and fed into a Transformer encoder to model long-range contextual dependencies between the features. While this method combines the strengths of CNNs and Transformers and achieves good detection performance, it ignores multi-scale information, which undoubtedly limits its performance in remote sensing imagery of varying scales.

[0005] Zheng Jianwei et al. proposed a CNN- and Transformer-based change detection method for remote sensing images in IEEE, 2022, 60:1-13. This method extracts hierarchical local and global information using CNN and Transformer. It then uses two strategies, intra-scale cross-interaction and inter-scale feature fusion, to obtain features that combine local fine-grained information with global long-range contextual dependencies. A three-layer fusion strategy is then used to obtain a change prediction map. While this method achieves good detection performance, it generates the change prediction map using only a difference method, which is insufficient for learning difference information, leaving room for further improvement. Summary of the Invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a remote sensing image change detection network and detection method based on dual twin branches to extract and integrate multi-scale features with global and local information, improve detection accuracy, and enhance generalization ability.

[0007] The technical ideas for achieving the purpose of the present invention are: improving detection accuracy through deformable ideas, difference enhancement and feature fusion; enhancing generalization ability through parallel integration of twin CNN and Transformer branches and skip connections between encoder and decoder.

[0008] According to the above technical ideas, the technical solutions of the present invention include the following:

[0009] 1. A remote sensing image change detection network based on twin branches, characterized by comprising:

[0010] The twin local information encoding branch, including several downsampling layers, several standard residual blocks and several deformable convolution units, is used to extract multi-level bi-temporal local fine-grained features.

[0011] The twin global information encoding branch, which includes several downsampling layers, several sliding window Transformer blocks, and deformable Transformer blocks, is used to extract multi-level bi-temporal global context features.

[0012] The difference enhancement module, which includes a subtraction unit, a splicing unit, and a convolution unit, is used to extract the same-level dual-time local fine-grained features from the twin local information encoding branch. Enhance difference information and generate local difference features The same-level bi-temporal global context features extracted from the twin global information encoding branch Enhance difference information and generate global difference features

[0013] The feature fusion module includes an addition unit, a channel attention unit, a multiplication unit, a convolution unit, and a splicing unit, which is used to fuse multi-level features step by step to generate the final fused features;

[0014] The classifier consists of a first 3×3 convolutional layer, a batch normalization layer, a ReLU activation function layer, a second 3×3 convolutional layer, a softmax function, and an argmax function cascaded in sequence. It is used to classify the final fused features and generate a binary change prediction map, where 0 represents unchanged pixels and 1 represents changed pixels.

[0015] Furthermore, the twin local information encoding branch includes two weight-sharing local feature extractors:

[0016] Each local feature extractor consists of a downsampling layer, four standard residual blocks, and four deformable convolution units connected in sequence, where the four standard residual blocks and four deformable convolution units divide the local feature extractor into four levels;

[0017] Each downsampling layer consists of a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer connected in sequence to reduce the spatial dimension of the input image. The stride of the 7×7 convolutional layer and the maximum pooling layer are both 2.

[0018] Each standard residual block consists of two sequentially connected basic residual units, each of which consists of two 3×3 convolutional layers, two batch normalization layers, and two ReLU activation function layers;

[0019] Each deformable convolution unit includes 1 3×3 convolution layer, 1 3×3 deformable convolution layer, 1 batch normalization layer and 1 ReLU activation function layer.

[0020] Furthermore, the twin global information encoding branch includes two weight-sharing global feature extractors:

[0021] Each global feature extractor consists of four downsampling layers, two sliding window Transformer blocks, and four deformable Transformer blocks connected sequentially, where the four downsampling layers divide the global feature extractor into four levels;

[0022] Each sliding window Transformer block consists of a sequentially connected window attention unit and a sliding window attention unit to extract shallow global context features;

[0023] Each deformable Transformer block consists of a local attention unit and a deformable global attention unit connected sequentially to extract deep global context features.

[0024] 2. A method for remote sensing image change detection using the above network, characterized in that it includes the following steps:

[0025] 1) Obtain the divided training sample set and test sample set from the optical remote sensing change detection database;

[0026] 2) Extract multi-level dual-phase local fine-grained features from dual-time images step by step through the twin local information encoding branch And input it into the difference enhancement module;

[0027] 3) Extract multi-level dual-phase global context features from dual-time images step by step through the twin global information encoding branch And input it into the difference enhancement module;

[0028] 4) Multi-level dual-time local fine-grained features and global context features are used to generate and enhance difference information using the difference enhancement module to obtain multi-level local difference enhancement features that carry change information. and global difference enhancement features And these features are layered and input into the feature fusion module;

[0029] 5) Feature fusion module enhances features of multi-level local differences and global difference enhancement features Perform fusion, interpret the change information in the dual-time remote sensing image, obtain the final fusion feature H1, and transmit it to the classifier;

[0030] 6) The classification module classifies the final fusion feature H1 to obtain a binary change prediction map M, where 0 represents unchanged pixels and 1 represents changed pixels;

[0031] 7) Using the training sample set, the change detection network is iteratively trained using back propagation to obtain a trained change detection network;

[0032] 8) Input the test sample set into the trained change detection network for forward propagation to obtain the change detection results of all test samples.

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] First, the twin local information encoding branch designed by the present invention adopts a hierarchical structure design and can extract multi-level local fine-grained features by combining its standard residual block with the deformable convolution unit.

[0035] Second, the twin global information encoding branch designed by the present invention adopts a hierarchical structure design and can extract multi-level global context features by combining its sliding window Transformer block with the deformable Transformer block.

[0036] Third, the present invention can fully interpret the rich change information in dual-time remote sensing images by combining the twin local information coding branch and the twin global information coding branch in parallel, thereby enhancing the generalization ability on remote sensing images of different scales.

[0037] Fourth, the change detection network designed by the present invention, due to the use of a difference enhancement module, can not only emphasize significant changes, but also gradually approach changes of interest in a learnable manner, and suppress the negative impact of irrelevant changes on detection accuracy; at the same time, because the network adopts a feature fusion module, it can not only integrate local, global and cross-level information, but also deeply explore inter-channel clues. The combined effect of these two modules improves the change detection accuracy of remote sensing images of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the specific implementation or the description of the prior art.

[0039] Figure 1 It is a schematic diagram of the network structure of the present invention;

[0040] Figure 2 Schematic diagram of the difference enhancement module structure in the network of the present invention;

[0041] Figure 3 It is a schematic diagram of the structure of the feature fusion module in the network of the present invention;

[0042] Figure 4 This is a flow chart of the present invention for performing remote sensing image change detection;

[0043] Figure 5 These are simulation results of change detection of remote sensing image datasets using the present invention and existing detection methods respectively. DETAILED DESCRIPTION

[0044] The examples and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0045] Reference Figure 1 , the change detection network in this example contains:

[0046] Twin local information encoding branch 1, twin global encoder branch 2, difference enhancement module 3, feature fusion module 4, classifier 5.

[0047] The twin local information encoding branch 1 includes two weight-sharing local feature extractors for extracting multi-level bi-temporal local fine-grained features. Each local feature extractor includes four downsampling layers 11, four standard residual blocks 12, and four deformable convolution units 13. The four downsampling layers divide the local feature extractor into four levels. The specific structure is: first downsampling layer → first standard residual block → first deformable convolution unit → second downsampling layer → second standard residual block → second deformable convolution unit → third downsampling layer → third standard residual block → third deformable convolution unit → fourth downsampling layer → fourth standard residual block → fourth deformable convolution unit. Wherein:

[0048] The first downsampling layer 11 includes a 7×7 convolutional layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer connected in sequence to reduce the spatial dimension of the input image. The stride of the 7×7 convolutional layer and the maximum pooling layer are both 2. The remaining downsampling layers 11 are composed of 3×3 convolutional layers with a stride of 2. In this example, the four downsampling layers 11 downsample the original image of spatial size 256×256 to 64×64, 32×32, 16×16, and 8×8, respectively.

[0049] Each standard residual block 12 includes two basic residual units connected in sequence. Each residual unit includes two 3×3 convolutional layers, two batch normalization layers, and two ReLU activation function layers. Its specific structure is: first 3×3 convolutional layer → first batch normalization layer → first ReLU activation function layer → second 3×3 convolutional layer → second batch normalization layer → second ReLU activation function layer, where the input of the first 3×3 convolutional layer is residually connected to the output of the second batch normalization layer;

[0050] Each deformable convolution unit 13 includes a 3×3 convolution layer, a 3×3 deformable convolution layer, a batch normalization layer, and a ReLU activation function layer. Its specific structure is: 3×3 convolution layer → batch normalization layer → 3×3 deformable convolution layer → ReLU activation function layer, where the input of the 3×3 convolution layer is jump-connected to the input of the 3×3 deformable convolution layer, and the input of the 3×3 convolution layer is residually connected to the output of the ReLU activation function layer;

[0051] The dimensions of the four levels of local fine-grained feature channels extracted in this example are 64, 128, 256, and 512 respectively.

[0052] The twin global information encoding branch 2 includes two weight-sharing global feature extractors for extracting multi-level bi-temporal global context features. Each global feature extractor contains four downsampling layers 21, two sliding window Transformer blocks 22, and four deformable Transformer blocks 23. The four downsampling layers divide the global feature extractor into four levels. The specific structure is: first downsampling layer → first sliding window Transformer block → second downsampling layer → second sliding window Transformer block → third downsampling layer → first deformable Transformer block → second deformable Transformer block → third deformable Transformer block → fourth downsampling layer → fourth deformable Transformer block.

[0053] Each downsampling layer 21 is composed of a patch embedding layer, and the patch embedding layer step size of the first downsampling layer is 4, and the patch embedding layer step size of the remaining downsampling layers is 2. In this example, the four downsampling layers downsample the original image with a spatial size of 256×256 to 64×64, 32×32, 16×16, and 8×8 respectively;

[0054] Each sliding window Transformer block 22 includes a window attention unit 221 and a sliding window attention unit 222 connected in sequence to extract shallow global context features. The window attention unit 221 includes two layer normalization layers, one window multi-head attention layer and one multi-layer perceptron, which is used to divide the original feature map into several windows so that the multi-head attention calculation is limited to each small window to reduce computational consumption. Its specific structure is: first layer normalization layer → window multi-head attention layer → second layer normalization layer → multi-layer perceptron, the input of the first layer normalization layer is connected to the output residual of the window multi-head attention layer, and the input of the second layer normalization layer is connected to the output residual of the multi-layer perceptron; the sliding window attention unit 222 includes two layer normalization layers, one sliding window multi-head attention layer and one multi-layer perceptron, which is used to analyze the connection between different windows and extract global features. Its structure is: first layer normalization layer → sliding window multi-head attention layer → second layer normalization layer → multi-layer perceptron, and the input of the first layer normalization layer is connected to the output residual of the sliding window multi-head attention layer, and the input of the second layer normalization layer is connected to the output residual of the multi-layer perceptron;

[0055] Each deformable Transformer block 23 includes a local attention unit 231 and a deformable global attention unit 232 connected sequentially to extract deep global context features. The local attention unit 231 includes two layer normalization layers, one window multi-head attention layer and one multi-layer perceptron, which is used to divide the original feature map into several local windows, so that the multi-head attention calculation is limited to each small block to reduce computational consumption. Its specific structure is: first layer normalization layer → local multi-head attention layer → second layer normalization layer → multi-layer perceptron, and the input of the first layer normalization layer is connected to the output residual of the local multi-head attention layer, and the input of the second layer normalization layer is connected to the output residual of the multi-layer perceptron; the deformable global attention unit 232 includes two layer normalization layers, one deformable multi-head attention layer and one multi-layer perceptron, which is used to calculate global attention and extract global features. Its specific structure is: first layer normalization layer → deformable global multi-head attention layer → second layer normalization layer → multi-layer perceptron, and the input of the first layer normalization layer is connected to the output residual of the deformable global multi-head attention layer, and the input of the second layer normalization layer is connected to the output residual of the multi-layer perceptron.

[0056] The channel dimensions of the four levels of global context features extracted in this example are 64, 128, 256, and 512 respectively.

[0057] Reference Figure 2 The difference enhancement module 3 includes a subtraction unit 31, a splicing unit 32 and a convolution unit 33, which is used to extract the same-level dual-phase local fine-grained features of the twin local information encoding branch. Enhance difference information and generate local difference enhancement features The same-level bi-temporal global context features extracted from the twin global information encoding branch Enhance difference information and generate global difference enhancement features in:

[0058] The subtraction unit 31 is used to perform a subtraction on the dual-time features extracted by the dual twin branches and take the absolute value to generate a difference feature, and transmit it to the splicing unit 32;

[0059] The splicing unit 32 is used to splice the difference features and the dual-time feature channel dimensions to enhance the difference information, generate splicing features, and transmit them to the convolution unit 33;

[0060] Convolution unit 33, which consists of 1×1 convolution layers and is used to reduce the channel dimension of the concatenated features to obtain difference-enhanced features;

[0061] In this example, local difference enhancement features and global difference enhancement features The dimensions in the four levels are 64×64×64, 128×32×32, 256×16×16, and 512×8×8.

[0062] Reference Figure 3 The feature fusion module 4 includes an addition unit 41, a channel attention unit 42, a multiplication unit 43, a convolution unit 44 and a splicing unit 45, which is used to fuse multi-level features step by step to generate the final fusion feature, wherein:

[0063] Addition unit 41, used to integrate the local difference features of the twin branches after difference enhancement and global difference features Transmitted to the channel attention unit 42;

[0064] The channel attention unit 42 is composed of an average pooling layer and a maximum pooling layer in parallel, a multi-layer perceptron layer and a sigmoid activation function layer in cascade, and is used to generate the channel attention weight S i , transmitted to the multiplication unit 43;

[0065] Multiplication unit 43 is used to enhance the local difference features after the difference is enhanced and global difference features and channel attention weight S respectively i Multiply them to obtain channel weighted features, which are transmitted to the splicing unit 45;

[0066] The convolution unit 44 is composed of a 1×1 convolution layer and is used to reduce the dimension of the fused features of the previous layer. The reduced features are transmitted to the concatenation unit 45.

[0067] Splicing unit 45, used to enhance the features of weighted local differences at the same level and global difference enhancement features After fusion with the previous level feature H i+1 Perform channel dimension splicing to complete feature fusion and generate fusion feature H i .

[0068] The classifier 5 is composed of a first 3×3 convolutional layer, a batch normalization layer, a ReLU activation function layer, a second 3×3 convolutional layer, a softmax function and an argmax function cascaded in sequence, and is used to classify the final fusion features and generate a binary change prediction map, where 0 represents unchanged pixels and 1 represents changed pixels.

[0069] Reference Figure 4 , the method of applying this example network to remote sensing image change detection is implemented as follows:

[0070] Step 1: Obtain the divided training sample set and test sample set from the optical remote sensing change detection database.

[0071] Obtain four different remote sensing image change detection datasets from the optical remote sensing change detection database. In this example, one of the datasets is selected, which contains a training sample set of 10,000 samples and a test sample set of 3,000 samples.

[0072] Each sample contains a pair of registered dual-time images and and a ground truth label Where ∈ means belonging to, represents the real number domain, H, W, and C represent the height, width, and number of channels of the image, respectively, which are 256, 256, and 3 in this example.

[0073] Step 2: Extract the multi-level dual-temporal local fine-grained features of the dual-time image in the sample step by step through the twin local information encoding branch 1 And input it to the difference enhancement module 3.

[0074] 2.1) Register the dual-time images and Input to the twin local information encoding branch 1 for local fine-grained feature extraction:

[0075] The dual-time image is downsampled through its downsampling layer 11 to reduce its spatial dimension to 64×64 and transmitted to the standard residual block 12;

[0076] The local features of the downsampled dual-time image are extracted through its standard residual block 12 and transmitted to the deformable convolution unit 13;

[0077] The deformable convolution unit 13 further learns the extracted local features and extracts local fine-grained features;

[0078] 2.2) Repeat 2.1) three times to finally obtain four levels of dual-temporal local fine-grained features Where i=1,2,3,4.

[0079] Step 3: Extract the multi-level dual-temporal global context features of the dual-time images in the sample step by step through the twin global information encoding branch 2 And input it to the difference enhancement module 3.

[0080] 3.1) Register the dual-time images and Input twin global information encoding branch 2;

[0081] 3.2) The downsampling layer 21, sliding window Transformer block 22, and deformable Transformer block 23 at each level of the twin global information encoding branch 2 extract global context features from the dual-time image step by step:

[0082] The input bi-temporal image is downsampled by the first-level downsampling layer 21 to reduce its spatial dimension to 64×64, and then the first-level global context features are extracted by the sliding window Transformer block 22 in the first level;

[0083] The extracted first-layer global context features are downsampled by the second-level downsampling layer 21 to reduce their spatial dimension to 32×32, and then the second-layer global context features are extracted by the sliding window Transformer block 22 in the second level;

[0084] The extracted second-layer global context features are downsampled by the third-level downsampling layer 21 to reduce their spatial dimension to 16×16, and then the third-layer global context features are extracted by the deformable Transformer block 23 in the third level;

[0085] The extracted third-layer global context features are downsampled by the fourth-level downsampling layer 21 to reduce their spatial dimension to 8×8, and then the fourth-layer global context features are extracted by the fourth-level deformable Transformer block 23;

[0086] Finally, four levels of dual-temporal global context features are obtained Where i=1,2,3,4.

[0087] Step 4: Multi-level dual-time local fine-grained features and global context features are used to generate and enhance difference information using the difference enhancement module 3 to obtain multi-level local difference enhancement features that carry change information. and global difference enhancement features These features are then layered and input into the feature fusion module 4.

[0088] 4.1) Input the dual-temporal local fine-grained features or dual-temporal global context features into the difference enhancement module 3, where each unit enhances the change information to generate difference enhancement features:

[0089] The subtraction unit 31 performs subtraction on the dual-time features of the layer and takes the absolute value to generate the difference feature of the layer, and transmits it to the splicing unit 32;

[0090] The difference features of the layer and the dual-time features are spliced ​​in the channel dimension by the splicing unit 32 to enhance the difference information, generate the splicing features of the layer, and transmit them to the convolution unit 33;

[0091] The convolution unit 33 is used to reduce the dimension of the splicing feature channel of this level to obtain the difference enhancement feature of this level;

[0092] 4.2) The local fine-grained features and global context features in the four levels are input into the hierarchical difference enhancement module 3 to obtain the local difference enhancement features and global difference enhancement features of the four levels.

[0093] Step 5: Feature fusion module 4 enhances features of multi-level local differences and global difference enhancement features Perform fusion and interpret the change information in the dual-time remote sensing image to obtain the final fusion feature H1, which is transmitted to the classifier 5.

[0094] 5.1) Enhance the local difference features of the 4th level and global difference enhancement features Only the four units of the addition unit 41, the channel attention unit 42, the multiplication unit 43 and the splicing unit 45 in the feature fusion module 4 of this layer are fused to obtain the fourth fused feature H4, which is transmitted to the feature fusion module 4 of the third layer;

[0095] 5.2) Enhance the local difference features of the third level and global difference enhancement features The fourth fused feature H4 after upsampling is fused through the five units of the addition unit 41, the channel attention unit 42, the multiplication unit 43, the convolution unit 44 and the splicing unit 45 in the feature fusion module 4 of this layer to obtain the third fused feature H3, which is transmitted to the feature fusion module 4 of the second layer;

[0096] 5.3) Enhance the local difference features of the second level and global difference enhancement features The up-sampled third fusion feature H3 is fused through the five units of the addition unit 41, the channel attention unit 42, the multiplication unit 43, the convolution unit 44 and the splicing unit 45 in the feature fusion module 4 of this layer to obtain the second fusion feature H2, which is transmitted to the feature fusion module 4 of the first layer;

[0097] 5.4) Enhance the local difference features of the first level and global difference enhancement features The second fused feature H2 after upsampling is fused through the five units of the addition unit 41, the channel attention unit 42, the multiplication unit 43, the convolution unit 44 and the splicing unit 45 in the feature fusion module 4 of this layer to obtain the final fused feature H1.

[0098] Step 6: Classifier 5 classifies the final fusion feature H1 and obtains the change probability prediction map of the two channels through two layers of convolution and softmax function Finally, the maximum value is declared pixel by pixel through the argmax function to obtain the binary change prediction map of the sample Among the binary values, 0 represents unchanged pixels and 1 represents changed pixels.

[0099] Step 7: Use the training sample set and back propagation to iteratively train the change detection network to obtain a trained change detection network.

[0100] 7.1) Define the overall network loss function: L = λL ce +(1-λ)L dice ,

[0101] Among them, λ represents the weight coefficient that controls the ratio of the two loss functions, 0.1≤λ≤0.9, which is 0.9 in this example.

[0102] represents the cross entropy loss function,

[0103] represents the dice loss function, N represents the total number of pixels, Y represents the ground truth, and M represents the change prediction map;

[0104] 7.2) Set the initial number of iterations to t and the maximum number of iterations to T, where T ≥ 200. In this example, T = 250 and the initial value t = 1.

[0105] 7.3) Input the dual-temporal remote sensing images from all training samples into the change detection network for forward propagation, outputting a change prediction map M corresponding to all training samples. This map, along with the corresponding ground truth values ​​Y from all training samples, is then used as the input to the network's overall loss function, which is used to calculate the loss of the change detection network.

[0106] 7.4) The loss value is back-propagated to calculate the network parameter gradient, and the network parameters are updated by gradient descent method to optimize the network. The update formula is:

[0107]

[0108] Among them, θ n represents the network parameters after the nth iteration, θ n+1 represents θ n The updated network results, γ represents the learning rate, 0<γ≤1e-2, L(θ n ) represents the overall loss value of the network after the nth iteration, Represents partial derivative calculation.

[0109] In this example, the initial learning rate γ is 0.0001 and remains unchanged in the first 125 iterations. It decays linearly in the next 125 iterations. The optimizer uses the Adam optimizer. The learning rate decays when the network iterates to 125 times to prevent the loss function from falling into the local minimum.

[0110] 7.5) Determine whether t ≥ T holds:

[0111] If so, obtain the trained change detection network;

[0112] Otherwise, set t=t+1 and return to 7.3).

[0113] Step 8: Input the test sample set into the trained change detection network for forward propagation to obtain the change detection results of all test samples in the test sample set.

[0114] The effect of the present invention can be further illustrated by the following simulation:

[0115] 1. Simulation conditions

[0116] The simulation environment of the present invention selected the framework of Python 3.7+PyTorch 1.7 and was completed on an NVIDIA Tesla V100 graphics card workstation with 32G video memory.

[0117] The four data sets used in the simulation are SVCD data set, WHU-CD data set, LEVIR-CD data set and SYSU-CD data set, among which,

[0118] The SVCD dataset consists of two sets of seasonal remote sensing image pairs collected from Google Earth. The first set contains seven pairs of images, each sized 4725 × 2700 pixels, with manually annotated objects showing changes. The second set contains four pairs of images, each sized 1900 × 1000 pixels, with minimal internal variation. The spatial resolution of the 11 image pairs in these two sets ranges from 3 cm to 100 cm per pixel. After preprocessing such as cropping and rotation, the final training and test sets contain 10,000 and 3,000 pairs of images, respectively, both sized 256 × 256 pixels.

[0119] The WHU-CD dataset consists of a pair of aerial images measuring 32,507 × 15,354 pixels with a resolution of 0.2 meters per pixel. Following common dataset segmentation rules, the original aerial images were split into non-overlapping 256 × 256 pixel images in this paper. These images were then randomly divided into 5,947 pairs for training and 744 pairs for testing.

[0120] The LEVIR dataset consists of 637 pairs of 1024×1024 pixel images obtained from Google Earth, with a spatial resolution of 0.5 meters per pixel. These images were collected from over 20 regions in Texas, USA, and contain 31,333 instances of building changes. Following a common processing method, the present invention slices the images into 256×256 pixel blocks without overlapping, constructing 7120 pairs of images for training and 2048 pairs for testing.

[0121] The SYSU dataset is a large-scale change detection dataset. It contains 20,000 pairs of 256×256 pixel images with a spatial resolution of 0.5 meters. Notably, the diverse range of changing objects in the SYSU-CD dataset, such as urban buildings, forest vegetation, and marine vessels, increases the difficulty of change detection. Consistent with common partitioning methods, 12,000 and 4,000 image pairs are used for training and testing, respectively.

[0122] 2. Simulation Content

[0123] Under the above simulation conditions, the present invention and six existing methods, DSAMNet, SNUNet, ISNet, BIT, SwinSUNet and ICIF-Net, were used to perform change detection simulations on four datasets. The numerical results are shown in Table 1. The change detection visualization results of the present invention and six existing methods on a total of eight test samples on four datasets are shown in Table 1. Figure 5 shown.

[0124] Table 1 Numerical results of the present invention and six existing methods on four data sets

[0125]

[0126] Pre, Rec, IoU, F1, and OA shown in Table 1 are numerical indicators used to quantitatively analyze the performance in the change detection task, representing precision, recall, intersection over union, F1 score, and overall accuracy, respectively.

[0127] The six existing methods in Table 1 are from the following sources:

[0128] DSAMNet is a method for remote sensing image change detection published by Shi Qian et al. in IEEE, namely: Q.Shi, M.Liu, S.Li, X.Liu, F.Wang, and L.Zhang, "A deeply supervised attention metric-based network and an open aerial image dataset for remote sensing change detection," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–16, 2022;

[0129] SNUNet is a method for change detection in remote sensing images published by Fang Sheng et al. in IEEE, namely: S. Fang, K. Li, J. Shao, and Z. Li, "SNUNet-CD: Adensely connected siamese network for change detection of vhr images," IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022;

[0130] ISNet is a method for remote sensing image change detection published by Cheng et al. in IEEE, namely: G. Cheng, G. Wang, and J. Han, "ISNet: toward improving separability for remote sensing image change detection," IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–11, 2022;

[0131] BIT is a method for remote sensing image change detection published by Chen Hao et al. in IEEE, namely: H.Chen, Z.Qi, and Z.Shi, “Remote sensing image change detection with transformers,” IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp.1–14, 2022.

[0132] SwinSUNet is a method for remote sensing image change detection published by Wang Liejun et al. in IEEE, namely: C. Zhang, L. Wang, S. Cheng, and Y. Li, “SwinSUNet: pure transformer network for remote sensing image change detection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.

[0133] ICIF-Net is a method for remote sensing image change detection published by Zheng Jianwei et al. in IEEE, namely: Y. Feng, H. Xu, J. Jiang, H. Liu, and J. Zheng, “ICIF-Net: intra-scale cross-interaction and inter-scale feature fusion network for bitemporal remote sensing imageschange detection,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–13, 2022.

[0134] It can be clearly seen from Table 1 that the present invention has higher detection accuracy and stronger generalization ability in four commonly used change detection datasets compared with the six existing methods, which further illustrates the superiority of the network and method proposed in the present invention.

[0135] from Figure 5 It can be seen that the present invention can effectively detect fine changes of interest in remote sensing images of various resolutions.

Claims

1. A remote sensing image change detection network based on twin branches, characterized in that: include: The twin local information encoding branch (1) consists of several downsampling layers (11), several standard residual blocks (12), and several deformable convolution units (13) to extract multi-level dual-phase local fine-grained features. ; The twin global information encoding branch (2) includes several downsampling layers (21), several sliding window Transformer blocks (22) and deformable Transformer blocks (23) to extract multi-level bi-temporal global context features. ; The difference enhancement module (3) includes a subtraction unit (31), a concatenation unit (32) and a convolution unit (33) for encoding the same-level bi-temporal local fine-grained features extracted by the twin local information encoding branch (1). Enhance difference information and generate local difference enhancement features , the same-level dual-phase global context features extracted by the twin global information encoding branch (2) Enhance difference information and generate global difference enhancement features ; A feature fusion module (4), which includes an addition unit (41), a channel attention unit (42), a multiplication unit (43), a convolution unit (44) and a splicing unit (45), and is used to fuse multi-level features step by step to generate a final fused feature; The adding unit (41) is used to integrate the local difference features of the twin branches after difference enhancement and global difference features , transmitted to the channel attention unit (42); The attention unit (42) is composed of an average pooling layer and a maximum pooling layer in parallel, a multi-layer perceptron layer and a sigmoid activation function layer in cascade, and is used to generate channel attention weights. , transmitted to the multiplication unit (43); The multiplication unit (43) is used to enhance the local difference features after the difference enhancement. and global difference features and channel attention weights respectively Multiply them to obtain channel weighted features and transmit them to the concatenation unit (45); The convolution unit (44) is composed of a 1×1 convolution layer and is used to reduce the dimension of the fused features of the previous layer, and transmit the reduced dimension features to the splicing unit (45); The splicing unit (45) is used to enhance the weighted local difference features at the same level. and global difference enhancement features After fusion with the previous level features Perform channel dimension splicing to complete feature fusion and generate fusion features ; The classifier (5) consists of a first 3×3 convolutional layer, a batch normalization layer, a ReLU activation function layer, a second 3×3 convolutional layer, a softmax function, and an argmax function cascaded in sequence. It is used to classify the final fusion features and generate a binary change prediction map, where 0 represents unchanged pixels and 1 represents changed pixels.

2. The network according to claim 1, wherein: The twin local information encoding branch (1) includes two weight-sharing local feature extractors: Each local feature extractor includes a downsampling layer (11), four standard residual blocks (12) and four deformable convolution units (13) connected in sequence, wherein the four standard residual blocks (12) and the four deformable convolution units (13) divide the local feature extractor into four levels; Each downsampling layer (11) includes a 7×7 convolution layer, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer connected in sequence, for reducing the spatial dimension of the input image, wherein the step size of the 7×7 convolution layer and the maximum pooling layer are both 2; Each standard residual block (12) includes two sequentially connected basic residual units, each residual unit includes two 3×3 convolutional layers, two batch normalization layers and two ReLU activation function layers; Each of the deformable convolution units (13) includes one 3×3 convolution layer, one 3×3 deformable convolution layer, one batch normalization layer and one ReLU activation function layer.

3. The network according to claim 1, wherein: The twin global information encoding branch (2) includes two weight-sharing global feature extractors: Each global feature extractor consists of four downsampling layers (21), two sliding window Transformer blocks (22), and four deformable Transformer blocks (23) connected sequentially, where the four downsampling layers divide the global feature extractor into four levels; Each sliding window Transformer block (22) includes a window attention unit (221) and a sliding window attention unit (222) connected in sequence, for extracting shallow global context features; Each of the deformable Transformer blocks (23) includes a local attention unit (231) and a deformable global attention unit (232) connected in sequence to extract deep global context features.

4. The network according to claim 3, characterized in that: The window attention unit (221) includes two residual-connected normalization layers, a window multi-head attention layer, and a multi-layer perceptron, and is used to divide the original feature map into a plurality of windows so that the multi-head attention calculation is limited to each small window to reduce the computational cost; The sliding window attention unit (222) includes two residual-connected normalization layers, a sliding window multi-head attention layer, and a multi-layer perceptron, for analyzing the connections between different windows and extracting global features; The local attention unit (231) includes two residual-connected normalization layers, a window multi-head attention layer, and a multi-layer perceptron, and is used to divide the original feature map into a number of local windows so that the multi-head attention calculation is limited to each small block to reduce the computational cost; The deformable global attention unit (232) includes two normalization layers connected in a residual form, a deformable multi-head attention layer and a multi-layer perceptron, which is used to calculate global attention and extract global features.

5. The network according to claim 1, wherein: The subtraction unit (31) is used to perform a subtraction on the dual-time features extracted by the dual twin branches and take the absolute value to generate a difference feature, which is transmitted to the splicing unit (32); The splicing unit (32) is used to splice the difference feature and the dual-time feature channel dimension to enhance the difference information, generate the splicing feature, and transmit it to the convolution unit (33); The convolution unit (33), which consists of a 1×1 convolution layer, is used to reduce the channel dimension of the spliced ​​feature to obtain difference enhancement features.

6. A method for detecting changes in remote sensing images using the network of claim 1, characterized in that: The steps include: 1) Obtain the divided training sample set and test sample set from the optical remote sensing change detection database; 2) Extract multi-level dual-phase local fine-grained features from dual-time images step by step through the twin local information encoding branch (1) , and input it into the difference enhancement module (3): 3) Extract multi-level dual-phase global context features from dual-time images step by step through the twin global information encoding branch (2) , and input it into the difference enhancement module (3): 4) Multi-level dual-time local fine-grained features and global context features are used to generate and enhance difference information using the difference enhancement module (3) to obtain multi-level local difference enhancement features that carry change information. and global difference enhancement features , and these features are hierarchically input into the feature fusion module (4): 5) Feature fusion module (4) enhances features of multi-level local differences and global difference enhancement features Perform fusion and interpret the change information in dual-time remote sensing images to obtain the final fusion features , transmitted to the classifier (5); 6) Classifier (5) will finally fuse the features Perform classification to obtain a binary change prediction map , where 0 represents unchanged pixels and 1 represents changed pixels; 7) Using the training sample set, the change detection network is iteratively trained using back propagation to obtain a trained change detection network; 8) Input the test sample set into the trained change detection network for forward propagation to obtain the change detection results of all test samples.

7. The method according to claim 6, characterized in that In step 2), multi-level bi-temporal local fine-grained features are extracted step by step, which is achieved as follows: 2a) Input the registered dual-time remote sensing image into the twin local information encoding branch (1); 2b) Downsample the dual-time image to reduce its spatial dimension and transfer it to the standard residual block (11); 2c) extracting local features of the downsampled dual-temporal remote sensing image through the standard residual block (11) and transmitting them to the deformable convolution unit (12); 2d) Further learning the extracted local features and extracting local fine-grained features through the deformable convolution unit (12); 2e) Stepwise downsampling of features reduces their spatial dimensions, and repeating steps 2c) and 2d) to obtain multi-level bi-temporal local fine-grained features.

8. The method according to claim 6, characterized in that The step-by-step extraction of multi-level bi-temporal global context features in step 3) is implemented as follows: 3a) Input the registered dual-time remote sensing images into the twin global information encoding branch (2); 3b) Downsample the bi-temporal image to reduce its spatial dimension and pass it to the sliding window Transformer block (21); 3c) Extracting shallow global context features of the downsampled dual-temporal remote sensing image through a sliding window Transformer block (21); 3d) Downsample the extracted shallow global context features to reduce their spatial dimensions, repeat step 3c), and then transfer the downsampled shallow features to the deformable Transformer block (22); 3e) Extracting deep global context features from the downsampled shallow features through the deformable Transformer block (22); 3f) Downsample the extracted deep global context features to reduce their spatial dimensions, repeat step 3e), and finally obtain multi-level bi-temporal global context features.

9. The method according to claim 6, characterized in that Step 5) Enhance the features of multi-level local differences and global difference enhancement features Fusion is to enhance the local differences of the same level step by step. and global difference enhancement features After fusion with the previous level features Perform channel dimension splicing, which is specifically implemented as follows: 5a) Enhance the local difference features of the 4th level and global difference enhancement features Only the addition unit (41), channel attention unit (42), multiplication unit (43) and splicing unit (45) in the feature fusion module (4) of this layer are fused to obtain the fourth fusion feature , transmitted to the third-level feature fusion module (4); 5b) Enhance the local difference features of the third level and global difference enhancement features And the fourth fusion feature after upsampling The third fusion feature is obtained by fusing the five units of the addition unit (41), channel attention unit (42), multiplication unit (43), convolution unit (44) and splicing unit (45) in the feature fusion module (4) of this layer. , transmitted to the second-level feature fusion module (4); 5c) Enhance the local difference features of the second level and global difference enhancement features And the third fusion feature after upsampling The second fusion feature is obtained by fusing the five units of the addition unit (41), channel attention unit (42), multiplication unit (43), convolution unit (44) and splicing unit (45) in the feature fusion module (4) of this layer. , transmitted to the first-level feature fusion module (4); 5d) Enhance the local difference features of the first level and global difference enhancement features And the second fusion feature after upsampling The final fusion feature is obtained by fusing the five units in the feature fusion module (4) of this layer, namely the addition unit (41), channel attention unit (42), multiplication unit (43), convolution unit (44) and splicing unit (45). .

10. The method according to claim 6, characterized in that In step 7), the change detection network is iteratively trained using the training sample set and back propagation, as follows: 7a) Define the overall network loss function ,in Represents the weight coefficient that controls the ratio of the two loss functions, , represents the cross entropy loss function, represents the dice loss function, where Indicates the total number of pixels, represents the ground truth, Representative change prediction map; 7b) Set the number of initialization iterations to , the maximum number of iterations is , , and order ; 7c) Input the dual-time remote sensing images of the training sample set into the change detection network for forward propagation and output the change prediction map , and compare it with the ground truth value in the training sample set As input to the network's overall loss function, calculate the change prediction map Ground truth The loss value between 7d) Backpropagate the loss value to calculate the network parameter gradient and update the network parameters using the gradient descent method; 7e) Judgment Is it established? If so, obtain the trained change detection network; Otherwise, let , return to steps 7c) and 7d).

Images