Remote sensing image detection method based on dual temporal interactive enhancement CNN-Transformer

Abstract

This invention discloses a remote sensing image detection method based on a dual-temporal interactive enhancement CNN-Transformer. By constructing a dual-temporal interactive enhancement subnetwork within the change detection network based on the dual-temporal interactive enhancement CNN-Transformer, this invention effectively increases the information content of changing targets in features, extracts more discriminative features, and significantly improves the accuracy of remote sensing image change detection, reducing the false positive and false negative rates. Furthermore, by utilizing a multi-scale difference fusion subnetwork within the dual-temporal interactive enhancement CNN-Transformer change detection network, this invention effectively fuses deep features with global information and shallow features with detailed information, further enhancing the accuracy of remote sensing image change detection.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically relates to a remote sensing image detection method based on a dual-temporal interactive enhanced convolutional neural network (CNN-Transformer) within the field of remote sensing image detection technology. This invention can be used to detect changes in buildings within urban optical remote sensing images. Background Technology

[0002] Remote sensing image change detection compares two remote sensing images of the same area acquired at different times to detect and identify changes in the state of objects or phenomena. This process is used to identify changes that occur in objects or phenomena at different times. Change detection algorithms are categorized into traditional method-based change detection algorithms and deep learning-based change detection algorithms based on their detection principles. Traditional change detection algorithms can be further divided into pixel-based change detection algorithms and object-based change detection algorithms. Traditional change detection algorithms generally utilize professional knowledge to extract features such as texture and structure, designing different feature extraction schemes for different images. Deep learning-based remote sensing image change detection can reduce manpower, material resources, and time, and can automatically perform change detection, automatically learning effective features using network models. However, deep learning-based remote sensing image change detection methods have limited information about changing targets in features extracted in complex backgrounds, resulting in poor accuracy and robustness, and a high number of false positives and false negatives. Furthermore, this method has a high computational cost, leading to poor timeliness in practical applications.

[0003] Harbin Institute of Technology and Shanghai Aerospace Measurement and Control Communication Research Institute proposed a method for detecting changes in remote sensing images based on feature enhancement and fusion in their patent application, "A Method and Device for Remote Sensing Image Change Detection Based on Feature Enhancement and Fusion" (Application No.: 202310723124.1, Publication No.: CN117079122 A). This method inputs dual-temporal images into a weighted shared feature extraction backbone to obtain three levels of feature maps. Then, global and local feature enhancements are performed on these three levels to obtain three corresponding differential features. These features are then fused and decoded using simple convolution, upsampling, and combination. Finally, a change map is obtained based on the obtained features. The method has shortcomings. Because there are differences between shallow and deep features in the weighted shared feature extraction backbone, it ignores the interrelationships between features from both temporal phases. Using the same feature enhancement module across different levels and temporal phases results in a limited amount of information about the changing targets in the extracted features. This makes it difficult to apply to complex remote sensing image scenarios, leading to poor detection accuracy and robustness, and poor performance in practical applications.

[0004] The University of Electronic Science and Technology of China (UESTC) proposed a method for detecting changes in optical remote sensing images in its patent application, "A Deep Learning-Based Method for Optical Remote Sensing Image Change Detection" (Application No.: 202311080420.0, Publication No.: CN 117152630 A). This method combines ResNet18, a Transformer encoder, and a Transformer decoder for feature extraction, employing a two-step progressive structure. First, ResNet18 is used for feature extraction, followed by the fusion of different features using a feature pyramid structure. Then, a Transformer encoder and decoder are used for further feature extraction, and a channel attention module is used to enhance the features. Finally, a difference map is extracted using a differential feature extraction module, and the data is input into a classifier for classification. The shortcomings of this method are that using Transformer encoders and decoders greatly increases the number of parameters and computational cost of the network model, thus affecting the timeliness of remote sensing image change detection. At the same time, this method directly uses the feature pyramid structure to fuse different features without considering multiple dimensions to enhance features at different levels. This results in less information about the changing targets in the features extracted by this method, making it difficult to apply to more complex remote sensing image scenes and resulting in poor detection accuracy. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the existing technologies. A remote sensing image detection method based on dual-temporal interactive enhanced CNN-Transformer is proposed to solve the technical problems of poor detection performance of existing remote sensing image change detection technologies, low accuracy in complex backgrounds, and high false positive and false negative rates.

[0006] The technical approach to achieve the objective of this invention is to construct a change detection network based on a CNN-Transformer with dual-temporal interactive enhancement, consisting of a feature extraction subnetwork, a dual-temporal interactive enhancement subnetwork, a multi-scale difference fusion subnetwork, and a prediction head connected sequentially. The dual-temporal interactive enhancement subnetwork and the multi-scale difference fusion subnetwork are part of this change detection network. Specifically, the dual-temporal interactive enhancement subnetwork fully considers the differences in the output feature maps of different convolutional groups in the feature extraction subnetwork. Based on the depth of the feature extraction subnetwork, it is divided into shallow and deep feature maps, and interactive channel-space enhancement modules and dual-temporal Transformer modules are designed for each. The interactive channel-space enhancement module enhances features for dual-temporal features by fully considering the channel dimension, spatial dimension, and temporal dimension. The dual-temporal Transformer module adopts a lightweight approach to introduce global information to enhance dual-temporal features. The multi-scale difference fusion subnetwork allows deep features with global information to gradually fuse with shallow features, enabling shallow features to gradually acquire global information, which is more conducive to the subsequent generation of change maps. This invention enables the effective extraction of changing targets in remote sensing images, effectively increases the amount of information about changing targets in features, and can extract more discriminative features. It overcomes the problems of low accuracy, low false detection rate, and low false negative rate in remote sensing image change detection in complex backgrounds in existing technologies.

[0007] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0008] Step 1: Construct a dual-temporal interactive enhancement subnetwork consisting of a first interactive channel-spatial enhancement module, a second interactive channel-spatial enhancement module, a third interactive channel-spatial enhancement module, and a dual-temporal Transformer module connected in parallel;

[0009] Step 2: Construct a multi-scale difference fusion subnetwork consisting of a subtraction layer group, a first combination module, and a second combination module connected in series; the subtraction layer group consists of four subtraction layers connected in parallel.

[0010] Step 3: Construct a change detection network based on bi-temporal interactive enhancement CNN-Transformer, consisting of a feature extraction subnetwork, a bi-temporal interactive enhancement subnetwork, a multi-scale difference fusion subnetwork, and a prediction head concatenation.

[0011] Step 4: Input the training set into the change detection network based on the dual temporal interaction enhancement CNN-Transformer, and use the gradient descent method to iteratively update the network weight parameters until the total loss function of the network converges, thus obtaining the trained change detection network based on the dual temporal interaction enhancement CNN-Transformer.

[0012] Step 5: Input the two optical remote sensing images to be detected into the trained change detection network based on dual temporal interaction enhancement CNN-Transformer to obtain the change map.

[0013] Compared with existing technologies, the present invention has the following advantages:

[0014] First, because the dual-temporal interactive enhancement subnetwork in the change detection network based on dual-temporal interactive enhancement CNN-Transformer constructed in this invention can effectively increase the amount of information about changing targets in the features, it can extract more discriminative features, overcoming the problems of low accuracy and low false detection and false negative rates in remote sensing image change detection tasks in complex backgrounds in the prior art, effectively improving the accuracy of remote sensing image change detection and reducing false detection and false negative rates.

[0015] Second, because the multi-scale difference fusion subnetwork in the change detection network based on dual temporal interaction enhancement CNN-Transformer constructed in this invention can effectively fuse deep features with global information and shallow features with detailed information, it overcomes the problem in the prior art that remote sensing image change detection tasks cannot fully fuse different features, and effectively improves the accuracy of remote sensing image change detection. Attached Figure Description

[0016] Figure 1 This is a flowchart of the present invention;

[0017] Figure 2 This is a schematic diagram of the structure of the change detection network based on dual-temporal interaction enhanced CNN-Transformer constructed in this invention;

[0018] Figure 3 This is a schematic diagram of the interactive channel-space enhancement module constructed in this invention;

[0019] Figure 4 This is a schematic diagram of the structure of the dual-temporal Transformer module constructed in this invention;

[0020] Figure 5 This is a schematic diagram of the multi-scale differential fusion network structure constructed in this invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Reference Figure 1 The specific steps for implementing the embodiments of the present invention will be described in further detail below.

[0023] Step 1: Divide the optical remote sensing image dataset and perform data preprocessing.

[0024] The dataset used in the embodiments of this invention is the LEVIR-CD optical remote sensing image dataset. It is a large, public dataset for detecting building changes, containing 637 pairs of high-resolution color remote sensing images (1024×1024×3) and 637 corresponding real-world change maps. The LEVIR-CD dataset is divided into a training set, a validation set, and a test set, with 445 sets in the training set, 64 sets in the validation set, and 128 sets in the test set.

[0025] For the training set of the LEVIR-CD optical remote sensing image dataset, input image pairs were randomly cropped to a size of 512×512×3. Each image was then randomly flipped vertically and horizontally with a 50% probability. Each image was also rotated with a 50% probability from an angle randomly selected from [90, 180, 270]. Finally, each image was randomly blurred using a Gaussian blur with a 50% probability. Finally, each image in both the training and test sets of the LEVIR-CD dataset was normalized to a mean of [0.5, 0.5, 0.5] and a variance of [0.5, 0.5, 0.5].

[0026] This invention uses the preprocessed training set to train the change detection network based on the dual-temporal interaction enhanced CNN-Transformer, and uses the preprocessed test set to test the change detection network based on the dual-temporal interaction enhanced CNN-Transformer.

[0027] Step 2: Construct a change detection network based on dual-temporal interaction-enhanced CNN-Transformer.

[0028] A change detection network based on bi-temporal interactive enhancement CNN-Transformer is constructed, which includes a feature extraction subnetwork, a bi-temporal interactive enhancement subnetwork, a multi-scale difference fusion subnetwork, and a prediction head.

[0029] The following is combined with Figure 2 A detailed description is given of the constructed change detection network based on dual-temporal interaction enhanced CNN-Transformer.

[0030] First, a feature extraction subnetwork is used to extract multi-scale features from the input two-temporal image pairs. Then, a two-temporal interactive enhancement subnetwork further enhances the multi-scale features. Next, a multi-scale difference fusion subnetwork is used to fuse the multi-scale features output by the two-temporal interactive enhancement subnetwork. Finally, a prediction head is used to output a predicted change map.

[0031] Step 2.1, construct the feature extraction sub-network.

[0032] In the embodiments of this invention, the feature extraction subnetwork uses the existing ResNet18, whose structure consists of sequentially connected convolutional layers, pooling layers, a first block unit, a second block unit, a third block unit, and a fourth block unit. The convolutional kernel size of the convolutional layers is set to 7×7, the number of kernels is set to 64, the stride of the pooling layers is 2, and the window size is 3×3. The first block unit, the second block unit, the third block unit, and the fourth block unit all have the same structure. Each block unit is composed of a first convolutional layer, a second convolutional layer, a first residual connection layer, a third convolutional layer, a fourth convolutional layer, and a second residual connection layer connected in series. In the first block unit, the kernel size of the first convolutional layer is set to 3×3, and the number of kernels is 64. Similarly, the kernel size of the second convolutional layer in the first block unit is set to 3×3, and the number of kernels is 64. The kernel size of the third convolutional layer in the first block unit is also set to 3×3, and the number of kernels is 64. The kernel size of the fourth convolutional layer in the first block unit is also set to 3×3, and the number of kernels is 64. In the second block unit, the kernel size of the first convolutional layer is set to 3×3, and the number of kernels is 128. The kernel size of the second convolutional layer in the second block unit is also set to 3×3, and the number of kernels is 128. The kernel size of the third convolutional layer in the second block unit is also set to 3×3, and the number of kernels is 128. The kernel size of the fourth convolutional layer in the second block is set to 3×3, and the kernel size is set to 128. The kernel size of the first convolutional layer in the third block is set to 3×3, and the kernel size is set to 256. The kernel size of the second convolutional layer in the third block is set to 3×3, and the kernel size is set to 256. The kernel size of the third convolutional layer in the third block is set to 3×3, and the kernel size is set to 256. The kernel size of the fourth convolutional layer in the third block is set to 3×3, and the kernel size is set to 256. The kernel size of the fourth convolutional layer in the third block is set to 3×3, and the kernel size is set to 256. The kernel size of the first convolutional layer in the fourth block is set to 3×3, and the kernel size is set to 512. The kernel size of the second convolutional layer in the fourth block is set to 3×3, and the kernel size is set to 512. The kernel size of the third convolutional layer in the fourth block is set to 3×3, and the kernel size is set to 512. The kernel size of the fourth convolutional layer in the fourth block is set to 3×3, and the kernel size is set to 512.

[0033] Two image pairs acquired from the same location at different times are referred to as Time Phase 1 and Time Phase 2, both with a size of h×w×3, where h is the image height, w is the image width, and 3 is the image channel dimension. The Time Phase 1 image is input into the feature extraction subnetwork, yielding output features at different scales from the convolutional layer and the first to fourth block units. The feature extraction subnetwork, which takes the two temporal images as input, produces output features at different scales from the convolutional layers and the first to fourth block units. These features are respectively... The output features of phase 1 and phase 2 are combined to form an output feature pair. and and and and and

[0034] The output features of the feature extraction subnetwork are denoted as... Here, the phase number t = {1, 2}, and J is the sequence number, J = {2, 3, 4, 5}; the original input image size is h × w × 3, then the output features of the block unit... and The sizes are h / 4×w / 4×64, h / 8×w / 8×128, h / 16×w / 16×256, and h / 32×w / 32×512, respectively, which are downsampling factors of 4, 8, 16, and 32. In this embodiment of the invention, the image height h and the image width w are both 512.

[0035] Step 2.2: Construct the first interactive channel - spatial enhancement module, the second interactive channel - spatial enhancement module, and the third interactive channel - spatial enhancement module.

[0036] The first to third interactive channel-spatial enhancement modules have the same structure, and each interactive channel-spatial module will incorporate dual-temporal features. and A pixel-by-pixel addition operation yields a superimposed feature. This superimposed feature is then passed through a global average pooling layer and input to two structurally identical fully connected layers and a sigmoid activation layer, respectively. Finally, it is combined with the dual-temporal feature. and The enhanced features of the two channels obtained by performing the multiplication operation Features enhanced by two channels After pixel-by-pixel addition, the data is input into two identical convolutional layers and a sigmoid activation layer, and then multiplied with the enhanced features of the two channels. The specific parameters for the first interactive channel-spatial module are: a 1×1 kernel size, 1 kernel, and 64 parameters for the fully connected layer. The specific parameters for the second interactive channel-spatial module are: a 1×1 kernel size, 1 kernel, and 128 parameters for the fully connected layer. The specific parameters for the third interactive channel-spatial module are: a 1×1 kernel size, 1 kernel, and 256 parameters for the fully connected layer.

[0037] The output feature pairs of the feature extraction subnetwork in the embodiment of the present invention and and and The inputs are respectively sent to the first interactive channel - spatial enhancement module, the second interactive channel - spatial enhancement module, and the third interactive channel - spatial enhancement module.

[0038] Output feature pairs of the feature extraction subnetwork The input feature pairs will be input into the first interactive channel-spatial enhancement module, the second interactive channel-spatial enhancement module, and the third interactive channel-spatial enhancement module, respectively, where the sequence number J = {2,3,4} and the phase sequence number t = {1,2}.

[0039] The following is combined with Figure 3 Each interactive channel-space enhancement module constructed is described in detail.

[0040] Each interactive channel-space enhancement module contains sequentially connected attention blocks for the two-phase interactive channels and attention blocks for the two-phase interactive spaces.

[0041] Dual-temporal interactive channel attention block pairs input feature pairs First, a pixel-by-pixel addition operation is performed. The superimposed features are then sequentially input into a global average pooling layer, a fully connected layer, and a sigmoid activation layer to obtain two features for each layer. These two features are then compared with the input features at their respective time points. Channel multiplication yields the output feature map of the dual-temporal interactive channel attention block. The specific operating formula is as follows:

[0042]

[0043] Where σ(·) represents the activation operation using the Sigmoid function, and AvgPool(·) represents the global average pooling operation. For channel multiplication operations, FC(·) represents a fully connected network operation.

[0044] The two output feature maps of the dual-temporal interactive channel attention block As the input feature pairs for the dual-temporal interactive spatial attention block, pixel-by-pixel addition is first performed to obtain the superimposed features. The superimposed features are then sequentially input into a 1×1 convolutional layer and a sigmoid activation layer to obtain two spatial weight features. These two spatial weight features are then multiplied by the input features of their respective temporal phases to obtain the output feature maps of the dual-temporal interactive spatial attention block. The 1×1 convolutional layer has a 1×1 kernel size and a single kernel. The specific operating formula is as follows:

[0045]

[0046] Where σ(·) represents the activation operation using the Sigmoid function, Conv 1×1 (·) represents a 1×1 convolution operation.

[0047] Output feature map of dual-temporal interactive spatial attention block That is, the output feature maps of the first interaction channel - spatial enhancement module, the second interaction channel - spatial enhancement module, and the third interaction channel - spatial enhancement module. and

[0048] Step 2.3: Construct the dual-temporal Transformer module.

[0049] The dual-temporal Transformer module consists of two channel-space mapping blocks and an interactive Transformer block connected in series.

[0050] The following is combined with Figure 4 The bi-temporal Transformer module, which is constructed from two channel-space mapping blocks and one interactive Transformer block, is described in detail.

[0051] Output feature pairs and The input is fed into two channel-space mapping blocks in the dual-temporal transformer module. Each channel-space mapping block includes two branches: a channel branch and a spatial branch, such as... Figure 4 As shown in the yellow dashed box on the left. The channel branches consist sequentially of a global average pooling layer, a fully connected layer, and a Sigmoid activation layer, with the specific operation formulas as follows:

[0052]

[0053] Where σ(·) represents the activation operation using the Sigmoid function, AvgPool(·) represents the global average pooling operation, FC(·) represents the fully connected operation of the network, and TC t This represents the output feature after channel weighting.

[0054] The spatial branch of the channel-space mapping block consists of a 1×1 convolutional layer and a Softmax activation layer. The specific operation formula is as follows:

[0055]

[0056] Among them, Conv 1×1 (·) represents a 1×1 convolution operation, Softmax(·) represents an activation operation using the Softmax function, TS t This is represented as the spatially weighted output feature.

[0057] The output feature TC of the channel branch t With input feature map Perform channel multiplication, then combine the result of channel multiplication with the output feature TS obtained from spatial branching. t Perform matrix multiplication to obtain the output features T of the channel-space mapping block. t .

[0058] Output feature T of the channel-space mapping block t As input features of the interactive Transformer block, their input positions are encoded to obtain right After performing layer normalization, three identical linear transformations are performed, mapping to K based on the different weights. t V t and Q t Q 1 and Q 2 Q is obtained by performing pixel-by-pixel addition. K t V t The input to the multi-head attention module is Q, and residual connections are performed. The output feature is M. t The specific operating formula is as follows:

[0059]

[0060] Where LN(·) is the layer normalization operation and MHA(·) is the multi-head attention operation, the specific formulas of which are as follows:

[0061] MHA(Q,K t V t = Concat(head1,...,head) i,...,head m W o

[0062] Where Concat(·) represents a channel cascading operation, W 0 The head represents the learnable weight matrix. i The output features of the single-head attention module are defined by the following formula:

[0063] head i =Attention(QW i Q ,K t W i K V t W i V )

[0064] Where Attention(·) represents a single-head attention operation, i represents the index of the single-head attention module, and the maximum value of i is m = 8.

[0065] M t Input layer normalization and a multilayer perceptron are incorporated, along with residual connections, to obtain the output features of the interactive Transformer block. The specific operating formula is as follows:

[0066]

[0067] Where LN(·) is the layer normalization operation, and MLP(·) is the multilayer perceptron operation. The multilayer perceptron consists of two fully connected layers and Gaussian error linear units (GELUs), and its specific formula is as follows:

[0068]

[0069] Where GELU(·) is the activation operation of the Gaussian error linear unit function. and Both represent learnable weight matrices.

[0070] Finally, the output features of the interactive Transformer block Input characteristics of the dual-temporal Transformer module Mapping back to pixel space yields the output features of the dual-temporal Transformer module. The specific operating formula is as follows:

[0071]

[0072] Here, reshape(·) represents the matrix rearrangement operation, and Softmax(·) represents the activation operation using the Softmax function. and Both represent learnable weight matrices.

[0073] Step 2.4: Construct a multi-scale differential fusion subnetwork.

[0074] The following is combined with Figure 5 The constructed multi-scale differential fusion subnetwork is described in detail.

[0075] The output feature map of the interactive channel-spatial enhancement module obtained in step 2.2 and The output feature map of the dual-temporal Transformer module obtained in step 2.3 Feature maps at different scales are input into the multi-scale differential fusion sub-network. The multi-scale differential fusion sub-network mainly consists of progressively connected 3×3 convolutional layers and 1×1 convolutional layers, such as... Figure 5 As shown. The specific process is as follows: First, and The pixels are subtracted one by one, and then convolved with a kernel size of 3×3 and a number of 256 to obtain A5; and The features obtained by subtracting each pixel from the features are concatted with A5, and then convolved with a kernel size of 3×3 and a number of 128 to obtain A4. and The features obtained by subtracting each pixel from the features are concatted with A4, and then convolved with a kernel size of 3×3 and a number of 64 to obtain A3. and The features obtained by subtracting each pixel from A3 are concatenated with A3, and then convolved with a 3×3 kernel of 64 kernels to obtain A2. The specific operation formula is shown below:

[0076]

[0077]

[0078]

[0079]

[0080] Where Concat(.) represents a channel cascading operation, Conv 3×3 (·) indicates a 3×3 convolution operation.

[0081] Subsequently, A5 undergoes a convolution with a kernel size of 1×1 and 128 kernels. The result is then added pixel-by-pixel to A4. Next, it undergoes a convolution with a kernel size of 1×1 and 64 kernels. The result is then added pixel-by-pixel to A3. Finally, it undergoes another convolution with a kernel size of 1×1 and 64 kernels. The result is then added pixel-by-pixel to A2. The final output feature map Z of the multi-scale difference fusion network is shown below.

[0082] Z = Conv 1×1 (Conv 1×1 (Conv 1×1 (A5)+A4)+A3)+A2

[0083] Among them, Conv 1×1 (·) represents a 1×1 convolution operation.

[0084] Step 2.5, predict the head.

[0085] The prediction head consists of a first convolutional layer and a second convolutional layer connected sequentially. The specific parameters are set as follows: the first convolutional layer has a kernel size of 3×3 and 64 kernels; the second convolutional layer has a kernel size of 3×3 and 2 kernels. The output feature map Z obtained from the multi-scale difference fusion network in step 2.4 is repeatedly upsampled until it reaches the same size as the true change map. Then, the convolutional layers are used to classify it pixel-by-pixel into change and invariant classes to obtain the change map.

[0086] Step 3: Iteratively train the change detection network based on dual-temporal interaction-enhanced CNN-Transformer.

[0087] Using the training set obtained in step 1, the gradient descent method is used to iteratively update the network weight parameters in the change detection network based on the dual temporal interaction enhancement CNN-Transformer constructed in step 2 until the total loss function of the network converges, thus obtaining the trained change detection network based on the dual temporal interaction enhancement CNN-Transformer.

[0088] The total network loss function is the sum of the binary cross-entropy loss function and the Dice loss function; the binary cross-entropy loss function L bce and Dice loss function L dice The calculation formulas are as follows:

[0089]

[0090]

[0091] Where N represents the total number of true change maps in the training set, y iy represents the change state of the i-th element in the change graph. i When y = 1, it means that the i-th element in the change graph is a change class, y i When a = 0, it means that the i-th element in the change graph is an invariant class, a i This represents the network output probability of the i-th pixel in the change graph. TP indicates that the prediction is of the change class and is correct, FP indicates that the prediction is of the change class but is incorrect, and FN indicates that the prediction is of the invariant class but is incorrect.

[0092] Step 4: Use the trained change detection network based on dual-temporal interaction-enhanced CNN-Transformer to obtain the change detection results.

[0093] The test remote sensing image pairs are input into a trained change detection network based on a dual-temporal interactive enhanced CNN-Transformer to obtain change maps.

[0094] The technical effects of the present invention will be further described below with reference to simulation experiments.

[0095] 1. Simulation conditions.

[0096] The simulation experiment hardware platform of this invention is as follows: the processor is an Intel Core i9-9900K CPU with a main frequency of 3.60GHz, the memory is 64GB, and the graphics card is an NVIDIA GeForce GTX 2080Ti;

[0097] The simulation experiment software platform of this invention is the Linux Ubuntu 18.04 operating system, Python 3.6 and PyTorch 1.6.

[0098] The simulation experiment of this invention uses the publicly available remote sensing image change detection dataset LEVIR-CD, with a resolution of 1024×1024. The data contains 31,333 changing building targets.

[0099] 2. Simulation experiment content and result analysis.

[0100] The simulation experiment of this invention uses a test set to test the method of this invention and existing methods to obtain accuracy, recall and F1 score.

[0101] The existing method refers to:

[0102] In their paper "Remote Sensing Image Change Detection With Transformers (IEEE Transactions on Geoscience and Remote Sensing, vol.60, pp.1-14, 2022)," Chen et al. combined Transformer with CNN to propose a change detection network model.

[0103] All simulation experiments in this invention were conducted on the LEVIR-CD dataset. The remote sensing image change detection network models of this invention and existing methods were applied to detect changes in input dual-temporal images, and the accuracy was calculated. In this invention, the change detection network was trained using the training set to obtain a trained change detection network model, which was then tested on the test set to obtain the final accuracy. The accuracy rates compared to existing methods are shown in Table 1.

[0104] Table 1 Comparison of detection accuracy on the LEVIR-CD dataset

[0105]

[0106] In summary, the remote sensing image detection method based on dual-temporal interactive enhanced CNN-Transformer proposed in this invention can detect images more effectively compared with existing methods.

Claims

1. A remote sensing image change detection method based on two-phase interaction enhanced CNN-Transformer, characterized in that, The detection method involves constructing a dual-temporal interactive enhancement subnetwork and a multi-scale difference fusion subnetwork, respectively. The steps of this method are as follows: Step 1: Construct a dual-temporal interactive enhancement subnetwork consisting of a first interactive channel-spatial enhancement module, a second interactive channel-spatial enhancement module, a third interactive channel-spatial enhancement module, and a dual-temporal Transformer module connected in parallel; The first to third interactive channel-spatial enhancement modules have the same structure, and each interactive channel-spatial module will incorporate dual-temporal features. and A pixel-by-pixel addition operation yields a superimposed feature. This superimposed feature is then passed through a global average pooling layer and subsequently fed into two structurally identical fully connected layers and a sigmoid activation layer. The outputs are then compared with the bi-temporal features. and Performing a multiplication operation yields the enhanced features of the two channels. ; After pixel-by-pixel addition, the results are fed into two convolutional layers and a sigmoid activation layer with the same structure, and then multiplied with the features enhanced in the two channels. The dual-temporal Transformer module includes two channel-space mapping blocks with the same structure but different weight parameters and an interactive Transformer block; Features and The input is fed into two channel-space mapping blocks of the dual-temporal transformer module to obtain the output feature Tt of the channel-space mapping block. The feature Tt is then position-encoded to obtain the position-encoded feature. This position-encoded feature is then fed into the first normalization layer and the multi-head attention layer, and added to the position-encoded feature to obtain the multi-head attention-enhanced feature. This multi-head attention-enhanced feature is then passed through the second normalization layer and the multilayer perceptron, and added again to the multi-head attention-enhanced feature before being combined with... Perform mapping together; obtain the output features of the dual-temporal Transformer module. Each channel space mapping block will feature The feature TCt is obtained after passing through a global average pooling layer, a fully connected layer, and a Sigmoid activation layer. The first convolutional layer and the softmax activation layer produce the feature TS. t , will feature TC t With features Perform channel multiplication, then combine the result of channel multiplication with the output feature TS obtained from spatial branching. t Perform matrix multiplication to obtain the output features T of the channel-space mapping block. t ; Step 2: Construct a multi-scale difference fusion subnetwork consisting of a subtraction layer group, a first combination module, and a second combination module connected in series; the subtraction layer group consists of four subtraction layers connected in parallel. The first combined module includes a first convolutional layer, a first channel cascaded layer, a second convolutional layer, a second channel cascaded layer, a third convolutional layer, a third channel cascaded layer, and a fourth convolutional layer connected in sequence; the first convolutional layer is connected to the first subtraction layer in the subtraction layer group, and the first to third channel cascaded layers are respectively connected to the second to fourth subtraction layers in the subtraction layer group. The second combined module includes a first convolutional layer, a first addition layer, a second convolutional layer, a second addition layer, a third convolutional layer, and a third addition layer connected in sequence; the first convolutional layer is connected to the first convolutional layer of the first combined module, and the first to third addition layers are respectively connected to the second to fourth convolutional layers of the first combined module. Step 3: Construct a change detection network based on bi-temporal interactive enhancement CNN-Transformer, consisting of a feature extraction subnetwork, a bi-temporal interactive enhancement subnetwork, a multi-scale difference fusion subnetwork, and a prediction head concatenation. The structure of the feature extraction sub-network is as follows: a convolutional layer, a pooling layer, a first block unit, a second block unit, a third block unit, and a fourth block unit connected in sequence; the first block unit, the second block unit, the third block unit, and the fourth block unit all have the same structure; each block unit is composed of a first convolutional layer, a second convolutional layer, a first residual connection layer, a third convolutional layer, a fourth convolutional layer, and a second residual connection layer connected in series. Step 4: Input the training set into the change detection network based on the dual temporal interaction enhancement CNN-Transformer, and use the gradient descent method to iteratively update the network weight parameters until the total loss function of the network converges, thus obtaining the trained change detection network based on the dual temporal interaction enhancement CNN-Transformer. Step 5: Input the two optical remote sensing images to be detected into the trained change detection network based on dual temporal interaction enhancement CNN-Transformer to obtain the change map.

2. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, In step 1, the kernel size of the convolutional layers in the first to third interactive channel-space modules is set to 1×1, the number of convolutional kernels is set to 1, and the parameters of the fully connected layers are set to 64, 128, and 256, respectively.

3. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, The specific parameters in the dual-temporal Transformer module described in step 1 are set as follows: the size of the convolution kernel of the first convolutional layer is 1×1, and the number of convolution kernels is set to 8.

4. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, In step 2, the kernel size of the first to third convolutional layers of the first combined module is set to 3×3, and the number is set to 128, 128, and 64 respectively.

5. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, In step 2, the kernel size of the first to third convolutional layers of the second combined module is set to 1×1, and the number is set to 128, 64, and 64 respectively.

6. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, In step 3, the kernel size of the convolutional layers in the feature extraction sub-network is set to 7×7, the number of kernels is set to 64, the stride of the pooling layers is 2, and the window size is 3×3. The kernel size of the first to fourth convolutional layers in the first block unit is set to 3×3, and the number of kernels is 64. The kernel size of the first to fourth convolutional layers in the second block unit is set to 3×3, and the number of kernels is 128. The kernel size of the first to fourth convolutional layers in the third block unit is set to 3×3, and the number of kernels is 256. The kernel size of the first to fourth convolutional layers in the fourth block unit is set to 3×3, and the number of kernels is 512.

7. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, The prediction head structure described in step 3 consists of a first convolutional layer and a second convolutional layer connected sequentially; the kernel size of the first and second convolutional layers is set to 3×3, and the number of kernels is set to 64 and 2 respectively.

8. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, The training set mentioned in step 4 refers to selecting at least 200 pairs of optical remote sensing images and their corresponding real change maps, randomly cropping each pair of optical remote sensing images and their corresponding real change maps to a size of 512×512, performing image enhancement on each pair of optical remote sensing images, and combining all the enhanced image pairs and their corresponding real change maps into a training set.

9. The remote sensing image change detection method based on dual-temporal interactive enhanced CNN-Transformer according to claim 1, characterized in that, The total loss function mentioned in step 4 is the sum of the binary cross-entropy loss function and the Dice loss function; the binary cross-entropy loss function and Dice loss function The calculation formulas are as follows: ； ； in, This represents the total number of true change graphs in the training set. This represents the change state of the i-th element in the change graph. When , it indicates that the i-th element in the change graph is a change class. When , it means that the i-th element in the change graph is an invariant class. This represents the network output probability of the i-th pixel in the transformation graph. This indicates that the prediction is of the change category and the prediction is correct. This indicates that the prediction was for a variable class but was incorrect. This indicates that the prediction was for an invariant class, but the prediction was incorrect.

Images