Remote sensing image change detection method based on edge auxiliary adaptation

Abstract

The application provides a remote sensing image change detection method based on edge assistance and self-adaption, and the implementation steps are as follows: an edge assistance and self-adaption change detection network EATDer is constructed; a corresponding training set and a verification set are generated; the EATDer is trained by using the training set and the verification set; and change detection is performed on remote sensing images. By using the SAVT module and the FRFM module, the global features of remote sensing images can be extracted at a reasonable calculation cost, and the influence of the uneven positive and negative samples of the data set itself can be controlled. By using the change detection module emphasizing the edge information and the joint loss function, the network can sufficiently pay attention to the edge detection accuracy of the target to be detected, so that the problems of high calculation cost, frequent edge detection adhesion and low accuracy in the prior art are solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and further relates to a remote sensing image change detection method based on edge auxiliary adaptation. The present application can be used for double-phase change detection of remote sensing images obtained in land cover analysis and disaster monitoring. BACKGROUND

[0002] Change detection is an operation of quantitative analysis and surface change of phenomena or objects from two different periods. Remote sensing image change detection plays a crucial role in the field of remote sensing and receives more and more attention due to its important application value. The main technical scheme for realizing remote sensing image change detection is a supervised scheme based on deep learning. With the emergence of convolutional neural network (CNN) technology, deep learning methods are gradually applied to change detection. CNN uses its stacked structure and hierarchical learning method to further explore the semantic features of remote sensing images, thereby generating accurate change detection feature maps. However, there is still a difficult problem to be solved, that is, CNN cannot capture long-distance context information hidden in remote sensing images. This shortcoming more or less limits the detection accuracy of change detection networks based on pure CNN structure. Considering the above limitations, researchers have introduced the Transformer technology into the field of remote sensing. Due to its global receptive field, Transformer can well make up for the lack of CNN in long-distance information extraction. However, there is still room for improvement in Transformer. First, since the ground cover types in remote sensing images are diverse and their scales are also diverse, the changed areas in multi-temporal remote sensing images are usually irregular. However, conventional Transformer-based models cannot accurately detect boundaries, which has a negative impact on the remote sensing change detection task. Second, current Transformer-based models only focus on remote sensing images themselves, ignoring the time clues between remote sensing images collected at different times, while these clues are also crucial for remote sensing change detection of remote sensing images. Finally, due to self-attention and multi-head mechanism, Transformer-based models are always heavy, which limits the efficiency of the remote sensing change detection process.

[0003] Zhongke Xingtu TT&C Technology (Hefei) Co., Ltd. proposed a remote sensing change detection method in its applied patent document "Algorithm and system for remote sensing change detection" (Patent application number: 202210941062.7, application publication number: CN 115019186 A). The method performs feature extraction and splicing on different time images, and uses a pyramid module based on the CNN structure for feature extraction, and finally inputs the fused feature map into the FCN-head module and the SPP-head module to obtain the final output. Although this method attempts to use new structures to improve the network's ability to extract context information, there are still two shortcomings in this method. First, the receptive field of the CNN structure's pyramid module is insufficient, and the global feature extraction capability is limited, which leads to false detection and missed detection problems in the network, thereby affecting the detection accuracy. Second, the feature extraction and network design of this method pay insufficient attention to edge information, resulting in insufficient edge detection accuracy of this method, and there are still a large number of phenomena of multiple different target edge adhesion false detection.

[0004] Harbin Engineering University proposed a remote sensing change detection method in its applied patent document "Remote sensing image change detection method based on multi-scale semantic label Transformer network" (Patent application number: 202211026042.3, application publication number: CN 115393317 A). The method uses a feature encoder and decoder to extract feature maps, uses a multi-scale semantic label encoder to convert feature maps of different scales from the feature encoder into semantic labels of different lengths, and then inputs them into a Transformer to obtain global semantic information. Then, the multi-scale semantic label decoder combines semantic labels of different scales containing rich semantic information and multi-scale feature maps containing rich spatial information to obtain semantic spatial joint features. Finally, the multi-scale semantic spatial joint features are aggregated in the feature decoder in a skip connection manner, and the final change result map is obtained through a classifier. The shortcomings of this method are as follows: first, the method directly uses the original Transformer structure, and the multi-head attention calculation in the original Transformer structure requires a large amount of calculation and memory resources. SUMMARY

[0005] The purpose of the present application is to address the shortcomings of the prior art, and to propose a remote sensing image change detection method based on edge auxiliary adaptation, to solve the three problems existing in the prior art: first, the problem of excessive resource consumption in the multi-head attention calculation of the original Transformer structure; second, the problem of limited global feature extraction capability of pure CNN structure neural network; third, the problem of false detection and missed detection caused by poor edge preservation effect in the change detection result.

[0006] The idea for achieving the object of the present application is that the present application re-designs the multi-head attention calculation in the Transformer, and constructs a self-adaptive multi-head attention method, which can adaptively reduce the size and scale of the input feature map for different sizes and scales of the input feature map, greatly reducing the consumption of computing resources. The present application uses the combination of CNN and Transformer, uses the ability of the Transformer structure to extract global features to make up for the shortcomings of pure CNN structure, and constructs a feature extractor composed of a self-adaptive Transformer feature extraction module SAVT (Self-adaption Vision Transformer) and a cross branch information interaction module FRFM (Full Range Fusion Module). The feature of the SAVT is that the structure first uses CNN for feature extraction, then sends the extracted features into the Transformer module of the self-adaptive multi-head attention, and then performs global feature extraction under the condition of controllable calculation amount. The FRFM module builds information communication between the two branches of the change detection network, enhances the anti-interference ability of the network for the uneven distribution of positive and negative samples of the original data set, and makes the network more easily distinguish the change detection target and background. The combination of the two modules improves the problem of limited global feature extraction capability in the prior art. The present application uses the Canny operator to extract edge information from the original supervised data label, and in the design of the decoder and the loss function, introduces the Transformer into the decoder for feature refinement to improve the detection accuracy. The original data and the edge data are sent into the back propagation process of the network through the specially designed loss function, so as to emphasize the edge information, so that the network continuously maintains enough attention to the edge information, and then improves the edge retention effect of the network detection result.

[0007] To achieve the above object, the technical scheme of the present application comprises the following steps:

[0008] Step 1, constructing an edge-assisted adaptive change detection network EATDer (Edge-Assisted Adaptive Transformer Detector):

[0009] Step 1.1, building a feature extractor composed of three first, second and third sub-extractors in series;

[0010] Each sub-extractor is composed of two SAVT modules in series and a FRFM module in series. After the input image of 256x256 is passed through the three sub-extractors, three feature maps of 64x64, 32x32 and 16x16 are outputted;

[0011] The SAVT module in each sub-extractor is composed of a feature map encoding layer and four SAVT encoders with the same structure of adaptive multi-head attention; the feature map encoding layer is implemented by a convolutional layer; the feature map dimensions of the first to third sub-extractors are set to 64, 128, and 256 respectively, the convolution kernel sizes are set to 7, 3, and 3 respectively, the strides are set to 4, 2, and 2 respectively, and the padding is set to 3, 1, and 1 respectively; the SAVT encoder with adaptive multi-head attention is composed of a Layer Norm layer, an adaptive reduction layer, a multi-head attention layer, a Layer Norm layer, and an MLP layer connected in sequence; the adaptive reduction layer is implemented by average pooling and convolution on the input data; for an image with a long, wide, and input dimension of HxWxC before the input adaptive reduction layer, the size after average pooling is wherein, is a rounding up operation; the convolution kernel size, stride, and padding of the convolution in the adaptive reduction layer are set to 1, 1, and 0 respectively, and the number of heads of the multi-head attention in different sub-extractors is fixed to 4;

[0012] The FRFM module in each sub-extractor is composed of two FRFM sub-modules A and B with the same structure connected in parallel, and each FRFM sub-module is composed of an information interaction branch and a convolutional block attention module (CBAM) structure connected in series, wherein the information interaction branch accepts two output feature maps {X, Y} processed by the parallel SAVT module as the input of the branch; the information interaction branch in the FRFM sub-module A is implemented by the following formula:

[0013]

[0014] wherein, X cf represents the output of the information interaction branch in the FRFM sub-module A, represents different parameter matrices available for neural network learning, and softmax(·) represents a softmax function, d kx represents a scaling factor whose value is equal to the dimension of The formula of the cross-fusion process of the B branch is:

[0015]

[0016] wherein, Y cf represents the output of the information interaction branch in the FRFM sub-module B, and represent different parameter matrices available for neural network learning, and d ky represents a scaling factor whose value is equal to the dimension of y Kthe dimensions of the two images are equal;

[0017] Step 1.2, a change detection module composed of a recovery submodule and a refinement submodule in series is established;

[0018] The recovery submodule is composed of a 3x3 convolution layer and an up-sampling layer in series; the refinement submodule is composed of a 3x3 convolution layer, an SAVT module, a 2x2 deconvolution layer, a Batch Norm layer, a 2x2 deconvolution layer, a Batch Norm layer, a 3x3 convolution layer, an SAVT module, a 3x3 convolution layer, a Batch Norm layer, and a 1x1 convolution layer in series;

[0019] Step 1.3, the feature extractor and the change detection module are connected in series to form an edge-assisted adaptive network EATDer;

[0020] Step 2, a training set and a validation set are generated:

[0021] Step 2.1, at least 1000 pairs of aligned double-time-phase remote sensing change detection image pairs are collected, all the aligned double-time-phase image pairs are cropped to 256x256 size, and then a binary label is made for each change detection image pair, the pixel points containing change targets in the image are marked as positive examples, and the pixel points not containing change targets are marked as negative examples;

[0022] Step 2.2, a Canny operator is used to process each binary label to obtain an edge label image of the target to be detected, and these edge labels form an edge label 1, and then an image dilation processing with a kernel size of 3x3 is performed on the edge label 1 to obtain an edge label 2;

[0023] Step 2.3, data augmentation is performed on the double-time-phase remote sensing image pairs and the change detection labels and the edge labels 1 and 2;

[0024] Step 2.4, all the images and labels after data augmentation are divided according to a ratio of 4:1, the double-time-phase remote sensing image pairs and the change detection labels and the edge label 2 are combined to form a training set 1 and a validation set 1, and the double-time-phase remote sensing image pairs and the change detection labels and the edge label 1 are combined to form a training set 2 and a validation set 2;

[0025] Step 3, the training set and the validation set are used to train the EATDer:

[0026] Step 3.1, the training set 1 is input into the EATDer, a joint loss function is used to calculate the loss value of the change detection prediction result and the target edge prediction result output by the EATDer and the corresponding change detection label and target edge label 2, the gradient descent method is used to iteratively update the parameters of the EATDer network until the joint loss function converges, and a pre-trained EATDer is obtained.

[0027] Step 3.2, input training set 2 into the pre-trained EATDer, calculate the loss value of the change detection prediction result and the target edge prediction result output by the EATDer and the corresponding change detection label and target edge label 1 using the joint loss function, update the parameters of the EATDer network iteratively using the gradient descent method, input the test set into the iterated EATDer for verification every iteration, and take the network parameters with the highest accuracy in the verification result in 100 times as the final trained network parameters, to obtain the trained EATDer;

[0028] Step 4, change detection of remote sensing images:

[0029] After the aligned remote sensing images to be detected are cut into 256*256 size images, they are input into the trained EATDer network, and the output of the EATDer is the prediction result of the change detection.

[0030] Compared with the prior art, the present application has the following advantages:

[0031] Firstly, the present application uses the SAVT neural network submodule based on the Transformer to adaptively compress the sizes of K and V matrices in multi-head attention calculation according to the size of the input feature map, thereby overcoming the large calculation cost of the prior art, greatly reducing the calculation and storage cost of multi-head attention calculation under controllable information loss when the present application detects changes in remote sensing images.

[0032] Secondly, the present application constructs a feature extraction module combined by FRFM and SAVT modules, which can more fully mine the global information of the image, and overcomes the defects of insufficient feature extraction capability and easy to miss detection and false detection of the prior art, thereby improving the accuracy of remote sensing image change detection.

[0033] Thirdly, the present application introduces an edge loss function in the final detection module and uses feature refinement based on the Transformer, thereby overcoming the poor edge retention effect of the detection result of the prior art, and making the detection result output by the network in the present application less likely to have false detection due to detection result sticking. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the present application;

[0035] Figure 2 is a general structure diagram of the EATDer network in the present application;

[0036] Figure 3 is a structure diagram of the SAVT module in the EATDer network of the present application;

[0037] Figure 4 is a structural schematic diagram of the FRFM module in the EATDer network of the present application;

[0038] Figure 5 is a structural schematic diagram of the change detection module in the EATDer network of the present application;

[0039] Figure 6 is a schematic diagram of the edge dataset generated by the present application;

[0040] Figure 7 is a simulation result diagram of remote sensing image change detection using the present application and the existing detection method respectively. DETAILED DESCRIPTION

[0041] The present application will be further described in detail below with reference to the accompanying drawings and examples.

[0042] Referring to Figure 1 , the implementation steps of the embodiments of the present application will be further described in detail.

[0043] Step 1, constructing a graph convolution enhanced convolutional network EATDer.

[0044] Step 1.1, referring to Figure 2 , a feature extractor composed of three first, second and third sub-extractors in series is built.

[0045] Each sub-extractor is composed of two SAVT modules with the same structure connected in series and then connected with a FRFM module. After the input image of 256x256 is passed through the three sub-extractors, three feature maps of different scales of 64x64, 32x32 and 16x16 are output, thereby improving the multi-scale feature extraction capability of the network. Subsequently, these multi-scale features extracted will be sent to the change detection module.

[0046] Referring to Figure 3 , the sub-module SAVT of the constructed EATDer network will be further described. The module includes a feature map encoding process and four layers of SAVT encoders, wherein the feature encoding process is realized by convolution, and each layer of SAVT encoder is composed of Layer Norm layer and adaptive reduction layer and multi-head attention layer in the first half, and Layer Norm layer and MLP layer in the second half:

[0047] For convenience of description, the input and output of the entire SAVT module are introduced first, and then the data processing process inside the SAVT encoder is described in detail.

[0048] Suppose the size of the feature map input to each SAVT module is The feature map is encoded using a convolutional operation with c′ kernels, where c′ is set to 64, 128, and 256 in the three sub-extractors, and the convolution size, stride, and padding are 2s⁻¹, s, and s⁻¹, respectively. The stride is designed to be 4, 2, and 2 according to the order of the sub-extractors. The image patch was then subjected to a dimensional deformation operation to transform the patch into... The size is then input into the SAVT encoder for processing. After processing by the SAVT encoder, the output feature map of this module is obtained. The feature map F out It will be sent to the FRFM module for further processing.

[0049] The input and output feature map sizes of each SAVT encoder are consistent. To facilitate understanding of the SAVT encoder's data processing method, the data processing flow of the first encoder in the four layers will be explained in detail here. The remaining L-layer encoders in the same module adopt the same processing method. The SAVT encoder processes the data P1′ input from the first SAVT encoder as follows:

[0050]

[0051]

[0052] LN(·), SAMSA(·), and MLP(·) represent the functions of the layer-norm layer, the adaptive multi-head attention layer, and the MLP layer, respectively. This represents the encoded features. The SAMSA part mainly includes an adaptive reduction and a regular multi-head attention. In this process, firstly, the P1′ after layer-norm processing is copied three times to generate the data to be processed. and We then used adaptive scaling to process K1 and V1.

[0053] In the adaptive reduction process, K1 is first reshaped into a two-dimensional form. And using average pooling and 1×1 convolution, we obtain: The purpose of average pooling is to reduce computational cost, and the purpose of 1×1 convolution is to reduce the loss of information in the feature maps. After reconstruction, reduced data can be obtained. Similarly, the simplified data of V1 can be obtained, that is... Then the newly obtained data Use a multi-head attention mechanism.

[0054] The multi-head attention mechanism is represented as follows:

[0055]

[0056]

[0057]

[0058] where W O , denote the learnable parameter matrix, concat(·), Attention(·), softmax(·) denote dimension concatenation, self-attention calculation, and softmax function, respectively. is a scaling factor, d k is the value of K dimension. In the embodiment of the present application, the number of heads of multi-head attention is fixed to 4, that is, n = 4. Compared with the standard multi-head attention, the algorithm has lower computational complexity.

[0059] The time complexity of the original multi-head attention and the reduced multi-head attention is calculated as follows,

[0060]

[0061]

[0062] where Ω(MSA) represents the time complexity of multi-head attention, and Ω(SAMSA) represents the time complexity of the adaptive multi-head attention in the present application.

[0063] Referring to Figure 4 , the FRFM submodule includes a cross fusion module and a CBAM module, and the FRFM submodule is the main module for completing information interaction of the twin network double branches.

[0064] The schematic diagram of the cross fusion submodule FRFM is shown in Figure 4 (b). When the outputs from the SAVT modules of the two different branches are input as inputs to the FRFM, the cross fusion submodule first fuses the information of the two different branches to generate For ease of understanding, for the input X, this process can be expressed as:

[0065]

[0066] where X cf denotes the output of the information interaction branch in the FRFM submodule A, denotes a learnable parameter matrix, is a scaling factor, d kx is the value of K x dimension of Q y , Kx ,V x The definition of is:

[0067]

[0068] where, and are learnable weights.

[0069] Similarly, the cross-fusion process of Y is as follows:

[0070]

[0071]

[0072] where, Y cf represents the output of the information interaction branch in the FRFM sub-module B, and represent different parameter matrices available for neural network learning, is a scaling factor, d ky The value of K x is the dimension of.

[0073] X cf and Y cf output by the cross-fusion module are input into the CBAM module, which concatenates channel attention and spatial attention. For X cf , CBAM derives channel attention scores and spatial attention scores

[0074] The obtained scores are multiplied back into X cf to generate refined feature data as follows:

[0075]

[0076]

[0077] where, represents element-wise multiplication. Similarly, the CBAM process of Y cf is as follows:

[0078]

[0079]

[0080] where, represents channel scores and attention scores. By mining significant information, the changing / invariant information in the RS image can be highlighted.

[0081] The combination of feature extraction modules The twin encoder designed in this application consists of three consecutive stages, each of which contains two SAVT blocks and one FRFM. The double temporal RS image {I A ,I B} of size HxWx3 is input into the encoder, and the two SAVT blocks in the first stage map it into feature maps and {F1 A ,F1 B} which will be sent into the FRFM to generate enhanced features After the second and third stages, we get and Set the channels C1, C2, C3 to 64, 128, 256 respectively. These feature maps obtained after the feature extraction module not only involve the complex content of various scales in the remote sensing image, but also contain a lot of temporal information. The enhanced features will be sent to the change detection module for change detection.

[0082] Step 1.2, establish the change detection module.

[0083] Referring to Figure 5 , the recovery submodule is composed of a 3x3 convolution and an upsampling, and the main purpose is to further fuse and splice the multi-scale features extracted by the feature extractor. This process can be described as follows:

[0084]

[0085]

[0086]

[0087] O i ′=Conv 3×3 (O i ),i=1,2,3,

[0088]

[0089] Where O1, O2, O3 represent the outputs of the first, second, and third sub-extractors respectively, O i ′ represents the output feature map after feature fusion by convolution, O m represents the output feature map of the recovery submodule, Conv 3×3 (·) and Up(·) represent 3x3 convolution and upsampling operation respectively.

[0090] Referring to Figure 5, the refining submodule is composed of four 3x3 convolutions, one 1x1 convolution, two refining SAVT modules (only containing one layer of SAVT encoder in the module), two deconvolutions and three Layer Norm layers. The purpose of the design of the refining submodule is to improve the feature map O m for change detection, so as to further highlight the multi-scale information and time clues hidden in O m .

[0091] In the refining submodule, the application captures local knowledge by convolution, two SAVT modules comprehensively analyze global context clues, and introduces deconvolution to complete up-sampling. Among them, O m first passes through 3x3 convolution and SAVT to obtain Then O m ' is sent into two deconvolution and Layer Norm layer combination deconvolution blocks in turn to obtain Finally, O m " applies another two 3x3 convolutions, one 1x1 convolution and one refining SAVT module to obtain the feature map Then the feature map is processed using softmax to obtain

[0092] In order to improve the detection accuracy of the edge, the application increases the edge detection task to ensure that the model can fully perceive the edge, thereby improving the detection performance. The channel dimension of M s is divided into two parts, including a change prediction result and an edge prediction result The joint loss function will be applied to narrow the gap between the change detection prediction result and the target edge prediction result and the corresponding change detection label and target edge label. Considering that the change detection task is a binary classification task, the binary cross-entropy loss is selected as the basic unit of the joint loss function.

[0093] The joint loss function is defined as follows:

[0094]

[0095] Wherein, λ represents a hyperparameter with a value of 0.3, log(·) represents a logarithm operation with a base of 10, G c represents the label value of change detection, M c represents the prediction result of change detection output by the network, G e represents the label value of edge detection, M e represents the prediction result of edge detection output by the network.

[0096] Step 2, generate training set and validation set.

[0097] Step 2.1, the embodiment of the application is to collect 1000 pairs of aligned double-phase remote sensing change detection image pairs, and after all the aligned double-phase image pairs are cropped to 256x256 size, binary labels are made for each change detection image pair, and the pixel points containing change targets in the image are marked as positive examples, and the pixel points not containing change targets are marked as negative examples.

[0098] Step 2.2, the Canny operator is used to process each binary label to obtain an edge label image of the target to be detected, and these edge labels constitute edge label 1, and then edge label 2 is obtained after using a kernel size of 3x3 image dilation processing on edge label 1. The process is as shown in Figure 6

[0099] Step 2.3, data augmentation is performed on the double-phase remote sensing image pairs and the change detection labels and the edge labels 1 and 2.

[0100] Step 2.4, all the images and labels after data augmentation are divided according to 4:1, and the double-phase remote sensing image pairs and the change detection labels and the edge label 2 are combined to form training set 1 and validation set 1, and the double-phase remote sensing image pairs and the change detection labels and the edge label 1 are combined to form training set 2 and validation set 2.

[0101] Step 3, the training set and the validation set are used to train the EATDer.

[0102] Step 3.1, the training set 1 is input into the EATDer, the joint loss function is used to calculate the loss value of the change detection prediction result and the target edge prediction result output by the EATDer and the corresponding change detection label and target edge label 2, the gradient descent method is used to iteratively update the parameters of the EATDer network under the condition that the learning rate is 0.001, until the joint loss function converges, and a pre-trained EATDer is obtained.

[0103] Step 3.2, the training set 2 is input into the pre-trained EATDer, the joint loss function is used to calculate the loss value of the change detection prediction result and the target edge prediction result output by the EATDer and the corresponding change detection label and target edge label 1, the gradient descent method is used to iteratively update the parameters of the EATDer network under the condition that the learning rate is 0.001, and the validation set is input into the iterated EATDer every iteration to verify, and the network parameters with the highest accuracy in the 100 validation results are used as the final trained network parameters, and a trained EATDer is obtained.

[0104] Step 4, obtain the change result image.

[0105] ​After the already aligned remote sensing image to be detected is cropped into a 256*256 size image, the image is input into the trained EATDer network, and the output of the EATDer is the prediction result of the change detection.

[0106] The effect of the present application can be further illustrated by the following simulation.

[0107] I. Simulation conditions.

[0108] The hardware platform of the simulation environment of the present application is that the processor is intel XEON 4214R with a main frequency of 2.4GHz, and the graphics card is GeForce RTX 3090*2 with a total of 48g video memory.

[0109] The software platform of the simulation environment of the present application is ubuntu16.04, the framework of python 3.9.5+pytorch 1.13.

[0110] The input images used in the simulation of the present application are LEVIR-CD remote sensing change detection images, CDD remote sensing change detection images, and WHU remote sensing change detection images.

[0111] LEVIR-CD contains 637 remote sensing images collected by Google Earth, with a size of 1024*1024 pixels. The pixel resolution of these images is 0.5m. LEVIR-CD contains double-phase images with a time span of 5 to 14 years, mainly focusing on the changes of various buildings, such as villa houses and high-rise apartments.

[0112] CDD contains 16000 seasonal remote sensing double-phase image pairs collected from Google Earth. The spatial size is 256*256, and the resolution varies from 0.03m to 1m. In CDD, the land cover changes contained are diverse, including buildings, cars, and roads.

[0113] WHU contains a single remote sensing image. Its spatial size and resolution are 32507*15354 and 0.2m respectively. The main object in the data set is a building.

[0114] II. Simulation content and result analysis.

[0115] Under the above simulation conditions, the present application and the four existing methods ICIF-Net, DSAMNet, and SNUNet are used for change detection simulation on the four data sets respectively. The results are shown in Figure 7 .

[0116] In the simulation experiment, the three existing technologies refer to:

[0117] The change detection method of prior art dense supervision attention metric network DSAMNet is the method for remote sensing image change detection published by Liu Mengxi et al. on IEEE, namely: Liu, Mengxi, and Qian Shi. "DSAMNET: A deeply supervised attention metric based network for change detection of high-resolution images." 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. IEEE, 2021. (DSAMNet method for short).

[0118] The change detection method of prior art densely connected Siamese network SNUNet is the method for remote sensing image change detection published by Fang Sheng et al. on IEEE, namely: Fang, Sheng, et al. "SNUNet-CD: A densely connected Siamese network for change detection of VHR images." IEEE Geoscience and Remote Sensing Letters 19 (2021): 1-5. (SNUNet method for short).

[0119] The change detection method of prior art intra-scale cross-interaction and inter-scale feature fusion network ICIF-Net is the method for remote sensing image change detection published by Feng Yucha et al. on IEEE, namely: Feng, Yuchao, et al. "ICIF-Net: Intra-scale cross-interaction and inter-scale feature fusion network for bitemporal remote sensing images change detection." IEEE Transactions on Geoscience and Remote Sensing 60 (2022): 1-13. (ICIF-Net method for short).

[0120] The simulation effect of the present application is further described below. Figure 7 The simulation effect of the present application is further described below.

[0121] Figure 7 (a), Figure 7 (b),Figure 7 (c) is a contrast algorithm visualization result graph of LEVIR-CD dataset.

[0122] Figure 7 (d), Figure 7 (e), Figure 7 (f) is a contrast algorithm visualization result graph of CDD dataset.

[0123] Figure 7 (g), Figure 7 (h), Figure 7 (i) is a contrast algorithm visualization result graph of WHU dataset.

[0124] From Figure 7 (b) can be seen that the existing DSAMNet method has a large number of detection result adhesion, which is mainly because the method does not emphasize the edge information of the target to be detected, resulting in the case of detection accuracy decline.

[0125] From Figure 7 (g) can be seen that the existing ICIF-Net method, SNUNet method has a large number of false positive detection, which is mainly because the method communicates the double branch information, the feature extraction ability is insufficient, and the background change is mistakenly considered as the change to be detected, resulting in the detection accuracy does not meet the expected case.

[0126] From the visualization result graph, it can be seen that the detection effect of the present application has high precision, less noise, less missed detection and misdiagnosis, and the boundary detection of the changed ground object is more accurate, which fully illustrates the superiority of the method proposed in the present application.

[0127] In order to quantitatively illustrate the performance of the proposed method, the numerical performance indicators commonly used in change detection tasks are selected to measure the difference between the above existing methods and the present application, the performance indicators include change detection precision (P), change detection recall rate (R), F1 score (F1), and total accuracy (OA), and the calculation cost indicators include Params and FLOPs The results are shown in the following table. Among them, Table 1 shows the accuracy comparison results of the present application and the contrast algorithm, and Table 2 shows the calculation cost index comparison results of the present application and the contrast algorithm.

[0128]

[0129]

[0130]

[0131]

[0132] Table 1

[0133]

[0134] Table 2

[0135]

[0136] From the above two tables, it can be seen that the application is relatively higher in four precision quantitative indexes, the overall error of change detection is smaller, the change detection precision is higher, and the two indexes of calculation cost are relatively lower, the calculation cost is relatively small, which further illustrates the superiority of the method proposed by the application.

[0137] The above simulation experiment shows that: the EATDer neural network built by the application can extract the global spatial features of the remote sensing image, and fully pay attention to maintaining the detection accuracy of the image edge, the SAVT module and the FRFM module built can extract the global features of the remote sensing image under reasonable calculation cost, and control the influence of the imbalance of positive and negative samples in the data set itself, the change detection module and the joint loss function which emphasize the edge information can make the network fully pay attention to the edge detection accuracy of the target to be detected, solve the problems of large calculation cost, frequent edge detection adhesion and low precision in the prior art method, and is a very practical remote sensing image change detection method.

Claims

1. A change detection method based on edge-assisted adaptive detection, characterized in that: The method constructs an SAVT module with adaptive reduction and a FRFM module with dual-path information communication, designs a transformation detection module for edge information and a corresponding loss function; the steps of this transformation detection method include the following: Step 1, construct the edge-assisted adaptive change detection network EADDer: Step 1.1: Construct a feature extractor consisting of three identical sub-extractors connected in series. Each sub-extractor consists of two identical SAVT modules connected in parallel and then connected in series with an FRFM module. After the 256×256 input image passes through the three sub-extractors, it outputs three feature maps of 64×64, 32×32, and 16×16. The SAVT module in each sub-extractor consists of a feature map encoding layer and four identically structured adaptive multi-head attention SAVT encoders. The feature map encoding layer is implemented using a convolutional layer. The feature map dimensions of the first to third sub-extractors are set to 64, 128, and 256, respectively; the convolutional kernel sizes are set to 7, 3, and 3, respectively; the stride is set to 4, 2, and 2, respectively; and the padding is set to 3, 1, and 1, respectively. The adaptive multi-head attention SAVT encoder is composed of a Layer Norm layer, an adaptive reduction layer, a multi-head attention layer, another Layer Norm layer, and an MLP layer connected in series. The adaptive reduction layer is formed by average pooling and convolution of the input data. For an image with dimensions H×W×C before the adaptive reduction layer, the size after average pooling is... in, For the round-up operation, the kernel size, stride, and padding of the convolutions in the adaptive shrinking layer are set to 1, 1, and 0 respectively, and the number of heads for multi-head attention in different sub-extractors is fixed at 4. Each sub-extractor's FRFM module consists of two identical FRFM sub-modules A and B connected in parallel. Each FRFM sub-module is composed of an information interaction branch connected in series with a CBAM structure. The information interaction branch receives two output feature maps {X,Y} processed by the parallel SAVT module as its input. The information interaction branch in FRFM sub-module A is implemented by the following formula: Among them, X cf This represents the output of the information exchange branch in FRFM submodule A. Let d represent different parameter matrices that can be learned by the neural network, softmax(·) denotes the softmax function, and d kx The scaling factor represents the value of the scaling factor. The dimensions are equal; similarly, the formula for the crossover and fusion process of branch B is expressed as: Among them, Y cf This indicates the output of the information exchange branch in FRFM submodule B. and d represents different parameter matrices that can be learned by neural networks. ky The scaling factor represents the value of the scaling factor. The dimensions are equal; Step 1.2: Establish a change detection module consisting of a recovery submodule and a refinement submodule connected in series; The recovery submodule consists of a 3×3 convolutional layer and an upsampling layer connected in series; the refinement submodule consists of a 3×3 convolutional layer, an SAVT module, a 2×2 deconvolutional layer, a Batch Norm layer, a 2×2 deconvolutional layer, a Batch Norm layer, a 3×3 convolutional layer, an SAVT module, a 3×3 convolutional layer, a Batch Norm layer, and a 1×1 convolutional layer connected in series. Step 1.3: Connect the feature extractor and the change detection module in series to form the edge-assisted adaptive network EADDer; Step 2, generate training and validation sets: Step 2.1: Collect at least 1000 pairs of aligned dual-temporal remote sensing change detection image pairs. After cropping all aligned dual-temporal image pairs to a size of 256×256, create a binarized label for each change detection image pair. Mark the pixels in the image containing the changed target as positive examples and the pixels without the changed target as negative examples. Step 2.2: The Canny operator is used to process each binarized label to obtain the edge label image of the target label to be detected. These edge labels are used to form edge label 1. Then, edge label 1 is subjected to image dilation processing with a kernel size of 3×3 to obtain edge label 2. Step 2.3: Perform data augmentation on the dual-temporal remote sensing image pairs, change detection labels, and edge labels 1 and 2; Step 2.4: Divide all the data-enhanced images and labels into a 4:1 ratio. Combine the dual-temporal remote sensing image pairs, change detection labels, and edge labels 2 to form training set 1 and validation set 1. Combine the dual-temporal remote sensing image pairs, change detection labels, and edge labels 1 to form training set 2 and validation set 2. Step 3, train EADDer using the training and validation sets: Step 3.1: Input training set 1 into EADDer, use the joint loss function to calculate the loss value of the change detection prediction result and target edge prediction result output by EADDer and the corresponding change detection label and target edge label 2, use gradient descent to iteratively update the parameters of the EADDer network until the joint loss function converges, and obtain the pre-trained EADDer. Step 3.2: Input training set 2 into the pre-trained EADDer, use the joint loss function to calculate the loss value of the change detection prediction result and target edge prediction result output by EADDer and the corresponding change detection label and target edge label 1, use gradient descent to iteratively update the parameters of the EADDer network, input the verification set into the iterated EADDer for verification in each iteration, and take the network parameters with the highest accuracy in 100 verification results as the final network parameters after training, and obtain the trained EADDer; Step 4: Perform change detection on the remote sensing image: After the aligned remote sensing image to be detected is cropped to a size of 256×256, it is input into the trained EADDer network. The output of EADDer is the prediction result of change detection.

2. The edge-assisted adaptive change detection method according to claim 1, characterized in that, The joint loss function described in steps 3.1 and 3.2 is as follows: Where λ represents a hyperparameter with a value of 0.3, log(·) represents a logarithmic operation to the base 10, and G c M represents the label value for change detection. c G represents the prediction result of the change detection in the network output. e M represents the label value for edge detection. e This represents the edge detection prediction result output by the network.

Images