Change detection method and system of deep feature forward difference extraction and backward backtracking correction

Abstract

The application discloses a kind of deep feature forward difference extraction and backward trace correction change detection method and system, method includes: in deep feature forward difference extraction module carries out difference feature extraction, trains deep twin network to extract double time phase image multi-scale feature, designs difference detection module to obtain multi-scale difference, simultaneously channel attention module and spatial attention module are applied to difference detection module, so that difference feature is more distinguishable;Difference is corrected in backward trace correction module, utilize difference correction unit to learn more accurate boundary feature, avoid boundary detection result and change area prediction result separation, from the deepest network layer starts backtracking correction, by applying multiple boundary information guide module to different convolution layer, finally generate more accurate change result.The application can effectively handle the problem that small change is not easy to detect, building boundary is fuzzy in change detection, improve the accuracy of change detection.

Description

TECHNICAL FIELD

[0001] The present application relates to remote sensing image change detection technology, and particularly to a change detection method and system of deep feature forward difference extraction and backward backtracking correction. BACKGROUND

[0002] Remote sensing image change detection is an important application in the field of remote sensing, which aims to obtain the difference of objects occurring at different times through algorithm, and is one of the landmark tasks in the field of computer vision. At present, change detection has been applied to many aspects, for example, urban expansion, disaster assessment, land cover investigation, ecosystem detection, resource management. Change detection has been proven to be an indispensable application in the field of remote sensing. With the development of satellite remote sensing and aerial photography technology, high-resolution images have received more and more attention and have been continuously improved in availability. High-resolution images contain rich texture, image, structure, neighborhood relationship and other information of ground objects, which makes the potential application range of high-resolution images in the field of change detection continuously expand, and has become the key source of change detection raw data. However, the lack of prior knowledge (such as boundary structure information) leads to inaccurate detection results, especially in the building boundary area. In addition, with the continuous improvement of the spatial resolution of remote sensing images, more and more ground object textures and spatial details are observed, which also leads to the difficulty in detecting small changes. The above factors have become the challenge that cannot be ignored in high-resolution remote sensing images.

[0003] In recent years, due to the strong ability of deep learning in capturing and representing image features, deep learning has been booming in the field of computer vision, and many researchers have successfully applied deep learning to change detection. Daudt compared the twin network FC-Siam-conc, FC-Siam-diff with the early fusion network FC-EF, and proved the effectiveness of the deep twin network. However, the extracted features are easily affected by factors such as light scale, noise, etc., resulting in serious false changes. DSAMNet adopts a multi-level feature fusion scheme, which combines shallow features containing local information with deep features representing abstract information, improving the detection performance. STANet designs a self-attention mechanism to utilize the rich dependencies in the spatial-temporal dimension, producing more discriminative features. Due to the complexity of the scene, how to accurately detect the changed pixels of the changed building is still a challenge, for example, the dense distribution of buildings, the influence of shadows and the similarity between buildings and roads will affect the results of change detection. Boundary structure information as a typical priori is very suitable for building change detection. Buildings, as a relatively fixed shape object, can be described using boundary information. First, boundary information not only reflects the geometric shape of the building, but also reflects the location information of the building. Effective use of boundary information will guide the process of building change detection and produce better prediction results. Second, the two tasks of change detection and boundary evaluation are closely related and can be jointly trained in a unified framework to improve the robustness of the deep learning network. Third, boundary detection operators such as Canny operator can quickly and accurately extract boundary prior information based on the training sample set of the boundary. Based on the above considerations, a deep feature forward difference extraction and backward backtracking correction change detection method and system is developed, which uses boundary information to guide the continuous correction of change detection results. SUMMARY

[0004] The purpose of the present application is to provide a deep feature forward difference extraction and backward backtracking correction change detection method and system, which introduces prior-based boundary structure to correct the wrongly classified change information, can avoid the problem of fuzzy building boundary, generate more accurate change boundary, and effectively detect the small changes in the dual-time remote sensing image, improve the change detection result.

[0005] The technical solution for achieving the purpose of the present application is:

[0006] In a first aspect, the present application provides a deep feature forward difference extraction and backward backtracking correction change detection method, comprising the following steps:

[0007] First, input the dual-time image x1, x2, and perform image preprocessing;

[0008] Second step, construct a deep twin feature extraction network, use VGG network as a deep twin network feature extractor, extract double time image features at multiple scales respectively, set two memories to save the obtained double time image multi-scale features;

[0009] Third step, the difference detection module obtains the image multi-scale difference feature, maps and splices the double time feature at each scale stored in the second step, simultaneously applies a convolution attention module composed of a channel attention module and a spatial attention module to obtain an image multi-scale difference feature map, saves it to the memory, and completes the construction of the deep feature forward difference extraction module.

[0010] Fourth step, the difference correction unit learns boundary features, inputs the deepest layer of difference feature map into the difference correction unit to obtain boundary feature information, and combines the change information and boundary structure priori information into a network.

[0011] Fifth step, the boundary information guiding module continuously corrects the difference feature, splices the obtained boundary feature information and the difference feature map of the previous layer, and then inputs it into the boundary information guiding module to continuously correct the difference feature; the obtained result is re-input into the difference correction unit, and the above steps are repeated to continuously backtrack to the front, and correct the difference feature of each layer; the output result of the last layer is taken as the final change detection result, and the construction of the deep feature backward trace correction module is completed.

[0012] Sixth step, each difference correction unit includes a change area and a boundary detection; for the change area, a contrast loss is applied, and for the boundary detection, a Dice loss is applied, and a combined loss is used to realize the training of the model.

[0013] Secondly, the application provides a deep feature forward difference extraction and backward trace correction change detection system, comprising:

[0014] (1) Deep feature forward difference extraction module:

[0015] The feature extraction module is composed of a feature extractor of VGG network, which extracts double time image features at multiple scales respectively.

[0016] The difference detection module splices the double time feature map at each scale as the input of the difference detection module, and simultaneously applies a convolution attention module composed of a channel attention module and a spatial attention module to obtain an image multi-scale difference feature map.

[0017] (2) Deep feature backward trace correction module:

[0018] The difference correction unit inputs the difference feature map into the difference correction unit to obtain boundary feature information.

[0019] The boundary information guiding module corrects the difference feature in a backward direction.

[0020] In a third aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0021] In a fourth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0022] Compared with the prior art, the present application has the following significant features: (1) a deep feature forward difference extraction module is designed, a VGG network is taken as a deep twin network feature extractor, double-time image features are extracted at multiple scales, a difference detection module is constructed to obtain multi-scale difference features; (2) channel attention and spatial attention are applied to the difference detection module, so that the difference features are more easily distinguished, thereby more image difference features are extracted; (3) change information and boundary structure prior information are combined into one network, avoiding separation of the boundary detection result and the change area prediction branch; (4) by applying multiple boundary information guiding modules to different convolution layers, the difference feature information is constantly corrected in a backward direction, and finally more accurate building change boundaries are generated, thereby improving the change detection result.

[0023] The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a structural diagram of the method of the present application.

[0025] Figure 2 is a structural diagram of the difference detection module.

[0026] Figure 3 is a structural diagram of the channel attention module.

[0027] Figure 4 is a structural diagram of the spatial attention module.

[0028] Figure 5 is a structural diagram of the difference correction unit.

[0029] Figure 6 is a structural diagram of the boundary information guiding module.

[0030] Figure 7 is a comparison result diagram of the CDD data set experiment.

[0031] Figure 8 is a comparison result diagram of the WHU data set experiment. DETAILED DESCRIPTION

[0032] Compared with the existing method in the background art, the present application proposes a change detection method of deep feature forward difference extraction and backward trace correction. In the deep feature forward difference extraction module, difference feature extraction is performed layer by layer. A deep twin network based on VGG is trained to extract multi-scale features of double temporal images. A difference detection module is designed to obtain the difference at each scale. At the same time, a channel attention module and a spatial attention module are applied to the difference detection module to extract more difference features. In the backward trace correction module, the difference is corrected. A difference correction unit is used to learn more accurate boundary features. The change information and the boundary structure priori information are combined into a network. Starting from the deepest layer, the boundary feature information is mapped and spliced with the difference features of the previous layer, and then input into a boundary information guiding module to correct the difference features. The obtained results are re-input into the difference correction unit, and the above steps are repeated to trace and correct the difference features of each layer, thereby improving the quality of the change detection results. The results output by the last layer are taken as the final change detection results.

[0033] The implementation process of the present application will be described in detail below. Figure 1

[0034] A change detection method of deep feature forward difference extraction and backward trace correction, comprising the following steps:

[0035] Firstly, input double temporal images x1 and x2 for image preprocessing. The specific process is as follows:

[0036] (1) The high-resolution remote sensing image used by us has a size of HxWx3, where H represents the picture height, W represents the picture width, and the channel number of the image is 3. In order to increase the diversity of training data, image augmentation is performed in the preprocessing operation: 1) image rotation: the double temporal images are rotated by 45°, 90°, 135° and 180° and 270° respectively; 2) image flipping: the double temporal images are horizontally flipped; 3) image noise addition: 200 salt and pepper noises are randomly added to the T1 image; 4) image blurring: a Gaussian blur filter is applied to the T1 image to generate a blurred image; 5) image smoothing: a smoothing filter is applied to the T1 image to generate a smoothed T1 image. Different operations are applied to T1 and T2 to enhance the robustness of the model.

[0037] Secondly, a deep twin feature extraction network is constructed. VGG network is used as a deep twin network feature extractor to extract double temporal image features at multiple scales. Two memories are set to save the obtained multi-scale features of double temporal images. The specific process is as follows:

[0038] ​(1) The application adopts VGG-16Net to constitute two branches of a deep twin network as our feature extractor. The dual-phase images are input into the network to obtain feature maps of different scales. The feature extractor is divided into five stages.

[0039] (2) The composition of the feature extractor is as follows:

[0040] The feature extractor has five layers and performs five-stage feature extraction. The first layer is subjected to twice convolution with a kernel of 3*3 and a step of 1, and the output feature layer is 64, obtaining F1, and the image size is (H, W, 64); after a down-sampling layer, the second stage is entered, and the output feature layer is subjected to twice convolution with a kernel of 3*3 and a step of 1, and the output feature layer is 128, obtaining F2, and the image size is (H / 2, W / 2, 128); after a down-sampling layer, the third stage is entered, and the output feature layer is subjected to twice convolution with a kernel of 3*3 and a step of 1, and the output feature layer is 256, obtaining F3, and the image size is (H / 4, W / 4, 256); after a down-sampling layer, the fourth stage is entered, and the output feature layer is subjected to three times convolution with a kernel of 3*3 and a step of 1, and the output feature layer is 512, obtaining F4, and the image size is (H / 8, W / 8, 512); and the fourth stage is entered, and the output feature layer is subjected to three times convolution with a kernel of 3*3 and a step of 1, and the output feature layer is 512, obtaining F5, and the image size is (H / 16, W / 16, 512).

[0041] (3) The obtained multi-scale feature maps are stored in two memories respectively as the input of the difference detection module, which is the basis for obtaining multi-scale difference feature maps.

[0042] The third step is that the difference detection module obtains the multi-scale difference features of the images, splices the dual-phase feature maps of each scale stored in the second step, simultaneously applies a convolution attention module composed of a channel attention module and a spatial attention module, makes the difference features more easily distinguishable, obtains the multi-scale difference feature maps of the images, saves them to the memory, completes the construction of the deep feature forward difference extraction module, and the specific process is as follows:

[0043] (1) First, in order to obtain more distinctive features, a convolution attention module is designed on the difference module. The convolution attention module is composed of two sub-modules, a channel attention module and a spatial attention module, which realize feature map refinement in the channel and spatial fields respectively.

[0044] The channel attention module is used to emphasize the target related channels while suppressing the irrelevant channels, and the channel attention module can be calculated by using the following module:

[0045] M c (F)=σ(MLP r (AvgPool(F))+MLP r(MaxPool(F))) (1)

[0046] F represents the input feature of size H x W x C, on which average pooling (AvgPool) and max pooling (MaxPool) are used to generate aggregated vectors of size 1 x 1 x C, and then a weight-shared multi-layer perception (MLP) module with a channel reduction rate r is applied to each vector to give each channel a weight. The MLP contains two 1 x 1 convolutional layers, where the first convolution reduces the channels of the input feature by r times, and the second convolution restores the channel number to the size of the original input image, and the two layers are connected by a Relu activation function. Channel attention map M c (F) is obtained by using the sum of the above two elements through a sigmoid activation function σ. Finally, the original feature is multiplied by the channel attention map to obtain the channel refined feature

[0047]

[0048] wherein represents element-wise multiplication.

[0049] (2) The channel-refined feature is further refined by a spatial attention module. Similarly, the importance of each pixel position in the feature map is encoded in the spatial attention map. Through the feedback of the ground truth iteratively accepted during the network training process, the spatial attention module is trained to have the power to adaptively recalibrate the weight of each pixel position, and finally output a spatial attention map in which the positions of changed and unchanged pixels are assigned to higher weights and lower weights. The changed pixels are emphasized by multiplying higher weights, while the unchanged pixels are suppressed by multiplying lower weights. In this way, the network can quickly approach the changed area. The spatial attention submodule also uses average pooling (AvgPool) and max pooling (MaxPool) as the first step of processing, compressing the input channel-refined feature F into two H x W x 1 matrices, connecting them together, and inputting them to a convolutional layer with a kernel of 7. Finally, a sigmoid activation function σ is used to obtain the final spatial attention map M s (F), which can be represented as follows:

[0050] M s (F) = σ (f (7×7) (AvgPool(F); MaxPool(F))) (3)

[0051] wherein f (7×7) represents a 7 x 7 convolution operation, and [;] represents a concatenation operation. Finally, the original feature is multiplied by the spatial attention map to obtain the spatial-refined feature

[0052]

[0053] (3) From the deepest fifth layer network, the double time phase image of each layer is spliced and input into the convolution attention module to obtain more discriminative difference features D i :

[0054]

[0055] D i =M s (Conv(M c (F i ))) (6)

[0056] Where represents splicing, x 1- C i and x 2- C i represent the double time phase feature map of the i-th layer, Conv represents a set of convolution operations to obtain the original channel number. After processing by the convolution attention module, the feature value distance of the change area and the unchanged area is enlarged, which is convenient for distinguishing.

[0057] (4) Finally, the difference feature map of each layer is unified to 64x64x64 size and saved to the memory, and the construction of the deep feature forward difference extraction module is completed.

[0058] The fourth step is to learn the boundary feature by inputting the difference feature map of the deepest layer into the difference correction unit to obtain the boundary feature information, combining the change information and the boundary structure priori information into one network, avoiding the separation of the boundary detection result and the change area prediction branch. The specific process is as follows:

[0059] (1) First, a 3x3 convolution layer is applied to the difference feature D5 obtained in the fifth layer to obtain semantic features:

[0060] S5=Conv3(D5) (7)

[0061] The channel number of S5 is consistent with D5, and then the semantic features are used to predict the area change map, and a 1x1 convolution layer is applied to S5 to obtain a binary area change map

[0062]

[0063] The boundary feature B5 is extracted directly in the change map to obtain the boundary information related to the difference object. This operation can be given by the following expression:

[0064]

[0065] where the number of feature maps output by each convolutional layer is 64. Similarly, the boundary map can be generated using the boundary features

[0066]

[0067] Equations 7-10 collectively form a difference correction unit that includes change region and boundary detection.

[0068] (2) The D4 is fused with the semantic and boundary features generated in the previous stage to obtain complementary features C4:

[0069] C4 = Conv3(Cat(D4, S5, B5)) (11)

[0070] In the fifth step, the boundary information guiding module continuously corrects the difference features, maps and concatenates the obtained boundary feature information and the difference features of the previous layer, and then inputs them into the boundary information guiding module to continuously correct the difference features. The obtained results are re-input into the difference correction unit, and the above steps are repeated to continuously backtrack to the front, and the difference features of each layer are corrected to improve the quality of the change detection results. The results output by the last layer are taken as the final change detection results, and the construction of the deep feature backtracking correction module is completed. The specific process is as follows:

[0071] (1) First, F is uniformly divided into N sub-layers along the channel dimension, i.e. F = {F1, F2, …, N}, where F i (i = 1, 2, … N) contains C / N feature maps, and C represents the number of feature maps of F. Although the N sub-layers have the same number of feature maps, their ability to express boundary information can be different. In order to select the sub-layers containing accurate boundary information from these sub-layers, the boundary information of each sub-layer is compared with the guide map to obtain the boundary confidence (K i ) of each sub-layer:

[0072] K i = σ(Conv1(Conv3(F i ))) (12)

[0073] where σ represents the Softmax activation function, Conv3 and Conv1 represent 3x3 and 1x1 convolutional layers, respectively, and output feature maps with channel numbers of C / N and 2, respectively.

[0074] (2) Subsequently, all confidence maps and guide Figure 1 maps are combined to calculate the accuracy of the boundary information contained in each confidence map, which can be described by the following expression:

[0075]

[0076] denotes element-wise multiplication, sim i denotes K i and B. sim i is the calculation of K i boundary information corresponding to the pixel, sim i the greater the value, K i the greater the likelihood of belonging to the boundary pixel. Formula (13) calculates the similarity between each pixel of K i and B, and then calculates the overall similarity OS:

[0077]

[0078] where H i and W i represent the height and width of the feature map respectively. The value of OS i summarizes the boundary information contained in a sub-layer as a whole, and when the sub-layer contains more abundant boundary information, the corresponding OS value is also higher. Therefore, we dynamically assign weights to all sub-layers according to the obtained OS value to emphasize the importance, and the calculation of the weight is

[0079]

[0080] (3) After obtaining the weight, combine each sub-layer and the corresponding weight to determine the contribution, and refine the feature layer as follows:

[0081]

[0082] where denotes that all pixel values in F i are multiplied by w i . Finally, combine all sub-layers together to generate as the output of the boundary guidance module.

[0083]

[0084] The output of the convolution operation consists of C channels, and Cat(*) is the concatenation of the feature map. After correcting the information in F and further extracting features, contains more object details than F, which is more conducive to the recovery of the boundary of the changing object.

[0085] (4) Input C4 obtained in the previous step to the designed boundary information guidance module (BIG) to refine the difference features. The generated in the previous stage is used as a boundary guidance map, which is concatenated with C4 to generate an enhanced difference feature layer, which is specifically represented as:

[0086]

[0087] where, represent the enhanced feature layers. Then, are sent into the difference correction unit to predict the change region and the boundary.

[0088] (5) The remaining difference features D3, D2 and D1 are applied with the same processing as D4 by constantly tracing back. When processing, the designed semantic and boundary features, and the boundary guidance map are all generated from their corresponding previous stage. Therefore, the proposed network contains five difference correction units, which constitutes a five-stage detection scheme. By learning the boundary information with the help of the boundary guidance module, the features are constantly refined, and the next detection stage becomes more accurate than the previous one. Finally, the change map predicted by the last difference correction unit is taken as the final output of the network, and the construction of the deep feature backward correction module is completed.

[0089] Step 6, each difference correction unit contains change region and boundary information. For the obtained change region, a contrast loss is applied for training, and for the boundary information, a Dice loss is applied for training, and the training of the model is realized by using the combined loss, and the specific process is as follows:

[0090] (1) Each difference correction unit contains change region and boundary information. For the obtained change region, considering that the unchanged region is much larger than the changed region, a contrast loss is applied for training, which solves the problem of unbalanced positive and negative training samples in change detection, and the contrast loss can be expressed as:

[0091]

[0092] where, y i，j , d i，j respectively represent the values of the ground truth and the change region at the point (x, y), n u , n c respectively represent the number of unchanged pixels and the number of changed pixels, and 0 represents unchanged pixels and 1 represents changed pixels. m represents the edge distance of the pixel pair filtered out.

[0093] (2) For the information features of the difference correction unit, Dice loss is used for training,

[0094]

[0095] where, y i，j respectively represent the boundary prediction map and the ground truth.

[0096] (3) The loss composed of the contrast loss and the Dice loss function is:

[0097]

[0098] where λ is the equilibrium factor, and are the change region prediction and boundary prediction results obtained by the difference correction unit, GT R is the ground truth of the change region, GT B is the boundary ground truth, and N represents the number of network layers.

[0099] Based on the same idea, the present application provides a change detection system for deep feature forward difference extraction and backward trace correction, like Figures 2-6 , comprising:

[0100] (1) a deep feature forward difference extraction module:

[0101] a feature extraction module, a feature extractor composed of a VGG network, which extracts double-time image features at multiple scales.

[0102] a difference detection module, which maps and splices the double-time features at each scale as the input of the difference detection module, and applies a convolution attention module composed of a channel attention module and a spatial attention module to make the features more distinguishable and obtain image multi-scale difference feature maps.

[0103] (2) a deep feature backward trace correction module:

[0104] a difference correction unit, which inputs the difference feature map into the difference correction unit to obtain boundary feature information;

[0105] a boundary information guiding module, which corrects the difference features in a backward manner to improve the quality of the change detection results.

[0106] The specific implementation of each module corresponds to the first to sixth steps of the method described above, and will not be repeated here.

[0107] Further, the present application also provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps of the aforementioned change detection method.

[0108] Further, the present application also provides a computer program product comprising a computer program, which is executed by a processor to implement the steps of the aforementioned change detection method.

[0109] As can be seen from the above, the innovation of the present application lies in:

[0110] 1) VGG network is adopted as a deep twin network feature extractor to extract double-time image features at multiple scales;

[0111] 2) Apply channel attention and spatial attention to the network to improve the network's ability to identify differences;

[0112] 3) Combine change information and boundary structure prior information into a network to avoid separation of boundary detection results and change area prediction branches;

[0113] 4) By applying multiple boundary information guiding modules to different convolutional layers, constantly correcting multi-scale difference feature information, and finally generating more accurate building change boundaries to improve change detection results.

[0114] The effects of the present application can be further illustrated by the following simulation experiment:

[0115] Simulation conditions

[0116] The simulation experiment uses two groups of high-resolution remote sensing image data: CDD data set and WHU data set. The CDD data set includes 16000 pairs of real seasonal remote sensing change images, each image size is 256*256, the spatial resolution of the image is 0.3 to 1 meter, 10000 pairs and 3000 pairs of images are used for model training and verification, and the remaining 3000 pairs of images are used for model testing. The WHU data set covers the area where a 6.3 magnitude earthquake occurred in Christchurch, New Zealand in February 2011, and contains a pair of images taken in the same area in 2012 and 2016. Each image is 32507*15354 pixels in size, then it is cropped into multiple non-overlapping image blocks with a size of 256*256 for subsequent experiments. The images are divided into training set, validation set and test set, the ratio is 7:1:2, that is, the training set is 5205, the validation set is 743, and the test set is 1486.

[0117] The two groups of experiments take precision, recall, IoU and F1 as evaluation indexes. In addition, the comparison methods include: FC-EF, FC-Siam-conc, FC-Siam-diff, STANet and DSAMNet.

[0118] The network optimizer in the experiment uses Adam optimizer, wherein beta1 is 0.9, beta2 is 0.999, weight decay rate is 0.01, learning rate is 0.0001, batch size is 16, and iteration number is set to 200. The simulation experiment is completed under Windows operating system using Python3.7+pytorch1.8+cuda11.2.

[0119] Analysis of simulation experiment results

[0120] Table 1 and Table 2 are the precision, recall, IoU and F1 (%) of the simulation experiment of the method of the application on the CDD data set and the WHU data set, Figure 7 and Figure 8 are the visualization results of the CDD data set and the WHU data set, respectively, wherein (a) and (b) are the images before and after the change, respectively, (c) is the ground truth map, (d)-(h) are the comparative methods shown in Table 1 and Table 2, respectively, and (i) is the change result map of the application.

[0121] Table 1 Classification results of different methods on the CDD data set

[0122] Comparative method Precision Recall IoU F1 FC-EF 83.90 70.61 62.18 76.68 FC-Siam-conc 93.27 74.71 70.89 82.97 FC-Siam-diff 93.60 72.60 69.16 81.77 STANet 86.27 90.42 79.05 88.30 DSAMNet 92.21 92.72 85.98 92.46 The present invention 96.59 93.57 90.58 95.06

[0123] Table 2 Classification results of different methods on the WHU data set

[0124] Comparative method Precision Recall IoU F1 FC-EF 91.71 73.85 69.23 81.82 FC-Siam-conc 83.35 88.34 82.90 85.77 FC-Siam-diff 83.26 84.43 72.18 83.84 STANet 91.18 89.19 82.11 90.18 DSAMNet 82.96 91.44 76.98 86.99 The present invention 94.72 91.67 87.22 93.17

[0125] From the experimental results, we can find that the accuracy of the detection of the two data sets can be significantly improved by using the method of the application. On the CDD data set, compared with other methods, the method of the application obtains the best results in terms of precision, recall, IoU and F1, and the F1 is 2.6% higher than the second highest DASMNet, and in Figure 7 The visualization results of the CDD data set and the WHU data set can be seen that the object boundaries detected by FC-EF, FC-Siam-conc and FC-Siam-diff are not complete due to the influence of shadows, and for some small buildings, STANet and DSAMNet are prone to adhesion phenomenon and the boundary is not clear, and the results of the application are closest to the ground truth map, which benefits from the combination of change information and boundary structure prior information in the network, and more accurate boundary information can be obtained by the method. On the WHU data set, the method of the application also obtains the best results in the four evaluation indexes, and the F1 is 2.99% higher than the second highest STANet, and in Figure 8 The visualization results of the CDD data set and the WHU data set can be seen that the method of the application can detect the micro changes that cannot be detected by general methods, which benefits from the application of attention mechanism to difference detection, improves the ability of difference recognition, and the application of multiple boundary information guiding modules to different convolution layers makes the multi-scale difference feature information constantly corrected, and improves the change detection result. The above results fully show that the method of the application can effectively learn the feature information of remote sensing images, and the best results are obtained on different data sets, which proves the effectiveness and robustness of the application.

Claims

1. A change detection method of deep feature forward difference extraction and backward trace correction, characterized in that, The method comprises the following steps: The first step is to input the dual-time images x1 and x2 and perform image preprocessing; The second step is to construct a deep twin feature extraction network, use a VGG network as a deep twin network feature extractor, extract dual-time image features at multiple scales, and set two memories to save the obtained dual-time image multi-scale features; The third step is for a difference detection module to obtain image multi-scale difference features, map and splice the dual-time features at each scale saved in the second step, apply a convolution attention module composed of a channel attention module and a spatial attention module, obtain image multi-scale difference feature maps, save them to the memory, and complete the construction of the deep feature forward difference extraction module; The fourth step is for a difference correction unit to learn boundary features, input the difference feature map at the deepest layer into the difference correction unit to obtain boundary feature information, combine the change information and the boundary structure priori information into one network, and the specific process is as follows: (1) first, a 3x3 convolution layer is applied to the difference feature D5 obtained at the fifth layer to obtain semantic features: S5=Conv3(D5) (7) The number of channels of S5 is consistent with D5, and then the semantic features are used to predict the region change graph, a 1x1 convolution layer is applied to S5 to obtain a two-class region change graph The boundary feature B5 is extracted directly in the change map to obtain boundary information related to the difference object; this operation can be given by the following expression: wherein the number of feature maps output by each convolutional layer is 64; a boundary feature is used to generate a boundary map Equations (7)-(10) jointly constitute a difference correction unit containing change area and boundary detection; (2) D4 is fused with the semantic and boundary features generated in the previous stage to obtain complementary features C4: C4=Conv3(Cat(D4,S5,B5)) (11) The fifth step is to continuously correct the difference features by a boundary information guiding module, splice the obtained boundary feature information and the difference feature map of the previous layer, and then input them into the boundary information guiding module to continuously correct the difference features; the obtained result is re-input into the difference correction unit, and the above steps are repeated to continuously backtrack to the front, and the difference features of each layer are corrected; the output result of the last layer is taken as the final change detection result, and the construction of the deep feature backward correction module is completed; the specific process is as follows: (1) First, F is uniformly divided into N sub-layers along the channel dimension, that is, F = {F1, F2, …, N}, where F i contains C / N feature maps, c represents the number of feature maps of F, i = 1, 2, … N; the boundary information of each sub-layer is compared with the guide map to obtain the boundary confidence K i of each sub-layer: K i = σ(Conv1(Conv3(F i ))) (12) Wherein, sigma represents a softmax activation function, Conv3 and Conv1 represent 3x3 and 1x1 convolution layers respectively, and output a feature map with a channel number of C / N and 2 respectively; (2) subsequently, all confidence maps and guide maps are combined one by one, the accuracy of the boundary information contained in each confidence map is calculated, and the following expression is used to describe it: represents element-wise multiplication, sim i represents K i and B; sim i is calculated by finding the pixel corresponding to the boundary information on K i ; sim i The larger the value, the greater the likelihood that K i belongs to the boundary pixel; formula (13) calculates the similarity of each pixel between K i and B, and then calculates the overall similarity OS: where H i and W i represent the height and width of the feature map respectively; OS i values the boundary information contained in one sub-layer as a whole; according to the obtained OS values, weights are dynamically assigned to all sub-layers, and the calculation of the weights is (3) After obtaining the weight, combine each sub-layer and the corresponding weight to determine the contribution degree, and refine the feature layer Generated as follows: where, ⊙ represents that all pixel values in F i are multiplied by w i at the same time; finally, all sub-layers are combined together to generate as the output of the boundary guiding module; The output of the convolution operation consists of C channels, where Cat(*) is the concatenation of feature maps; (4) input the C4 obtained in the previous step into the designed boundary information guiding module to refine the differential features; the boundary information guiding module is generated in the previous stage As a boundary guiding map, spliced with the C4, an enhanced differential feature layer is generated, which is specifically represented as: wherein, representing an enhanced feature layer; then, are fed into a difference correction unit to predict the change region and the boundary; (5) continuously backtracking to the front, the remaining difference features D3, D2 and D1 are applied with the same processing as D4; when processing, the designed semantic and boundary features, and the boundary guide map are all generated from the previous stage; therefore, the proposed network contains five difference correction units, and a five-stage detection scheme is constructed; by learning boundary information with the help of the boundary guiding module, the features are continuously refined, and the next detection stage becomes more accurate than the previous one; finally, the change map predicted by the last difference correction unit is taken as the final output of the network, and the construction of the deep feature backward correction module is completed; The sixth step, each difference correction unit includes a change area and a boundary detection; for the change area, a contrast loss is applied, for the boundary detection, a Dice loss is applied, and the training of the model is realized by using a combined loss.

2. The method of claim 1, wherein the method further comprises: The first step is to input the dual-phase images x1 and x2, and to perform image preprocessing, the specific process being: The high-resolution remote sensing image has a size of HxWx3, where H represents the height of the picture, W represents the width of the picture, and the number of channels of the image is 3; the preprocessing operation is to perform image augmentation: 1) image rotation: the dual-phase images are rotated by 45°, 90°, 135° and 180° and 270° respectively; 2) image flipping: the dual-phase images are horizontally flipped; 3) image noise addition: 200 salt and pepper noises are randomly added to the image x1; 4) image blurring: a Gaussian blur filter is applied to x1 to produce a blurred image; 5) image smoothing: a smoothing filter is applied to x1 to produce a smoothed x1 image; different operations are applied to x1 and x2 to enhance the robustness of the model.

3. The deep feature forward difference extraction and backward backtracking correction change detection method according to claim 1, characterized in that, The second step is to construct a deep twin feature extraction network, using a VGG network as a deep twin network feature extractor to extract dual-phase image features at multiple scales, and setting two memories to save the obtained dual-phase image multi-scale features, the specific process being: (1) using VGG-16Net to constitute two branches of the deep twin network as the feature extractor; inputting the dual-phase images into the network to obtain feature maps of different scales; the feature extractor is divided into five stages; (2) the composition of the feature extractor is as follows: The feature extractor has five layers and performs five-stage feature extraction; the first layer is subjected to two times of convolution with a kernel of 3x3 and a step of 1, the output feature layer is 64, F1 is obtained, and the image size is (H, W, 64); after passing through a downsampling layer, it enters the second stage, is subjected to two times of convolution with a kernel of 3x3 and a step of 1, the output feature layer is 128, F2 is obtained, and the image size is (H / 2, W / 2, 128); after passing through a downsampling layer, it enters the third stage, is subjected to two times of convolution with a kernel of 3x3 and a step of 1, the output feature layer is 256, F3 is obtained, and the image size is (H / 4, W / 4, 256); after passing through a downsampling layer, it enters the fourth stage, is subjected to three times of convolution with a kernel of 3x3 and a step of 1, the output feature layer is 512, F4 is obtained, and the image size is (H / 8, W / 8, 512); After entering the fourth stage, it is subjected to three times of convolution with a kernel of 3x3 and a step of 1, the output feature layer is 512, F5 is obtained, and the image size is (H / 16, W / 16, 512); (3) the obtained multi-scale feature maps are stored in two memories as inputs of the difference detection module.

4. The deep feature forward difference extraction and backward backtracking correction change detection method according to claim 1, characterized in that, Thirdly, the difference detection module obtains the image multi-scale difference feature, maps and splices the double-phase feature at each scale stored in the second step, applies the convolution attention module composed of the channel attention module and the spatial attention module at the same time, makes the difference feature easier to distinguish, obtains the image multi-scale difference feature map, saves it to the memory, and completes the construction of the deep feature forward difference extraction module. The specific process is as follows: (1) a convolution attention module is designed on the difference module; the convolution attention module is composed of two sub-modules, namely the channel attention module and the spatial attention module, which realize feature map refinement in the channel and spatial fields respectively; The channel attention module is used to emphasize the target related channel while suppressing the irrelevant channel. The channel attention module can be calculated by the following formula: M c (F) = σ(MLP r (AvgPool(F)) + MLP r (MaxPool(F)) (1) F represents an input feature of size HxWxC, on which average pooling and max pooling are used to generate aggregated vectors of size 1x1xC, and then a weight-shared multi-layer perception module with a channel reduction rate r is applied to each vector to give each channel a weight; the MLP contains two 1x1 convolutional layers, where the first convolution reduces the channels of the input feature by r times, and the second convolution restores the channels to the size of the original input image, and the two layers are connected by a Relu activation function; the channel attention map Mc(F) is obtained by using the sum of the above two elements through a sigmoid activation function σ; finally, the original feature is multiplied by the channel attention map to obtain a channel refined feature wherein denotes element-wise multiplication; (2) Further refine the channel aspect of the refined features by a spatial attention module. The spatial attention submodule also uses average pooling and max pooling as the first step of processing, compressing the input channel refined features F into two HxWx1 matrices, which are concatenated together and input into a convolutional layer with a kernel of 7; finally, a sigmoid activation function σ is used to obtain the final spatial attention map M s (F), which can be expressed as follows: M s (F) = σ(f (7×7) (AvgPool(F); MaxPool(F)) (3) where f (7×7) represents a 7x7 convolution operation, [;] represents a concatenation operation; finally, the original feature is multiplied with the spatial attention map to obtain the spatial refined feature (3) Starting from the deepest fifth layer network, the double-phase images of each layer are spliced and input into the convolution attention module, so as to obtain more discriminative difference features D i : D i = M s (Conv(Mc(F i ))) (6) wherein denotes concatenation, x1_C i and x2_C i denotes the i-th layer of the dual-time image feature map, Conv denotes a set of convolution operations resulting in the original number of channels; (4) finally, the difference feature map of each layer is unified to 64*64*64 size and saved to the memory, and the construction of the deep feature forward difference extraction module is completed.

5. The deep feature forward difference extraction and backward backtracking revision change detection method according to claim 1, characterized in that, In the sixth step, each difference correction unit contains change area and boundary information. For the obtained change area, a contrast loss is applied for training, and for the boundary information, a Dice loss is applied for training. The combined loss is used to train the model. The specific process is as follows: (1) each difference correction unit contains change area and boundary information; for the obtained change area, a contrast loss is applied for training, and the contrast loss is represented as: wherein y i,j , d i,j respectively represent the value of the ground truth and the change region at point (x, y), n u , n c respectively represent the number of unchanged pixels and the number of changed pixels, 0 represents an unchanged pixel, and 1 represents a changed pixel; and m represents the edge distance of the pixel pair filtered out. (2) for the information feature of the difference correction unit, a Dice loss is used for training, wherein, y i,j respectively denote the boundary prediction map and the ground truth. (3) the loss composed of the contrast loss and the Dice loss function is: where λ is a balancing factor, and are the change region prediction and boundary prediction results obtained by the difference correction unit, respectively, GT R is the ground truth of the change region, GT B is the boundary ground truth, and N denotes the number of network layers.

6. A system for change detection by deep feature forward difference extraction and backward trace correction, characterized in that, The system is used to realize the method of claim 1, and comprises a deep feature forward difference extraction module and a deep feature backward backtracking correction module. The deep feature forward difference extraction module comprises: The feature extraction module comprises a feature extractor composed of a VGG network, which extracts double-phase image features at multiple scales; The difference detection module maps and splices the double-phase feature at each scale as the input of the difference detection module, and applies the convolution attention module composed of the channel attention module and the spatial attention module at the same time to obtain the image multi-scale difference feature map; The deep feature backward backtracking correction module comprises: The difference correction unit inputs the difference feature map into the difference correction unit to obtain boundary feature information; The boundary information guiding module corrects the difference feature backtracking.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps of the method of any one of claims 1-5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1-5.

Images