A high-resolution remote sensing image change detection deep network and a detection method

Abstract

The application belongs to the technical field of remote sensing change detection, and discloses a high-resolution remote sensing image change detection deep network and a detection method. The deep network is an attention-guided multi-level feature fusion network (AGMFFNet). In the feature coding stage, early fusion structure and late fusion structure are simultaneously used to extract the context features and local correlation features of double-phase images. In the feature fusion stage, an early-late hybrid feature fusion module (ELMFFM) is proposed between each layer of network structure to fuse and enhance the difference features of images. In addition, in the feature decoding stage, a multi-level auxiliary feature reconstruction module (MAFRM) is further proposed to make up for the spatial information loss caused by the traditional direct up-sampling method. The effectiveness of the proposed network and method is proved in two public data sets.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing change detection, and particularly relates to a high-resolution remote sensing image change detection deep network and a detection method. BACKGROUND

[0002] Remote sensing change detection is a process of identifying changes on the earth's surface by jointly processing dual-time or multi-time images collected in the same geographic area. It has been widely applied in the fields of urban planning, disaster assessment, vegetation coverage, etc. With the rapid development of satellite imaging technology, various types of sensors have emerged in an endless stream. High-resolution remote sensing images have been proved to be the main data source for change detection due to their wide coverage and high spatio-temporal resolution.

[0003] Traditional change detection methods can be divided into pixel-based methods and object-based methods. Pixel-based change detection methods measure the changes of images by comparing the differences between pixels or regions in adjacent phases, such as change vector analysis (CVA), principal component analysis (PCA), etc. However, such methods have large computational complexity and only rely on the spectral value changes of single pixels, which are difficult to handle images with complex data sources and features. Object-based change detection methods segment images into different categories using spectral, texture, etc. feature information, and then obtain change detection results by comparing corresponding regions, such as support vector machine (SVM), Markov random field (MRF), etc. Such methods consider the spectral and spatial information of images comprehensively, but the final detection results depend on the results of each feature extraction and classification unit, which is easy to cause accumulation of false information, thereby limiting the detection accuracy. With the rapid development of satellite imaging technology and change detection technology, traditional change detection methods have been unable to meet the requirements of detection accuracy. Deep learning methods have been widely applied in change detection due to their strong feature extraction ability and superior performance, and have shown good performance in change detection tasks.

[0004] The change detection method based on deep learning can be roughly divided into early fusion method and late fusion method, which respectively fuse features from different data sources at a specific position. The early fusion method fuses the two time-series images before extracting the change features, and performs feature extraction and change map generation operations on the fused image. This method can adaptively extract information before and after the image change, and can also obtain the spatio-temporal context information of the change region. However, after fusing two images into one, the image detail information may be lost or new noise may be generated, thereby reducing the accuracy of the change map. Therefore, the late fusion method realizes change detection by extracting features and fusing at the same time or extracting features first and then fusing. Specifically, the two time-series images are respectively sent into two same encoder networks, and then the change map is obtained by difference measurement or fusion method. The method based on difference measurement usually adopts the method of calculating the difference value between the corresponding feature pixels to judge the change region. This method is simple to calculate, but only calculates the difference between local pixels, ignoring the influence of neighborhood context information on the change region, thereby causing the loss of related information between the two time-series images when generating the final change map. The method based on fusion usually fuses the semantic features of the corresponding layers, and then generates the change map through a series of convolution operations. However, this method still faces the problem of insufficient spatio-temporal context information. SUMMARY

[0005] Change detection aims to identify changes occurring on the ground surface from two time-series images. In recent years, deep learning-based methods have made breakthrough progress in the field of change detection. Most existing methods extract multi-scale features of two time-series images through a single early fusion structure or late fusion structure. However, the late fusion method does not consider the context information of the neighborhood, and can only obtain the difference map through fusion or measurement, resulting in relatively single difference features. The early fusion method lacks the ability to extract deep features of a single original image, and is prone to lack of detailed features when reconstructing the change map. Therefore, in view of the above problems, the present application provides a high-resolution remote sensing image change detection deep network and detection method.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The application provides a high-resolution remote sensing image change detection deep network, which is an attention-guided multi-level feature fusion network (AGMFFNet) and mainly consists of three parts: a feature extraction network (FE), an early-late mixed feature fusion module (ELMFFM) and a multi-level auxiliary feature reconstruction module (MAFRM); the FE is used for extracting context features and local relevant feature information of an image; the ELMFFM is used for fusing and enhancing the difference features of the image extracted by the FE; and the MAFRM is used for recovering the size of a feature map and making up for the spatial information loss caused by a traditional direct up-sampling method.

[0008] Further, the FE includes an early fusion sub-network (FE1) and a late fusion sub-network (FE2) and is used for extracting context features and local relevant features of an image, respectively, and is composed of a hole convolution module, a maximum pooling layer and four residual modules, and the hole convolution module is composed of four consecutive hole convolutions.

[0009] Further, the ELMFFM has one between each layer after the pooling layer of the FE and is used for effectively fusing the semantic feature information extracted by the FE, and the ELMFFM includes a neighborhood self-attention module (NSAM) and a channel self-attention module (CSAM), the NSAM is used for fusing the extracted difference information and global information, and the CSAM is used for enhancing the channel information useful for a change target in the fusion information of the NSAM.

[0010] Further, the MAFRM is used for feature decoding and recovering the size of a feature map and includes a spatial attention module (SAM) and a channel attention module (CAM), which are respectively used for integrating local feature information and channel dependency relationship to achieve better feature reconstruction effect.

[0011] The application also provides a high-resolution remote sensing image change detection method based on the deep network, which comprises the following steps:

[0012] Step 1, training the deep network described above;

[0013] Step 2, sending an image into the feature extraction network to extract multi-level deep features;

[0014] Step 3: The outputs of the second layer and each subsequent layer in the feature extraction network are taken as inputs to the early-late mixed feature fusion module, and multi-level mixed feature fusion is performed hierarchically to generate effective difference features.

[0015] Step 4: The difference features of each layer of the early-late mixed feature fusion module are combined, and a multi-level auxiliary feature reconstruction module is used to recover the feature map layer by layer to the size of the input image in an auxiliary upsampling manner to predict the final change map. Direct upsampling may cause loss of spatial hierarchical information.

[0016] Further, the training of the deep network in step 1 is to optimize the objective function of the precise network using training images and corresponding true values, and the specific process is as follows:

[0017] First, use the cross-entropy loss function to measure the difference between the predicted value and the true value, and ensure that the change map is as close to the true value as possible, which is defined as follows:

[0018]

[0019] Where n represents the number of samples, y k represents the true value of a given sample, represents the predicted probability value;

[0020] Then add the dice loss to overcome the problem of sample imbalance, which is defined as follows:

[0021]

[0022] Where, and y represent the predicted image and the true label, respectively;

[0023] Finally, the two loss functions are mixed as the optimization target of the network, represented as:

[0024] L = L CE + L Dice (3).

[0025] In change detection, the number of unchanged region pixels is often greater than the number of changed region pixels, and under the influence of sample class imbalance, the training direction of the network model is usually guided by the majority of unchanged pixels, ignoring the information of the minority changed pixels, resulting in low efficiency. Therefore, we added the dice loss to overcome the problem of sample imbalance.

[0026] Further, the specific process of step 2 is as follows: first, set the hole rates of the 4 consecutive hole convolutions in the feature extraction network hole convolution module to 1, 2, 1, and 2, change the stride of the max pooling layer to 1, and change the stride of the first residual module to 2, so that the size of the feature map is compressed to 1 / 2 of the previous layer after passing through each residual module; then input the dual-phase images T0 and T1 into the early fusion sub-network FE1 and the late fusion sub-network FE2 respectively to extract multi-level deep features, which are early fusion features F E and paired late fusion features F0 L and F1 L .

[0027] Further, the specific process of step 3 is as follows:

[0028] Input the early fusion features F E and the paired late fusion features F0 L and F1 L into the ELMFFM, and perform addition operation on F0 L and F1 L to obtain the relevant information in the dual-phase images; similarly, perform subtraction operation on F0 L and F1 L to obtain the difference information F diff in the dual-phase images, then perform concatenation operation on the features after addition of F0 L and F1 L and F E to obtain the global information F C of the dual-phase images, so as to enhance the relevance of the local information.

[0029] In order to further fuse the difference information and the global information, NSAM is used to explore the similarities and differences between the global information and the difference information, and to obtain enhanced difference information F’;

[0030] CSAM is used to enhance the channel information useful for the change target in F’, so as to generate the weighted fused difference features F”.

[0031] Further, the specific process of using NSAM to explore the similarities and differences between the global information and the difference information, and to obtain the enhanced difference information F’ is as follows:

[0032] First, perform 1x1 convolution operation on the global information F C and the difference information F diff to map them to three different feature spaces K, Q, and V;

[0033] K and Q are respectively subjected to 3*3 convolution operation to obtain features K' and Q' of the gathered neighborhood information, then, multiplication operation is performed on K' and Q' and the attention weight W is obtained through the softmax function; since the size of the target object is uncertain, only the pixel point is used to calculate the attention weight of the global feature and the difference feature, which is easy to be disturbed by the noise data, therefore, the neighborhood information of 3*3 is used to replace the center pixel point, so that the confidence of the attention weight is improved and the influence of the salt and pepper noise is removed.

[0034] In order to highlight the change area, the inverse operation is performed on W, and W is multiplied by V, and then the enhanced difference information F' is generated by inputting it into the 1*1 convolution layer.

[0035] Further, the specific process of the step 4 is as follows:

[0036] Firstly, the difference feature F of the current layer is obtained by performing the difference operation on the feature V and the reconstructed feature F of the previous layer. c ”∈R H×W×C1 ; L up ∈R H×W×C1 ; trans 2H×2W×C2 ;

[0037] Then, the difference feature F of the previous layer is obtained by performing the difference operation on the feature V and the reconstructed feature F of the previous layer. L ”∈R 2H×2W×C2 ; trans trans ;

[0038] Finally, F, F and F are concatenated, and the reconstructed feature F is obtained by inputting it into the 1*1 convolution layer. trans CAM SAM C up ∈R 2H ×2W×C2 .

[0039] Compared with the prior art, the present application has the following advantages:

[0040] ​​​​​1) The application proposes an attention-guided multi-level feature fusion network (AGMFFNet), which can more perfectly extract the context features and local relevant features of the image by combining early fusion structure and late fusion structure. Meanwhile, an early-late hybrid feature fusion module (ELMFFM) is proposed, which fully utilizes the global information, local information and difference information of the image, and effectively and perfectly extracts the change information in the multi-level features.

[0041] 2) In the early-late hybrid feature fusion module (ELMFFM), the application proposes a neighborhood self-attention module (NSAM), which can enhance the features of the change area by performing self-attention operation on global features and difference features; meanwhile, a channel self-attention module (CSAM) is introduced after NSAM, which helps to enhance the channel features related to change information.

[0042] 3) The application proposes a multi-level auxiliary feature reconstruction module (MAFRM). MAFRM enables each level of feature to have high-resolution features as auxiliary features when up-sampling, thereby making up for the spatial information loss caused by direct up-sampling of low-resolution features.

[0043] 4) The network and method of the application well solve the shortcomings of existing early fusion methods and late fusion methods. Comparative experiments on LEVIR-CD and CDD show that AGMFFNet obtains the best results in three comprehensive evaluation indexes F1, OA and mIoU on the two data sets, which proves the adaptability of AGMFFNet to different types of change targets. Finally, the effectiveness of each module in AGMFFNet is verified by ablation experiments on LEVIR-CD. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 It is the structure diagram of the attention-guided multi-level feature fusion network AGMFFNet of the application.

[0045] Figure 2 It is the structure of the early-late hybrid feature fusion module in the network of the application; wherein (a) ELMFFM, (b) NSAM, (c) CSAM.

[0046] Figure 3 It is a 3x3 convolution receptive field diagram.

[0047] Figure 4 It is the structure of the multi-level auxiliary feature reconstruction module MAFRM.

[0048] Figure 5 It is the detection result of different methods on LEVIR-CD data set.

[0049] Figure 6Detection results of different methods on the CDD dataset.

[0050] Figure 7 Feature visualization results of key modules in the attention-guided multi-level feature fusion network AGMFFNet of the application. DETAILED DESCRIPTION

[0051] The technical solutions of the application will be described in detail below with reference to the embodiments of the application and the accompanying drawings. It should be noted that, for those skilled in the art, without departing from the principles of the application, a number of variations and improvements can be made, which should also be considered as falling within the scope of protection of the application.

[0052] Embodiment 1

[0053] The structure of a high-resolution remote sensing image change detection deep network, i.e., an attention-guided multi-level feature fusion network AGMFFNet, is shown in Figure 1 and mainly consists of three parts: a feature extraction network, an early-late hybrid feature fusion module, and a multi-level auxiliary feature reconstruction module.

[0054] The feature extraction network includes an early fusion sub-network and a late fusion sub-network, which are respectively used to extract the context features and the local relevant features of the image, and is constructed based on SE-ResNet50, as shown in Figure 1 The feature extraction network includes an early fusion sub-network and a late fusion sub-network, which are respectively used to extract the context features and the local relevant features of the image, and is constructed based on SE-ResNet50, as shown in

[0055] The early-late hybrid feature fusion module has one between each layer after the pooling layer of the feature extraction network and the pooling layer, which is used to effectively fuse the semantic feature information extracted by the feature extraction network. The early-late hybrid feature fusion module includes a neighborhood self-attention module and a channel self-attention module. The neighborhood self-attention module is used to fuse the difference information and the global information extracted, and the channel self-attention module is used to enhance the channel information useful for the change target in the fused information of the neighborhood self-attention module.

[0056] The multi-level auxiliary feature reconstruction module is used for feature decoding to restore the size of the feature map, and includes a spatial attention module and a channel attention module, which are respectively used to integrate the local feature information and the channel dependency relationship to achieve better feature reconstruction effect.

[0057] Embodiment 2

[0058] Network-based high-resolution remote sensing image change detection method

[0059] 1. Training of deep network:

[0060] Given a set of training images and corresponding ground truths, the goal is to optimize the objective function of the precise change detection network. Since the change detection task can be regarded as a pixelized binary classification task, we use the commonly used cross-entropy loss function in the training stage, which can measure the difference between the predicted value and the true value to ensure that the change map is as close to the true value as possible, which is defined as follows:

[0061]

[0062] where n represents the number of samples, yk represents the true value of the given sample, represents the predicted probability value;

[0063] In addition, in change detection, the number of unchanged area pixels is usually greater than the number of changed area pixels. Under the influence of sample class imbalance, the training direction of the model is usually guided by the majority of unchanged pixels, ignoring the information of the minority of changed pixels, resulting in low efficiency. Therefore, the dice loss is added to overcome the problem of sample imbalance, which is defined as follows:

[0064]

[0065] where, and y represent the predicted map and the true label, respectively;

[0066] Finally, the two loss functions are mixed as the optimization target of the network, denoted as:

[0067] L=L CE +L Dice (3)。

[0068] 2. Feature encoding

[0069] Firstly, the hole rates of the four consecutive hole convolutions in the hole convolution module of the feature extraction network are set to 1, 2, 1, and 2, the stride of the max pooling layer is changed to 1, and the stride of the first residual module is changed to 2, so that the size of the feature map is compressed to 1 / 2 of the previous layer after passing through each residual module; Then the dual-phase images T0 and T1 are sent into the early fusion sub-network FE1 and the late fusion sub-network FE2 respectively to extract multi-level deep features, which are early fusion features F E and paired late fusion features F0 L and F1 L .

[0070] 3. Feature fusion

[0071] Most of the current feature fusion methods in change detection networks are late fusion, i.e., the features of the two temporal images are first extracted by a twin network structure, and then the paired features between different layers are fused. Using late fusion can integrate the local correlation information between paired features, but it ignores the neighborhood context information, while early fusion can extract the spatio-temporal context information. Therefore, we introduce both early fusion structure (FE1) and late fusion structure (FE2) to obtain the context features and local correlation features in parallel. We propose an early-late hybrid feature fusion module between the pooling layer and each layer after it in the feature extraction network to effectively fuse the three kinds of multi-scale semantic feature information, and the structure of ELMFFM is as shown in Figure 2 (a).

[0072] The early fusion features F E and the paired late fusion features F0 L and F1 L are input into the early-late hybrid feature fusion module. For the input of the three kinds of features, three different operations are designed respectively to extract the local information, global information and difference information of the image. As shown in Figure 2 (a), the addition operation is performed on F0 L and F1 L to obtain the relevant information in the two temporal images; similarly, the subtraction operation is performed on F0 L and F1 L to obtain the difference information F diff of the two temporal images, and then the features after the addition of F0 L and F1 L are concatenated with F E to obtain the global information F C of the two temporal images, so as to enhance the relevance of the local information.

[0073] In order to further fuse the difference information and the global information, the neighborhood self-attention module is used to explore the similarities and differences between the global information and the difference information, and to obtain the enhanced difference information F’. The structure of NSAM is as shown in Figure 2 (b). First, the global information F C and the difference information F diff are subjected to 1x1 convolution operation to map them to three different feature spaces K, Q and V. Since the size of the target object is uncertain, calculating the attention weight of the global feature and the difference feature by only the pixel point is easy to be disturbed by the noise data, therefore, using the neighborhood information of 3x3 instead of the center pixel point can improve the confidence of the attention weight while removing the influence of salt and pepper noise. Specifically, we use 3x3 convolution operation to aggregate the feature information of the 3x3 neighborhood of each pixel point. As shown in Figure 3As shown, the feature map can obtain a 3x3 receptive field after 3x3 convolution, that is, each pixel point in the feature map after 3x3 convolution can represent the feature information of a 3x3 size region in the original feature map. Therefore, 3x3 convolution operation is performed on K and Q respectively to obtain the features K' and Q' of the aggregated neighborhood information, and then the multiplication operation is performed on K' and Q' and the attention weight W is obtained through the softmax function; since K' and Q' are global features and difference features respectively, the weight of the unchanged region is enhanced after multiplication. However, for the change detection task, we prefer to highlight the change region. Therefore, the reverse operation is performed on W and multiplied by V to highlight the information of the change region. Then it is sent to the 1x1 convolution layer to generate the enhanced difference feature.

[0074] At the same time, the channel self-attention module is used to enhance the channel information useful for the change target in F', so as to generate the weighted fused difference feature F". Figure 2 (c) as shown.

[0075] 4. Feature decoding

[0076] In the feature decoding stage, the most direct method is to recover the size of the feature map layer by layer through upsampling. However, direct upsampling may cause loss of spatial hierarchical information. Therefore, a multi-level auxiliary feature reconstruction module is proposed, which provides high-resolution features of the previous layer as auxiliary information when upsampling each layer of features, so as to reduce the loss of spatial information caused by direct upsampling. At the same time, we introduce a spatial attention module and a channel attention module, which are used to integrate local feature information and channel dependency respectively, so as to achieve better feature reconstruction effect.

[0077] The structure of MAFRM is shown in Figure 4 First, the difference feature (F c ”∈R H×W×C1 ) of the current layer is concatenated with the reconstructed feature (F L up ∈R H×W×C1 ) of the previous layer; in order to avoid the problem that traditional interpolation method is easy to enhance noise and generate low-quality pictures when upsampling, the concatenated feature is upsampled through 3x3 transpose convolution to obtain the feature F trans ∈R 2H×2W×C2 ; then, the difference feature (F L ”∈R 2H×2W×C2 ) of the previous layer is added to F trans , and sent to the spatial attention module to make up for the loss of spatial information caused by upsampling and enhance the position information of the target object; at the same time, F trans is sent to the channel attention module to enhance the feature channel information related to the target object; finally, Ftrans , F CAM and F SAM are concatenated and reconstructed features (F C up ∈R 2H×2W×C2 ).

[0078] Example 3

[0079] Validation of network and detection method

[0080] 1. Datasets

[0081] To verify the effectiveness of AGMFFNet, we conduct experiments on two representative high-resolution remote sensing image change detection datasets. Each dataset contains a change map and two high-resolution remote sensing images taken at different times in the same area. The detailed information of the two datasets is as follows:

[0082] (1) LEVIR-CD dataset: The LEVIR-CD dataset consists of 637 pairs of high-resolution remote sensing images with a size of 1024x1024, which come from 20 different areas in several cities in Texas, with a spatial resolution of 0.5 meters and the main change type being building change. In the experiment, we crop each image into 256x256 non-overlapping image blocks, and select 7120, 1024, and 2048 pairs of images for training, validation, and testing, respectively.

[0083] (2) CDD dataset: The CDD dataset consists of 16000 pairs of images with a size of 256x256, including 10000 pairs of training set, 3000 pairs of validation set and 3000 pairs of test set. The spatial resolution of CDD is 0.3-1 meter, which contains changes of different objects such as buildings, roads, vehicles, and ignores changes caused by seasonal differences, brightness and other factors.

[0084] 2. Experimental setup

[0085] (1) Comparison methods

[0086] To verify the effectiveness of the proposed AGMFFNet, we select seven state-of-the-art change detection methods for comparison, including FC-EF, FC-Siam-conc, FC-Siam-diff, STANet, DTCDSCN, SRCDNet and MSPSNet. The introduction of each method is as follows.

[0087] FC-EF is a fully convolutional early fusion network based on U-Net model, in which the dual-phase images are concatenated as the input of the network. It uses a skip connection mechanism to transmit multi-scale features from the encoder to the decoder to recover the spatial feature information at each level.

[0088] FC-Siam-conc is a variant of FC-EF, which adopts late fusion structure and extends the encoder to a weight-shared twin network to extract features of dual temporal images in parallel. It uses a skip connection to input dual temporal features of different levels to the decoder to integrate spatio-temporal features of different scales.

[0089] FC-Siam-diff adopts the same twin network structure as FC-Siam-conc, but the difference is that the features of the same level in the twin network structure are first subjected to a difference operation before being transmitted to the decoder to obtain multi-scale difference features of dual temporal images.

[0090] STANet is a change detection network model based on spatio-temporal attention mechanism, which uses spatio-temporal attention mechanism to obtain long-term spatio-temporal features that are robust to illumination changes and registration errors, and uses a metric-based method to learn the change map of dual temporal images.

[0091] DTCDSCN is a deep twin convolutional network model with dual-task constraints, which uses SE-ResNet as the backbone network and introduces a dual attention module to further improve the discriminability of features. At the same time, the network proposes an improved focal loss function to solve the problem of sample imbalance.

[0092] SRCDNet is a change detection network based on super-resolution with a stacked attention module (SAM), which integrates SAM composed of 5 convolution block attention modules (CBAM) into the feature extractor, and obtains the change map through a change strategy based on metric learning.

[0093] MSPSNet is a deep multi-scale twin network with parallel convolution structure and self-attention, which integrates different temporal features through parallel convolution structure and uses self-attention module to improve the representation ability of features.

[0094] (2) Implementation Details

[0095] Our network model is implemented on PyTorch and trained and tested using a single NVIDIA RTX3090 GPU. We use the Adam optimizer with a learning rate of 0.001 to optimize the model, and set the training period to 100 and the batch size to 4. After each training period, validation is performed, and the best model on the validation set is used for evaluation on the test set.

[0096] (3) Evaluation Metrics

[0097] We adopt five evaluation metrics, precision (P), recall (Re), F1, overall accuracy (OA) and mean intersection over union (mIoU) to evaluate the experimental results. The definitions of the above metrics are as follows:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] where TP, FP, TN and FN represent the number of true positives, false positives, true negatives and false negatives, respectively.

[0104] 3. Performance comparison

[0105] (1) Results and discussion on LEVIR-CD dataset

[0106] The detection results and evaluation metrics of each method on LEVIR-CD dataset are shown in Table 1 and Table 2. To analyze the detection effect of each method, we select different size of change targets for comparison. As shown in Fig. 2, (a) and (b) are small target regions. Due to the influence of lighting conditions and shadow angles, all the compared methods except DTCDSCN have missed detection and false detection, and the positioning of the contour information of small targets is also relatively fuzzy. In contrast, AGMFFNet performs more significantly in detecting small targets and their contour information. In (d), when the target is a large and continuous region, FC-Siam-diff and STANet have a large area of missing, and the rest of the methods also have missing to varying degrees. For the buildings arranged more complex and dense, whether the boundary of each building is complete and smooth can directly reflect the performance of the detection method. As shown in (c), (e) and (f), all the compared methods have missed detection and false detection, and STANet and SRCDNet have building adhesion, especially SRCDNet. In general, AGMFFNet performs more superior performance in the integrity of the target and the accuracy of the boundary. Figure 5 Figure 5 Figure 5 Figure 5

[0107] ​​​​Meanwhile, from Table 1, it can be seen that the evaluation indexes of each method show similar conclusions as the detection result maps. The three comprehensive evaluation indexes F1, OA and mIoU of SRCDNet are the lowest among all the comparative methods, which are 85.72%, 98.46% and 86.70%, respectively. The recall rate of STANet is higher, which is 90.45%, but due to the existence of target object adhesion, the precision is only 83.43%. And the three comprehensive evaluation indexes F1, OA and mIoU of STANet are also only higher than SRCDNet, which are 86.80%, 98.60% and 87.60, respectively. The three comprehensive evaluation indexes F1, OA and mIoU of MSPSNet reach 89.72%, 98.96% and 90.13%, which are higher than all other comparative methods. Among all the methods, AGMFFNet achieves the best results in the five evaluation indexes, and F1 and mIoU reach 91.2% and 91.53%, respectively, which are 1.48% and 1.4% higher than MSPSNet, respectively. This also proves that AGMFFNet has more superior change detection performance compared with other comparative methods.

[0108] Table 1 Experimental results on LEVIR-CD dataset

[0109]

[0110] Note: The optimal value is in bold

[0111] (2) Results and discussion on CDD dataset

[0112] The detection results and evaluation indexes of each method in the CDD dataset are shown in Figure 6 and Table 2. Since the change types of CDD are more and more complex than LEVIR-CD, some comparative methods show relatively poor detection performance in LEVIR-CD. As shown in Figure 6 (a) and (b), for small target regions, FC-EF, FC-Siam-conc, FC-Siam-diff and MSPSNet have obvious missed detection. There are false detections in STANet, DTCDSCN and SRCDNet, and there are change target adhesion in STANet and DTCDSCN. For more complex change targets, due to the sparse or uneven distribution of change regions, each method is more susceptible to noise. As shown in Figure 6In (c) and (d), FC-EF, FC-Siam-conc, FC-Siam-diff, STANet and MSPSNet have obvious false detection, missed detection and adhesion. Compared with other networks, the boundary of sparse target in AGMFFNet is clearer, and it has less false detection and missed detection area, which shows better detection results. In addition, in (e) and (f), due to the influence of factors such as light and season, the target area detected in these methods is missing. In contrast, AGMFFNet shows better adaptability to detect varying types of changing targets. Figure 6

[0113] In addition, from Table 2, it can also be seen that the evaluation indexes of FC-EF, FC-Siam-conc, FC-Siam-diff and MSPSNet are significantly lower than LEVIR-CD. Among them, the values of the three comprehensive evaluation indexes F1, OA and mIoU of FC-Siam-conc are the lowest, which are 70.21%, 94.23% and 73.95% respectively. The three comprehensive evaluation indexes of STANet, DTCDSCN, SRCDNet and AGMFFNet have improved compared with LEVIR-CD, which shows better adaptability to the data set with more complex change types. Among them, the proposed AGMFFNet achieves the best results in P, F1, OA and mIoU, which are 1.1%, 0.66%, 0.2% and 0.68% higher than the second best value respectively. In terms of recall rate, although the Re of AGMFFNet is lower than that of DTCDSCN, the precision of AGMFFNet is 3.07% higher than that of DTCDSCN. This proves the effectiveness of AGMFFNet in difference enhancement and multi-level auxiliary up-sampling.

[0114] Table 2 Experimental results of CDD data set

[0115]

[0116] Note: The optimal value is in bold

[0117] Example 4

[0118] To verify the effectiveness of each module in AGMFFNet, we conducted a series of ablation experiments on the LEVIR-CD data set, and the experimental results are shown in Table 3. Among them, Baseline represents using SE-ResNet50 as the feature extraction network in early and late fusion structure, and DCM represents the hollow convolution module.

[0119] ​In the ablation experiment of Baseline, we replace the ELMFFM module and the MAFRM module with cascade operation and bilinear interpolation method. As shown in Table 3, Baseline performs the worst in LEVIR-CD, and compared with AGMFFNet, the three comprehensive evaluation indicators F1, OA and mIoU of Baseline decrease by 2.11%, 0.19% and 1.84% respectively. At the same time, it can also be seen that the change detection performance of each module in the network compared with Baseline has improved, which indirectly proves the effectiveness of each module proposed. In order to study the influence of early fusion structure on the network, we delete FE1 and only use the late fusion subnetwork as the feature extraction network. It can be seen that AGMFFNet obtains higher values in the three comprehensive evaluation indicators F1, OA and mIoU, which increase by 0.51%, 0.04% and 0.44% respectively, which proves the effectiveness of combining early and late fusion structures. In the feature fusion stage, we delete NSAM and CSAM in the ELMFFM module respectively, and it can be seen that each evaluation index decreases, which proves the effectiveness of the two attention modules in enhancing the difference information and related feature channel information.

[0120] In the feature decoding stage, we replace the MAFRM module with several consecutive bilinear interpolation operations. As can be seen from Table 3, AGMFFNet improves the three comprehensive evaluation indicators F1, OA and mIoU by 0.45%, 0.2% and 0.48% respectively, which proves that the MAFRM module plays an auxiliary role in promoting the direct upsampling method. In addition, in order to prove that DCM does not have a negative impact on the network performance, we replace DCM with the first layer convolutional layer in the original SE-ResNet50. It can be seen that DCM maintains the receptive field and resolution while also having a certain positive effect on the improvement of network performance, and AGMFFNet improves in each evaluation index. Through the above ablation experiment, it can be seen that AGMFFNet has significantly improved change detection performance, which further proves the effectiveness of the proposed method.

[0121] Table 3 Ablation experiment results of LEVIR-CD dataset

[0122]

[0123] Note: The optimal value is in bold

[0124] Example 5

[0125] Learning effect detection of AGMFFNet in each stage: In order to better show the learning effect of AGMFFNet in each stage, we use a sample in the test set to visualize the heat map of each stage in AGMFFNet. Through the heat map, the learning situation of each stage of the network to the changing target in the image can be intuitively explained, and the visualization result is as shown in Figure 7 As shown in the figure. Given the dual-phase images, FE1 and FE2 generate high-level semantic feature maps containing context information and local information layer by layer respectively. Then, ELMFFM fuses and enhances the feature maps of each layer of FE1 and FE2, thereby generating refined difference features. It can be observed that with the increase of network depth, the difference features extracted by ELMFFM are gradually enhanced, and the positioning of the target contour and position information in the difference map is also gradually refined, which indicates that ELMFFM can further enhance the region where the changing target is located. In the feature decoding stage, the highlighting of the changing region by MAFRM gradually becomes clear as the decoding deepens. At the same time, the attention of MAFRM to the unchanged target also gradually decreases layer by layer. Finally, the change prediction map and the change map accurately show the difference between the dual-phase images. Overall, each stage of AGMFFNet is learning to distinguish the changing target to complete the highlighting of the changing region.

[0126] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be realized in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. A deep network based high resolution remote sensing image change detection method, characterized in that, The method comprises the following steps: Step 1, training a deep network, the deep network is an attention-guided multi-level feature fusion network, mainly composed of three parts: a feature extraction network, an early-late mixed feature fusion module and a multi-level auxiliary feature reconstruction module; the feature extraction network is used for extracting the context features and local relevant feature information of an image; the early-late mixed feature fusion module is used for fusing and enhancing the difference features of the image extracted by the feature extraction network; and the multi-level auxiliary feature reconstruction module is used for restoring the size of the feature map and making up for the spatial information loss caused by the traditional direct up-sampling method; The early-late mixed feature fusion module has one between each layer residual module after the pooling layer of the feature extraction network, which is used for effectively fusing the semantic feature information extracted by the feature extraction network, and the early-late mixed feature fusion module comprises a neighborhood self-attention module and a channel self-attention module, the neighborhood self-attention module is used for fusing the difference information and global information, and the channel self-attention module is used for enhancing the channel information useful for the change target in the fused information of the neighborhood self-attention module; The multi-level auxiliary feature reconstruction module is used for feature decoding to restore the size of the feature map, and comprises a spatial attention module and a channel attention module, which are respectively used for integrating local feature information and channel dependency to achieve better feature reconstruction effect; Step 2, sending the image into the feature extraction network to extract multi-level deep features; Step 3, taking the output of the second layer maximum pooling layer and each layer residual module after the feature extraction network as the input of the early-late mixed feature fusion module, and generating effective difference features by performing multi-level mixed feature fusion in layers; Step 4, combining the difference features of each layer of the early-late mixed feature fusion module, using the multi-level auxiliary feature reconstruction module to restore the feature map layer by layer to the size of the input image in an auxiliary up-sampling manner, and predicting the final change map.

2. The high-resolution remote sensing image change detection method based on a deep network according to claim 1, characterized in that: The feature extraction network comprises an early fusion sub-network and a late fusion sub-network, which are respectively used for extracting the context features and local relevant features of an image, and the feature extraction network is composed of a hollow convolution module, a maximum pooling layer and four residual modules, and the hollow convolution module is composed of four consecutive hollow convolutions. 3.The high-resolution remote sensing image change detection method based on deep network according to claim 1, characterized in that, The training of the deep network in step 1 is to optimize the objective function of the precise network by using training images and corresponding true values, and the specific process is as follows: Firstly, a cross-entropy loss function is used to measure the difference between the predicted value and the true value, so as to ensure that the change map is as close to the true value as possible, and the definition is as follows: ； wherein, denotes the number of samples, denotes the true value of a given sample, denotes the predicted probability value; Secondly, a dice loss is added to overcome the problem of sample imbalance, and the definition is as follows: ； wherein, and pred and ytrue represent the predicted map and the true label, respectively; Finally, the two kinds of loss functions are mixed as the optimization target of the network, and is expressed as: 。 4. The high-resolution remote sensing image change detection method of claim 1, wherein, The specific process of step 2 is as follows: first, the hole rates of the four consecutive hole convolution modules of the feature extraction network are set to 1, 2, 1 and 2, the step of the maximum pooling layer is changed to 1, and the step of the first residual module is changed to 2, so that the size of the feature map is compressed to 1 / 2 of the previous layer after passing through each residual module; then the dual-phase images T0 and T1 are respectively sent into the early fusion sub-network FE1 and the late fusion sub-network FE2 to extract multi-level deep features, which are early fusion features F E and paired late fusion features F0 L and F1 L .

5. The high-resolution remote sensing image change detection method according to claim 4, wherein, The specific process of step 3 is as follows: Early fusion features F E and pair late fusion features F0 L and F1 L Input early and late mixed feature fusion module, add F0 L and F1 L to obtain relevant information in dual-phase images; similarly, subtract F0 L and F1 L to obtain difference information F diff of dual-phase images, then add the features of F0 L and F1 L to obtain global information F E of dual-phase images by cascading F C , so as to enhance the relevance of local information; The neighborhood self-attention module is used to explore the similarities and differences between the global information and the difference information, and enhanced difference information F' is obtained; The channel self-attention module is used to enhance the channel information useful for the change target in F', so as to generate weighted fused difference features F''.

6. The high-resolution remote sensing image change detection method according to claim 5, wherein, The specific process of obtaining the enhanced difference information F' by using the neighborhood self-attention module to explore the similarity and difference between the global information and the difference information is as follows: First, the global information F C and the difference information F diff are respectively subjected to a 1x1 convolution operation to be mapped to three different feature spaces K, Q, V; K and Q are subjected to 3*3 convolution operations respectively to obtain the features K' and Q' of the aggregated neighborhood information, then, the multiplication operation is performed on K' and Q', and the attention weight W is obtained through the softmax function; W is subjected to the reverse operation and multiplied by V, and then is sent into the 1*1 convolution layer to generate the enhanced difference information F'.

7. The high-resolution remote sensing image change detection method according to claim 1, wherein, The specific process of step 4 is as follows: First, the difference feature F of the current layer is obtained c ” ∈R H×W×C1 and the reconstructed feature F of the previous layer is obtained L up ∈R H×W×C1 are concatenated; the concatenated features are up-sampled by a 3x3 transpose convolution to obtain the feature F trans ∈R 2H×2W×C2 ; Then, the difference feature F L ” ∈R 2H×2W×C2 is added to F trans and is sent to the spatial attention module to make up for the loss of spatial information caused by upsampling and enhance the location information of the target object; at the same time, F trans is sent to the channel attention module to enhance the feature channel information related to the target object; Finally, F trans , F CAM , and F SAM are concatenated and passed through a 1x1 convolution to obtain the reconstructed feature F C up ∈ R 2H×2W×C2 .

Images