Remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network

Abstract

The present invention relates to the field of remote sensing image change detection, specifically to a remote sensing image change detection method based on a hierarchical cross-scale global feature fusion deep network. First, the network hierarchically extracts information of different scales of dual-phase high-resolution remote sensing images; then, for feature maps of different scales (such as the same level, adjacent level, and global), difference information enhancement and fusion modules are constructed respectively to strengthen the local feature representation ability of the fusion network and the relevance of contextual information; finally, the whole process combined loss function is proposed to supervise the learning of the intermediate feature layer to enhance the feature representation ability of each module. Experiments have shown that this paper has achieved significant results in both qualitative and quantitative indicators.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image change detection, and specifically to a remote sensing image change detection method based on a hierarchical cross-scale global feature fusion deep network. Background Art

[0002] Remote sensing image change detection aims to compare the image differences of the same area at different time periods and mark the changed areas. It has been widely used in urban expansion analysis, disaster assessment, military strikes, vegetation cover detection and other fields. At present, the field of change detection is facing the following challenges: first, the shooting conditions at different times (such as lighting, seasons, etc.) may cause objects with the same semantic information in dual-phase images to appear differently; second, the interference of unchanged areas and noise often affects the recognition of real change features.

[0003] Traditional change detection methods can be roughly divided into three categories: image arithmetic, image transformation, and post-classification methods. Image arithmetic methods include image difference, image ratio, change vector analysis (CVA), etc. These algorithms use relevant arithmetic (such as subtraction or division) to obtain feature maps, and then use segmentation thresholds to distinguish between changed and unchanged information. Obviously, the segmentation threshold is the difficulty and key point of this type of method. Image transformation methods include principal component analysis (PCA) and multivariate alteration detection (MAD). They transform images into specific feature spaces and improve the accuracy of change detection by highlighting the changed areas and suppressing the unchanged areas. The post-classification method first classifies the two phase images, and then compares and analyzes the classification results to generate a change result map. The accuracy of the post-classification method depends on the classification accuracy, so its error accumulation effect is serious.

[0004] The development of machine learning has improved the accuracy of change detection to a certain extent, such as Support Vector Machine (SVM) and Random Forest (RF). However, facing different scenarios, the selection of machine learning methods and the generalization ability of models pose challenges to practical applications.

[0005] Deep learning methods represented by Convolutional Neural Networks (CNN) have demonstrated their powerful feature extraction methods and nonlinear representation capabilities. They have been widely used in change detection tasks and have shown good performance. This is attributed to the fact that various deep networks can break through the appearance interference of change noise, mine high-level semantic information in the region of interest, and thus achieve intelligent target recognition.

[0006] According to the feature map extraction and fusion methods, the deep learning-based change detection framework can be roughly divided into three categories: early feature fusion, late feature fusion, and multi-level feature fusion (also known as correlation fusion). The change detection deep network based on early fusion cascades the input paired images at the beginning, performs feature extraction, encoding, decoding and other operations to obtain the final change detection results. The late-stage change detection network often adopts a two-stream structure to extract the deep features of the original image respectively, and finally performs feature fusion and pixel classification. However, the above two methods lack hierarchical interaction and global fusion between multi-scale deep features, which will have a certain impact on the extraction of high-level semantic information for change detection.

[0007] Recently, more research work has been devoted to extracting deep information through the twin neural network structure and fusing the corresponding layer feature pairs to determine the change area. This is the change detection method based on multi-level feature fusion. Obviously, this method can fully integrate shallow features (such as target texture, corners, edges, etc.) and deep semantic information (such as image content, semantic concepts), alleviate or even eliminate the semantic gap between different feature layers, and thus improve the change detection performance.

[0008] However, some current works tend to study the potential correlation between feature maps of the same scale; obviously, the fusion of features of different scales can take into account high-level semantic features and image resolution, making the extracted feature information more recognizable. Many quantitative and visual results show that there is still a huge research space for relationship mining between multi-scale feature maps. On the other hand, the existing multi-level feature fusion methods often adopt a relatively simple method, namely cascading or adding. This overly simple fusion mode is difficult to cross the semantic gap between different feature maps, and then comprehensively extract effective information. To this end, some research works use attention mechanisms to identify interesting changes in dual-phase remote sensing images. However, most existing methods only apply attention to each time period image to enhance its feature representation; or simply use attention mechanisms to re-weight and fuse dual-temporal feature maps in channel or spatial dimensions. Although these operations have learned the correlation between feature maps in a fixed neighborhood, the limited sampling range makes it impossible to model the relationship between long-distance information. Recently, some studies have achieved context modeling in space and time through self-attention mechanisms, effectively improving the globality of change detection in high-resolution remote sensing images.

[0009] In addition, most change detection networks based on deep learning only use loss function supervision models on the prediction value and label side, such as cross entropy, dice loss, contrast loss and their simple linear combination. However, through testing, it is found that multi-level supervision of the training process is conducive to the improvement of model convergence and detection performance, so in-depth research on the supervision of the whole process of the model is very necessary. Summary of the invention

[0010] When acquiring high-level semantic features, most change detection algorithms fail to fully consider the importance of multi-level feature fusion and long-distance relationship modeling; in addition, the deep supervision method for the change detection model training process cannot be ignored. In response to the above problems, the present invention provides a remote sensing image change detection method based on a hierarchical cross-scale global feature fusion deep network.

[0011] The present invention is implemented by adopting the following technical scheme: a remote sensing image change detection method based on a hierarchical cross-scale global feature fusion deep network is proposed, and a hierarchical multi-scale deep information extraction and fusion model framework for remote sensing image change detection is proposed, including a feature extraction network and a same-level difference feature enhancement module SLDFEM, a cross-scale adjacent level feature fusion module CSALFFM and a Transformer-based multi-scale global feature fusion module MSGFFM;

[0012] First, the dual-phase remote sensing images T1 and T2 are input into the framework, and the original multi-scale local features are obtained in parallel through the feature extraction network.

[0013] Then, the i-th stage pairwise features They are input into SLDFEM respectively, and their significant features and differential features are further extracted as Transformer's Q, K and V. This operation forms a new feature

[0014] At the same time, the adjacent feature tensors Input to CSALFFM for multi-scale information aggregation

[0015] The multi-scale global feature fusion module MSGFFM uses the Transformer decoder to enhance the multi-scale features mentioned above. Refine and decode; the output F of Transformer decoder o It is the tensor prototype of the change detection result;

[0016] Finally, the network detection head detects the output F of the Transformer decoder o Convolution is performed to generate the change result prediction value PV, and the change result prediction value PV is then binarized to obtain the final prediction result.

[0017] The above remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network, the feature of the same-level difference feature enhancement module SLDFEM The specific formation process is as follows:

[0018] For the i-th layer paired input features and SLDFEM cascades it and sends it to the spatial-channel parallel attention module S-CPAM, S-CPAM cascades and Forming input features And perform feature aggregation through convolution; then The spatial correlation enhancement features are obtained by convolution, spatial attention operation and convolution in sequence. At the same time, the convolution After the channel attention operation Multiply to get the output Then we can get the spatial and channel joint consistency features

[0019] for Respectively and Perform multiplication and residual connection, and then obtain the relevant features of depth enhancement and Respectively and Perform element-wise addition and subtraction operations to obtain the salient information and difference information in the dual-phase image. The entire attention process is expressed as: Final Output Right now

[0020] The above remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network, the feature The specific formation process is:

[0021] The input of CSALFFM is two sets of adjacent feature pairs. First, the deep features Perform convolution and transposed convolution operations. The whole process is as follows:

[0022] in, Represented as a transposed convolution operation;

[0023] right Perform convolution operation to compress it to the same size as The same number of channels is used, and the global average pooling is used to fuse and compress the entire spatial dimension information, and then linear operations and activation functions are used to convert Converted into channel weights, the whole process is as follows:

[0024] Finally, according to the residual connection paradigm, Add the above spatial correlation features and channel correlation features to obtain the fused adjacent layer features

[0025] Then for the deep features Perform convolution and transposed convolution operations. The whole process is as follows:

[0026] in, Represented as a transposed convolution operation;

[0027] right Perform convolution operation to compress it to the same size as The same number of channels is used, and the global average pooling is used to fuse and compress the entire spatial dimension information, and then linear operations and activation functions are used to convert Converted into channel weights, the whole process is as follows:

[0028] Finally, according to the residual connection paradigm, Add the above spatial correlation features and channel correlation features to obtain the fused adjacent layer features

[0029] After fully fusing adjacent features, CSALFFM further extracts their global information, significant information, and difference information, and performs self-attention operations on them to fully mine the potential high-level semantic information of the current adjacent feature layer. The whole process is as follows: Final Output Right now

[0030] The above remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network, the multi-scale global feature fusion module MSGFFM outputs F o The specific process is:

[0031] First, large-scale features Multi-scale information fusion is achieved through step-by-step information enhancement, as shown in the following formula: Where g2(·) is a multi-scale information enhancement function;

[0032] Then, the features of different scales are Adjust to the same size and send directly to the fusion end as a jumper. Where C(·) represents the cascade function, d n (X) represents n-fold downsampling on tensor X;

[0033] Secondly, large-scale features Multi-scale information fusion is achieved through step-by-step information enhancement, as shown in the following formula:

[0034] In this way, the multi-scale fusion results extracted by SLDFEM and CSALFFM can be obtained. and In order to further model the global contextual relationship, a multi-head cross-attention mechanism is implemented on the multi-scale fusion results. The large-scale feature maps that appear in the entire feature extraction and enhancement process are regarded as Q, that is: where u n (X) represents the n-fold upsampling of the tensor X; the entire attention process can be expressed as follows:

[0035] Final Output Right now

[0036] The above remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network designs the main loss function and auxiliary loss function.

[0037] The main loss function is as follows:

[0038] L main =λ1*L bce +λ2*L dice

[0039] L bce =-y·logy′-(1-y)·log(1-y′)

[0040] L dice =1-(2·y·softmax(y′)) / (y+softmax(y′))

[0041] , where y represents the true label and y′ represents the predicted value of the model;

[0042] The auxiliary loss function is as follows:

[0043]

[0044] Where Z i represents multi-scale labels, L i Represents the loss value of the predicted value of the i-th same-level fusion module and its corresponding scale label; Represents the loss value of the predicted value of the i-th neighboring fusion module and its corresponding scale label;

[0045] Final loss

[0046] L total =λ1*L bce +λ2*Ldice +λ3*L AUX

[0047] Here λ i , i∈{1,2,3} is the regularization coefficient.

[0048] The method of the present invention firstly builds a paired hierarchical multi-scale information fusion deep learning network architecture for remote sensing image change detection, aiming to extract robust high-level semantic features to identify changed areas and unchanged areas; then, in order to realize the enhancement of relevant information of dual-temporal remote sensing images, the dual-temporal image same-level difference feature enhancement module (SLDFEM) and the cross-scale adjacent level feature fusion module (CSALFFM) are proposed; secondly, in order to further enhance the relevance of global contextual information, the present invention extracts the global dependency of dual-temporal images based on the cross-attention mechanism, and proposes a multiscale global feature fusion module (MSGFFM); finally, the present invention designs a full-process supervised loss function to accelerate the convergence of the model and alleviate the problems of gradient disappearance. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is the structural diagram of the hierarchical cross-scale global feature fusion change detection network architecture HCGFFNet.

[0050] Figure 2 This is the overall structure diagram of the SLDFEM module.

[0051] Figure 3 Schematic diagram of the attention module.

[0052] Figure 4 This is the structure diagram of the cross-scale neighbor-level feature fusion module CSALFFM.

[0053] Figure 5 This is the structure diagram of the multi-scale global feature fusion module MSGFFM.

[0054] Figure 6 The following are the detection results of different methods on the LEVIR-CD dataset.

[0055] Figure 7 Figure 2 shows the detection results of different methods on the CDD dataset.

[0056] Figure 8 The following are the detection results of different methods on the SYSUCD dataset.

[0057] Fig. 9 Schematic diagram of the convergence and accuracy of HCGFFNet on LEVIR-CD during training / validation sets, where (a) shows the overall trend of the loss values ​​of the training / validation sets, and (b) shows the overall trend of the F1 scores of the training / validation sets.

[0058] Fig.10 This is a visualization diagram of the key modules of HCGFFNet. Each heat map is generated by adding and normalizing the elements of the same channel of the corresponding feature tensor. For easy observation, each visualization image is upsampled to 256x256 through bilinear interpolation. DETAILED DESCRIPTION

[0059] Due to differences in imaging conditions such as seasons, lighting, sensors, and the complexity of surface scenes, objects with the same semantic information in dual-temporal remote sensing images often show different characteristics in different time and space. In order to bridge this semantic gap, this paper proposes a remote sensing image change detection deep network framework to achieve robust recognition of high-level semantic information in complex scenes. The model makes full use of CNN local feature extraction capabilities and Transformer long-distance relationship modeling.

[0060] A. Overall Architecture

[0061] The HCGFFNet network structure is as follows Figure 1 As shown in the figure, intuitively it adopts the Siamese Convolutional neural network (SCNN) architecture, which consists of a feature extraction network and three auxiliary modules (SLDFEM, CSALFFM, MSGFFM).

[0062] The HCGFFNet framework can be summarized as Algorithm 1:

[0063] First, the dual-temporal remote sensing images Input the twin neural network for feature extraction, and obtain the original multi-scale local features in parallel through the backbone network (such as Resnet34) in H stands for height, W stands for width, represents the number of channels of the feature map of the i-th layer of the backbone network, and M is the total number of layers of the feature map;

[0064] Then, the i-th stage pairwise features They are input into SLDFEM respectively, and their significant features and differential features are further extracted as Transformer's Q, K and V. This operation forms a new feature representation It effectively models the global dependencies of local features extracted by CNN. is the number of feature map channels of the i-th layer of the same-level feature fusion network;

[0065] At the same time, the adjacent feature tensors Input to CSALFFM for multi-scale information aggregation This module helps to enhance the contextual correlation between adjacent features. is the number of feature map channels of the i-th layer of the neighboring feature fusion network;

[0066] Next, Transformer is used to refine and decode the enhanced multi-scale features. The output of Transformer decoder is the tensor prototype of the change detection result, C o Output feature map channels for transformer decoder;

[0067] Finally, the network detection head generates the predicted value of the change result It is worth noting that the intermediate results of all modules in the entire network are supervised by multi-scale labels, which are obtained by downsampling the source labels.

[0068]

[0069] B. Same-level difference feature enhancement module

[0070] Generally, the change detection difference map is obtained by directly subtracting the two-phase images, and then performing a series of operations such as convolution, pooling or attention mechanism to enhance the difference features. However, there are often noise interferences such as illumination, shadow and unchanged information in the two-phase images. Direct subtraction may cause the accumulation of erroneous information, which is not conducive to distinguishing the change detection target. In addition, since the twin network structure extracts high-level semantic features of the two-phase images by weight sharing, the unchanged areas in the same-level paired feature maps should have the same semantic features. However, due to the influence of factors such as noise, some unchanged areas show large differences at the pixel level, which interferes with the final change detection.

[0071] To solve the above problems, the present invention proposes a same-level difference feature enhancement module SLDFEM to improve the deep semantic expression of local difference features, such as Figure 2 SLDFEM first cascades the input information and performs feature enhancement from the spatial and channel aspects, then uses the fused features to extract the effective information of the original image, and finally obtains the difference features and significant features through element-level addition and subtraction operations, and uses cross self-attention to fully mine the useful information of the feature maps of the same level.

[0072] like Figure 2As shown, for the i-th layer paired input features and SLDFEM cascades it and sends it to the space-channel parallel attention module (S-CPAM) to select and focus on the important information of deep features. Figure 3 (a) shows the S-CPAM cascade and Forming input features And perform feature aggregation through 1×1 convolution; then, the dimensionality reduction result The spatial correlation enhancement feature is obtained by sequentially performing 3×3 convolution, spatial attention operation and 3×3 convolution. At the same time, After the channel attention operation Multiply to get the output Then the spatial and channel joint consistency features are obtained.

[0073] The entire S-CPAM process can be expressed by the following formula:

[0074]

[0075] Here, x is represented as a tensor.

[0076] for Respectively and Perform multiplication (feature weighting) and residual connection to obtain the relevant features of depth enhancement and In order to further explore the global correlation of the bi-temporal level features, SLDFEM is used to enhance the features. Perform self-attention operation. Figure 2 As shown, respectively and Perform element-wise addition and subtraction operations to obtain the salient information and difference information in the dual-phase image. It is worth noting that in the design of the self-attention mechanism, the salient features As Q, the differential feature As K and V, they aim to match the key information of salient features and differential features, and thereby enhance the significant difference information, that is, the semantic change area.

[0077] The entire attention process can be expressed as follows:

[0078]

[0079] Final Output Right now

[0080]

[0081] In the above formula, W q , W k , W v is the linear transformation matrix to be learned.

[0082] C. Cross-scale neighbor-level feature fusion module

[0083] As the depth increases, the information extracted by the network gradually transitions from low-level features (such as edges, textures, etc.) to high-level semantic information. In view of the complexity of objects in remote sensing images, strengthening the correlation between feature maps is obviously beneficial for the model to understand the semantic information of changing features. Looking at the entire feature extraction process, it is obvious that the correlation between feature maps of adjacent layers is the strongest. Inspired by this, the present invention proposes a cross-scale neighboring level feature fusion module (CSALFFM) to extract potential useful information from feature maps of adjacent layers.

[0084] like Figure 4 As shown, the input of CSALFFM is two sets of adjacent feature pairs. First, the deep features 1×1 convolution and transposed convolution operations are performed to adjust the two layers of feature maps to the same number of channels, and to adjust adjacent feature maps to the same size and enhance their salient features through the spatial attention mechanism.

[0085] The whole process is as follows:

[0086]

[0087] in, Represented as a transposed convolution operation.

[0088] At the same time, since the deep features have a wider receptive field, converting them into channel weights can highlight the effective information of shallow features. Perform a 1×1 convolution operation to compress it to the same size as The same number of channels is used, and the global average pooling is used to fuse and compress the entire spatial dimension information, and then linear operations and activation functions are used to convert Converted to channel weights.

[0089] The whole process is as follows:

[0090]

[0091] Finally, according to the residual connection paradigm, Add the above spatial correlation features and channel correlation features to obtain the fused adjacent layer features

[0092]

[0093] The same can be said

[0094] After fully fusing adjacent features, CSALFFM further extracts their global information (cascade), significant information (addition), and difference information (subtraction), and performs self-attention operations on them to fully explore the potential high-level semantic information of the current adjacent feature layer.

[0095] The whole process is as follows:

[0096]

[0097] Final Output Right now

[0098]

[0099] D. Multi-scale global feature fusion module

[0100] The SLDFEM and CSALFFM modules enhance local correlation and local context information respectively. In order to achieve a balance between spatial and semantic information and establish global dependencies between dual-phase images, the present invention proposes a Transformer-based multi-scale global feature fusion module (MSGFFM).

[0101] In view of the multi-scale features extracted by the SLDFEM and CSALFFM modules, the present invention intends to fuse and enhance them by adopting the paradigm of Resnet network residual blocks.

[0102] First, large-scale features realize multi-scale information fusion through step-by-step information enhancement, as shown in the following formula:

[0103]

[0104] Where g2(·) is a multi-scale information enhancement function, using a specially designed different-scale spatial channel parallel attention module (DSS-CPAM), as Figure 3 (b) The structure of DSS-CPAM is similar to that of S-CPAM, but the difference is that DSS-CPAM adds a maximum pooling layer before the 3×3 convolution to downsample the features and unify the feature scale.

[0105] Then, the features of different scales are Adjust to the same size and send directly to the fusion end as a jump connection (i.e. original features);

[0106]

[0107] Where C(·) represents the cascade function, d n (x) represents downsampling by a factor of n on the tensor x.

[0108] Secondly, large-scale features are further enhanced to achieve multi-scale information fusion.

[0109]

[0110]

[0111] In this way, the multi-scale fusion results extracted by SLDFEM and CSALFFM can be obtained. and In order to further model the global contextual relationship, the present invention implements a multi-head cross attention mechanism on the multi-scale fusion results. Specifically, firstly, and Cascaded and fed into the Transformer decoder as K and V. The decoder's Q is crucial for the Transformer, as it needs to pay attention to both the feature scale (consistent with the Transformer output size) and the multi-level physical and semantic meanings, which helps to achieve a balance between deep and shallow information. Therefore, the large-scale feature maps that appear throughout the feature extraction and enhancement process are regarded as Q, that is:

[0112]

[0113] where u n (x) represents upsampling the tensor x by a factor of n.

[0114] The entire attention process can be expressed as follows:

[0115]

[0116] Final Output Right now

[0117]

[0118]

[0119]

[0120] in, and is a linear projection operation, and h is the number of attention heads. In addition, the MLP consists of two linear layers, and there is a GELU activation function between the two linear layers.

[0121] In order to obtain the final change detection result PV, the detection head outputs F of the Transformer decoder o Perform 3×3 convolution, that is The predicted value PV of the change result is then binarized to obtain the final prediction result.

[0122] E. Loss Function

[0123] In general, the change detection loss function is often only located at the end of the network, but the final decision-making ability of the model is generally heavily dependent on the level of feature representation in the middle layer. Inspired by this, the present invention adopts a multi-layer side output strategy to design the main loss function and the auxiliary loss function. On the one hand, it can effectively supervise the feature learning of each layer of the deep network, and on the other hand, it can alleviate the gradient vanishing problem in the training process to a certain extent.

[0124] 1) Main loss function

[0125] The inherent problem of change detection is the imbalance of positive and negative sample classes. Regardless of which data set, the number of negative samples (unchanged pixels) almost always exceeds the number of positive samples (changed pixels). This will cause the neural network to ignore the learning of positive samples and instead learn the unimportant negative sample information. In order to solve this problem, the main loss function of the present invention uses a loss function that is a linear combination of binary cross entropy (BCE) and dice coefficient (DICE) to guide the network training process. Among them, binary cross entropy can measure the difference between the probability distributions of two given random variables; the dice coefficient can be used to test the similarity between different sets of variables (here, pixels of images in two different time periods).

[0126] The main loss function is as follows:

[0127] L main =λ1*L bce +λ2*L dice (18)

[0128] L bce =-y·logy′-(1-y)·log(1-y′) (19)

[0129] L dice =1-(2·y·softmax(y′)) / (y+softmax(y′)) (20)

[0130] Where y represents the true label and y′ represents the predicted value of the model.

[0131] 2) Auxiliary loss function

[0132] In order to supervise the feature learning of the intermediate layers of the deep network, a cross entropy loss function is set after each fusion module. Considering that the sizes of different intermediate features are different, the change detection labels are downsampled as samples of the intermediate layers.

[0133] The helper function looks like this:

[0134]

[0135] Where Z i Represents a multi-scale label, which is obtained by downsampling the true label according to the size of the intermediate prediction result, L i Represents the loss value of the predicted value of the i-th same-level fusion module and its corresponding scale label; Represents the loss value of the prediction value of the i-th neighboring fusion module and its corresponding scale label.

[0136] 3) Final loss

[0137] L total =λ1*L bce +λ2*L dice +λ3*L AUX (twenty two)

[0138] Here λ i , i∈{1,2,3} is the regularization coefficient.

[0139] Experimental verification

[0140] A. Dataset

[0141] To verify the effectiveness of HCGFFNet, the present invention conducted experiments on three representative high-resolution remote sensing image change detection datasets (i.e., LEVIR-CD, CDD, and SYSUCD). Each dataset consists of a change map and two high-resolution remote sensing images taken at different times in the same area. The details of the three datasets are as follows:

[0142] LEVIR-CD dataset

[0143] The LEVIR-CD dataset consists of 637 pairs of high-resolution remote sensing images of size 1024×1024, which are from 20 different areas in several cities in Texas, with a spatial resolution of 0.5 meters, and the main change type is building changes. In the experiment, each image is cropped into 256×256 non-overlapping image blocks, and 7120, 1024, and 2048 pairs of images are selected for training, verification, and testing, respectively.

[0144] CDD Dataset

[0145] The CDD dataset consists of 16,000 pairs of images of size 256×256, including 10,000 pairs of training sets, 3,000 pairs of validation sets, and 3,000 pairs of test sets. The spatial resolution of CDD is 0.3-1 meter, which includes the changes of different objects such as buildings, roads, and vehicles, and ignores the changes caused by seasonal differences, brightness and other factors.

[0146] SYSUCD Dataset

[0147] The dataset contains 20,000 pairs of 256×256 0.5-meter aerial images taken at a certain location between 2007 and 2014. The main types of changes in the dataset include: (a) new urban construction; (b) suburban expansion; (c) pre-construction foundation work; (d) vegetation change; (e) road expansion; (f) offshore construction. In addition, the 20,000 image pairs are divided into a training set (10,000 pairs), a validation set (4,000 pairs), and a test set (4,000 pairs).

[0148] B. Experimental Parameters

[0149] HCGFFNet is trained and tested using a single NVIDIA RTX3090 GPU. The Adam optimizer is used to transform the model so that the initial learning rate of 0.001 decays linearly to 0. In addition, validation is performed after each training cycle, and the best model on the validation set is used to evaluate the test set. The number of feature extraction backbone network layers M in the present invention is uniformly set to 5, and the number of feature layers in the i-th layer is

[0150] C. Evaluation Indicators

[0151] In order to fully reflect the performance of the model, five indicators including precision (P), recall (Re), F1, overall accuracy (OA) and mean intersection over union (mIoU) are used to evaluate the experimental results. The above indicators are defined as follows:

[0152]

[0153]

[0154]

[0155]

[0156]

[0157] Experimental Results

[0158] A. Comparison Method

[0159] In order to verify the effectiveness of the proposed HCGFFNet, this paper selects several advanced change detection methods for comparison, including FC-EF, FC-Siam-conc, FC-Siam-diff, STANet, DTCDSCN, SRCDNet, MSPSNet, ChangeFormer and BiT. For the sake of fairness, the above change detection networks are trained and tested on a unified platform using their public codes and default hyperparameters. The comparison of relevant indicators of the dataset is shown in Table 1; Figure 6-8 The model visualization comparison results are shown in Figure 2. In order to comprehensively evaluate the model performance, different sizes and numbers of change targets are selected for comparison.

[0160] B. Quantitative comparison

[0161] Table 1 Performance comparison of different algorithms on different data sets

[0162]

[0163] *All numerical values ​​are expressed as percentages (%). In addition, italics and bold italics indicate the best and second best, and bold black indicates the third best.

[0164] Optimum indicates the optimal parameter configuration of the model.

[0165] Table 1 shows the overall performance index comparison results of three datasets (LEVIR-CD, CDD, SYSUCD). Quantitative indicators show that the model of HCGFFNet after regularization is almost consistently better than other algorithms. For example, F1 and mIoU are 2.29 / 0.98 / 4.42 percentage points and 1.98 / 1.01 / 3.93 percentage points better than the BIT model in the three datasets, respectively. In addition, the CNN backbone only uses resnet34, and does not involve more complex structures such as resnet50, FPN, and UneT. This may be due to the model's ability to fuse multi-scale spatiotemporal information and enhance the model's feature representation capabilities through global context information modeling. C. Results and discussion on the LEVIR-CD dataset

[0166] The visual comparison of the detection results of each method on the LEVIR-CD dataset is shown in Figure 6As shown. For small targets, many algorithms are prone to missed detection, such as FC-EF, FC-CONC, FC-iff, SRCD, MSPS, changeformer, etc. When there are multiple change detection targets, the algorithm will suffer from different degrees of false detection near the edge, especially STANet and SRCD. For buildings with more complex and dense arrangements, the problems of false detection (MSPS, BiT, changeformer) and missed detection (STANet, SRCD) generally still occur near the edge. Therefore, whether the boundaries of each building are complete and smooth can intuitively reflect the performance of the model. Complex scenes will also interfere with the detection performance of some algorithms, such as MSPS, BiT, and changeformer. Relatively speaking, regardless of the size of the detection target, HCGFFNet shows superior performance in target integrity and boundary accuracy, which may be due to the fact that the model in this paper fully integrates information at each scale.

[0167] D. Results and Discussion on the CDD Dataset

[0168] The detection results on the CDD dataset are as follows Figure 7 As shown. Compared with the other two datasets, the CDD dataset scene is more complex and the change area is more disturbed. In addition, the change area is unevenly distributed, and the types and sizes of the change targets are different. Visually, FC-Siam-conc, FC-Siam-diff, and STANet show obvious false negatives overall; MSPSNet and FC-Siam-conc show certain false positives at different scales. Models such as FC-Siam-conc, FC-Siam-diff, MSPS, BiT, and Changeformer can easily lose some small targets with scattered distribution. For larger change areas, many methods show missed detection (e.g., FC-Siam-conc, FC-Siam-diff, MSPS, etc.) or detection errors (e.g., MSPS, STANet, FC-Siam-conc, etc.) at the edge of the target. In contrast, HCGFFNet shows better adaptability to change targets with a larger detection range.

[0169] E. Results and Discussion on the SYSUCD Dataset

[0170] Figure 8The visualized change detection results of the algorithms on the SYSUCD dataset are shown. Due to the greater number and complexity of change types in the SYSUCD dataset, the change detection performance of some models is relatively poor compared to LEVIR-CD. Under similar lighting conditions, there are obvious false negatives in small target areas of FC-Siam-conc, FC-Siam-diff, and DTCDNet. For larger targets, the complexity of the scene causes the detected target boundaries to be affected to varying degrees. When disturbed by light, FC-Siam-conc, STANet, DTCDNet, SRCDNet, MSPS, and BiT all show obvious false positives, while FC-Siam-diff, STANet, and ChangeFormer have more serious false negatives. Overall, the boundaries of the change areas detected by HCGFFNet are clear, with fewer false positives and false negatives, and it shows better adaptability for detecting target areas with richer change types. This may be because HCGFFNet can extract high-level semantic information through multi-scale feature fusion, making the model more robust.

[0171] F. Regularization parameter setting

[0172] Table 2: Experimental results of regularization coefficients

[0173]

[0174] *All values ​​are expressed as percentages (%), and bold black indicates the best.

[0175] In order to evaluate the contribution of each loss term, ablation experiments were performed on the hyperparameters λ1, λ2, and λ3, and the experimental results are shown in Table 2. For the LEVIR-CD dataset, the overall performance of the model under different trade-off parameters is relatively small. It can be seen that the cross entropy, dice loss, and deep supervision auxiliary loss have a relatively balanced impact on the model and dataset. The weight ratio with the highest recall rate (1:1:1) is selected as the optimal parameter, and its recall rate is 2.2% higher than the worst value (90.6VS 88.4). For CDD, the best parameter configuration (2:2:1) is more significant. Its cross entropy and dice loss have the same proportion, both higher than the deep supervision loss. Similarly, the optimal regularization coefficient ratio for the SYSUCD dataset is (2:2:3). They are obviously better than other configurations in terms of comprehensive indicator evaluation (F1, mIoU). This shows that for different datasets, the model has different requirements for deep supervision.

[0176] G. Ablation Experiment

[0177] As shown in Table 3, ablation experiments are performed to verify the effectiveness of the modules. The experiments show that no matter which dataset, each module makes its own contribution to improving the model performance.

[0178] Table 3: Module ablation experiment

[0179]

[0180]

[0181] *All values ​​are expressed as percentages (%), and bold black indicates the best.

[0182] H. Convergence Analysis

[0183] To visualize the training process, the convergence and accuracy of HCGFFNet are tested on LEVIR-CD. Fig. 9 (a) shows that the model loss decreases rapidly within the first 25 epochs for both the test set and the validation set; the validation set loss tends to stabilize after 100 epochs, indicating that HCGFFNet has good convergence.

[0184] Similarly, the F1 index rises rapidly within the first 25 epochs, and the validation set remains stable after 100 epochs. The results show that HCGFFNet has certain convergence, stability, and effectiveness. This may be attributed to the fact that HCGFFNet can learn effective multi-scale features and global context information, thereby accurately representing the detected change area of ​​interest.

[0185] In order to better show the learning effect of HCGFFNet at each stage, a sample in the test set is used to visualize the heat map of each stage in HCGFFNet. The heat map can intuitively explain the learning of the network at each stage for the changing target in the image. The visualization results are as follows: Fig.10 As shown in the figure. Given a dual-phase image, Resnet is first used to extract multi-level feature information from shallow to deep. Then, the same-level and adjacent-level feature fusion modules designed in this paper focus the attention mechanism on the change area of ​​interest. Obviously, the same-level fusion features are more refined, while the adjacent-level fusion features are richer in scale, which fully demonstrates the effectiveness of the module. In the feature decoding stage, Q shows the contour and position information of the target in the change area, which fully reflects the required goal of change detection, while K and V mainly contain the possible spatial and high-level semantic information of the change area. At the same time, the attention to unchanged targets in the decoding stage is also significantly reduced. In general, HCGFFNet effectively learns semantic features level by level to highlight the change targets.

[0186] This paper proposes a new high-resolution remote sensing image change detection network HCGFFNet. First, HCGFFNet uses ResNet34 as the backbone network to fully extract the multi-scale local features of dual-phase remote sensing images; then, this paper proposes modules such as SLDFEM, CSALFFM and MSGFFM to achieve multi-scale feature fusion and enhance the global context information representation capability, and finally introduces an auxiliary loss function to supervise the learning of intermediate layer features. In terms of experiments, HCGFFNet has achieved good results in comprehensive evaluation indicators (F1, mIoU) and qualitative comparison on three datasets including LEVIR-CD, CDD and SYSUCD, which proves that HCGFFNet has strong adaptability to different types of change target detection.

Claims

1. A remote sensing image change detection method based on a hierarchical cross-scale global feature fusion deep network, characterized by: A hierarchical multi-scale deep information extraction and fusion model framework for remote sensing image change detection is proposed, which includes a feature extraction network and the same-level difference feature enhancement module SLDFEM, the cross-scale neighbor-level feature fusion module CSALFFM and the Transformer-based multi-scale global feature fusion module MSGFFM. First, the dual-phase remote sensing images T1 and T2 are input into the framework, and the original multi-scale local features are obtained in parallel through the feature extraction network. Then, the i-th stage pairwise features They are input into SLDFEM respectively, and their significant features and differential features are further extracted as Q, K and V of Transformer respectively. This operation forms a new feature At the same time, the adjacent feature tensors Input to CSALFFM for multi-scale information aggregation The multi-scale global feature fusion module MSGFFM uses the Transformer decoder to enhance the multi-scale features mentioned above. Refine and decode; the output F of Transformer decoder o It is the tensor prototype of the change detection result; Finally, the network detection head detects the output F of the Transformer decoder o Convolution is performed to generate the change result prediction value PV, and the change result prediction value PV is then binarized to obtain the final prediction result; Features in the same-level difference feature enhancement module SLDFEM The specific formation process is as follows: For the i-th layer paired input features and SLDFEM cascades it and sends it to the spatial-channel parallel attention module S-CPAM, S-CPAM cascades and Forming input features And perform feature aggregation through convolution; then The spatial correlation enhancement features are obtained by convolution, spatial attention operation and convolution in sequence. At the same time, the convolution After the channel attention operation Multiply to get the output Then we can get the spatial and channel joint consistency features for Respectively and Perform multiplication and residual connection, and then obtain the relevant features of depth enhancement and Respectively and Perform element-wise addition and subtraction operations to obtain the salient information and difference information in the dual-phase image. The entire attention process is expressed as: Final Output Right now 2. The remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network according to claim 1 is characterized by: Features in the cross-scale neighboring feature fusion module CSALFFM The specific formation process is: The input of CSALFFM is two sets of adjacent feature pairs. First, the deep features Perform convolution and transposed convolution operations. The whole process is as follows: in, Represented as a transposed convolution operation; right Perform convolution operation to compress it to the same size as The same number of channels is used, and the global average pooling is used to fuse and compress the entire spatial dimension information, and then linear operations and activation functions are used to convert Converted into channel weights, the whole process is as follows: Finally, according to the residual connection paradigm, Add the above spatial correlation features and channel correlation features to obtain the fused adjacent layer features Then for the deep features Perform convolution and transposed convolution operations. The whole process is as follows: in, Represented as a transposed convolution operation; right Perform convolution operation to compress it to the same size as The same number of channels is used, and the global average pooling is used to fuse and compress the entire spatial dimension information, and then linear operations and activation functions are used to convert Converted into channel weights, the whole process is as follows: Finally, according to the residual connection paradigm, Add the above spatial correlation features and channel correlation features to obtain the fused adjacent layer features After fully fusing adjacent features, CSALFFM further extracts their global information, significant information, and difference information, and performs self-attention operations on them to fully mine the potential high-level semantic information of the current adjacent feature layer. The whole process is as follows: Final Output Right now 3. The remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network according to claim 2 is characterized by: The output of the multi-scale global feature fusion module MSGFFM is F o The specific process is: First, large-scale features Multi-scale information fusion is achieved through step-by-step information enhancement, as shown in the following formula: Where g2(·) is a multi-scale information enhancement function; Then, the features of different scales are Adjust to the same size and send directly to the fusion end as a jumper. Where C(·) represents the cascade function, Represented in tensor Perform n-fold downsampling on top; Secondly, large-scale features Multi-scale information fusion is achieved through step-by-step information enhancement, as shown in the following formula: In this way, the multi-scale fusion results extracted by SLDFEM and CSALFFM can be obtained. and In order to further model the global contextual relationship, a multi-head cross-attention mechanism is implemented on the multi-scale fusion results. The large-scale feature maps that appear in the entire feature extraction and enhancement process are regarded as Q, that is: in Represents a tensor Perform n-fold upsampling; the entire attention process can be expressed as follows: Final Output Right now 4. The remote sensing image change detection method based on hierarchical cross-scale global feature fusion deep network according to claim 3 is characterized by: The main loss function and auxiliary loss function are designed. The main loss function is as follows: L main =λ1*L bce +λ2*L dice L bce =-y·logy′-(1-y)·log(1-y′) L dice =1-(2·y·softmax(y′)) / (y+softmax(y′)) Where y represents the true label and y′ represents the predicted value of the model; The auxiliary loss function is as follows: Where Z i represents multi-scale labels, L i represents the loss value of the predicted value of the i-th same-level fusion module and its corresponding scale label, represents the loss value of the predicted value of the i-th neighboring fusion module and its corresponding scale label; the final loss L total =λ1*L bce +λ2*L dice +λ3*L AUX Here λ i , i∈{1,2,3} is the regularization coefficient.

Images