A multi-scale feature fusion deep network with adjustable recall for remote sensing change interpretation

Abstract

The present invention relates to the field of remote sensing image change detection, and specifically to a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation. Remote sensing image change detection aims to identify the change area of ​​interest in dual-phase images. However, the diversity of shapes and scales of change areas in remote sensing images in complex scenes can easily lead to serious missed detection problems in related change detection algorithms. In response to the above problems, the present invention first extracts hierarchical multi-scale feature information of dual-phase remote sensing images; then, CNN and Transformer are used to realize the fusion of local and global effective information of same-scale, cross-scale and multi-scale features; finally, an over-expectation push-pull loss regularization term is proposed to address the problem of imbalanced change detection samples. The loss function can obtain a deep learning model with a high recall rate by overall raising the prediction expectation of positive samples. Experiments have shown that the present invention has achieved significant results in both qualitative and quantitative indicators.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image change detection, and in particular to a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation. Background Art

[0002] Remote sensing image change detection aims to compare the image differences of the same area at different time periods and mark the changed areas. It has been widely used in urban expansion analysis, disaster assessment, military strikes, vegetation cover detection and other fields. At present, remote sensing image change detection often faces the following challenges: First, different lighting conditions often cause the same scene to show different characteristics, and because the dual-phase images are collected from different periods or even seasons, it is difficult to avoid the impact of light disturbances due to the satellite period remote sensing images; second, the change area is relatively complex, and its size and number are often irregular. Figure 1 As shown in the figure, sometimes the change area is small, sometimes it is large, and multiple adjacent change areas often show consistent characteristics. To address the above problems, integrating multi-scale features and extracting high-level semantic information to resist illumination perturbations is an effective solution.

[0003] Traditional change detection methods can be roughly divided into three categories: methods based on image arithmetic, image transformation, and post-classification. Methods based on image arithmetic include image difference, image ratio, change vector analysis, etc. These algorithms use relevant arithmetic (such as subtraction or division) to obtain feature maps, and then use segmentation thresholds to distinguish between changed and unchanged information. Obviously, the segmentation threshold is the difficulty and key point of this type of method. Methods based on image transformation include principal component analysis and multivariate change detection. They transform images into specific feature spaces, and improve the accuracy of change detection by highlighting changed areas and suppressing unchanged areas. Post-classification methods first classify two phase images, and then compare and analyze the classification results to generate a change result map. The accuracy of post-classification methods depends on the classification accuracy, so its error accumulation effect is serious.

[0004] The development of machine learning has improved the accuracy of change detection to a certain extent, such as support vector machines and random forests. However, in the face of different scenarios, the selection of machine learning methods and the generalization ability of models pose challenges to practical applications. Due to its powerful feature extraction method and nonlinear representation ability, deep learning methods have been widely used in change detection tasks and have shown good performance. This is attributed to the fact that various deep networks can break through the appearance interference of change noise, mine high-level semantic information of the region of interest, and thus achieve intelligent target recognition.

[0005] According to the feature map extraction and fusion methods, the change detection framework based on deep learning can be roughly divided into three categories: early feature fusion, late feature fusion, and multi-level feature fusion. The change detection deep network based on early fusion cascades the input paired images at the beginning, and then obtains the detection results through feature extraction, encoding, decoding and other operations; the change detection network based on the late stage often adopts a dual-stream structure to synchronously extract the deep features of the original image, and finally performs feature fusion and pixel classification. However, both of the above methods lack the hierarchical interaction and global information fusion of multi-scale deep features, which is obviously not conducive to the acquisition of high-level semantic information for change detection.

[0006] Recently, more research works are devoted to extracting deep information through the twin network structure and fusing the corresponding layer feature pairs to determine the change area, which is called the change detection method based on multi-level feature fusion. Obviously, this method can fully integrate shallow features (such as target texture, corners, edges, etc.) and deep semantic information (such as image content, semantic concepts), alleviate or even eliminate the semantic gap between different feature layers, and thus improve the change detection performance. However, some current works tend to study the potential relationship between feature maps of the same scale; obviously, the fusion of features of different scales can take into account both high-level semantic features and image resolution, making the extracted feature information more recognizable. Many quantitative and visualization results show that there is still huge room for research in the mining of relationships between multi-scale feature maps.

[0007] In addition, some simple multi-level feature fusion methods (such as cascading or addition) are difficult to cross the semantic gap caused by illumination disturbances and correctly interpret complex remote sensing scenes. To this end, some research works identify interesting changes in high-resolution remote sensing images based on attention mechanisms. Common spatial and channel attention mechanisms only re-weight and fuse bi-temporal feature maps in two dimensions: space and channel. Although these operations learn the correlation between feature maps in fixed neighborhoods or dimensions, the limited sampling range makes it impossible to model the relationship between long-distance information. Therefore, some studies have achieved context modeling in space and time through self-attention mechanisms, effectively improving the globality of remote sensing image change detection models.

[0008] In addition, the loss functions of most supervised learning models for change detection tasks use cross entropy, dice loss, contrast loss and their simple linear combinations. They all attempt to measure the probability distribution of two types of pixels: changed / unchanged, while balancing the inequality in the number of positive and negative samples of changed pixels, and ultimately improve the overall performance of the model. However, in some situations where high recall rates are required, more attention is paid to positive samples rather than comprehensive indicators, such as some illegal building detection scenarios. Summary of the invention

[0009] In order to identify the changing regions of different sizes under light disturbance, the present invention proposes a multi-scale feature fusion deep network with high recall rate (such as Figure 2 As shown in the figure, the aim is to extract high-level semantic features to cross the semantic gap caused by illumination noise and correctly identify the changed area. Firstly, hierarchical multi-scale features of bi-temporal images are extracted based on the Resnet backbone network, and a same-layer feature fusion module (SsFFM) is proposed; then, to enhance the neighborhood information of bi-temporal remote sensing images, an adjacent layer feature information enhancement and fusion module (CsFFM) is proposed; secondly, to further enhance the relevance of global contextual information, the global dependency of bi-temporal images (MsFFM) is extracted based on the cross-attention mechanism; finally, the super-expected push-pull loss is designed and combined with the cross entropy to form a hybrid loss function, and the recall rate of the model can be controlled by adjusting its regularization coefficient.

[0010] The present invention is implemented by adopting the following technical scheme: a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation, including a feature extraction network and a same-scale feature fusion module SsFFM, a cross-scale feature fusion module CsFFM and a multi-scale global feature fusion module MsFFM;

[0011] The specific algorithm steps of the network are:

[0012] First, the dual-temporal remote sensing images Input feature extraction network to extract features and obtain original multi-scale local features in parallel in H and W represent the height and width of the image respectively, and C represents the channel dimension of the image. represents the set of real numbers;

[0013] Secondly, the multi-scale local feature pair of the mth stage Input into SsFFM, and share the difference features as Q of the multi-head attention mechanism, while K and V are extracted from their respective feature tensors. This aims to establish its global feature representation based on the local information extracted by the feature extraction network.

[0014] At the same time, adjacent multi-scale local features The data are input into CsFFM together, and the adjacent but different scale information aggregation is realized in a non-sampling form. Further strengthen the contextual correlation between adjacent features;

[0015] Then, the module MsFFM scales the enhanced multi-scale features and constructs the Q, K, and V of the Transformer decoder. The output of the Transformer decoder is the change detection result tensor. Finally, by applying the tensor F o Convolutional fusion is performed to generate the predicted change detection mask.

[0016] The algorithm steps of the multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation and the same-scale feature fusion module SsFFM are as follows:

[0017] Get the m-th layer paired input features and The difference between the characteristics of As the q of the shared Transformer, each phase remote sensing feature tensor extracts its k and v, and the contextual relationship of the local feature tensor is modeled through the multi-head attention mechanism; the entire multi-head attention process can be expressed by the following formula:

[0018]

[0019] in The convolution kernel size of the depth-wise separable convolution is 3×3, and the stride d=1;

[0020] Right now Where σ(·) represents the Softmax operation implemented in the channel dimension, and d represents the scaling factor;

[0021] Final multi-head cross attention mechanism output in

[0022] The local features and global information of a single image are aggregated through residual connection, and finally the same-scale feature fusion of dual-phase images is achieved based on cascade operation. where ∪(·) is the cumulative cascade function.

[0023] The algorithm steps of the above-mentioned multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation and the cross-scale feature fusion module CsFFM are as follows:

[0024] The weight vector of the original feature map is obtained by concatenating the features of the same scale and fusing them through convolution. Then, the semantic tokens of the features of different scales are obtained by matrix multiplication.

[0025]

[0026] Where n represents the cross-scale operation of the nth neighboring layer, σ(·) represents the softmax function, which normalizes each semantic group to obtain the weight of the original feature;

[0027] In order to enhance the information fusion of different visual semantic tokens, the information flow between visual semantic tokens is realized by sliding the feature fusion window. This method requires cascading semantic tokens of different scales and arranging them in spatial order, and then realizing semantic interaction by sliding the window, and finally obtaining a new token group, namely

[0028] The process can be expressed by the following formula: Where TF(·) represents the feature fusion operation on tokens;

[0029] The Transformer encoder is used to model the global relationship between adjacent scale features, and the original multi-scale local features are used as the Q of the Transformer encoder. The fused semantic tokens are shared as the KV of encoders of different scales, further realizing the cross-flow of information in the encoding process. The entire Transformer encoder can be expressed by the following formula:

[0030]

[0031] Single-head attention output The characteristics of its remodeling can be specifically expressed as follows:

[0032] Final encoder output

[0033] in

[0034] The algorithm process of the above-mentioned multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation and the multi-scale global feature fusion module MsFFM is as follows:

[0035] First, the multi-scale features are downsampled to a uniform size and used as K and V of the Transformer decoder. Where Maxpool(·) represents downsampling through the maximum pooling layer;

[0036] Then, the multi-scale features are upsampled to a uniform size and used as Q of the Transformer decoder. Where Up(·) means passing through the upsampling layer;

[0037] The entire multi-scale global cross-attention process can be expressed as follows:

[0038]

[0039] Single-head attention output Right now:

[0040] Multi-head attention output Single head output

[0041] The final change detection head of the multi-scale global fusion feature obtains the prediction map PV through a 1×1 convolution operation, namely:

[0042] The above-mentioned multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation has a final loss function L T By binary cross entropy L BCE and the super-expected push-pull loss L HEPP Mixed composition, L T =λ1*L BCE +λ2*L HEPP , Where 0 represents unchanged pixels, N nc Indicates the number of unchanged pixels; 1 indicates the number of changed pixels, N c Indicates the number of changed pixels; Y hw represents the pixel value of the label at position (h,w), P hw It represents the probability value of the model's prediction at the position (h, w) as a changed pixel, and λ1 and λ2 represent the regularization coefficients. During the training phase, the predicted change detection mask and the true value GT need to be calculated through the proposed hybrid loss function to calculate the loss value, and then adjust the weight coefficient of the multi-scale feature fusion deep network.

[0043] The main advantages of the present invention are as follows:

[0044] 1) A high-recall hierarchical multi-scale remote sensing image feature fusion deep network is proposed. On the one hand, the network effectively combines CNN and transformer to achieve local and global information fusion of multi-scale features. On the other hand, it realizes adjustable recall rate, which is very suitable for those occasions that pay more attention to positive samples;

[0045] 2) The proposed same-level feature fusion module (SsFFM) realizes the fusion of the same-size deep feature maps by sharing Q; the proposed cross-scale neighboring feature fusion module (CsFFM) realizes the fusion of different scales in a non-sampling manner; the proposed multi-scale feature fusion module (MsFFM) realizes the global feature relationship modeling;

[0046] 3) An ultra-expected push-pull loss regularization term is proposed. This loss function can alleviate the problem of change detection data imbalance by predicting the value of positive samples through the push-pull model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1This is a schematic diagram of the effect of the regularization term coefficient in the loss function of the present invention on the model change detection results. For the convenience of intuitive comparison, white, black, red and green represent TP (true positive), TN (true negative), FP (false positive) and FN (false negative), respectively. Obviously, the greater the proportion of the super-expected push-pull loss proposed in the present invention (such as 0:1), the fewer missed detection areas, that is, the higher the recall rate. However, the false detection rate also increases. For this reason, the overall performance can be improved by adding or even increasing the cross-entropy regularization coefficient (such as 20:1). Experiments show that adjusting the super-expected push-pull loss coefficient can effectively achieve adjustable recall rate, which makes the model of the present invention suitable for a wider range of change detection tasks.

[0048] Figure 2 This is the structural diagram of the multi-scale feature fusion deep network HRMsFF.

[0049] Figure 3 This is the overall structure diagram of the module SsFFM.

[0050] Figure 4 This is the overall structure diagram of the module CsFFM.

[0051] Figure 5 This is the overall structure diagram of the module MsFFM.

[0052] Figure 6 Visual comparison of the detection results of each method on the LEVIR-CD dataset, (a) T1 Image, (b) T2Image, (c) Ground Truth, (d) FC-EF, (e) FC-Siam-Conc, (f) FC-Siam-Diff, (g) DTCDSCN, (h) BIT, (i) ChangeFormer, (j) ICIFNet, (k) DMINet, (l) USSFC-Net, (m) Our HRMsFF.

[0053] Figure 7 The detection results of each method on the WHU-CD dataset are as follows: (a) T1 Image, (b) T2 Image, (c) Ground Truth, (d) FC-EF, (e) FC-Siam-Conc, (f) FC-Siam-Diff, (g) DTCDSCN, (h) BIT, (i) ChangeFormer, (j) ICIFNet, (k) DMINet, (l) USSFC-Net, and (m) Our HRMsFF.

[0054] Figure 8Visual detection results of each method on the CDD dataset: (a) T1 Image, (b) T2Image, (c) Ground Truth, (d) FC-EF, (e) FC-Siam-Conc, (f) FC-Siam-Diff, (g) DTCDSCN, (h) BIT, (i) ChangeFormer, (j) ICIFNet, (k) DMINet, (l) USSFC-Net, (m) Our HRMsFF.

[0055] Fig. 9 Schematic diagram of the training and analysis process of HRMsFF on three datasets: (a) LEVIR-CD, (b) WHU-CD, and (c) CDD.

[0056] Fig.10 A visualization diagram of the key modules of the network, taking an image from the LEVIR-CD dataset as an example. (a) Input dual-time remote sensing image, (b) Extract multi-layer feature maps of the original image using CNN as the backbone (c) Feature map after selecting the same-layer feature fusion module (SsFFM) (d) Using CNN as the backbone to extract multi-layer feature maps of the original image (e) Feature map after selecting the cross-scale neighboring feature fusion module (CsFFM) and (f) Select the input of Transform in the multi-scale feature fusion module (MsFFM) and (g) Change probability map, (h) the final change detection result generated. DETAILED DESCRIPTION

[0057] like Figure 2 As shown in the figure, the multi-scale feature fusion deep network HRMsFFM adopts the SCNN structure as a whole, which consists of a feature extraction network and three auxiliary modules (SsFFM, CsFFM, MsFFM).

[0058] The HRMsFFM network framework algorithm is as follows:

[0059] First, the dual-temporal remote sensing images Input SCNN for feature extraction, and obtain original multi-scale local features in parallel through the backbone network (such as Resnet18) in H and W represent height and width respectively, C represents the channel dimension, represents the set of real numbers, represents the number of channels of the m-th layer feature map extracted by the backbone network;

[0060] Secondly, the corresponding multi-scale local features of the mth stage are The difference features are input into SsFFM, and the difference features are shared as the query vector Q of the multi-head attention mechanism. The key vector K and value vector V are extracted from their respective feature tensors. This is to establish its global feature representation based on the local information extracted by CNN. Indicates the number of channels of the m-th layer feature map processed by the same-layer feature fusion module;

[0061] At the same time, the local features of adjacent scales The data are input into CsFFM together, and the adjacent but different scale information aggregation is realized in a non-sampling form. Further strengthen the contextual correlation between adjacent features;

[0062] Then, the module MsFFM scales the enhanced multi-scale features and constructs the Q, K, and V of the Transformer decoder. The output of the Transformer decoder is the change detection result tensor. Finally, by applying the tensor F o Convolutional fusion is performed to generate predicted change detection masks C o Represents the number of channels of the feature map after being processed by the multi-scale feature fusion module;

[0063] Note that during the training phase, the predicted change detection mask and the true value GT need to be calculated through the proposed hybrid loss function (cross entropy and super-expected push-pull loss) to calculate the loss value, and then adjust the deep network weight coefficient.

[0064] Same-Scale Feature Fusion Module (SsFFM)

[0065] When humans observe the difference in changes in two-phase images, they often identify the difference features by directly fusing the information of the two images. Inspired by this, the present invention completes change detection based on the idea of ​​feature fusion. Affected by various noises such as illumination, it is obvious that simple methods such as direct addition and subtraction operations as fusion methods cannot effectively identify the true change features.

[0066] For the deep feature map of the dual-phase image, the same-scale features have the most direct relationship in terms of the corresponding pixel positions. Considering the contextual relevance of the change area, the present invention further extracts global information based on the local feature tensor extracted by CNN. First, obtain the m-th layer multi-scale local feature pair and As we all know, the difference semantic feature is the change detection requirement, so the present invention uses the difference feature As the Q of the shared Transformer, each phase remote sensing feature tensor extracts its K and V by itself, and the contextual relationship of the local feature tensor is modeled through the multi-head attention mechanism.

[0067] The entire multi-head attention process can be expressed as follows:

[0068]

[0069]

[0070] in The convolution kernel size of the depthwise separable convolution is 3×3, the stride d=1, and W q , W k and W v represents the learnable matrix parameters.

[0071] Right now

[0072] where σ(·) represents the Softmax operation implemented in the channel dimension and d represents the scaling factor.

[0073] Final multi-head cross attention mechanism output

[0074] in represents the jth head number of multi-head attention, h represents the total number of multi-head attention heads, and W O represents the learnable matrix parameters,

[0075] In order to improve the stability and comprehensive expression ability of model feature extraction, the present invention aggregates the local features (extracted by CNN) and global information (extracted by Transformer) of a single image through residual connection, and finally realizes the same-scale feature fusion of dual-phase images based on cascade operation.

[0076]

[0077] where ∪(·) is the cumulative cascade function.

[0078] Cross-Scale Feature Fusion Module (CsFFM)

[0079] Affected by human activities and the laws of objective world development, the relevant features of remote sensing change areas are often reflected in similar scales, such as buildings of the same shape but different areas. Inspired by this, this paper proposes a cross-scale feature fusion module (CsFFM) to mine the potential useful information of feature maps of adjacent layers, so as to help the model understand the semantic information of change features of similar scales.

[0080] When it comes to multi-scale feature fusion, usually the larger scale features are downsampled or the smaller scale features are upsampled to match the size. However, interpolation will inevitably lose information or blur the boundaries. For this reason, the present invention proposes a non-sampling method to achieve information flow at different scales. This method decomposes the feature map into multiple visual semantic tokens, where each tag vector represents a visual semantic unit, and then fuses these tokens to achieve the integration of features at different scales.

[0081] To this end, the present invention concatenates the same-scale features and obtains the weight vector of the original feature map through convolution fusion, and then obtains the semantic tokens of different-scale features through matrix multiplication.

[0082]

[0083] Where n represents the cross-scale operation of the nth neighboring layer, σ(·) represents the softmax function, Represents a cascade, which normalizes each semantic group to obtain the weight of the original feature.

[0084] In order to enhance the information fusion of different visual semantic tokens, the information flow between visual semantic tokens is realized by sliding the feature fusion window. This method requires cascading semantic tokens of different scales and arranging them in spatial order, and then realizing semantic interaction by sliding the window. Finally, the new token group is obtained, namely

[0085] The process can be expressed by the following formula:

[0086]

[0087] Where TF(·) represents the feature fusion operation on tokens.

[0088] Next, the Transformer encoder is used to model the global relationship between features at adjacent scales. Considering factors such as the size of the feature map, the original feature map is used as the Q of the Transformer encoder, and the fused semantic tokens are shared as the KV of encoders at different scales, further realizing information cross-flow in the encoding process.

[0089] The entire Transformer encoder can be expressed as follows:

[0090]

[0091] Single-head attention output The characteristics of its remodeling can be specifically expressed as follows:

[0092] Final encoder output

[0093] in

[0094] Multi-scale global feature fusion module (MsFFM)

[0095] Overall, both SsFFM and CsFFM modules enhance the global feature representation of local and neighborhood information. Figure 1 This reflects two major characteristics of remote sensing image change detection: one is the diversity of the size of the change area; the other is the correlation of certain change areas in the image context. To this end, the present invention proposes a multi-scale global feature fusion module for extracting multi-scale information and establishing its global dependency.

[0096] Enhanced multi-scale features for SsFFM and CsFFM The present invention obtains the predicted change detection results through the cross-attention mechanism of the Transformer decoder. First, the multi-scale features are downsampled to a uniform size (8×8) and used as K and V of the Transformer decoder, as shown in the following formula. Reducing the size of KV can not only efficiently obtain its main features, but also greatly reduce the amount of calculation.

[0097]

[0098] Where Maxpool(·) represents downsampling through the maximum pooling layer.

[0099] Then, the multi-scale features are upsampled to a uniform size (128×128) and used as Q of the Transformer decoder, as shown in the following equation. Increasing the size of Q can not only obtain all its detailed information, but also ensure that the decoder output size is as close to the original image as possible.

[0100]

[0101] Where Up(·) means passing through the upsampling layer.

[0102] The entire multi-scale global cross-attention process can be expressed as follows:

[0103]

[0104] Single-head attention output Right now:

[0105] Multi-head attention output Single head output

[0106] The final change detection head of the multi-scale global fusion feature obtains the prediction map PV through a 1×1 convolution operation, namely:

[0107] Loss Function

[0108] At present, most remote sensing change detection models tend to demonstrate their superior performance through comprehensive indicators (i.e., F1 and MIOU), but some scenarios have higher requirements for recall rate, such as medical diagnosis, financial fraud detection, security monitoring and other fields. A high recall rate means that the model can find as many positive examples as possible, while a low recall rate means that the model has serious missed detections. In the remote sensing change interpretation process, scene complexity and imbalanced change sample data may lead to higher missed detection rates, which is undesirable in monitoring scenarios such as illegal buildings and vegetation destruction.

[0109] To address this problem, the present invention proposes an over-expectation push-pull loss function. Most loss functions have a predicted expectation of 1 for positive samples and a predicted expectation of 0 for negative samples; the present invention uses an over-expectation push-pull loss function to push and pull the predicted probabilities of positive and negative examples to the positive probability threshold T and the negative probability threshold τ, respectively. When T>1 or τ<0, it is obvious that the predicted probability cannot reach the corresponding expected value, but a larger loss value will be calculated for each iteration, which will help improve model performance and accelerate convergence within a certain range. The loss function is shown below:

[0110]

[0111] Where 0 represents unchanged pixels, N nc Indicates the number of unchanged pixels; 1 indicates the number of changed pixels, N c Indicates the number of changed pixels. hw represents the pixel value of the label at position (h,w), P hw Represents the probability value of the model's prediction at position (h,w) being a changed pixel.

[0112] In this loss function, if the prediction expectation for changed pixels is increased, that is, the positive probability threshold T is increased, the model will focus on the learning of changed pixels and thus alleviate the data imbalance problem of remote sensing image change detection, and ultimately significantly improve the recall rate of the model.

[0113] When false positives or missed positives occur, that is, when the model prediction value and the true value GT differ greatly, the cross entropy loss function obviously calculates a larger loss value, which is more conducive to model convergence and even performance improvement. When the prediction result is relatively close to the expected value, the loss value calculated by the cross entropy loss function drops sharply. At this time, the over-expectation push-pull loss function is expected to improve model performance by increasing the predicted expectation to obtain a larger loss value.

[0114] To this end, the final loss function L T By binary cross entropy L BCE and the super-expected push-pull loss L HEPP Mixed composition.

[0115]

[0116] L T =λ1*L BCE +λ2*L HEPP

[0117] Experimental setup

[0118] A. Dataset

[0119] In order to verify the effectiveness of HRMsFF, experiments were conducted on three representative remote sensing image change detection datasets. Each dataset contains two remote sensing images taken at different times in the same area and their change detection results. The detailed information of the three datasets is as follows.

[0120] LEVIR-CD dataset

[0121] LEVIR-CD is a public large-scale building CD dataset, which contains 637 pairs of ultra-high resolution (0.5 m / pixel) RS images of size 1024×1024, covering a time span of 5-14 years. These images are from 20 different areas in multiple cities in Texas, USA, covering 31,333 changing building instances. It is worth noting that LEVIR-CD covers various types of buildings with significant land use changes, such as villa residences, high-rise apartments, small garages, and large warehouses. The present invention crops each image pair into 16 image pairs of size 256×256 without overlapping, and uses 7120 / 1024 / 2048 pairs of images for training / validation / testing, respectively.

[0122] WHU-CD dataset

[0123] WHU-CD records the changes in buildings after the 6.3 magnitude earthquake in Christchurch, New Zealand in February 2011. Specifically, the dataset contains a pair of aerial images with a spatial size of 32507×15354 and a resolution of 0.2 m / pixel taken in the same area in 2012 and 2016. The present invention cuts the large-scale image pairs into 256×256 patch pairs and randomly divides them into 6096 / 762 / 762 pairs for training / validation / testing respectively.

[0124] CDD Dataset

[0125] CDD is a public CD dataset consisting of seasonal satellite images with a spatial resolution of 0.03m-1m / pixel. The size of the changing areas in CDD is different, and it includes buildings, roads, vehicles, etc. The present invention divides the image data into 10000 / 2998 / 3000 pairs of 256×256 patches for training / validation / testing respectively.

[0126] Experimental parameters

[0127] The model HRMsFF is implemented in PyTorch and trained and tested using a single NVIDIA RTX4090 GPU. The SGD optimizer is used to optimize the model, with momentum set to 0.98 and weight loss set to 5e. -4 , the learning rate is initially set to 0.002. The optimizer learning rate is dynamically adjusted, that is, the learning rate decays by 0.7 times every 10 training cycles. In addition, the training cycles of the dataset are all 150, and the batch size is 24. In addition, a validation is performed after each training cycle, and the best model on the validation set is used to evaluate the test set. Backbone's M layer is set to 5, where

[0128] Evaluation indicators

[0129] The experimental results are comprehensively evaluated using five evaluation indicators: precision (P), recall (Re), F1, overall accuracy (OA) and mean intersection over union (mIoU). The definitions of the above indicators are as follows:

[0130]

[0131]

[0132] Among them, TP, FP, TN and FN refer to the number of true positives, false positives, true negatives and false negatives respectively.

[0133] Experimental Results

[0134] AComparison method (quantitative indicators / qualitative visual comparison)

[0135] To verify the effectiveness of HRMsFF on dual-temporal RS images, it is compared with several state-of-the-art CD methods, including FC-EF, FC-Siam-conc, FC-Siam-diff, DTCDSCN, BIT, Change Former, ICIFNet, DMINet, and USSFC-Net.

[0136] TABLE I

[0137] PERFORMANCE COMPARISON OF DIFFERENT METHODS ON VARIOUS DATASETS

[0138]

[0139] *All values ​​are reported in percentage(%).Color convention:best, 2nd best .

[0140] Table 1 Performance comparison of different methods on different datasets

[0141] Table 1 shows the quantitative indicators of different models on three data sets (i.e., LEVIR, WHU, and CDD). It is not difficult to find that the recall rate of the proposed model HRMsFF is ahead of other models, and is 2.7 / 2.6 / 2.2 points better than the suboptimal value respectively. In addition, while maintaining a high recall rate, the model of the present invention shows obvious advantages over other models in the three comprehensive indicators of F1 score, OA and mIoU of these data sets. Moreover, the CNN backbone of the present invention only uses resnet18, and does not involve more complex structures, such as resnet34, resnet50, FPN and UneT. This may be due to the fact that the model of the present invention integrates multi-scale spatiotemporal information, and can enhance the model feature representation capability through global context information modeling.

[0142] In order to facilitate intuitive comparison, the present invention makes a visual comparison of the experimental results of various algorithms, such as Figure 6-8 .

[0143] The visual comparison of the detection results of each method on the LEVIR-CD dataset is shown in Figure 6As shown. Obviously, the dual-time image is affected by different illumination differences, and the change area covers a wide range from small to large. For small target areas, FC-EF, BIT, ICIFNet, DMINet, and USSFC-Net have serious missed detections. For large target areas, the complexity of the scene causes the detected target area boundaries to be affected to varying degrees. For example, USSFC-Net has very large false detections. For buildings with more complex and dense arrangements, the problems of false detection (BIT) and missed detection (DMINet, USSFC-Net) generally still occur near the edges, which may be attributed to the weak context feature extraction capabilities of the above two models. When affected by light interference, DTCDNet has obvious false detections; while BIT, DMINet, and USSFC-Net have obvious missed detections. Overall, the HRMSFF algorithm shows excellent performance in detecting changed areas, with clear boundaries and low false detection and missed detection rates. This is mainly because HRMSFF focuses on the fusion of multi-scale semantic information of local details and global information in high-resolution remote sensing images, thereby more accurately identifying various types of changes and adapting to different target areas.

[0144] The detection results of each method on the WHU-CD dataset are as follows Figure 7 As shown. Missed detection is more common under light disturbance, especially for areas with small changes, such as FC-EF, FC-Siam-Diff, BIT, and Change Former. Obviously, DTCDSCN has serious false detection. When there are multiple change detection targets, the algorithm will suffer from different degrees of false detection near the edge, especially FC-EF, FC-Siam-Conc, and DTCDSCN. For change detection of large-scale targets in complex scenes, it is difficult for the algorithm to distinguish the change area, and it is very easy to produce false detection (FC-Siam-Conc, BIT) and missed detection (DTCDSCN, DMINet, USSFC-Net). DTCDSCN and BIT have poor change detection performance in complex scenes. Obviously, the scene complexity is positively correlated with the model's deep semantic feature extraction ability. Relatively speaking, regardless of the size of the detection target, the model proposed in the present invention shows superior performance in terms of target integrity and boundary accuracy, which may be attributed to the fact that the MsFFM module fully integrates information of various scales.

[0145] The visual detection results of each method on the CDD dataset are as follows: Figure 8As shown in the figure. The CDD dataset particularly emphasizes the impact of seasonal changes in remote sensing images on change detection. From the results, it can be seen that FC-EF, FC-Siam-Conc and FC-Siam-Diff show more missed detection problems under the influence of illumination and seasonal changes. Although BIT, ChangeFormer and ICIFNet can identify significant change areas, smaller change areas are often ignored. DMINet has made progress in improving change detection performance, but its shortcomings in multi-scale global-local modeling limit its performance on subtle changes. In addition, for larger change areas, ChangeFormer, ICIFNet and USSFCNet show false detection of edge targets and loss of unchanged objects. Compared with the above models, HRMsFF not only shows better detection performance in large and small scale change areas, but also shows excellent performance when facing seasonal changes. The HRMsFF model reveals the integrity of regions with different scales, which further verifies its effectiveness in multi-scale global and local feature extraction and fusion.

[0146] Ablation test

[0147] Table 2 Ablation experiments of each module of the model

[0148] TABLE II

[0149] MODULE ABLATION EXPERIMENT

[0150]

[0151] The present invention verifies the effectiveness of the proposed modules (i.e., SsFFM, CsFFM, and MsFFM) through ablation experiments. As shown in Table 2, “√” indicates that the corresponding module is adopted. The experiment shows that no matter on which dataset, each module makes its own contribution to improving the model performance.

[0152] Parameter analysis

[0153] Token length

[0154] Table 3. The impact of different token length changes in TF. The performance of HRMsFFM is evaluated on the LEVIR-CD dataset. Note that the ratio of cross entropy loss to contrast loss is set to 1:0.

[0155] TABLE III

[0156] EFFECT OF VARYING TOKEN LENGTHS IN TF.THIS EXPERIMENT EVALUATES THEPERFORMANCE OF HRMSFNET BASED ON THE LEVIR-CD DATASET.NOTE THAT AT THISPOINT,THE LOSS FUNCTION OF THE MODEL ONLY INCLUDES CROSS ENTROPY

[0157]

[0158] The present invention aggregates the deep semantic features of the image into a compact token group through a feature fusion window, so the token length is one of the important hyperparameters that affect the performance of the model. The present invention tests tokens of different lengths on the LEVIR-CD dataset and analyzes their impact on the model, where l∈{4, 9, 16, 25, 36, 49, 64, 81}. The experimental results are shown in Table 3. When l is shorter (such as 4 or 9), the token group is not enough to represent all the features of the model; when l is longer (such as 81 or 64), redundant information may distract the model from paying attention to effective features. When l=36, the model comprehensive indicators F1 and mIou reach the best.

[0159] Determination of hyperparameters T and τ for super-expected push-pull loss

[0160] TABLE IV

[0161] IMPACTS OF DIFFERENT HYPERPARAMETERS T AND τ ON THE MODEL WHEN USING THE HEPP LOSS FUNCTION ON THE LEVIR-CD DATASET.CAREFULLY, THE RATIO OF THE BCELOSS TO THE HEPP LOSS IS SET TO 1:1.

[0162]

[0163] Table 4 shows the impact of using different parameters τ and T of the super expected push-pull loss function on the model on LEVIR-CD. Note that the ratio of cross entropy loss to super expected push-pull loss is set to 1:1.

[0164] In order to solve the problem of sample imbalance in remote sensing change detection, the super-expected push-pull loss function proposed in the present invention mainly focuses on the detection of changed pixels. Under the premise of keeping τ=0, the influence of different T values ​​on the model performance is tested to explore the maximum target threshold to which the positive sample can be pushed and pulled. Through this series of experiments, it is finally determined that the comprehensive performance of the model reaches a better solution when T=2.5. Then, fix T=2.5 and gradually change the value of τ. The study found that when τ<0, the indicators of the model in various aspects have not been significantly improved. This may be due to the large number of negative samples (unchanged pixels), and the push and pull of negative samples causes the loss values ​​of the prediction probability and τ to be too large, which makes the model unable to be better tuned.

[0165] Loss function regularization term

[0166] TABLE V

[0167] THE INFLUENCE OF REGULARIZATION COEFFICIENT ON QUANTITATIVEEXPERIMENTAL RESULTS REGARDING DIFFERENT DATASET

[0168]

[0169] In order to verify the effectiveness of the super-expected push-pull loss function, the present invention conducted regularization experiments on the hyper-parameter cross-entropy loss coefficient λ1 and the contrast loss coefficient λ2 on the three data sets of LEVIR-CD, WHU-CD, and CDD. When there is only a super-expected push-pull loss term, its model recall rate is far ahead of other mixed loss functions. This rule proves that the super-expected push-pull loss function can significantly improve the model recall rate. At this time, the model is very practical for those occasions with high requirements for recall rate. With the increase of the cross-entropy coefficient, the comprehensive indicators F1 and MIOU of the model gradually change. When the comprehensive indicators of the model reach the optimal level, the model is more practical for those occasions with high requirements for comprehensive performance. In short, by adjusting the regularization term coefficient, the model can be adapted to changing scenarios and tasks.

[0170] Convergence analysis

[0171] The present invention evaluates the performance of HRMsFF on three datasets by tracking the F1 score and loss, such as Fig. 9As shown. It can be observed that when the number of epochs is less than 25, the model loss on the three data sets drops rapidly, and the F1 score value increases significantly with the increase of epochs. When the epoch exceeds 75, the model loss basically tends to be stable, indicating that the model of the present invention is stable, convergent and efficient, which may be attributed to the model learning effective multi-scale and global context information, and this information can accurately and quickly represent the changing area of ​​interest.

[0172] Visualization

[0173] In order to understand the HRMsFF model more intuitively, this paper selects a set of representative samples from the LEVIR-CD test set to visualize the feature maps generated by the HRMsFF model at different stages, such as Fig.10 As shown. Given a dual-phase image [ Fig.10 (a)] First, CNN is used as the backbone (i.e., resnet18) to extract multi-scale feature information from shallow to deep and [ Fig.10 (b) and (d)]. Then, the same-layer features are and Perform information fusion to generate a fusion feature map [ Fig.10 (c)] shows that the model focuses on the interesting change area and suppresses the interference of the background irrelevant to the task. At the same time, the adjacent feature layers and as well as and After channel concatenation, they are sent to CsFFM, which condenses these feature maps into more compact semantic tokens and cleverly realizes the cross-flow of information between feature maps of different sizes, thereby obtaining high-level semantic feature maps. and [ Fig.10 (e)]. In addition, in the feature decoding stage, [ Fig.10 (f)#1] shows the changing building outline and location information and ignores the existing building area, which fully reflects the demand goal of change detection; [ Fig.10 (f)#2] mainly contains the potential spatial and high-level semantic information of the change area. Finally, the change probability map after convolution [ Fig.10 (g)] shows accurate object localization and fine boundaries, demonstrating that HRMsFF captures the changing objects through feature fusion and effective supervision.

[0174] This paper proposes a new remote sensing image change detection network HRMsFF, which makes full use of CNN local feature extraction capabilities and Transformer long-distance relationship modeling. First, HRMsFF uses ResNet18 as the backbone network to fully extract multi-scale local features of dual-phase remote sensing images; then, the modules such as SsFFM, CsFFM and MsFFM proposed in this paper realize multi-scale feature fusion; finally, an over-expectation push-pull loss regularization term is proposed to address the missed detection problem, and a high-recall change detection model is realized. In addition, comparative experiments on LEVIR-CD, CDD, WHU and other datasets show that HRMsFF has achieved good results in comprehensive evaluation indicators (F1, mIoU) and qualitative comparisons on the three datasets, which proves that HRMsFF has strong adaptability to change detection of different types of targets.

Claims

1. A multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation, characterized by: It includes feature extraction network and same-scale feature fusion module SsFFM, cross-scale feature fusion module CsFFM and multi-scale global feature fusion module MsFFM; The specific algorithm steps of the network are: First, the dual-phase remote sensing image T1, Input feature extraction network to extract features and obtain original multi-scale local features in parallel in H and W represent height and width respectively, C represents the channel dimension, represents the set of real numbers, represents the number of channels of the m-th layer feature map extracted by the backbone network; Secondly, the multi-scale local feature pair of the mth stage Input into SsFFM, and share the difference features as Q of the multi-head attention mechanism, while K and V are extracted from their respective feature tensors. This aims to establish its global feature representation based on the local information extracted by the feature extraction network. Indicates the number of channels of the m-th layer feature map processed by the same-layer feature fusion module; At the same time, adjacent multi-scale local features The data are input into CsFFM together, and the adjacent but different scale information aggregation is realized in a non-sampling form. Further strengthen the contextual correlation between adjacent features; Then, the module MsFFM enhances the multi-scale features mentioned above and The scale integration is performed and the Q, K, and V of the Transformer decoder are constructed. The output of the Transformer decoder is the change detection result tensor Finally, by applying the tensor F o Convolutional fusion is performed to generate the predicted change detection mask; C o Represents the number of channels of the feature map after being processed by the multi-scale feature fusion module; The specific algorithm steps of the same-scale feature fusion module SsFFM are as follows: Get the m-th layer paired input features and The difference between the characteristics of As the q of the shared Transformer, each phase remote sensing feature tensor extracts its k and v, and the contextual relationship of the local feature tensor is modeled through the multi-head attention mechanism; the entire multi-head attention process can be expressed by the following formula: in The convolution kernel size of the depthwise separable convolution is 3×3, and the stride d=1; W q , W k , W v represents the learnable matrix parameters, and m represents the number of layers; Right now Where σ(·) represents the Softmax operation implemented in the channel dimension, and d represents the scaling factor; Final multi-head cross attention mechanism output in W O represents the learnable matrix parameters; The local features and global information of a single image are aggregated through residual connection, and finally the same-scale feature fusion of dual-phase images is achieved based on cascade operation. where ∪(·) is the cumulative cascade function.

2. According to claim 1, a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation is characterized by: The algorithm steps of the cross-scale feature fusion module CsFFM are as follows: The same-scale features are concatenated and fused through convolution to obtain the weight vector of the original feature map, and then the semantics of features of different scales are obtained through matrix multiplication. Where n represents the cross-scale operation of the nth neighboring layer, σ(·) represents the softmax function, which normalizes each semantic group to obtain the weight of the original feature; In order to enhance the information fusion of different visual semantic tokens, the information flow between visual semantic tokens is realized by sliding the feature fusion window; the cross-scale feature fusion module CsFFM needs to cascade semantic tokens of different scales and arrange them in spatial order, and then realize semantic interaction through the sliding window, and finally obtain a new token group, namely The process can be expressed by the following formula: Where TF(·) represents the feature fusion operation on tokens; The Transformer encoder is used to model the global relationship between adjacent scale features, and the original multi-scale local features are used as the Q of the Transformer encoder. The fused semantic tokens are shared as the KV of encoders of different scales, further realizing the cross-flow of information in the encoding process. The entire Transformer encoder can be expressed by the following formula: Single-head attention output The characteristics of its remodeling can be specifically expressed as follows: Final encoder output in 3. According to claim 2, a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation is characterized by: The algorithm process of the multi-scale global feature fusion module MsFFM is: First, the multi-scale features are downsampled to a uniform size and used as K and V of the Transformer decoder. Where Maxpool(·) means downsampling through the maximum pooling layer; Then, the multi-scale features are upsampled to a uniform size and used as Q of the Transformer decoder. Where Up(·) means passing through the upsampling layer; The entire multi-scale global cross-attention process can be expressed as follows: Single-head attention output Right now: Multi-head attention output Single head output The final change detection head of the multi-scale global fusion feature obtains the prediction map PV through a 1×1 convolution operation, namely:

4. According to claim 3, a multi-scale feature fusion deep network with adjustable recall rate for remote sensing change interpretation is characterized by: The final loss function L T By binary cross entropy L BCE and the super-expected push-pull loss L HEPP Mixed composition, L T =λ1*L BCE +λ2*L HEPP , Where 0 represents unchanged pixels, N nc Indicates the number of unchanged pixels; 1 indicates a changed pixel, N c Indicates the number of changed pixels; Y hw represents the pixel value of the label at position (h,w), P hw represents the probability value of the model's prediction at position (h, w) as a changed pixel, and λ1 and λ2 represent the regularization coefficients; During the training phase, the predicted change detection mask and the true value GT need to be calculated through the proposed hybrid loss function to calculate the loss value, and then adjust the weight coefficient of the multi-scale feature fusion deep network.

Images