A Remote Sensing Image Change Detection System and Method Based on Deep Supervision Networks

Abstract

This invention provides a remote sensing image change detection system and method based on a deep supervised network. It utilizes a Unet++ model-based end-to-end encoder-decoder network (DSNet), including an encoder, decoder, and classifier. Two temporal images, Image1 and Image2, serve as inputs to the Siamese network. A multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information of the two temporal images acquired at each layer, along with the outputs of preceding nodes, is concatenated and used as input to the decoder. The decoder fuses feature maps from different scales, and the resulting feature maps are then classified by the deep supervised network to obtain the predicted change map. The asymmetric multi-scale residual feature extraction module (MultiRes block) designed in this system improves the F1-Score by 1.4%, and the deep fusion supervision part improves the F1-Score by 2.1% with a small increase in the number of parameters.

Description

Technical Field

[0001] This invention provides a remote sensing image change detection system and method based on deep supervised networks, belonging to the field of remote sensing image technology. Background Technology

[0002] Remote sensing image change detection is a technique that analyzes pixels that change across multiple temporal remote sensing images of the same area acquired at different times. This technique is widely used in land use, urban sprawl, and disaster assessment. With the development of machine learning, methods such as support vector machines, random forests, and decision trees are increasingly being applied to remote sensing images. Because traditional ground pixel classification often leads to the salt-and-pepper effect, the basic unit of change detection has shifted from pixels to objects. Simultaneously, many methods considering the relationships between adjacent pixels have been introduced into object-level change detection. Linear weights are used to classify invariant and uncertain pixels in differential images, and an improved Markov random field method is employed for change detection.

[0003] Convolutional Neural Networks (CNNs) have seen significant development in remote sensing image research, with representative algorithms including FC-EF, FC-Siam-conc, and FC-Siam-diff, enabling end-to-end training. Change detection methods based on fully convolutional networks can be broadly categorized into pre-fusion and post-fusion methods. Pre-fusion methods primarily use concatenated temporal images as input to a single-branch network, directly generating a binary change map. For example, an improved Unet++ network utilizes global and fine-grained information to generate a prediction map with high spatial accuracy. Then, it fuses prediction maps from different levels to generate a high-precision final change map, validating its effectiveness and reliability on high-resolution (VHR) satellite imagery. However, the improved Unet++ network structure requires a large amount of change image data and cannot effectively generate multi-scale change features, which limits its widespread application to some extent.

[0004] Post-fusion involves a dual-branch network with shared weights simultaneously extracting features from an image, mapping them to high-dimensional feature maps for fusion, and finally upsampling to obtain a change map. Representative algorithms include DifUnet++, SNUNet-CD, and Sima-DeepLabv3+ networks. However, remote sensing images are typically complex and information-rich. Most existing networks, when extracting features from them, do not fully extract the original image information and ignore the semantic relationships between different levels. They rely solely on training a single model to match a single output, resulting in the incomplete utilization of the image's spatial and spectral features, thus limiting the improvement of change detection capabilities.

[0005] Although existing remote sensing image change detection methods can extract features well, some problems still exist in the feature extraction process:

[0006] First: The original image information was not fully extracted.

[0007] Second: The semantic relationships between different levels are ignored, and the spatial and spectral features of the image cannot be fully utilized.

[0008] Third: Remote sensing images acquired at different times are easily affected by external conditions such as lighting, making it difficult to detect changes. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention provides a remote sensing image change detection system and method based on deep supervised networks.

[0010] Specifically:

[0011] A remote sensing image change detection system based on a deep supervised network, which is based on the Unet++ model end-to-end encoder-decoder network DSNet, including an encoder, decoder and classifier;

[0012] The two temporal images Image1 and Image2 are used as inputs to the Siamese network. The multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information of the two temporal images obtained from each layer and the output of the preceding node are concatenated and used as inputs to the decoder. The decoder fuses feature maps of different scales. The resulting feature maps are then classified by a deep supervised network to obtain the predicted change map.

[0013] The encoder, serving as the backbone of the network, is used to extract the difference information from the two temporal images. The two temporal images, Image1 and Image2, are used as inputs to the Siamese network. A multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information from each layer's two temporal images, along with the outputs of preceding nodes, is concatenated and used as the input to all subsequent nodes. Horizontally, the encoder contains five paths, named L1-L5 from top to bottom. Convolutional units within the same path have the same feature output mapping. Different paths contain different numbers of convolutional units; the output channels after convolutional units in paths L1-L5 are 32, 64, 128, 256, and 512, respectively.

[0014] The structure diagram of the multi-scale residual feature extraction module is as follows:

[0015] First, shallow information and detailed features of the image are extracted using standard 3×3 convolutional layers. Second, asymmetric convolutional kernels of 1×3 and 3×1 sizes are applied. Then, a separable 3×3 convolutional layer is introduced to perform channel-by-channel convolution on the features output from the asymmetric convolution in a two-dimensional plane, and the resulting independent channels are weighted and combined along the depth direction. Next, the output features of the three convolutional layers are concatenated to extract semantic information at different scales. Simultaneously, two 1×1 convolutional layers are used, one for residual connections and the other for adjusting the number of channels after concatenation. Finally, the concatenated output features are added to the residually connected features to obtain additional spatial information.

[0016] The decoder is used to generate the feature difference map:

[0017] Feature maps X0,4, X1,3, X2,2, and X3,1 are processed by a multi-scale residual feature extraction module and a transposed convolution operation, mapping the fused features of each path into feature maps F1, F2, F3, and F4, each with 32 channels. Different upsampling rates are used for each path: X0,4->U1×1, X1,3->U2×2, X2,2->U4×4, and X3,1->U8×8. This means that different strides are used in the transposed convolution to achieve a 256×256 dimension for the output feature maps F1, F2, F3, and F4. The Normalized Attention (NAM) method concatenates the feature maps of different paths with the same size into a fused feature map Fc with 128 channels. This is first processed by batch normalization (BN), then multiplied by the weight coefficient Wγ, and finally processed by an activation function to obtain the feature output FD.

[0018] Based on the above system, the present invention also provides a method for detecting changes in remote sensing images based on deep supervised networks, comprising the following steps:

[0019] Step 1: Collect image data to build a dataset;

[0020] Step 2: Image preprocessing, eliminating irrelevant information in the image, restoring useful and realistic information, enhancing the detectability of relevant information, and simplifying the data to the maximum extent to make it more suitable for model input.

[0021] Step 3: Evaluation indicators and parameter settings.

[0022] Based on the combination of the predicted result and the true value, each pixel is categorized into four cases: predicted as a positive sample but false (FP), predicted as a positive sample and true (TP), predicted as a negative sample but false (FN), and predicted as a negative sample but true (TN). To evaluate the model's performance, the following metrics are used: recall, precision, and F1-Score. In change detection tasks, a higher recall indicates a stronger ability of the model to detect more changed pixels, while higher precision indicates more accurate detection of changed pixels. The F1-Score combines the results of precision and recall, and is considered a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0. The evaluation metric formulas are shown below:

[0023]

[0024]

[0025]

[0026]

[0027] Where N represents the total number of pixels, P represents the ratio between the true and predicted values ​​under a given class distribution, and OA represents the overall classification accuracy. In the DSNet network, the training epochs are 100, and the batch size is set to 16. The learning rate uses StepLR, with an initial learning rate of 0.001 that decays by 0.5 every 10 epochs.

[0028] This invention has the following features:

[0029] First, to address the issue that existing models cannot fully extract information from the original image, a multi-scale residual feature extraction module (MultiRes block) is introduced in the encoder to replace the ordinary convolutional blocks in the original Unet++. The two temporal images Image1 and Image2 are used as inputs to the Siamese network. The multi-scale residual feature extraction module performs continuous feature extraction and downsampling, and the feature difference information from each layer's two temporal images, along with the outputs of the preceding nodes, are concatenated and used as inputs to all subsequent nodes. First, shallow information and detailed features of the image are extracted using standard 3×3 convolutional layers. Second, asymmetric convolutional kernels of 1×3 and 3×1 sizes effectively enhance the network's feature representation ability and robustness, while reducing parameter computation compared to standard convolutional layers. Then, a separable 3×3 convolutional layer is introduced to perform channel-by-channel convolution on the features output by the asymmetric convolution in a two-dimensional plane. The resulting independent channels are then weighted and combined in the depth direction, effectively utilizing feature information from different channels at the same spatial location and reducing the network's computational cost. Simultaneously, two 1×1 convolutional layers are used, one for residual connections and the other for adjusting the number of channels after concatenation. Finally, the concatenated output features are added to the residual-connected features to obtain additional spatial information. The output mapping of the convolutional unit is determined by X. ij Let i represent the i-th path in the downsampling process, j represent the j-th convolutional unit and the direction of the skip connection, C(.) represent the convolution operation of the multi-scale residual feature extraction module, D(.) represent the downsampling operation, U(.) represent the upsampling operation, and [.] represent the concatenation operation. The output of each node of the encoder is:

[0030]

[0031] Second: To address the issue of poor semantic correlation between different levels and the inability to fully utilize the spatial and spectral features of the image, the decoder fuses feature maps of different scales. Four feature maps X of different scales are constructed through the lateral output layer encoder. 0,4 X 1,3 X 2,2 X 3,1 This completes the deep supervision process of aggregating low-level features into high-level features at the output end. Feature map X 0 , 4 X 1,3 X 2,2 X 3,1 After multi-scale residual feature extraction and transposed convolution, the fused features of each path are mapped to feature maps F1, F2, F3, and F4, each with 32 channels. To ensure that the output feature maps can be restored to the original image size, a different upsampling rate (X) is used for each path. 0,4->U1×1,X 1,3 ->U2×2,X 2,2 ->U4×,X 3 ,1 ->U8×8), meaning that different strides are used in the transposed convolution to achieve output feature maps F1, F2, F3, and F4, all with dimensions of 256×256. The training process of different upsampling branches is supervised by an objective function, and the learning information from different supervisory layers is integrated into a highly fused branch. This reduces the feature differences between the encoder and decoder, improving the detection performance of remote sensing images.

[0032] Third: To address the issue that remote sensing images acquired at different times are easily affected by external conditions such as lighting, making change detection difficult, a normalization-based attention module (NAM) based on the training model is introduced in the decoder section. This method concatenates feature maps of different paths and the same size into a fused feature map F with 128 channels. C First, it undergoes batch normalization (BN), then is multiplied by the weight coefficient Wγ, and finally passes through an activation function to obtain the feature output FD. The specific formula is as follows:

[0033]

[0034] F D =sigmoid(W γ (BN(F C )))

[0035]

[0036] Fourth: To address the issue of declining learning capacity in deep learning and improve network robustness, this system employs deep supervision in the classifier section. A multi-scale prediction method is designed to enhance the network's segmentation accuracy, improve its ability to detect pseudo-changes, and thus enhance its learning ability for changed regions. Deep supervision primarily involves monitoring and backpropagating the hidden layers by calculating the true labels and the loss values ​​of the hidden layer classifier. The entire network not only monitors the fused feature map F... D The loss values ​​were calculated and backpropagated for different levels of feature maps F1, F2, F3, and F4 obtained from the decoder. That is, F1, F2, F3, F4, F... D Two-dimensional predicted change maps M1, M2, M3, M4, and M5 are generated using 1×1 convolutional layers and the Softmax activation function. Finally, the loss of each predicted change map is calculated against the ground truth values, and supervised training is performed. The loss calculation formula for the entire network is as follows:

[0037]

[0038] Compared with the prior art, the advantages of the present invention are:

[0039] First: In the ablation experiment, the asymmetric multi-scale residual feature extraction module (MultiResblock) designed in this system improved the F1-Score by 1.4%, and the deep fusion supervision part improved the F1-Score by 2.1% with a small increase in the number of parameters.

[0040] Second: In the comparative experiment of scene segmentation of large, medium and small target scenes and scenes with large differences in images before and after different lighting conditions, this system did not have any missed detections, false detections or edge blurring compared with other network models.

[0041] Third: By comparison, the floating-point operation value (FLOPs) of this system is only 60G, and the model performance is improved, which can achieve a balance between network performance and computational load. Attached Figure Description

[0042] Figure 1 Here is the overall structure diagram of the algorithm model of this invention:

[0043] Figure 2 Here is a structural diagram of the encoder of this invention system:

[0044] Figure 3 Here is a structural diagram of the multi-scale residual feature extraction module of this invention:

[0045] Figure 4 This is the feature difference map decoder generated by the decoder of this invention:

[0046] Figure 5 This is a diagram of the depth-supervision structure of the present invention:

[0047] Figure 6a This is one of the data processing diagrams of the embodiment;

[0048] Figure 6b This is the second schematic diagram of data processing in the embodiment;

[0049] Figure 6c This is the third schematic diagram of data processing in the embodiment;

[0050] Figure 7 This is a schematic diagram of the confusion matrix in an embodiment;

[0051] Figure 8 This is a visualization of the different change detection methods in the embodiments;

[0052] Figure 9 This is the F1-Score graph of the training and validation sets in this embodiment. Detailed Implementation

[0053] The specific technical solutions of the present invention will be described with reference to the embodiments.

[0054] A remote sensing image change detection system based on deep supervised networks, such as Figure 1 As shown, a novel end-to-end encoding / decoding network, DSNet, based on the Unet++ model, is designed, mainly consisting of an encoder, a decoder, and a classifier. Two temporal images, Image1 and Image2, are used as inputs to the Siamese network. A multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information of the two temporal images obtained from each layer, along with the outputs of the preceding nodes, are concatenated and used as input to the decoder. The decoder fuses feature maps from different scales, and the resulting feature maps are then classified by a deep supervised network to obtain the predicted change map.

[0055] like Figure 2 As shown, the encoder, serving as the backbone of the network, is used to extract the difference information from the two temporal images. The two temporal images, Image1 and Image2, are used as inputs to the Siamese network. A multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information from each layer's two temporal images, along with the outputs of preceding nodes, is concatenated and used as the input to all subsequent nodes. Horizontally, the encoder contains five paths, named L1-L5 from top to bottom. Convolutional units within the same path have the same feature output mapping. Different paths contain different numbers of convolutional units; the number of output channels after convolutional units in L1-L5 are 32, 64, 128, 256, and 512, respectively.

[0056] like Figure 3 As shown, firstly, shallow information and detailed features of the image are extracted using a standard 3×3 convolutional layer. Secondly, asymmetric convolutional kernels of 1×3 and 3×1 sizes are used. This has been proven to effectively enhance the network's feature representation ability and robustness, and reduces the computational cost compared to standard convolutional layers. Then, a 3×3 separable convolutional layer is introduced to perform channel-wise convolution on the features output by the asymmetric convolution in a two-dimensional plane. The resulting independent channels are then weighted and combined in the depth direction, effectively utilizing feature information from different channels at the same spatial location and reducing the network's computational cost. Next, the output features of the three convolutional layers are concatenated to extract semantic information at different scales. Simultaneously, two 1×1 convolutional layers are used in the module, one for residual connections and the other for adjusting the number of channels after concatenation. Finally, the concatenated output features are added to the residually connected features to obtain additional spatial information.

[0057] Decoder, the decoder used to generate the feature difference map, such as Figure 4As shown: Feature maps X0,4, X1,3, X2,2, X3,1 are processed by a multi-scale residual feature extraction module and transposed convolution to map the fused features of each path into feature maps F1, F2, F3, F4, each with 32 channels. To ensure that the output feature maps can be restored to the original image size, different upsampling rates are used for each path (X0,4->U1×1, X1,3->U2×2, X2,2->U4×4, X3,1->U8×8), that is, different strides are used in the transposed convolution to achieve a dimension of 256×256 for the output feature maps F1, F2, F3, F4. The Normalized Attention (NAM) method concatenates feature maps of different paths and the same size into a fused feature map Fc with 128 channels. It first undergoes batch normalization (BN), then multiplies with the weight coefficient Wγ, and finally passes through an activation function to obtain the feature output FD.

[0058] like Figure 5 The deep supervised architecture diagram shown illustrates that the entire network not only calculates the loss value for the fused feature map FD, but also calculates and backpropagates the loss values ​​for the different layers of feature maps F1, F2, F3, and F4 obtained from the decoder. Specifically, F1, F2, F3, F4, and FD are processed by 1×1 convolutional layers and the Softmax activation function to generate two-dimensional predicted transformation maps M1, M2, M3, M4, and M5, respectively. Finally, the loss of each predicted transformation map is calculated against the ground truth value, and supervised training is performed.

[0059] This system can be used in fields such as land use, urban expansion, and disaster assessment. It improves the detection effect of remote sensing images, which are easily affected by external conditions such as lighting, even when images acquired at different times are susceptible to changes. It achieves high detection accuracy, low false negative rate for change detection, and good detection results. The specific implementation steps are as follows.

[0060] Step 1: Collect image data to build the dataset. The experiment used the LEVIR-CD dataset. LEVIR-CD is an annotated dataset with image pairs annotated by professionals. It contains 0.5-resolution hyperspectral images taken by Google Earth in different seasons. The images span from 2002 to 2018 and come from 20 different areas in several cities in Texas, USA. It is primarily used to identify significant urban changes over 5 to 14 years, containing a wealth of information on seasonal and lighting variations, which helps train the change detection model.

[0061] Step 2: Image preprocessing. This involves removing irrelevant information from the image, restoring useful and accurate information, enhancing the detectability of relevant information, and simplifying the data to the maximum extent possible to make it more suitable for model input. For example... Figure 6a , Figure 6b and Figure 6cThe original dataset contained 637 pairs of 1024*1024 images. Through cropping and rotation operations, 10192 pairs of 256*256 pixel images were generated from the original images. The dataset was randomly divided into three parts: 70% for training, 20% for validation, and 10% for testing.

[0062] Step 3: Evaluation indicators and parameter settings. For example... Figure 7 The ground truth (blue) is obtained through manual labeling. The model's prediction is represented by the blue portion. Based on the combination of the predicted result and the ground truth, each pixel can be categorized into four cases: predicted as a positive sample but false (FP), predicted as a positive sample and true (TP), predicted as a negative sample but false (FN), and predicted as a negative sample but true (TN). Figure 8 To more intuitively evaluate the performance of DSNet, Table 1 further visualizes the segmentation results of representative networks on the LEVIR dataset. This experiment selected four typical aviation scene images, mainly involving large, medium, and small target scenes, as well as scenes with significant differences in images from different periods due to varying lighting conditions. The areas marked with circles and squares in the figures show local segmentation details, facilitating observation and comparison. To evaluate model performance, the following metrics were used: recall, precision, and F1-Score. In change detection tasks, a higher recall indicates a stronger ability of the model to detect more changed pixels, while higher precision indicates more accurate detection of changed pixels. Figure 9 The F1-Score combines the results of precision and recall, and can be seen as a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0. The evaluation metric formula is shown below:

[0063]

[0064]

[0065]

[0066]

[0067] Where N represents the total number of pixels, P represents the ratio between the true and predicted values ​​under a given class distribution, and OA represents the overall classification accuracy. In the DSNet network, the training epochs are 100, and the batch size is set to 16. The learning rate uses StepLR, with an initial learning rate of 0.001 that decays by 0.5 every 10 epochs.

[0068] Step 4: The method of this system was compared with existing deep learning-based change detection methods. The results are shown in Figure 1, which shows that the system has achieved excellent performance in change detection in complex scenes.

[0069] Table 1. Comparison of experimental results for change detection tasks using different methods.

[0070]

[0071] For large and medium-sized target scenarios, compared to other models, there were no missed detections, false detections, or blurred edges. For complex scenarios where large and small targets coexist, this system can effectively extract large targets and completely detect small targets. This system introduces an attention mechanism to reweight the fused feature maps, eliminating some redundant noise information and emphasizing important information about the target object. This allows the network to overcome the influence of irrelevant changes in lighting or climate on the model to the greatest extent. Experimental comparisons show that the proposed MultiResblock in this system improves the F1-Score by 1.4%, and the deep fusion supervision part improves the F1-Score by 2.1% with a small increase in parameters. The proposed method has a FLOPs value of only 60G and its model performance is superior to other methods, achieving a balance between network performance and computational cost.

Claims

1. A remote sensing image change detection system based on a deep supervised network, characterized in that, DSNet, an end-to-end encoder-decoder network based on the Unet++ model, includes an encoder, decoder, and classifier. The dual-temporal images Image1 and Image2 are used as inputs to the Siamese network. The multi-scale residual feature extraction module performs continuous feature extraction and downsampling. The feature difference information of the dual-temporal images obtained from each layer and the output of the preceding node are concatenated and used as inputs to the decoder. The decoder fuses feature maps of different scales. The resulting feature maps are then classified by a deep supervised network to obtain the predicted change map. The multi-scale residual feature extraction module first extracts shallow information and detail features of the image through a standard 3×3 convolutional layer; second, it uses 1×3 and 3×1 asymmetric convolutional kernels; then, it introduces a 3×3 separable convolutional layer to perform channel-by-channel convolution on the features output by the asymmetric convolution in a two-dimensional plane, and weights the resulting independent channels in the depth direction; next, it concatenates the output features of the above three convolutional layers to extract semantic information at different scales; simultaneously, the module also uses two 1×1 convolutional layers, one for residual concatenation and the other for adjusting the number of channels after concatenation; finally, it adds the concatenated output features to the residual concatenation features to obtain additional spatial information.

2. The remote sensing image change detection system based on a deep supervised network according to claim 1, characterized in that, The encoder, serving as the backbone of the network, is used to extract the difference information of the two temporal images. The two temporal images Image1 and Image2 serve as the input to the Siamese network. Through the multi-scale residual feature extraction module, continuous feature extraction and downsampling are performed. The feature difference information of the two temporal images obtained from each layer and the output of the preceding node are concatenated and used as the input of all subsequent nodes. From a horizontal structure perspective, the encoder contains five paths, named L1-L5 from top to bottom. The convolutional units in the same path have the same feature output mapping. Different paths contain different numbers of convolutional units. The number of output channels after the convolutional units of L1-L5 are 32, 64, 128, 256, and 512, respectively.

3. The remote sensing image change detection system based on a deep supervised network according to claim 1, characterized in that, The decoder is used to generate feature difference maps: feature maps X0,4, X1,3, X2,2, X3,1 are processed by a multi-scale residual feature extraction module and transposed convolution operation to map the fused features of each path into feature maps F1, F2, F3, F4, each with 32 channels. Each path uses a different upsampling rate: X0,4->U1×1, X1,3->U2×2, X2,2->U4×4, X3,1->U8×8. That is, different strides are used in the transposed convolution to achieve the output feature maps F1, F2, F3, and F4, all with dimensions of 256×256. The Normalized Attention (NAM) method concatenates the feature maps of different paths with the same size into a fused feature map Fc with 128 channels. It first undergoes batch normalization (BN), then multiplies with the weight coefficient Wγ, and finally passes through an activation function to obtain the feature output FD.

4. A method for detecting changes in remote sensing images based on deep supervised networks, characterized in that, A remote sensing image change detection system based on a deep supervised network as described in any one of claims 1 to 3 includes the following steps: Step 1: Collect image data to build a dataset; Step 2: Image preprocessing, removing irrelevant information from the image, restoring useful and realistic information, enhancing the detectability of relevant information, and simplifying the data to the maximum extent to make it more suitable for model input; Step 3: Setting evaluation indicators and parameters; Based on the combination of the predicted result (PredictedResult) and the true value, each pixel is categorized into four cases: FP (predicted as a positive sample but false), TP (predicted as a positive sample and true), FN (predicted as a negative sample but false), and TN (predicted as a negative sample but true). To evaluate the model's performance, the following metrics are used: recall, precision, and F1-Score. In change detection tasks, a higher recall indicates a stronger ability of the model to detect more changed pixels, while higher precision indicates more accurate detection of changed pixels. The F1-Score combines the results of precision and recall, and is considered a harmonic average of the model's precision and recall, with a maximum value of 1 and a minimum value of 0. The evaluation metric formulas are shown below: ； ； ； ； Where N represents the total number of pixels, P represents the ratio between the true and predicted values ​​under a given class distribution, and OA represents the overall classification accuracy; in the DSNet network, the training cycle is 100 rounds, and the batch size is set to 16; the learning rate adopts interval-adjusted StepLR, with an initial learning rate of 0.001 and a decay of 0.5 every 10 rounds.

Images