A remote sensing image change detection method and system based on a recurrent UNet network

Abstract

The application discloses a remote sensing image change detection method and system based on a cyclic UNet network, and the method comprises the following steps: constructing a densely connected and cyclic UNet network DRUNet based on a UNet network; the DRUNet is constructed into a pair of cyclic units, and parameters and a codec are reused by utilizing forward and backward jump connections, wherein the forward jump connection transmits the coded features to the decoder, the backward jump connection transmits the decoded high-level semantic features from the decoder to the encoder, and recursion is continuously performed between the encoding layer and the decoding layer to realize better feature refinement; network architecture search is performed on the DRUNet network, an optimal substructure is searched, a change detection model is obtained, and the change detection model is trained; and the trained model is used for remote sensing image change detection. The application can improve the precision of the remote sensing image change detection model, reduce the complexity of the model, and improve the performance of the model from the perspective of cyclic utilization of features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and change detection, and particularly relates to a remote sensing image change detection method and system based on a recurrent UNet network. BACKGROUND

[0002] With the rapid development of earth observation technology, it becomes easier to obtain a large number of high-resolution remote sensing images, so it has become a research hotspot in the field of remote sensing to automatically extract effective information from remote sensing images. Remote sensing image change detection refers to assigning binary labels to pairs of pixels in dual temporal remote sensing images to define the ground surface changes caused by human activities or natural evolution. The ultimate goal of the change detection algorithm is to train a model similar to the human brain, which can autonomously identify changes in the same geographical area at different times. The existing various models are all through different ways to simulate the learning process of the human brain. Remote sensing image change detection has been successfully used in urban planning, agricultural monitoring and disaster emergency response.

[0003] Firstly, the change of imaging conditions at different times (such as seasonal alternation and light intensity) causes the same object in a given dual temporal remote sensing image to have different spectral characteristics, and different spectral characteristics lead to the misclassification of confusing features (unchanged areas are identified as changed, and changed areas are judged as unchanged). Secondly, the change area often has irregular and complex shapes, especially the changes of vegetation and forest. Therefore, it is very difficult to accurately identify the complex and irregular change areas in the dual temporal remote sensing image.

[0004] In recent years, deep learning methods (especially convolutional neural networks) have achieved remarkable success in a variety of remote sensing tasks. Due to the powerful feature representation and strong generalization ability of convolutional neural networks, many researchers have tried to apply them to the remote sensing image change detection task. Change detection can be regarded as a pixel-level binary classification task similar to semantic segmentation. Given the high requirements of the application field of remote sensing image change detection on detection accuracy and model performance, people also try to develop more change detection networks based on deep neural networks.

[0005] The existing deep learning-based remote sensing image change detection mainly consists of two steps: feature extraction and change discrimination. The feature extraction and background information fusion of remote sensing images directly affect the accuracy of change detection in the later stage. In view of the excellent effect of full convolutional neural network (FCN) in the pixel-level binary classification task, many remote sensing image change detection models have followed the detection idea of the FCN framework. For example, Ronneberger et al. proposed an end-to-end change detection network (UNet), which uses a deep fully convolutional network instead of traditional image classification methods to realize end-to-end remote sensing change detection and indirectly improve the accuracy of change detection. However, this method also has the same defects as traditional methods (first classification and then change detection), and the precision of change detection is very sensitive to pixel classification results. In order to avoid the error propagation caused by pixel classification, scholars mostly use deep neural networks to directly detect changes in dual-time remote sensing images. Zhang et al. proposed FC-Siam-Conc and FC-Siam-Diff, which use Siamese networks as part of feature extraction, use two parallel encoding streams, and achieve better weight reuse through the weight sharing process to reduce the number of parameters. Although simple connection can help recover spatial information in the up-sampling stage, the feature extraction of each change object is single, which is difficult to meet the needs of multi-scale change objects. Therefore, Peng et al. proposed UNet++ to realize multi-scale feature extraction through dense skip connection, reduce the pseudo-change caused by scale variance, realize the sampling of different depth feature output layers, and better restore the details of the image through the fusion of multiple results. Fang et al. proposed SNUNet (combination of Siamese network and Nested UNet), which maintains high-resolution, fine-grained representation through dense skip connection between the encoder and the decoder. In recent years, some research has introduced metric learning into change detection to replace the up-sampling process of the decoder, and directly obtains the change map by calculating the distance between the features of the dual-time images. For example, STANet and DASNet use Resnet18 and Resnet50 as the backbone network, respectively, combined with metric learning to calculate the difference, which has stronger feature extraction ability.

[0006] Although the deep learning-based remote sensing image change detection method has achieved good detection effect, due to the high resolution of remote sensing image, the influence of complex factors such as satellite imaging and seasonal style, the complex network is also difficult to effectively extract and distinguish the change of features, so that the detection precision is difficult to improve and the performance is difficult to optimize. Although the above method improves the ability of the change detection network to a certain extent, due to the negative influence of adding new modules and new branches, the features cannot be extracted and distinguished efficiently and accurately, which limits the precision and performance of the change network, so that the remote sensing image change detection method cannot obtain very accurate results in various data sets or multi-type change scenes. SUMMARY

[0007] The application provides a remote sensing image change detection method and system based on a recurrent UNet network to solve the technical problem that the existing remote sensing image change detection method cannot obtain very accurate results in multi-type change scenes.

[0008] To solve the above technical problems, the application provides the following technical solutions:

[0009] In one aspect, the application provides a remote sensing image change detection method based on a recurrent UNet network, which comprises:

[0010] A densely connected and recurrent UNet network DRUNet is constructed based on a UNet network; wherein the DRUNet is constructed into a pair of recurrent units, and parameters and encoders and decoders are reused by using forward and backward skip connections, wherein the forward skip connection transmits the encoded features to the decoder, the backward skip connection transmits the decoded high-level semantic features from the decoder to the encoder, and recursion is continuously performed between the encoding layer and the decoding layer to achieve better feature refinement;

[0011] The network architecture of the DRUNet network is searched to search for an optimal substructure and obtain a change detection model;

[0012] The change detection model is trained to obtain a trained change detection model;

[0013] The trained change detection model is used for remote sensing image change detection to obtain a change detection result.

[0014] Further, the DRUNet is a recurrent iterative network nested by multiple pairs of recurrent units at different levels; a pair of recurrent units contains a pair of encoder and decoder, and has forward and backward skip connections on the same and different levels; wherein the forward skip connections connect the retrieved features with the features from the encoders at the same and different levels, and the backward skip connections connect the retrieved features with the features from the decoders at the same and different levels.

[0015] Further, in the DRUNet, the size of the feature map is changed by using max-pooling or interpolation with different parameters; and the feature concatenation scheme is optimized to element average; the previous encoding / decoding features of all layers are connected with each next decoding / encoding layer through forward and backward skip connections to fuse multi-scale features.

[0016] Further, the network architecture search is performed on the DRUNet network to search for an optimal substructure to obtain a change detection model, comprising:

[0017] A two-stage search algorithm, K-continuous evolutionary algorithm, is used to find the sparse connection sub-architecture of the DRUNet; in the first stage, k skip connections are selected to narrow the search space; in the second stage, a continuous evolutionary algorithm is introduced to gradually discover the optimal substructure, thereby obtaining a change detection model without redundant connections or blocks.

[0018] Further, one extraction block in the DRUNet contains L layers and C skip connections;

[0019] The K-continuous evolutionary algorithm is used to find the sparse connection sub-architecture of the DRUNet, comprising:

[0020] Compress a learnable selection matrix to model the mapping between L input skip connections and k selected skip connections, and represent Φ(.) as a fully differentiable equation:

[0021] Φ(C,M)=Matmul(C,Gumbel_SoftMax(M))

[0022] wherein C={C1,…,C L} represents the complete set of skip connections in the extraction block, Matmul represents matrix multiplication, and Gumbel_SoftMax forces each column in the k columns of the selection matrix M to be a one-hot vector, which votes for one of the L input skip connections; the above formula generates k selected skip connections that allow repetition, to realize the dynamic selection of candidate skip connections and reduce the search space for searching for the optimal substructure;

[0023] After realizing the dynamic selection of candidate jump connections and narrowing the search space, a continuous evolution algorithm is used to gradually determine the current optimal substructure in subsequent iterations.

[0024] Further, the change detection model is trained to obtain a trained change detection model, comprising:

[0025] A balanced gradient contribution loss function is constructed, and the change detection model is trained using a gradient coordination mechanism to solve the class imbalance problem caused by the small number of changed pixels in remote sensing images in the change detection task.

[0026] Further, the expression of the balanced gradient contribution loss function is as follows:

[0027]

[0028] Wherein, L GHM (p,p * ) represents the balanced gradient contribution loss function, L CE (p i ,p i * ) is the cross-entropy loss, GD(g i ) represents the number of pixel points in the unit length region centered on g i , g i is the gradient module length of the i-th pixel, L CE represents the cross-entropy loss, p=sigmoid(x) is the probability predicted by the model, x is the output of the model, p * is the corresponding label, and N represents the total number of pixels.

[0029] On the other hand, the application also provides a remote sensing image change detection system based on a cyclic UNet network, comprising:

[0030] A DRUNet network construction module is used to construct a densely connected and cyclic UNet network DRUNet based on a UNet network; wherein the DRUNet is constructed into a pair of cyclic units, and the parameters and the encoder-decoder are reused by using forward and backward jump connections, wherein the forward jump connection transmits the encoded features to the decoder, and the backward jump connection transmits the decoded high-level semantic features from the decoder to the encoder, and recursion is continuously performed between the encoding layer and the decoding layer to achieve better feature refinement.

[0031] A network architecture search module is used to search the optimal substructure by searching the DRUNet network constructed by the DRUNet network construction module, and obtain a change detection model;

[0032] The change detection model training module is configured to train the change detection model obtained by the network architecture search module to obtain a trained change detection model.

[0033] The change detection module is configured to perform remote sensing image change detection by using the change detection model trained by the change detection model training module to obtain a change detection result.

[0034] In another aspect, the present application also provides an electronic device comprising a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0035] In another aspect, the present application also provides a computer-readable storage medium, which stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0036] The technical scheme provided by the present application has at least the following beneficial effects:

[0037] The present application adds the connection cycle idea and the neural network search algorithm to the remote sensing image change detection method, and solves the sample imbalance problem in remote sensing image change detection by balancing the gradient contribution. Specifically, the present application proposes an end-to-end dense recursive convolutional neural network DRUNet for remote sensing image change detection, which constructs the encoder and the decoder in a cyclic manner without introducing any additional parameters, can effectively aggregate multi-scale features and multi-level context information, and can reuse information with minimal additional memory requirements. Our DRUNet reduces the loss of image information in the learning process while utilizing multi-scale image features, enhancing the expression ability, which is superior to UNet and other extended methods that introduce auxiliary parameters. In addition, a two-stage search algorithm is used to find the sparse connection sub-architecture of DRUNet, and the optimized model realizes a competitive precision / complexity trade-off. At the same time, we use GHM loss to coordinate the imbalance of change / invariant pixels in change detection, and achieve improved results.

[0038] In summary, the present application can improve the accuracy of the existing remote sensing image change detection model, reduce the model complexity, and improve the performance of the remote sensing image change detection model from the perspective of recycling features. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 is an execution flow schematic diagram of a remote sensing image change detection method based on a cyclic UNet network provided by an embodiment of the present application;

[0041] Figure 2 is a ReUNet architecture schematic diagram provided by an embodiment of the present application;

[0042] Figure 3 is a comparison schematic diagram of the ReUNet and DRUNet architectures provided by an embodiment of the present application;

[0043] Figure 4 is an optimized sparse connection DRUNet schematic diagram provided by an embodiment of the present application;

[0044] Figure 5 is a comparison of evaluation indexes of the DRUNet trained by the CE loss and GHM loss on different data sets provided by an embodiment of the present application;wherein (a) is a comparison of evaluation indexes on the LEVIR-CD data set, (b) is a comparison of evaluation indexes on the WHU-CD data set, (c) is a comparison of evaluation indexes on the CDD data set, and (d) is a comparison of evaluation indexes on the SYSU-CD data set. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present application clearer, the following will further describe the embodiments of the present application in combination with the drawings.

[0046] First embodiment

[0047] In order to alleviate the contradiction between rich image representation and insufficient feature extraction in remote sensing image change detection, the present embodiment proposes a densely connected and cyclic UNet network (DRUNet), which maps the decoded features back to the encoder through backward jump connection, and explores the recursion between the encoding layer and the decoding layer, helping the change detection model to better represent the features and distinguish the changes. In addition, the dense connection may bring redundancy of model parameters, and it is necessary to filter and discard some invalid multi-scale features and redundant connections in different layers and cycles to search for the optimal substructure. At the same time, the few changed pixels and many unchanged pixels are a significant sample imbalance problem existing in remote sensing image change detection, and effective balancing of sample contribution can optimize the training and parameter optimization of the model. In view of this, the present embodiment proposes a remote sensing image change detection method based on a cyclic UNet, which can be implemented by an electronic device, and the main contributions of the method include the following three aspects:

[0048] (1) In order to integrate multi-scale features and multi-level semantic context information, this method proposes a new densely connected recurrent UNet network (DRUNet). The network constructs paired recurrent units (PRUs) and reuses parameters and encoders and decoders using forward and backward skip connections. The forward skip connection passes the encoded features to the decoder, and the backward skip connection passes the decoded high-level semantic features from the decoder to the encoder. Recursion is continuously performed between the encoding layer and the decoding layer to achieve better feature refinement. The encoding layer contains an encoder + DOWN module, and the decoding layer contains a decoder + UP module. The input of the encoder is spliced ​​from the output of the previous encoding layer and the output of the decoder at the same layer, and the input of the decoding layer is spliced ​​from the output of the previous decoding layer and the output of the encoder at the same layer.

[0049] (2) The model proposed by this method is lightweight. Considering that dense forward and backward jump connections will lead to computational redundancy of the DRUNet model, an improved two-stage progressive search algorithm (K-continuous evolutionary algorithm) is designed to perform network architecture search to obtain the simplest and most efficient change detection model, achieving a competitive accuracy and efficiency trade-off.

[0050] (3) In order to solve the class imbalance problem caused by the small number of changed pixels in remote sensing images in the change detection task, this method adopts the gradient harmony mechanism (GHM) for training. Without adding additional hyperparameters, the effectiveness of balanced gradient in solving the class imbalance problem of remote sensing images is verified, thereby improving the change detection accuracy.

[0051] Based on the above, the execution process of this method is as follows Figure 1 As shown, the following steps are included:

[0052] S1, based on the UNet network, builds a densely connected and recurrent UNet network DRUNet. DRUNet builds paired recurrent units and reuses parameters and encoder-decoders using forward and backward skip connections. The forward skip connection passes the encoded features to the decoder, and the backward skip connection passes the decoded high-level semantic features from the decoder to the encoder. Recursion is continuously performed between the encoding and decoding layers to achieve better feature refinement.

[0053] Specifically, this embodiment combines Figure 2 and Figure 3 The model proposed in this method is explained.

[0054] like Figure 2The rotation "U" shaped recurrent UNet network with paired bidirectional connections (ReUNet) proposed in the embodiment is shown, the bidirectional connections of which include forward skip connections from the encoder to the decoder and backward skip connections from the decoder to the encoder, which not only maintains the compactness of the UNet structure, but also reuses the potential features of all decoding layers, and realizes more intermediate information aggregation. For the pixel change detection task, the model can obtain more sufficient feature extraction and representation, and can effectively capture changes of different sizes in complex scenes.

[0055] (1) Basic network structure ReUNet: ReUNet is a recurrent iterative network nested by a plurality of paired recurrent units (PRUs) of different levels, and the iteration is self-defined. A paired recurrent unit (PRU) contains a pair of encoder and decoder, forward skip connection and backward skip connection. In the PRU, the forward skip connection connects the retrieved features with the features from the same layer encoder, and the backward skip connection connects the retrieved features with the features from the same layer decoder. We rely on a UNet design, and when initialized, the encoder is replaced by a pre-trained network ResNet34, which has strong feature extraction capability in the first iteration. Input the two-phase remote sensing images into the pre-trained backbone network for feature extraction, and assign to the encoder at initialization. The recursion starts from the output of the decoder of the deepest PRU, and then up-samples the abstracted cascaded features of the two-phase remote sensing images extracted from the pre-trained encoder network. In the iterative update process, the parameters are reused through the paired forward and backward connections, and the PRU is recursively updated. The features are passed in the PRUs of different levels, and along the direction of data flow, the features are then passed to a series of decoders to recover the encoded details using the transpose convolution operation. In order to formalize our recursive unit, let's consider the process of iteration t of recursion. All the encoders and decoders of the UNet are taken as the recurrent unit, and the activation value generated by the previous layer iteration and the output of the previous layer are taken as the input of the next cycle update. As shown in the encoder and decoder in Figure 2 , we use to represent the layer encoder activation value at iteration t, which is the feature map of the encoding layer, and is used to represent the layer decoder activation value corresponding to the decoding layer. ReUNet takes the extracted features as the initialization of , and then iteratively updates and T times.

[0056] Forward skip connection. The forward skip connection connecting the encoder and the decoder in the same layer can preserve the bottom layer visual features of the layer encoding such as object boundaries and colors. In our iterative update, the forward skip connection is reused. In ReUNet, the decoder in PRU generates the low-level decoded layers from the high-level encoded layers As input, the activation generated by its symmetric encoder at iteration t is propagated through the decoder convolution to generate further recovered to higher resolution by the UP module. This process can be defined as:

[0057]

[0058]

[0059] where [·] denotes the concatenation operation, and UP denotes the up-sampling (transpose convolution).

[0060] Backward skip connection. The backward skip connection passes the decoded high-level semantic features from the decoder to the encoder, which is a new operation based on the experience of traditional forward skip connection. The backward skip connection is simple and easy to implement, without introducing additional parameters, and realizes the flexible aggregation of low-level visual features and high-level semantic features, which makes our model more discriminative when dealing with various types of remote sensing image changes. In the current iteration t, our encoder can combine with the original input x in produced by the previous encoding layer as a new input. Similar to the decoding path enhanced by the forward skip connection, our encoding process can be represented by the encoding convolution as:

[0061]

[0062]

[0063] DOWN module represents the down-sampling (composed of CONV and max-pooling), and feeds to the subsequent encoder for cyclic and deeper feature extraction.

[0064] (2) Dense connection cyclic UNet network DRUNet: the first layer is not only related to the output of the first layer, but also related to the output of all previous layers. For more clarity, we denote the sequential encoding or decoding process as a feature extraction stage. A ReUNet has four feature extraction stages, iterated twice, as shown in Figure 3 DRUNet is proposed under the ReUNet structure, which contains encoders / decoders, and has forward and backward skip connections on the same layer and different layers. Figure 3The specific architecture of DRUNet is shown, and the success of the multi-scale method shows the rationality of the multi-scale skip connection that fuses fine-grained features and coarse-grained semantics. Feature reuse is a good feature extraction method without learning redundant features multiple times. For this purpose, we introduce multi-scale dense forward / backward skip connections in ReUNet, enriching feature aggregation in change detection.

[0065] For DRUNet, the feature maps of different layers need to be spliced, so the same size must be maintained, which limits the implementation of down-sampling and up-sampling in the network. In DRUNet, we change the size of the feature map by using max-pooling or interpolation with different parameters. For convenience of explanation, we use to represent these size transformation operations. In summary, at iteration t times, for the layer features, the expressions of the forward skip connection and the backward skip connection in DRUNet should be updated as:

[0066]

[0067]

[0068] where L is the total number of layers. At the same time, we optimize the feature splicing scheme to element average to initially reduce the total complexity. In order to fuse multi-scale features, the previous encoding / decoding features of all layers are connected to each next decoding / encoding layer through bidirectional skip.

[0069] S2, network architecture search is performed on DRUNet to search for the optimal substructure and obtain a change detection model;

[0070] It should be noted that although the above design promotes multi-scale feature fusion and reduces the network size, we have found from experience that this dense connection only brings marginal improvement in overall performance, while the computational cost increases. In order to seek a more sparse connection sub-architecture in DRUNet, we use a two-stage search algorithm: K-continuous evolutionary algorithm to find the sparse connection sub-architecture of DRUNet. In the first stage, we select k(k<L) skip connections to reduce the search space; in the second stage, we introduce a continuous evolutionary algorithm to gradually discover the optimal substructure. Finally, we get the DRUNet architecture without redundant connections or blocks as shown in Figure 4 .

[0071] Stage 1: Skip connection search. A feature extraction block in DRUNet contains L layers and C skip connections. We compress a learnable selection matrix to model the mapping between L input skip connections and k selected skip connections, and represent Φ(.) as a fully differentiable equation:

[0072] Φ(C, M) = Matmul(C, Gumbel_SoftMax(M)) (7)

[0073] where C = {C1, …, C L} denotes the full set of skip connections in the extraction block, Matmul represents matrix multiplication, and Gumbel_SoftMax forces each column in the k columns of the matrix M to be a one-hot vector, which votes for one of the L input skip connections. Our formula generates k selected skip connections that are allowed to repeat, which realizes the dynamic selection of candidate skips and reduces the search space for searching the optimal substructure.

[0074] Stage 2: Finding the optimal substructure. We use a continuous evolution algorithm to find a best subset of the DRUNet that is preliminarily cut. First, initialize a SuperNet (the DRUNet searched in the first stage) with different feature extraction layers and different skip connections, then select several excellent sub-models of different sizes and accuracies using a non-dominated sorting strategy, then train the subnets and update the blocks in the SuperNet corresponding to the subnets. We simultaneously search all layers between a pair of adjacent feature extraction blocks for candidate skips, and then move to the next pair step by step after the current search is completed, so we can sample different subnets for different blocks in the first iteration of the SuperNet, and determine the discarded blocks in the first iteration by calculating the average loss. Then fix the network parameters of the first iteration to update the SuperNet and gradually determine the current optimal iteration substructure in subsequent iterations.

[0075] In summary, we manually select k skip connections obtained in stage 1 to narrow the search space of stage 2, and then use the continuous evolution algorithm of stage 2 to gradually determine the current optimal iteration substructure in subsequent iterations. By using this two-stage k-continuous evolution algorithm to search for the optimal precision / complexity trade-off model structure on different datasets, as Figure 4 shown.

[0076] S3, training the change detection model to obtain the trained change detection model;

[0077] It should be noted that in the traditional cross-entropy loss, there is an imbalance problem when punishing the feature pairs that change and the feature pairs that do not change during training. The traditional cross-entropy loss is defined as:

[0078]

[0079] where p = sigmoid(x) is the probability predicted by the network, x is the output of the network, and p * is the corresponding label. Define the gradient magnitude g:

[0080]

[0081] where L CE represents the cross-entropy loss, the value of g is proportional to the difficulty of detection, and its range is [0, 1]. The greater g is, the more difficult it is to determine whether a change has occurred in the current pixel.

[0082] To address the incoordination problem of gradient magnitude distribution, a gradient density coordination method is proposed. According to the physical definition of density (mass per unit volume), the gradient density function is defined as:

[0083]

[0084] where g i is the gradient magnitude of the i-th pixel, N represents the total number of pixels, and d represents the gradient density function argument gradient magnitude, which ranges from 0 to 1.

[0085]

[0086]

[0087] where represents the number of pixels whose gradient magnitude falls within the interval , and l ∈ (d) represents the length of the interval , and ∈ is a self-defined size. The physical meaning of the gradient density GD(g i ) is the number of pixels within the region centered on the gradient density argument g i , with a length of ∈, and normalized by the effective length of the region. Therefore, for each pixel, the gradient density coordination parameter of each pixel is defined as Consider using the gradient coordination mechanism (GHM) idea to weight the cross-entropy loss function, i.e., multiply the cross-entropy loss by the inverse of the gradient density of the pixel, which is defined as follows:

[0088]

[0089] According to the calculation formula of the GHM loss function, the candidate weights of simple negative samples and outliers are reduced, and the influence of normal difficult samples is improved, so that the model focuses on more effective normal difficult samples. The loss and difficulty of training will be greatly reduced, thereby improving the performance of the model.

[0090] S4, using the trained change detection model to perform remote sensing image change detection to obtain the detection result.

[0091] Below, for the purpose of illustrating the superiority of the present method, the present embodiment carries out experiment effectiveness and strategy effectiveness proof.

[0092] To evaluate the performance of the method, we compare several change detection methods on four remote sensing image datasets, LEVIR-CD, WHU-CD, CDD and SYSU-CD, and design ablation experiments to evaluate the optimized structure and newly designed loss function.

[0093] 1) LEVIR-CD dataset: The LEVIR-CD dataset contains 637 pairs of VHR Google Earth images with a resolution of 0.5m, size 1024x1024, and collection time from 2002 to 2018, covering various changes in buildings such as garages, villas, warehouses, etc. The fully annotated LEVIR-CD contains a total of 31,333 individual changed buildings, most of the changed areas are represented as rectangles or polygons with clear edges. The dataset is divided into training, validation, and test sets, including 445 pairs, 64 pairs, and 128 pairs. We crop each sample into 16 small images of size 256x256, generating 7120 image patch pairs for training, 1024 image patch pairs for validation, and keeping the test size as 1024x1024 in our experiments.

[0094] 2) WHU-CD dataset: The WHU-CD dataset covers the area of reconstruction after the 6.3 earthquake. It contains a pair of high-resolution (0.075m) aerial images with a size of 32507x15354, taken in 2012 and 2016. The dataset contains 12796 buildings. We crop the images into small blocks of size 256x256 without overlap, and randomly divide them into 6096 / 762 / 762 parts for training / validation / testing.

[0095] 3) CDD dataset: The CDD dataset consists of 11 pairs of original images, 7 pairs of 4725x2200 pixel seasonal change images and 4 pairs of 1900x1000 pixel images. The resolution varies from 3 centimeters to 100 centimeters per pixel, and the season changes greatly in the double-time images. The original data is processed to generate a training set of size 10000 and a test and validation set of size 3000 by cutting and rotating, respectively.

[0096] 4) SYSU-CD dataset: SYSU-CD dataset contains 20,000 pairs of 256x256, 0.5m resolution aerial images taken from 2007 to 2014, mainly from the construction and maintenance of Hong Kong's ports, sea routes, marine and coastal engineering, and shipping hubs in the international and Asia-Pacific regions. Due to the influence of bias and shadow, high-rise buildings are difficult to mark in high-resolution images, this dataset greatly supplements the change instances of high-rise buildings, and compared with the CDD dataset, it has more port-related change information. In our experiments, the samples for training / validation / testing are divided into 12000:4000:4000.

[0097] In the experiment, the present application is compared with UNet, UNet++, DeepLabV3, DASNet, SNUNet five benchmark models one by one, and the experimental results in table 1 show that DRUNet obtains the best change detection results on different scale building size, different change type change detection datasets, which fully proves the effectiveness of our cycle structure, and also shows the advantages of our ReUNet and DRUNet in restoring the spatial information of high-level feature loss and fusing context features. At the same time, in order to verify the search effect of the search algorithm on DRUNet, we make statistics on the parameter quantity and computational quantity of the benchmark model and the searched model, and the experimental results in table 2 show that the DRUNet of stage 1 and stage 2 searched by us has the optimal trade-off, and its performance is equivalent to the original DRUNet, and the complexity is greatly reduced, which shows that they have enough feature representation ability in complex remote sensing scenes. In order to verify the effectiveness of GHM loss in solving the imbalance problem of remote sensing image change detection task, we carry out comparative experiments with CE loss based on DRUNet on LEVIR-CD, WHU-CD, CDD and SYSU-CD datasets. Figure 5 is the improvement of the model trained by GHM loss. We can conclude that the proposed GHM loss can improve the performance of model change detection on all datasets.

[0098] Table 1 Quantitative performance comparison of LEVIR-CD, WHU, CDD and SYSU-CD datasets. The optimal result is represented in bold, and the suboptimal result is represented in italic and underline

[0099]

[0100] Table 2 Performance and complexity of LEVIR-CD. The optimal result is represented in bold, and the suboptimal result is represented in italic and underline

[0101]

[0102] In summary, the embodiment provides a remote sensing image change detection method based on a cyclic UNet network. In order to integrate multi-scale features and multi-level semantic context information, a new dense cyclic UNet (DRUNet) is designed. The network transmits the features of the decoder to the encoder through backward skip connection, and recursively between the encoding layer and the decoding layer, thereby achieving better feature refinement. Moreover, the two-stage progressive search algorithm is applied to the DRUNet for architecture search, and the simplest and most efficient change detection model is obtained. The phased DRUNet achieves a competitive accuracy and efficiency trade-off. In addition, in view of the class imbalance problem caused by the small number of changed pixels in remote sensing images in the change detection task, the GHM loss is used to prove the effectiveness of balancing gradient contribution in solving the sample imbalance problem. The DRUNet trained by the gradient coordination mechanism (GHM) loss achieves good accuracy. Through the above design, the method can improve the accuracy of the existing remote sensing image change detection model and reduce the model complexity, and improve the performance of the remote sensing image change detection model from the perspective of recycling features.

[0103] Second embodiment

[0104] The embodiment provides a remote sensing image change detection system based on a cyclic UNet network, comprising:

[0105] A DRUNet network construction module is configured to construct a dense and cyclic UNet network DRUNet based on a UNet network. The DRUNet is constructed into a pair of cyclic units, and parameters and encoders and decoders are reused by using forward and backward skip connections. The forward skip connection transmits the encoded features to the decoder, and the backward skip connection transmits the decoded high-level semantic features from the decoder to the encoder. Recursive operations are continuously performed between the encoding layer and the decoding layer to achieve better feature refinement.

[0106] A network architecture search module is configured to perform network architecture search on the DRUNet network constructed by the DRUNet network construction module, search for an optimal substructure, and obtain a change detection model.

[0107] A change detection model training module is configured to train the change detection model obtained by the network architecture search module, and obtain a trained change detection model.

[0108] A change detection module is configured to perform remote sensing image change detection by using the change detection model trained by the change detection model training module, and obtain a change detection result.

[0109] The remote sensing image change detection system based on the cyclic UNet network of the embodiment corresponds to the remote sensing image change detection method based on the cyclic UNet network of the above embodiment; wherein the functions realized by each functional module in the remote sensing image change detection system based on the cyclic UNet network correspond one by one to each process step in the remote sensing image change detection method based on the cyclic UNet network; therefore, it is not repeated here.

[0110] Third embodiment

[0111] The electronic device provided in the embodiment includes a processor and a memory; wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment.

[0112] The electronic device can have great differences due to different configurations or performances, and can include one or more processors (central processing units, CPUs) and one or more memories, wherein the memory stores at least one instruction, which is loaded and executed by the processor to implement the above method.

[0113] Fourth embodiment

[0114] The embodiment provides a computer readable storage medium, which stores at least one instruction, which is loaded and executed by a processor to implement the method of the above first embodiment. The computer readable storage medium can be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein can be loaded and executed by the processor in the terminal to implement the above method.

[0115] In addition, it should be noted that the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code.

[0116] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a machine that implements the functions described in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks.

[0117] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart or multiple flows and / or blocks. Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks. These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or multiple flows and / or blocks. Figure 1 one or more processes and / or blocks Figure 1 an apparatus for performing the functions specified in the flowchart or multiple flows and / or blocks. These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart or multiple flows and / or blocks.

[0118] It is also noted that the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0119] Finally, it should be noted that the above-mentioned preferred embodiments illustrate rather than limit the application, since various modifications and improvements can be suggested to one skilled in the art and it is intended that the scope of the application claimed should include all such modifications and improvements when they fall within the scope of the principles and new concepts of the application. Accordingly, the text is to be regarded as illustrative only and not restrictive, since modifications will be obvious to those skilled in the art. Therefore, the scope of the application is indicated not by the foregoing description but by the claims that follow.

Claims

1. A remote sensing image change detection method based on a cyclic UNet network, characterized in that: include: Based on the UNet network, a densely connected and recurrent UNet network DRUNet is constructed. DRUNet builds paired recurrent units and reuses parameters and encoder-decoders using forward and backward skip connections. The forward skip connection passes the encoded features to the decoder, and the backward skip connection passes the decoded high-level semantic features from the decoder to the encoder. Recursion is continuously performed between the encoding and decoding layers to achieve better feature refinement. Perform network architecture search on the DRUNet network, search for the optimal substructure, and obtain the change detection model; Training the change detection model to obtain a trained change detection model; Use the trained change detection model to perform remote sensing image change detection and obtain change detection results; The network architecture search is performed on the DRUNet network to search for the optimal substructure and obtain a change detection model, including: A two-stage search algorithm, K-continuous evolutionary algorithm, is used to find sparsely connected sub-architectures of DRUNet. In the first stage, k skip connections are selected to narrow the search space. In the second stage, continuous evolutionary algorithm is introduced to gradually discover the optimal sub-structure, thus obtaining a change detection model without redundant connections or blocks. An extraction block in DRUNet contains L layers and C skip connections; Use the K-continuous evolutionary algorithm to find the sparsely connected sub-architecture of DRUNet, including: Compressing a learnable selection matrix to model the mapping between L input skip connections and k selected skip connections, and express Φ(.) as a fully differentiable equation: Φ(C,M)=Matmul(C,Gumbel_SoftMax(M)) Where C={C1,…,C L } represents the complete set of skip connections in the extraction block, Matmul represents matrix multiplication, and Gumbel_SoftMax forces each of the k columns of the selection matrix M to be a one-hot vector, which votes for one of the L input skip connections; the above formula generates k repeated selection skip connections to achieve dynamic selection of candidate skip connections and reduce the search space for the subsequent search for the optimal substructure; After implementing dynamic selection of candidate skip connections and narrowing the search space, a continuous evolutionary algorithm is used to gradually determine the current optimal substructure in subsequent iterations.

2. The remote sensing image change detection method based on the cyclic UNet network according to claim 1, characterized in that: DRUNet is a recurrent iterative network composed of multiple nested paired recurrent units at different levels; a paired recurrent unit contains a pair of encoders and decoders, and has forward jump connections and backward jump connections on the same layer and different layers; wherein the forward jump connection connects the retrieved features with the features of the encoder from the same layer and different layers, and the backward jump connection connects the retrieved features with the features of the decoder from the same layer and different layers.

3. The remote sensing image change detection method based on the cyclic UNet network according to claim 2, characterized in that: In DRUNet, the size of the feature map is changed by using maximum pooling or interpolation with different parameters; and the feature splicing scheme is optimized to element-wise averaging; the previous encoding / decoding features of all layers are densely connected to each next decoding / encoding layer through forward and backward bidirectional skip connections to fuse multi-scale features.

4. The remote sensing image change detection method based on a cyclic UNet network according to claim 1, characterized in that: Training the change detection model to obtain a trained change detection model includes: A balanced gradient contribution loss function is constructed, and a gradient coordination mechanism is adopted to train the change detection model.

5. The remote sensing image change detection method based on the cyclic UNet network according to claim 4 is characterized in that: The expression of the balanced gradient contribution loss function is as follows: Among them, L GHM (p,p * ) represents the balanced gradient contribution loss function, L CE (p i ,p i * ) is the cross entropy loss, GD(g i ) indicates g i The number of pixels in the unit length area centered, g i is the gradient modulus of the i-th pixel, L CE represents the cross entropy loss, p = sigmoid(x) is the probability predicted by the model, x is the output of the model, p * is the corresponding label, and N represents the total number of pixels.

6. A remote sensing image change detection system based on a cyclic UNet network, characterized in that: include: The DRUNet network building module is used to build a densely connected and recurrent UNet network DRUNet based on the UNet network. Among them, DRUNet builds paired recurrent units and reuses parameters and encoders and decoders using forward and backward skip connections. The forward skip connection passes the encoded features to the decoder, and the backward skip connection passes the decoded high-level semantic features from the decoder to the encoder. Recursion is continuously performed between the encoding and decoding layers to achieve better feature refinement. A network architecture search module is used to perform a network architecture search on the DRUNet network constructed by the DRUNet network construction module, search for the optimal substructure, and obtain a change detection model; a change detection model training module, configured to train the change detection model obtained by the network architecture search module to obtain a trained change detection model; a change detection module, configured to perform remote sensing image change detection using the change detection model trained by the change detection model training module to obtain a change detection result; Performing a network architecture search on the DRUNet network constructed by the DRUNet network building module to search for the optimal substructure and obtain a change detection model, including: A two-stage search algorithm, K-continuous evolutionary algorithm, is used to find sparsely connected sub-architectures of DRUNet. In the first stage, k skip connections are selected to narrow the search space. In the second stage, continuous evolutionary algorithm is introduced to gradually discover the optimal sub-structure, thus obtaining a change detection model without redundant connections or blocks. An extraction block in DRUNet contains L layers and C skip connections; Use the K-continuous evolutionary algorithm to find the sparsely connected sub-architecture of DRUNet, including: Compressing a learnable selection matrix to model the mapping between L input skip connections and k selected skip connections, and express Φ(.) as a fully differentiable equation: Φ(C,M)=Matmul(C,Gumbel_SoftMax(M)) Where C={C1,…,C L } represents the complete set of skip connections in the extraction block, Matmul represents matrix multiplication, and Gumbel_SoftMax forces each of the k columns of the selection matrix M to be a one-hot vector, which votes for one of the L input skip connections; the above formula generates k repeated selection skip connections to achieve dynamic selection of candidate skip connections and reduce the search space for the subsequent search for the optimal substructure; After implementing dynamic selection of candidate skip connections and narrowing the search space, a continuous evolutionary algorithm is used to gradually determine the current optimal substructure in subsequent iterations.

Images