Deep neural network change detection method suitable for dual-temporal remote sensing image

Abstract

The invention discloses a deep neural network change detection method suitable for a dual-temporal remote sensing image. The method mainly comprises the following steps: constructing a pyramid space-time attention module introducing coordinate attention in a feature extraction layer; constructing an upper sampling layer according to the pyramid space-time attention module obtained in the previous step; according to the pyramid space-time attention module obtained in the first step and the upper sampling layer obtained in the second step, a network output layer is constructed, and a change detection deep neural network model suitable for the dual-temporal remote sensing image is obtained; using the training data set, and introducing a deep supervision mechanism to design a loss function combination to train the deep neural network model constructed in the third step; and inputting the dual-time-phase remote sensing image into the deep neural network model trained in the step 4 to obtain a change detection feature image. The deep neural network change detection method for the dual-temporal remote sensing image has a good effect in change detection of the remote sensing image change, and is an effective deep neural network change detection method for the dual-temporal remote sensing image.

Description

Technical Field

[0001] The present invention relates to image change detection technology, and in particular to a deep neural network change detection method suitable for dual-temporal remote sensing images, belonging to the field of change detection. Background Art

[0002] Remote sensing change detection is the process of identifying significant differences between multi-temporal remote sensing images. Significant differences usually depend on specific applications, such as urbanization monitoring, disaster assessment, and environmental monitoring. In recent years, automated change detection technology has promoted the development of remote sensing applications and has received widespread attention in recent years.

[0003] With the development of deep learning, many network models for change detection have been proposed, and change detection models for remote sensing images have also emerged in an endless stream. Due to its end-to-end network structure, deep learning directly outputs the corresponding feature map at the output end after the input dual-phase image is extracted, which not only simplifies the complexity of the change detection task, but also greatly improves the detection accuracy. Chao-Peng Chen et al. proposed a new network SARAS-Net with some mechanisms, using a relationship perception module before feature subtraction, and a ratio perception module and a cross transformer module after subtraction. Finally, the cross Transformer module integrates multi-level features, pays more attention to spatial information, and is easy to separate foreground and background, thereby reducing false positives. Experiments show that SARAS-Net has achieved good results on different data sets. The BIT network proposed by HaoChen et al. is a Transformer-based method that uses a Transformer encoder-decoder network to enhance the contextual information of ConvNet features through semantic labeling, and then performs feature difference to obtain a change map. Zhuo Zheng et al. proposed a model of single temporal supervised learning (STAR) to train high-precision change detectors using unpaired labeled images and generalize them to real-world bi-temporal images. In addition, to evaluate the effectiveness of STAR, a simple and effective change detector ChangeStar was designed, which can reuse any deep semantic segmentation architecture through the ChangeMixin module. Comprehensive experimental results show that ChangeStar outperforms the baseline by a large margin under single temporal supervision and achieves excellent performance under dual temporal supervision.

[0004] STANet is a network proposed by Hao Chen et al. that integrates a new change detection self-attention module. STANet processes feature maps in time and space scales, calculates attention weights for different regions, and ultimately achieves good results. However, in the final processing of feature maps, STANet takes the Euclidean distance of the dual-time distance feature map and then performs linear interpolation, which will result in blurred gaps in the changed areas of the image and a high false alarm rate. Although the location of the changed pixels is identified, the detection accuracy is lost. Although the design of PAM calculates attention weights for different areas of the image, the computational overhead is large. In addition, the feature maps before PAM are not better utilized. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a deep neural network change detection method applicable to dual-phase remote sensing images to solve the problems existing in the above-mentioned prior art.

[0006] The present invention proposes a deep neural network change detection method applicable to dual-temporal remote sensing images, which mainly includes the following steps:

[0007] (1) In the feature extraction layer, a pyramid spatiotemporal attention module (PCA) is constructed that introduces coordinate attention (CA);

[0008] (2) constructing an upsampling layer based on the feature extraction layer obtained in step (1);

[0009] (3) constructing a network output layer according to the feature extraction layer obtained in step (1) and the upsampling layer obtained in step (2), thereby obtaining a deep neural network change detection method suitable for dual-temporal remote sensing images;

[0010] (4) Using the training data set, introduce a deep supervision mechanism to design a loss function combination to train the deep neural network model constructed in step 3;

[0011] (5) The dual-phase remote sensing image is input into the deep neural network model trained in step 4 to obtain the change detection feature image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is the coordinate attention module.

[0013] Figure 2 It is an upsampling module.

[0014] Figure 3 It is the deep neural network model proposed in the present invention.

[0015] Figure 4 This is a diagram showing the effect of the STANet method on remote sensing image change detection.

[0016] Figure 5 It is an effect diagram of remote sensing image detection according to the present invention.

[0017] Figure 6 is the real label map. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings:

[0019] A deep neural network change detection method suitable for dual-temporal remote sensing images can be divided into the following steps:

[0020] (1) In the feature extraction layer, a pyramid spatiotemporal attention module (PCA) is constructed that introduces coordinate attention (CA);

[0021] (2) constructing an upsampling layer based on the feature extraction layer obtained in step (1);

[0022] (3) constructing a network output layer according to the feature extraction layer obtained in step (1) and the upsampling layer obtained in step (2), thereby obtaining a deep neural network change detection method suitable for dual-temporal remote sensing images;

[0023] (4) Using the training data set, introduce a deep supervision mechanism to design a loss function combination to train the deep neural network model constructed in step 3;

[0024] (5) The dual-phase remote sensing image is input into the deep neural network model trained in step 4 to obtain the change detection feature image.

[0025] Specifically, if Figure 1 As shown in FIG. 1 , in step (1), after the ResNet network extracts the features, the feature map is input into the spatiotemporal attention module. Among them, CA extracts two feature maps with directional features in the vertical and horizontal directions from the feature map of the image through two one-dimensional global pooling layers. Afterwards, the two directional image features are respectively encoded through a convolutional layer to obtain two attention features in different directions. The two attention features are the attention feature maps generated by capturing the long-term dependency of the image features along the two directions.

[0026] The global pooling of image features by the pooling layer is generally used to encode global spatial information, but this will lose some spatial information in the features. Therefore, the CA attention module uses precise spatial position information to capture long-term dependencies in image features and decomposes global pooling into two one-dimensional encoding operations. Given the input image features, CA uses pooling kernels (H, 1) and (1, W) in two different spatial ranges to channel encode the feature map in the vertical and horizontal directions. Therefore, the output of c channels and height h is as shown in Formula 1:

[0027]

[0028] Similarly, the output of the cth channel at width w is as shown in Formula 2:

[0029]

[0030] The above two formulas encode features in the vertical and horizontal directions respectively, aggregate image features, and thus generate a pair of directional attention feature maps. In addition, the above two transformation formulas enable the CA module to effectively capture the long-term dependency in one direction and retain the precise spatial coordinate information in this direction, so that CA can more accurately and effectively calculate the area of ​​interest in the image features.

[0031] Formula 1 and Formula 2 extract global spatial information, increase the receptive field and obtain more accurate spatial coordinate encoding information. After that, CA performs coordinate attention generation to effectively utilize the obtained encoding information. After calculation by Formula 1 and Formula 2, the obtained feature map is input into the shared 1×1 convolution layer, that is, the F1 transformation function, and Formula 3 is obtained:

[0032] f=δ(F1([z h , z w ])) (3)

[0033] Among them, [·,·] means that the two parts of feature z h and z w Connect along the spatial dimension, δ is the Sigmoid nonlinear activation function, and the generated It is the spatial encoding feature in both vertical and horizontal directions. Among them, r is used to implement SE

[11] The control of the block size reduction rate in the module. Afterwards, the obtained features are decomposed along the spatial dimension to obtain two independent tensors f h and f w . It is fed into two more 1×1 convolutional layers F h and F w , these two convolutional layers are used to transform f h and f w Converted to features with the same number of channels as the input features, as shown in Formula 4 and Formula 5:

[0034] g h =σ(F h (f h )) (4)

[0035] g w =σ(F w (f w )) (5)

[0036] Among them, δ is still the Sigmoid nonlinear activation function. The adjustment of the reduction rate r can control the number of channels of f, which is conducive to reducing the computational overhead of the model. The output feature g h and g w This is the required attention score. Finally, the output y of the coordinate attention mechanism CA is c It can be expressed as formula 6:

[0037]

[0038] Figure 3 The architecture of PCA is given in . Given a bi-temporal feature map X (1) , X (2) ∈R C×H×W , before entering the PCA module, the feature map is interpolated to generate X (3) ∈R C×H×W , and then stack the three feature maps into a feature tensor X∈R C×H×W×3 Then there are four parallel branches; each branch divides the feature tensor into s×s subregions equally, where s∈S, S∈{1, 2, 4, 8}, defining four pyramid scales. In the branch of scale s, each region is defined as Four CAs are used for each of these four branches. In each pyramid branch, all sub-regions R s，i，j Apply CA to generate the updated residual feature tensor Y s ∈R C×H×W×3 Then, these feature tensors Y s (s∈S) and input them into the convolutional layer to generate the final feature tensor Y∈R C×H×W×3 Finally, we add Y to X to get a new tensor Z∈R C ×H×W×3 .

[0039] like Figure 2 In step (2), in order to improve the accuracy of change detection, an upsampling module is constructed. Each convolution block contains two convolution layers, where Cov i , i∈(1,3,5), are two convolution layers with a kernel of 3×3, a stride of 1, and a padding of 1, followed by batch normalization (BatchNorm2d) and ReLU activation function. i , i∈(1,3,5) will be added to the input feature map and then input into the next Cov i, i∈(2, 4, 6), which represents a convolutional layer with a kernel size of 1×1, a stride of 1, and a padding of 1, followed by batch normalization (BatchNorm2d) and ReLU activation function. The three convolutional blocks extract distance feature maps of different scales and channels respectively, and then stack them up and input them into Cov7, which is a convolutional layer with a kernel size of 1×1 and a stride of 1. Finally, after the upsampling module, a distance feature map of the same size as the input image is output.

[0040] like Figure 3 As shown, according to the feature extraction layer obtained in step (1) and the upsampling layer obtained in step (2), a network output layer is constructed to obtain a deep neural network change detection method suitable for dual-temporal remote sensing images.

[0041] like Figure 3 As shown in the figure, during the training process, two branches are performed after the feature extraction layer for remote sensing image change detection. The number of changed samples and unchanged samples varies greatly. In many cases, the changed pixels only account for a small part of all pixels, which will bring some deviations to the network during the training stage. In order to reduce the impact of class imbalance, batch balanced contrast loss (BCL) is used and a deep supervision mechanism is added. The calculation formula of the loss function is as follows:

[0042]

[0043]

[0044]

[0045] loss=loss_1+α(1-epoch / n_epoch)×loss_2 (10)

[0046] Among them, D * Represents the Euclidean distance feature map between two feature maps, M * It is a binary mapping of the true label, b represents the training set, i and j represent the height and width. u 、n c They represent the number of unchanged pixel pairs and the number of changed pixel pairs respectively.

[0047] like Figure 3 As shown in the figure, loss_1 indicates that the features obtained by the feature extraction layer are directly input into the upsampling to obtain the change detection features (Loss1 in the figure) without passing through the pyramid spatiotemporal attention module, and then the loss value is calculated using formula 9. loss_2 indicates the loss value calculated by the change detection features of the entire network (Loss2 in the figure), epoch indicates the number of iterations currently reached, n_epoch indicates the total number of iterations of training, and α is the weight parameter for controlling attenuation.

[0048] In the step (5), the dual-phase remote sensing image is input into the deep neural network model trained in the step (4) to obtain a change feature map.

[0049] The present invention proposes a deep neural network extraction method suitable for core image particles, which uses a spatiotemporal attention module to obtain spatiotemporal attention features and an upsampling module to improve detection accuracy. It accelerates the convergence of the network through a deep supervision mechanism, effectively improving the accuracy of remote sensing image change detection. Figure 4-5 The STANet method and the effect diagram of the invention on remote sensing image change detection are given. Figure 6 is the real label map.

[0050] At the same time, the present invention selects precision (Precision, Pr), recall (Recall, Re) and F1 to evaluate the training effect of the algorithm of the present invention on the remote sensing image data set. And compared with the STANet method, the experimental results are shown in Table 1. It can be seen that the present invention has improved all indicators of remote sensing image change detection compared with the STANet method. Among them, the improvement of the F1 score is more obvious, because the present invention greatly improves the accuracy of remote sensing image change detection and effectively reduces the false alarm rate.

[0051] Table 1 Comparative experiment

[0052]

[0053] In summary, compared with the STANet method, the present invention has obvious advantages in the effect of remote sensing image change detection and objective evaluation. Therefore, the present invention is an effective dual-temporal remote sensing image change detection method.

[0054] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A deep neural network change detection method for dual-temporal remote sensing images, characterized in that The following steps are involved: Step 1: In the feature extraction layer, construct a pyramid spatiotemporal attention module (PCA) that introduces coordinate attention (CA), as shown in the following formula: f=δ(F1([z h ,z w ])) (3) g h =σ(F h (f h )) (4) g w =σ(F w (f w )) (5) in, represents the output of the cth channel with height h, represents the output of the cth channel at width w, and [·,·] represents the combination of the two features z h and z w Connect along the spatial dimension, δ is the Sigmoid nonlinear activation function, and the generated is the spatial encoding feature in both vertical and horizontal directions, r is the reduction rate, and f h and f w The obtained features are decomposed along the spatial dimension to obtain two independent tensors, F h and F w represents two 1×1 convolutional layers, σ is still the Sigmoid nonlinear activation function, g h and g w That is the required attention score. Specifically, given the dual-time feature map X (1) , X (2) ∈R C×H×W , before entering the PCA module, the feature map is interpolated to generate X (3) ∈R C×H×W , and then stack the three feature maps into a feature tensor X∈R C×H×W×3 , then there are four parallel branches; each branch divides the feature tensor into s×s sub-regions equally, where s∈S, S∈{1, 2, 4, 8}, and four pyramid scales are defined. In the branch of scale s, each region is defined as Four CAs are used for each of the four branches. In each pyramid branch, all sub-regions R s，i，j Apply CA to generate the updated residual feature tensor Y S ∈R C×H×W×3 , then, these feature tensors Y s (s∈S) and input them into the convolutional layer to generate the final feature tensor Y∈R C×H×W×3 Finally, we add Y to X to get a new tensor Z∈R C×H×W×3 ; Step 2: Construct an upsampling layer based on the pyramid spatiotemporal attention module obtained in step 1; Step 3: Based on the pyramid spatiotemporal attention module obtained in step 1 and the upsampling layer obtained in step 2, a network output layer is constructed to obtain a deep neural network model suitable for change detection of dual-temporal remote sensing images; Step 4: Use the training data set and introduce a deep supervision mechanism to design a loss function combination to train the deep neural network model constructed in step 3. The calculation formula of the loss function is as follows: loss=loss_1+α(1-epoch / n_epoch)×loss_2 (10) Among them, D * Represents the Euclidean distance feature map between two feature maps, M * is a binary mapping of the true label, b represents the number of training sets, i and j represent height and width, and n u 、n u They represent the number of unchanged pixel pairs and the number of changed pixel pairs respectively. loss_1 represents the loss value calculated by using formula 9 after directly inputting the features obtained by the feature extraction layer into the upsampling to obtain the change detection features without passing through the pyramid spatiotemporal attention module. loss_2 represents the loss value calculated by the change detection features of the entire network. epoch represents the number of iterations currently reached, n_epoch represents the total number of iterations of training, and α is the weight parameter for controlling attenuation. Step 5: The dual-phase remote sensing image is input into the deep neural network model trained in step 4 to obtain the change detection feature image.

2. A deep neural network change detection method for dual-temporal remote sensing images according to claim 1, characterized in that In step 1, a coordinate attention module is added to calculate the spatiotemporal attention scores of the feature map after the feature extraction layer.

3. The deep neural network change detection method applicable to dual-temporal remote sensing images according to claim 1, characterized in that The upsampling layer constructed in step 2 upsamples the obtained bi-phase feature map using the residual network to obtain an accurate change detection feature map.

4. The deep neural network change detection method applicable to dual-temporal remote sensing images according to claim 1, characterized in that The deep neural network model proposed in step 3, in order to improve its change detection accuracy, increases the weight of the change area through the spatiotemporal attention module, deep supervision mechanism and upsampling operation, and finally obtains an accurate change detection map.

5. The deep neural network change detection method applicable to dual-temporal remote sensing images according to claim 1, characterized in that The loss function combination designed in step 4 uses a deep supervision mechanism to solve the problems of slow convergence and gradient vanishing in deep neural networks. A branching strategy is added to the hidden layer of the deep neural network to supervise the entire network and play a regularization role, promoting faster convergence of the network.

Images