Remote sensing image change detection method based on multi-scale feature complementation and verification attention network

Abstract

The application discloses a remote sensing image change detection method based on multi-scale feature complementation and verification attention network, comprising the following steps: 1. inputting a double-time RGB image pair of the same region at different times for feature extraction; 2. inputting the double-time feature pair of each stage into a decoder, wherein the decoder comprises a multi-scale feature complementation module and a subtraction operation on the complementary feature pair; 3. performing progressive cross-scale feature fusion through a field of view expansion module and a verification attention module, and obtaining an optimal remote sensing image change detection network through network training and optimization, which is used for high-precision change detection on any double-time remote sensing image. The application can weaken the influence of redundant information between features at multiple scales, and obtain a change range image with better detection effect through progressive fusion, so as to better mine the feature information of the change and improve the change area detection effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of remote sensing image change detection, and particularly relates to a remote sensing image change detection method based on multi-scale feature complementation and verification attention network. BACKGROUND

[0002] The purpose of remote sensing change detection (RSCD) is to locate and identify the surface changed regions of the common area in different time remote sensing image data. It is an important problem in the remote sensing perception understanding task and a key step for many real tasks such as resource monitoring, land change detection, damage assessment and urban construction management. Therefore, how to accurately find the changed region of the area is particularly important.

[0003] There are a large number of research works in the field of remote sensing image change detection. The initial RSCD method mainly uses hand-crafted features to obtain the change result. However, the hand-designed features have poor robustness and insufficient semantic information, and are only suitable for low or medium resolution RSCD tasks. With the development of sensor technology, high resolution (HR) and very high resolution (VHR) remote sensing images have become more and more common in many practical applications. For HR and VHR remote sensing images, the texture of the ground object is more complex and rich, which makes it more challenging to identify changes.

[0004] In recent years, deep convolutional neural networks (CNN) have achieved excellent results in various computer vision tasks due to their strong feature learning ability. However, although CNN can achieve high detection accuracy in the detection task, the cost is usually high computational cost and large model size. These methods contain a large number of model parameters and huge computational amount, which leads to an unbearable computational overhead on devices such as unmanned aerial vehicles.

[0005] The proposal of lightweight change detection network is a major breakthrough in the field of change detection, especially in the RSCD task with limited energy consumption and computing power. However, the existing lightweight detection network reduces the detection accuracy of the network while reducing the computational overhead of the model. The possible reason is that these models reduce the depth of the network too much in order to be lightweight, without considering the impact of the number of network layers on the detection accuracy. For example, FC-diff, FC-Ef and FC-cat, although they achieve the effect of lightweight in remote sensing change detection, they cannot guarantee the detection accuracy and other objective evaluation indicators of the model. SUMMARY

[0006] This invention addresses the shortcomings of existing technologies by proposing a remote sensing image change detection method based on multi-scale feature complementarity and a verification attention network. The method aims to reduce the influence of redundant information between features through multi-scale feature subtraction and to obtain a change detection model with better detection performance and lower energy consumption by progressively fusing multi-scale features. This will better serve low-power detection tools to better mine feature information of changes and improve the detection effect of changed areas.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The remote sensing image change detection method based on multi-scale feature complementarity and verification attention network of the present invention is characterized by the following steps:

[0009] Step 1: Acquire two remote sensing RGB images pic1 and pic2 from the same region at different times to form a dual-temporal remote sensing RGB image pair pic; and use the difference between the true masks of pic1 and pic2 as the true change mask of pic, denoted as . ;

[0010] Step 2: Construct a remote sensing image change detection network, including an encoder and a decoder;

[0011] Step 3: The encoder includes a multi-scale feature extraction module and a multi-scale feature complementation module;

[0012] Step 3.1: The multi-scale feature extraction module extracts features from the image pair pic to obtain M scale feature pairs {L1, L2, ..., L...}. i ,…,L M}, where L i Represents the i-th scale feature pair;

[0013] Step 3.2: The multi-scale feature complementarity module is used for L i Processing is performed to obtain the i-th complementary feature pair P. i Thus, the i-th change feature is obtained according to equation (2). :

[0014] (2)

[0015] In equation (2), This represents the i-th complementary feature corresponding to the first remote sensing RGB image pic1. This represents the i-th complementary feature corresponding to the second remote sensing RGB image pic2;

[0016] Step 4: The decoder uses equation (4) to... Decode the i-th decoded feature. , thereby obtaining the i-th decoding feature After convolution and up-sampling operations, the i-th mask is obtained

[0017] c i = V { F [ C o n v ( d i + U p ( c i + 1 ) ) ] } , i = 1 , 2 , ⋯ , M (4)

[0018] In formula (4), Up represents a bilinear up-sampling operation, F represents a field of expansion function, V represents a verification attention function, Conv represents a convolution operation, represents the i+1-th decoding feature;

[0019] Step 5, constructing a loss function of the remote sensing image change detection network:

[0020] Step 5.1, constructing the i-th binary cross-entropy loss using formula (7) :

[0021] L B C E i = − [ m T × l o g m i + ( 1 − m T ) × l o g ( 1 − m i ) ] (7)

[0022] Step 5.2, constructing the i-th dice loss using formula (8) :

[0023] (8)

[0024] Step 5.3, constructing the total loss L using formula (9)

[0025] (9)

[0026] Step 6, using the Adam optimizer to train the remote sensing image change detection network, while the real mask supervises the mask generated by the decoder and guides the generation of the mask of the next decoder.

[0027] Step 7, calculating the total loss L to update the network parameters, when the number of training iterations reaches the set number or the error of L is less than the set threshold, the training stops, thereby obtaining the optimal remote sensing image change detection model for detecting the change area in the remote sensing image.

[0028] The remote sensing image change detection method based on the multi-scale feature complementary and verification attention network has the characteristics that the multi-scale feature complementary module is composed of M layers of structures; wherein the i-th layer of structure comprises M-i pairs of multi-scale subtraction units;

[0029] The output feature pair of the j+1-th pair of multi-scale subtraction units on the i-th layer of structure is obtained using formula (3) , thereby obtaining the output feature pair of the i-th layer of structure​ Thus, the output feature pairs of the M-1 layer structure are obtained. ;

[0030] k i j + 1 = C o n v [ f i j − U p ( f i + 1 j ) ] , i = 1 , 2 , ⋯ , M ; j = 1 , ⋯ , M − i (3)

[0031] In equation (3), This indicates a bilinear upsampling operation. This represents the convolution operation. This represents the input feature pair of the (j+1)th multi-scale subtraction unit. Let represent the input feature pair of the j-th multi-scale subtraction unit. When j=1, let =L i ;

[0032] Will The output feature pair of the last multi-scale subtraction unit in the i-th layer structure The extracted feature is the i-th complementary feature pair P. i .

[0033] The decoder in step 4 includes: a field-of-view expansion module and a verification attention module;

[0034] Step 4.1: The field-of-view expansion module calculates the i-th fused feature. Thus, using equation (5) to Perform field-of-view enhancement to obtain the i-th dilated feature. :

[0035] e i = t i + C o n v d 1 { t i + C o n v d 2 [ t i + 1 + C o n v d 3 ( t i ) ] } , i = 1 , 2 , ⋯ , M (5)

[0036] In equation (5), , , These represent three different dilation rates of dilated convolutions;

[0037] Step 4.2: The verification attention module includes two branches, one of which focuses on the i-th dilated feature. After performing average pooling, a convolution operation is performed to obtain the i-th mean feature. Then to After upsampling, the input is processed by the Sigmoid function to obtain the i-th dilated feature. weight ;

[0038] Another branch for the i-th dilated feature After convolution processing, the features before the i-th verification are obtained. And use equation (6) to The re-verification is performed to obtain the i-th verified feature The i-th verified feature is obtained by performing the re-verification The i-th decoded feature is obtained by performing the convolution operation

[0039] (6)

[0040] In formula (6), Indicates an element-wise multiplication.

[0041] The electronic device comprises a memory and a processor, and the memory is used for storing a program supporting the processor to execute the remote sensing image change detection method, and the processor is configured to execute the program stored in the memory.

[0042] The computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the remote sensing image change detection method are executed.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] 1、The whole adopts multi-scale features to perform change detection, which can obtain more rich feature information, so that the semantic information and edge information of the picture can be better utilized, thereby improving the change area detection effect.

[0045] 2、The multi-scale feature extraction is increased in the present application, and the multi-scale feature complementary module is added to ablate the redundant information between the multi-scale features, so that the problem of information redundancy after the multi-scale features are fused is solved, and the model can accurately identify the change area in the image during detection.

[0046] 3、The present application designs a visual field expansion module after the multi-scale feature fusion, which uses a hollow convolution with different expansion rates to expand the receptive field of the fused feature, so that the model can better associate the context information during detection to ensure the detection accuracy of the model. In addition, a verification attention module is proposed, so that the fused feature is reweighted to meet various performance indicators. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is the overall structure diagram of the detection network of the present application;

[0048] Figure 2 is the multi-scale subtraction unit structure diagram in the multi-scale feature complementary module in the present application;

[0049] Figure 3 is the visual field expansion module structure diagram in the present application;​

[0050] Figure 4 This is a structural diagram of the verification attention module in this invention. Detailed Implementation

[0051] In this embodiment, a remote sensing image change detection method based on a multi-scale feature complementarity module and a verification attention network is described in the following process: Figure 1 As shown, it includes the following steps:

[0052] Step 1: Acquire two remote sensing RGB images pic1 and pic2 from the same region at different times to form a dual-temporal remote sensing RGB image pair pic; and use the difference between the true masks of pic1 and pic2 as the true change mask of pic, denoted as . .

[0053] Step 2: Construct a remote sensing image change detection network, including: an encoder and a decoder, such as... Figure 1 As shown;

[0054] Step 3: The encoder includes a multi-scale feature extraction module and a multi-scale feature complementation module;

[0055] Step 3.1: In this embodiment, the multi-scale feature extraction module uses the latest lightweight visual Transform to extract bitemporal feature pairs. Each feature extraction layer contains a convolutional layer with a stride of 2. Therefore, the feature map of each layer is downsampled to half the resolution of the previous layer. The multi-scale feature extraction module extracts from the image pair pic to obtain M multi-scale feature pairs {L1, L2, ..., L...}, each with 64 channels. i ,…,L M}, where L i This represents the i-th scale feature pair. The uniformity of the number of channels not only ensures the smooth operation of subsequent subtraction but also compresses features with a large number of channels, reducing the computational load of the model.

[0056] Step 3.2: In this embodiment, considering the information redundancy problem between features of different layers, which becomes particularly prominent when multi-level features are fused in the encoder part, this invention designs a multi-scale complementary module in the network encoder part to make features of adjacent layers complementary, so that it can better serve the decoder part of the network. The overall structure of the multi-scale complementary module is as follows: Figure 1 The multi-scale complementary module is shown in the diagram. The multi-scale feature complementary module is used for L... i Processing is performed to obtain the i-th complementary feature pair P. i Thus, the i-th change feature is obtained according to equation (2). :

[0057] (2)

[0058] In equation (2), This represents the i-th complementary feature corresponding to the first remote sensing RGB image pic1. This represents the i-th complementary feature corresponding to the second remote sensing RGB image pic2.

[0059] In specific implementation, the multi-scale feature complementarity module consists of an M-layer structure; wherein the i-th layer contains Mi pairs of multi-scale subtraction units; the structure of the multi-scale subtraction unit is as follows: Figure 2 As shown;

[0060] Using equation (3), we obtain the output feature pairs of the (j+1)th multi-scale subtraction units on the i-th layer structure. Thus, the output feature pairs of the i-th layer structure are obtained. Thus, the output feature pairs of the M-1 layer structure are obtained. ;

[0061] k i j + 1 = C o n v [ f i j − U p ( f i + 1 j ) ] , i = 1 , 2 , ⋯ , M ; j = 1 , ⋯ , M − i (3)

[0062] In equation (3), This indicates a bilinear upsampling operation. This represents the convolution operation. This represents the input feature pair of the (j+1)th multi-scale subtraction unit. Let represent the input feature pair of the j-th multi-scale subtraction unit. When j=1, let =L i ;

[0063] Will The output feature pair of the last multi-scale subtraction unit in the i-th layer structure The extracted feature is the i-th complementary feature pair P. i .

[0064] Step 4: The decoder includes a vision expansion module and a verification attention module. The vision expansion module uses dilated convolutions with different dilation rates and residual connections to increase the receptive field of the fused features. This is mainly to expand the receptive field of the fused features to enhance the network's ability to acquire contextual information. Its structure diagram is shown below. Figure 3 As shown. Since the decoder uses a feature pyramid network, it provides a classic architecture for multi-level feature fusion, fusing deep features to shallow features; however, deep features lack context guidance and are prone to generating a lot of noise, making multi-level feature fusion difficult; therefore, this invention designs a verification attention module to reweight verification features to better achieve multi-level feature fusion.

[0065] The overall working principle of the decoder is shown in equation (4). decoding, obtaining the ith decoding feature , thereby performing convolution and up-sampling operations on the ith decoding feature , obtaining the ith mask ;

[0066] c i = V { F [ C o n v ( d i + U p ( c i + 1 ) ) ] } , i = 1 , 2 , ⋯ , M (4)

[0067] In formula (4), Up represents a bilinear up-sampling operation, F represents a field of expansion function, V represents a verification attention function, Conv represents a convolution operation, represents the ith decoding feature.

[0068] Step 4.1, the field of expansion module calculates the ith fusion feature , thereby performing field enhancement on using formula (5) to obtain the ith expansion feature :

[0069] e i = t i + C o n v d 1 { t i + C o n v d 2 [ t i + 1 + C o n v d 3 ( t i ) ] } , i = 1 , 2 , ⋯ , M (5)

[0070] In formula (5), , , respectively represent three different expansion rates of the hole convolution;

[0071] Step 4.2, the structure of the verification attention module is shown in Figure 4 , which includes two branches; one branch performs an average pooling operation on the ith expansion feature , and then performs a convolution operation to obtain the ith mean feature , and then performs an up-sampling operation on and inputs it into a Sigmoid function for processing to obtain the weight of the ith expansion feature .

[0072] The other branch performs a convolution operation on the ith expansion feature to obtain the ith feature before verification , and uses formula (6) to re-verify to obtain the ith feature after verification , thereby performing a convolution operation on the ith feature after verification to obtain the ith decoding feature ;

[0073] (6)

[0074] In formula (6), represents element-wise multiplication.

[0075] Step 5, constructing the loss function of the remote sensing image change detection network:

[0076] Step 5.1, constructing the i-th binary cross-entropy loss using formula (7) :

[0077] L B C E i = − [ m T × l o g m i + ( 1 − m T ) × l o g ( 1 − m i ) ] , i = 1 , 2 , ⋯ , M (7)

[0078] Step 5.2, constructing the i-th dice loss using formula (8) :

[0079] (8)

[0080] Step 5.3, constructing the total loss L using formula (9)

[0081] (9)

[0082] Step 6, training the remote sensing image change detection network using the Adam optimizer, while the real mask supervises the mask produced by the decoder and guides the production of the mask of the next decoder.

[0083] Step 7, calculating the total loss L to update the network parameters, when the number of training iterations reaches the set number or the error of L is less than the set threshold, the training stops, thereby obtaining the optimal remote sensing image change detection model, which is used to detect the changed area in the remote sensing image.

[0084] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0085] In this embodiment, a computer readable storage medium stores a computer program on the computer readable storage medium, and the computer program is run by a processor to execute the steps of the above method.

Claims

1. A remote sensing image change detection method based on multi-scale feature complementarity and verification attention network, characterized in that, The method comprises the following steps: Step 1, obtaining two remote sensing RGB images pic1 and pic2 of the same region at different times to form a double-time remote sensing RGB image pair pic; and the difference between the true masks of pic1 and pic2 is taken as the true change mask of pic, denoted as ; Step 2, constructing a remote sensing image change detection network, comprising: an encoder and a decoder; Step 3, the encoder comprises: a multi-scale feature extraction module and a multi-scale feature complementary module; Step 3.1, the multi-scale feature extraction module extracts the image pair pic to obtain M scale feature pairs {L1, L2, …, L i ,…,L M}, wherein L i represents the i-th scale feature pair; Step 3.2, the multi-scale feature complementary module is used to process L i to obtain the i-th complementary feature pair P i , so as to obtain the i-th change feature according to formula (2) (2) in formula (2), denotes the i-th complementary feature corresponding to the first remote sensing RGB image pic1, denotes the i-th complementary feature corresponding to the second remote sensing RGB image pic2; Step 4, the decoder decodes the i-th decoded feature using formula (4) to obtain the i-th decoded feature , and then performs convolution and up-sampling operations on the i-th decoded feature to obtain the i-th mask . (4) In formula (4), Up represents a bilinear up-sampling operation, F represents a field of expansion function, V represents a verification attention function, Conv represents a convolution operation, represents the i+1th decoding feature; Step 5, constructing a loss function of the remote sensing image change detection network: Step 5.1, Constructing the ith binary cross-entropy loss with formula (7) : (7) Step 5.2, building the ith die loss with formula (8) : (8) Step 5.3, constructing a total loss L using formula (9): (9) Step 6, training the remote sensing image change detection network with Adam optimizer, while the real mask The supervised decoder generates a mask and guides the generation of the next decoder mask. Step 7, calculating the total loss L to update the network parameters, when the number of training iterations reaches a set number or the error of L is less than a set threshold, the training stops, thereby obtaining an optimal remote sensing image change detection model for detecting the changed area in the remote sensing image.

2. The remote sensing image change detection method based on multi-scale feature complementary and verification attention network according to claim 1, characterized in that, The multi-scale feature complementary module is composed of M layers of structures; wherein the i-th layer of structure contains M-i pairs of multi-scale subtraction units; The output feature pair of the j+1th pair of multi-scale subtraction units on the ith layer structure is obtained by using formula (3) , thereby obtaining the output feature pair of the ith layer structure , and further obtaining the output feature pair of the M-1th layer structure ; (3) in formula (3), denotes a bilinear up-sampling operation, denotes a convolution operation, denotes an input feature pair of the j+1-th multi-scale subtraction unit, denotes an input feature pair of the j-th multi-scale subtraction unit, when j = 1, let = L i ; Will The output feature pair of the last multi-scale subtraction unit in the i-th layer structure The extracted feature is the i-th complementary feature pair P. i . 3.The remote sensing image change detection method based on multi-scale feature complementary and verification attention network according to claim 1, characterized in that, The decoder in step 4 comprises: a field of view expansion module and a verification attention module; Step 4.1, the field expansion module calculates the i-th fused feature , thereby performing field enhancement on the i-th fused feature using equation (5) , to obtain the i-th expanded feature : (5) In formula (5), , , respectively represent three different dilated rates of the cavity convolution; Step 4.2: The verification attention module includes two branches, one of which focuses on the i-th dilated feature. After performing average pooling, a convolution operation is performed to obtain the i-th mean feature. Then to After upsampling, the input is processed by the Sigmoid function to obtain the i-th dilated feature. weight ; Another branch for the i-th dilated feature After convolution processing, the features before the i-th verification are obtained. And use equation (6) to Perform a re-verification to obtain the i-th verified feature. Thus, for the i-th verified feature After performing the convolution operation, the i-th decoded feature is obtained. ; (6) In formula (6), represents element-wise multiplication.

4. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the remote sensing image change detection method in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to execute the steps of the remote sensing image change detection method in any one of claims 1-3.

Images