Twin multi-scale depth fusion remote sensing image change detection method

Abstract

The invention relates to the technical field of change detection, and discloses a twinborn multi-scale depth fusion remote sensing image change detection method, which comprises the following steps that: input images are two remote sensing images T1 and T2 at the same place and in different time periods respectively, the lengths and widths of the input images are marked as H and W, C is the number of channels, and the size is H * W * C; the method comprises the following steps: respectively inputting T1 and T2, firstly entering an encoder layer, and respectively obtaining features E (0, 1) and E (0, 2) through a starting convolution module consisting of a 3 * 3 convolution layer, a batch normalization layer and a ReLU activation function; the twinborn multi-scale deep fusion remote sensing image change detection method aims to solve the current problems that feature extraction in a coding stage is imperfect, details are easy to lose in a decoding stage and the like, the detection precision is improved, and the problem of missing detection of small targets is also improved.

Description

Technical Field

[0001] The present invention relates to the field of change detection technology, and specifically to a twin multi-scale deep fusion remote sensing image change detection method. Background Art

[0002] Remote sensing image change detection is an efficient remote sensing data analysis technology. It identifies and quantifies changes in surface features by comparing images covering the same surface in different time periods. With the rapid advancement of remote sensing technology, especially the development of aerospace monitoring technology and the popularization of high-resolution imaging sensors, modern remote sensing image change detection can not only provide higher spatial and spectral resolution, but also process rich remote sensing data from multiple sources and different scales. These technological innovations have promoted the widespread application of remote sensing image change detection in many fields, such as urban change monitoring, land use and land cover change analysis, natural disaster assessment, and environmental monitoring. Since this method can effectively reduce the noise interference in traditional technologies and accurately identify the change area of ​​semantic information, remote sensing image change detection has become an important research direction in the field of remote sensing.

[0003] With the rapid rise of deep learning technology, this technology has also been widely used in the interpretation tasks of high-resolution remote sensing images. Compared with traditional change detection methods, change detection methods based on deep learning have significant advantages. First, deep learning methods do not rely on prior knowledge, but can automatically learn rich contextual features and complex high-level semantic features directly from large amounts of data. This enables deep learning models to more effectively process and parse high-resolution remote sensing images, and maintain high accuracy even in environments with complex scenes and variable landforms. Although the technology based on deep learning has made certain progress, many existing research methods still face the following problems: The existing Chinese patent document CN111047551B discloses a remote sensing image change detection method and system based on the improved U-net algorithm, which uses Resnet-101 in the encoder part to complete the feature extraction of the image, and adds ASPP void space pyramid pooling to the connection between the encoder and the decoder, realizing the role of multi-scale extraction of data context; Compared with the prior art, the present invention uses a single aggregation module in the encoder part to perform multi-scale feature fusion on the data, and introduces a lightweight multi-scale fusion module in the decoding stage so that the feature map can retain more details after decoding. The present invention makes up for the deficiencies in the prior art in multi-scale feature fusion and detail retention, improves the detection accuracy and also improves the problem of missed detection of small targets. Summary of the invention

[0004] The purpose of the present invention is to solve the current problems of imperfect feature extraction in the encoding stage and easy loss of details in the decoding stage, improve the detection accuracy and improve the problem of missed detection of small targets, and propose a twin multi-scale deep fusion remote sensing image change detection method.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A twin multi-scale deep fusion remote sensing image change detection method includes the following steps:

[0007] S1: The input images are two remote sensing images of the same location at different time periods, T 1 , T 2 , its length and width are recorded as H and W, C is the number of channels, and the size is H×W×C;

[0008] S2: T 1 , T 2 The inputs are first fed into the encoder layer, and then go through a startup convolution module consisting of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the features E(0, 1) and E(0, 2), and then go through the startup convolution module once more to obtain the features E(1, 1) and E(1, 2);

[0009] S3: Use a single aggregation module to perform multi-scale fusion on E(1, 1) and E(1, 2) to obtain E(2, 1) and E(2, 2) respectively. The two features E(2, 1) and E(2, 2) are further subjected to a single aggregation module to obtain features E(3, 1) and E(3, 2). Finally, E(3, 2) is subjected to a single aggregation module to obtain E(4, 2). After that, the encoding stage is completed.

[0010] S4: In order to enhance the change information and provide contextual comparison, the features of each level in S3 are spliced ​​together, and the spliced ​​results are represented by C m Indicates that m = 1, 2, 3, 4;

[0011] S5: The model uses a lightweight multi-scale fusion module to perform decoding operations, and uses existing information to generate more feature maps so that the upsampling process can retain as many details as possible. The feature maps are passed through a lightweight multi-scale fusion module through a main convolution and a cheap operation to obtain two feature maps. The main convolution consists of a 1×1 convolution, a batch normalization layer, and a ReLU activation function. This part has a large amount of calculation but can retain the main features of the input. The cheap operation consists of a deep convolution with a 3×3 convolution kernel size, a batch normalization layer, and a ReLU activation function. This part has a small amount of calculation, but it can still generate useful feature maps. Finally, the two feature maps are spliced ​​to obtain the final output feature map;

[0012] S6: For E(1,2) and C 1 The splicing operation is performed and then lightweight multi-scale fusion is performed to obtain D(0,0), E(2,2) and C 2 After splicing, lightweight multi-scale fusion is performed to obtain D(1,0), E(3,2) and C 3 After the splicing operation, lightweight multi-scale fusion is performed to obtain D(2,0), E(4,2) and C 4 After the splicing operation, lightweight multi-scale fusion is performed to obtain D(3, 0);

[0013] S7: D(0,0) and D(1,0) are concatenated after upsampling and then pass through the multi-scale attention module to obtain D(0,1). The multi-scale attention module can convert the input feature F∈R C×H×W Split into X independent subsets, that is, = [F 0 , F 1 , …, F x-1 ], F i ∈R C / / X×H×W , and then the grouped feature maps will be obtained through three parallel subnetworks respectively;

[0014] S8: D(2,0) and C 2 and D(1,0) and then pass through the lightweight multi-scale fusion module to D(1,1). D(0,1) passes through the multi-scale attention module and is combined with C 1 The feature maps D(0,2), D(3,0) and D(2,0) are concatenated with D(0,0) and D(1,1) to obtain the feature maps D(0,2), D(3,0) and D(2,0) and C 3 After the splicing operation, lightweight multi-scale fusion is performed to obtain the feature maps D(2, 1), D(2, 1) and D(1, 0), D(1, 1), C 2 After the splicing operation, a lightweight multi-scale fusion is performed to obtain the feature map D(1, 2). After g(0, 2) passes through the multi-scale attention module, it is combined with D(1, 2), D(0, 1), D(0, 0), C 1 After splicing, lightweight multi-scale fusion is performed to obtain the feature D(0, 3). Finally, D(0, 3) is output through the multi-scale attention module to output the prediction map of the corresponding scale, and then through the weighted negative log-likelihood function, and the network parameters are optimized during the model training process;

[0015] S9: The change detection task is actually a binary classification problem. The weighted negative log-likelihood loss function can specify different weights for each category through the weight parameter. In this case, the weights of both the unchanged and changed categories can be set to 0.5. This setting is very effective in solving the problem of sample imbalance. The specific formula is as follows;

[0016]

[0017] in is the weight of each category, N represents the size of the batch, and the specific formula of ln is as follows:

[0018]

[0019] in Represents the probability of the target.

[0020] Based on the above technical solution, the present invention can also be improved as follows.

[0021] Furthermore, the evaluation parameters include precision, recall, F1 score, and overall accuracy. The specific formula is as follows:

[0022]

[0023]

[0024]

[0025]

[0026] Among them, TP represents the number of samples that are actually positive and predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative and predicted to be negative.

[0027] Furthermore, the twin multi-scale deep fusion model includes a single aggregation module, a multi-scale attention module and a lightweight multi-scale fusion module.

[0028] Furthermore, the single aggregation module performs five 3×3 convolutions on the input feature map to output five feature maps respectively, and then splices the feature maps output by the five convolutions together through a splicing operation, and finally adjusts the number of channels through a 1×1 convolution. The multi-scale attention module as claimed in claim 1 efficiently processes multi-scale information on the obtained feature map while maintaining independence and correlation between channels. The lightweight multi-scale fusion module generates more feature maps while maintaining less calculation.

[0029] Compared with the prior art, the technical solution of this application has the following beneficial technical effects:

[0030] 1. The present invention uses multi-scale fusion and deep supervision mechanisms to capture details in remote sensing images more accurately. At each scale, the model can better extract feature information at different levels, especially through the multi-scale attention module to strengthen the change information of key areas, avoiding the situation of ignoring details or false detection in traditional methods, thereby effectively improving the accuracy of change detection.

[0031] 2. In the decoding stage, the model uses a lightweight multi-scale fusion module. This design can reduce computational overhead while ensuring model performance. By combining the main convolution with the deep convolution, the main convolution retains the main features, while the deep convolution can provide efficient feature extraction, reduce the amount of computation, and help accelerate the inference process in practical applications.

[0032] 3. Through the multi-scale attention module, the model can adaptively adjust the spatial importance of input features. Two parallel subnetworks extract channel attention maps from the horizontal and vertical directions respectively, and combine them through multiplication operations to generate a more accurate spatial attention map. This mechanism can further improve the model's understanding of spatial features and attention to changing areas, allowing the model to more accurately identify changing areas in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the network model structure of the present invention;

[0034] Figure 2 Schematic diagram of the multi-scale attention module structure;

[0035] Figure 3 This is a schematic diagram of the remote sensing image of the first phase;

[0036] Figure 4 This is a schematic diagram of the remote sensing image of the second phase;

[0037] Figure 5 It is a schematic diagram of remote sensing image change detection results in a specific embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of the actual change area in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Combination Figure 1-Figure 6As shown, a twin multi-scale deep fusion remote sensing image change detection method of the present invention comprises the following steps:

[0041] S1: The input images are two remote sensing images of the same location at different time periods, T 1 , T 2 , its length and width are recorded as H and W, C is the number of channels, and the size is H×W×C;

[0042] S2: T 1 , T 2 The inputs are first fed into the encoder layer, and then go through a startup convolution module consisting of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function to obtain the features E(0, 1) and E(0, 2), and then go through the startup convolution module once more to obtain the features E(1, 1) and E(1, 2);

[0043] S3: Use a single aggregation module to perform multi-scale fusion on E(1, 1) and E(1, 2) to obtain E(2, 1) and E(2, 2) respectively. The two features E(2, 1) and E(2, 2) are further subjected to a single aggregation module to obtain features E(3, 1) and E(3, 2). Finally, E(3, 2) is subjected to a single aggregation module to obtain E(4, 2). After that, the encoding stage is completed.

[0044] S4: In order to enhance the change information and provide contextual comparison, the features of each level in S3 are spliced ​​together, and the spliced ​​results are represented by C m Indicates that m = 1, 2, 3, 4;

[0045] S5: The model uses a lightweight multi-scale fusion module to perform decoding operations, and uses existing information to generate more feature maps so that the upsampling process can retain as many details as possible. The feature maps are passed through a lightweight multi-scale fusion module through a main convolution and a cheap operation to obtain two feature maps. The main convolution consists of a 1×1 convolution, a batch normalization layer, and a ReLU activation function. This part has a large amount of calculation but can retain the main features of the input. The cheap operation consists of a deep convolution with a 3×3 convolution kernel size, a batch normalization layer, and a ReLU activation function. This part has a small amount of calculation, but it can still generate useful feature maps. Finally, the two feature maps are spliced ​​to obtain the final output feature map;

[0046] S6: For E(1,2) and C 1 The splicing operation is performed and then lightweight multi-scale fusion is performed to obtain D(0,0), E(2,2) and C 2 After splicing, lightweight multi-scale fusion is performed to obtain D(1,0), E(3,2) and C 3After the splicing operation, lightweight multi-scale fusion is performed to obtain D(2,0), E(4,2) and C 4 After the splicing operation, lightweight multi-scale fusion is performed to obtain D(3, 0);

[0047] S7: D(0,0) and D(1,0) are concatenated after upsampling and then pass through the multi-scale attention module to obtain D(0,1). The multi-scale attention module can convert the input feature F∈R C×H×W Split into X independent subsets, that is, = [F 0 , F 1 , …, F X-1 ], F i∈R C / / x×H×W , and then the grouped feature maps will be obtained through three parallel subnetworks respectively;

[0048] S8: D(2,0) and C 2 and D(1,0) and then pass through the lightweight multi-scale fusion module to D(1,1). D(0,1) passes through the multi-scale attention module and is combined with C 1 The feature maps D(0,2), D(3,0) and D(2,0) are concatenated with D(0,0) and D(1,1) to obtain the feature maps D(0,2), D(3,0) and D(2,0) and C 3 After the splicing operation, lightweight multi-scale fusion is performed to obtain the feature maps D(2, 1), D(2, 1) and D(1, 0), D(1, 1), C 2 After the splicing operation, lightweight multi-scale fusion is performed to obtain the feature map D(1, 2). After D(0, 2) passes through the multi-scale attention module, it is combined with D(1, 2), D(0, 1), D(0, 0), C 1 After splicing, lightweight multi-scale fusion is performed to obtain the feature D(0, 3). Finally, D(0, 3) is output through the multi-scale attention module to output the prediction map of the corresponding scale, and then through the weighted negative log-likelihood function, and the network parameters are optimized during the model training process;

[0049] S9: The change detection task is actually a binary classification problem. The weighted negative log-likelihood loss function can specify different weights for each category through the weight parameter. In this case, the weights of both the unchanged and changed categories can be set to 0.5. This setting is very effective in solving the problem of sample imbalance. The specific formula is as follows;

[0050]

[0051] in is the weight of each category, N represents the size of the batch, and the specific formula for 1n is as follows:

[0052]

[0053] in Represents the probability of the target.

[0054] In a preferred embodiment, the present invention can be further configured as follows: the evaluation parameters include precision, recall rate, F1 score, and overall accuracy. The specific formula is as follows:

[0055]

[0056]

[0057]

[0058]

[0059] Among them, TP represents the number of samples that are actually positive and predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative and predicted to be negative.

[0060] In a preferred embodiment, the present invention can be further configured as follows: the twin multi-scale deep fusion model includes a single aggregation module, a multi-scale attention module and a lightweight multi-scale fusion module.

[0061] In a preferred embodiment of the present invention, the single aggregation module can be further configured as follows: the single aggregation module performs five 3×3 convolutions on the input feature map to output five feature maps respectively, and then splices the feature maps output by the five convolutions together through a splicing operation, and finally adjusts the number of channels through a 1×1 convolution. The multi-scale attention module as described in claim 1 efficiently processes multi-scale information on the obtained feature map while maintaining independence and correlation between channels. The lightweight multi-scale fusion module generates more feature maps while maintaining less calculation.

[0062] The data used in the present invention is the LEVIR-CD dataset, which is a remote sensing image collection specifically used for building change detection. It consists of 637 pairs of high-resolution multispectral images, each with a resolution of 0.5 meters and a size of 1024×1024 pixels. These image pairs cover 20 different areas in multiple cities in Texas, USA, with shooting times spanning from 2002 to 2018, covering various types of building changes. The dataset is divided into training set, validation set and test set, which are used for model training, adjustment and evaluation respectively. In order to adapt to the limitations of computing resources, the original 1024×1024 pixel image is slidably cropped into small blocks of 256×256 pixels without overlap. The cropped training set, validation set and test set contain 7120, 1024 and 2048 pairs of images respectively. The changed objects in the LEVIR-CD dataset are more and more concentrated, but the change area of ​​a single changed object is relatively small, indicating that most of the changes occur within a small scale, which provides challenging samples for the task of building change detection.

[0063] Comparative experiment and evaluation

[0064] The present invention is compared with the models of remote sensing image change detection in recent years, which verifies the effectiveness of the present invention. The models used for comparison are CDNet, FC-Siam-conc, FC-Siam-diff, and IFNet.

[0065] The quantitative results of the invention are shown in Table 1

[0066] Table 1: Quantitative experimental results of LEVIR-CD dataset

[0067]

[0068]

[0069] The accuracy of the present invention is 0.18% lower than the best FC-Siam-diff, which means that the performance of the present invention is comparable to other excellent models in positive class prediction (change area) and has strong prediction accuracy. The Recall of the present invention is the highest among all models, 1.31% and 1.96% higher than IFNet and CDNet respectively. Its advantage in recall rate shows that it can better detect the actual change area and reduce the possibility of missed detection. The F1-score of the present invention is 90.86%, which is the best performance among all models, which means that its comprehensive performance is the best. The overall accuracy of the present invention is the highest among all methods, indicating that the present invention has obvious advantages in the accuracy of the entire image classification.

[0070] In summary, the present invention proposes a twin multi-scale deep fusion remote sensing image change detection method. The present invention adopts a twin network structure, and fuses features of different scales through a single aggregation module in the encoding stage. In the decoding stage, a lightweight multi-scale fusion module is used to generate more feature maps at a lower computational cost, thereby effectively reducing resource consumption. At the same time, a multi-scale attention module is used to perform a pooling operation on the input feature map, calculate and apply the weight of each position, highlight important features, further optimize the detection effect, improve the detection accuracy and solve the problem of missed detection of small targets.

[0071] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A twin multi-scale deep fusion remote sensing image change detection method, characterized in that: The following steps are involved: S1: The input images are two remote sensing images of the same location at different time periods, T1 and T2, with length and width denoted as H and W, C as the number of channels, and the size is H×W×C; S2: Input T1 and T2 respectively, first enter the encoder layer, and pass through a startup convolution module consisting of a 3×3 convolution layer, a batch normalization layer, and a ReLU activation function to obtain features E(0, 1) and E(0, 2) respectively, and then pass through the startup convolution module again to obtain features E(1, 1) and E(1, 2) respectively; S3: Use a single aggregation module to perform multi-scale fusion on E(1, 1) and E(1, 2) to obtain E(2, 1) and E(2, 2) respectively. The two features E(2, 1) and E(2, 2) are further subjected to a single aggregation module to obtain features E(3, 1) and E(3, 2). Finally, E(3, 2) is subjected to a single aggregation module to obtain E(4, 2). After that, the encoding stage is completed. S4: In order to enhance the change information and provide contextual comparison, the features of each level in S3 are spliced ​​together, and the spliced ​​results are represented by C m Indicates that m = 1, 2, 3, 4; S5: The model uses a lightweight multi-scale fusion module to perform decoding operations, and uses existing information to generate more feature maps so that the upsampling process can retain as many details as possible. The feature maps are passed through a lightweight multi-scale fusion module through a main convolution and a cheap operation to obtain two feature maps. The main convolution consists of a 1×1 convolution, a batch normalization layer, and a ReLU activation function. This part has a large amount of calculation but can retain the main features of the input. The cheap operation consists of a deep convolution with a 3×3 convolution kernel size, a batch normalization layer, and a ReLU activation function. This part has a small amount of calculation, but it can still generate useful feature maps. Finally, the two feature maps are spliced ​​to obtain the final output feature map; S6: E(1, 2) is concatenated with C1 and then subjected to lightweight multi-scale fusion to obtain D(0, 0); E(2, 2) is concatenated with C2 and then subjected to lightweight multi-scale fusion to obtain E(1, 0); E(3, 2) is concatenated with C3 and then subjected to lightweight multi-scale fusion to obtain D(2, 0); E(4, 2) is concatenated with C4 and then subjected to lightweight multi-scale fusion to obtain D(3, 0); S7: D(0,0) and D(1,0) are concatenated after upsampling and then pass through the multi-scale attention module to obtain D(0,1). The multi-scale attention module can convert the input feature F∈R C×H×W Divide into X independent subsets, that is, F = [F0, F1, ..., F X-1 ], F i ∈R C / / X×H×W , and then the grouped feature maps will be obtained through three parallel subnetworks respectively; S8: D(2,0) is concatenated with C2 and D(1,0) and then passes through a lightweight multi-scale fusion module to D(1,1). D(0,1) passes through a multi-scale attention module and is concatenated with C1, D(0,0) and D(1,1) to obtain a feature map D(0,2). D(3,0) is concatenated with D(2,0) and C3 and then undergoes lightweight multi-scale fusion to obtain a feature map D(2,1). D(2,1) is concatenated with D(1,0), D(1,1), C2 After the splicing operation, lightweight multi-scale fusion is performed to obtain the feature map D(1, 2). After D(0, 2) passes through the multi-scale attention module, it is spliced ​​with D(1, 2), D(0, 1), D(0, 0), and C1, and then lightweight multi-scale fusion is performed to obtain the feature D(0, 3). Finally, D(0, 3) passes through the multi-scale attention module to output the prediction map of the corresponding scale, and then passes through the weighted negative log-likelihood function, and the network parameters are optimized during the model training process; S9: The change detection task is actually a binary classification problem. The weighted negative log-likelihood loss function can specify different weights for each category through the weight parameter. In this case, the weights of both the unchanged and changed categories can be set to 0.

5. This setting is very effective in solving the problem of sample imbalance. The specific formula is as follows; in is the weight of each category, N represents the size of the batch, and the specific formula of ln is as follows: in Represents the probability of the target.

2. According to claim 1, a twin multi-scale deep fusion remote sensing image change detection method is characterized in that: The evaluation parameters include precision, recall rate, F1 score, and overall accuracy. The specific formula is as follows: Among them, TP represents the number of samples that are actually positive and predicted to be positive, FP represents the number of samples that are actually negative but predicted to be positive, FN represents the number of samples that are actually positive but predicted to be negative, and TN represents the number of samples that are actually negative and predicted to be negative.

3. According to claim 2, a twin multi-scale deep fusion remote sensing image change detection method is characterized in that: The twin multi-scale deep fusion model includes a single aggregation module, a multi-scale attention module and a lightweight multi-scale fusion module.

4. According to claim 3, a twin multi-scale deep fusion remote sensing image change detection method is characterized in that: The single aggregation module performs five 3×3 convolutions on the input feature map and outputs five feature maps respectively, then splices the feature maps output by the five convolutions together through a splicing operation, and finally adjusts the number of channels through a 1×1 convolution. The multi-scale attention module as claimed in claim 1 efficiently processes multi-scale information on the obtained feature map while maintaining independence and correlation between channels. The lightweight multi-scale fusion module generates more feature maps while maintaining less calculation.

Images