A multimodal remote sensing image change detection method based on multi-level feature fusion

Abstract

The present invention discloses a multimodal remote sensing image change detection method based on multi-level feature fusion, which belongs to the technical field of remote sensing image analysis, and specifically comprises: firstly, preprocessing the multimodal remote sensing image change detection data, and dividing it into a training set and a test set; using a convolutional block attention model to perform space-channel joint feature extraction on images of different phases, and performing multi-level feature splicing and dimensionality reduction processing; then, using a mean square error loss function to calculate the distance between an input image and the reduced-dimensional features, and analyzing effective change information through a spatiotemporal difference information transmission module; calculating the total network loss and using a stochastic gradient descent algorithm to update the network parameters until the number of iterations is reached and the network model is saved; finally, using the saved network model to predict the test set and generate a detection result graph; the present invention can obtain competitive detection results under lightweight conditions through multi-level feature reconstruction and spatiotemporal difference information transmission modules.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image analysis, and in particular to a multimodal remote sensing image change detection method based on multi-level feature fusion. Background Art

[0002] With the development of sensing and communication technology, the integrated space-air-ground remote sensing monitoring platform has made significant progress, realizing the efficient collection of multimodal data from the ground, air and space. However, the limitations brought by the inability of traditional single data sources to provide surface change information in a short period of time, as well as the impact of harsh environmental detection conditions (such as rain, snow and long-term image acquisition) on remote sensing images, make how to achieve real-time and high-precision remote sensing image change detection a key issue in this field.

[0003] In the field of remote sensing image change detection, existing technologies mainly focus on using deep learning networks to process and analyze the difference information between synthetic aperture radar (SAR) and optical remote sensing images. Specifically, researchers have developed a model based on conditional generative adversarial networks, which converts optical images into SAR styles through a translation network and reduces the difference between the two. At the same time, a discriminator is used to verify the consistency of image features and extract change information through a differential method. In addition, there are studies that use deep translation networks to achieve the conversion of optical images to SAR subspaces through a cyclic structure, and then identify changed pixels through a change detection network. These methods aim to improve the accuracy and efficiency of change detection, especially in the processing of heterogeneous images.

[0004] However, due to the unique inherent imaging principle of SAR, coherent speckle noise is generated in the image. Obviously, this noise information seriously affects the network's extraction of effective information and the improvement of detection accuracy; optical images are easily blocked by rain, snow and clouds, resulting in the loss of spatial information. Therefore, how to extract as much effective information as possible from the image has become one of the key issues in solving this field.

[0005] In addition, existing detection schemes usually design two translation and detection networks with too many layers to achieve heterogeneous change detection. Although this scheme can achieve high detection accuracy, it is not timely, especially in practical applications such as military and disasters that require real-time detection results. Therefore, how to improve detection accuracy while constructing a lighter network is also a key and difficult problem that needs to be solved in this field. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal remote sensing image change detection method based on multi-level feature fusion to solve the following technical problems:

[0007] From the perspective of high robustness and high timeliness, the multimodal remote sensing images of any area acquired by sensors mounted on different remote sensing platforms at different times are analyzed.

[0008] The purpose of the present invention can be achieved through the following technical solutions:

[0009] A multimodal remote sensing image change detection method based on multi-level feature fusion includes the following steps:

[0010] S1. Preprocess the multimodal remote sensing image change data and divide it into a training set and a test set;

[0011] S2, collect two image blocks X in the test set t , where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model to X t Perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0012] S3, yes and Perform spatial-channel joint feature extraction respectively, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0013] S4, yes and Similarly, space-channel joint feature extraction is performed separately, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0014] S5. Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X1, denoted as F 12 ; Splicing , and , and the dimension is also reduced to the same channel dimension as X2, marked as F 21 ;

[0015] S6. Calculate F using the mean square error loss function 21 The distance from X2 is marked as l 21 , use the mean square error loss function to calculate F 12 The distance from X1 is marked as l 12 ;

[0016] S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Analyze the effective change information and mark the results as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred;

[0017] S8. Use the mean square error loss function to calculate the distance between Pred and the input human-labeled true value map GT, marked as l0; calculate the total network loss L=l 21 +l 12 +l0, and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model;

[0018] S9. Use the saved network model to predict the test set and obtain the final detection result graph.

[0019] As a further solution of the present invention: in S1, the pretreatment step is:

[0020] S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images;

[0021] S1.2: Calculate the mean mu1 and standard deviation std1 of each band of T1, and calculate the result t1 after standardization of T1 according to the following formula, where n is the pixel value in each band;

[0022] ;

[0023] S1.3: Calculate the average value for each band and standard deviation , the result after normalization of T2 is calculated according to the following formula ;

[0024] ;

[0025] S1.4: Cut t1 into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value;

[0026] S1.5: Cut t2 into m×m blocks with a step size of 1×1, and the result is recorded as ;

[0027] S1.6: Randomly select the same position of X1 and X2 The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

[0028] As a further solution of the present invention: in S2, X t The formula for space-channel joint feature extraction is:

[0029] ;

[0030] in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

[0031] As a further solution of the present invention: in S6, and The process of analyzing effective change information is:

[0032] Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

[0033] As a further solution of the present invention: in S6, the operation formula for convolution, activation and separation of the splicing result is:

[0034] .

[0035] As a further solution of the present invention: In S6, the effective change characteristics are calculated for the previous and next phases respectively. The formula is:

[0036] .

[0037] As a further solution of the present invention: in S6, and Convolution is performed on the concatenated result of and activation The formula is:

[0038] ;

[0039] in, Represents the features after convolution.

[0040] Beneficial effects of the present invention:

[0041] (1) The present invention can fully extract the spatiotemporal information of dual-phase images through multi-level fusion and reconstruction of heterogeneous domain features, overcome the shortcomings of existing technologies in detecting heterogeneous images, and make the detection results more reliable. Compared with the traditional complex network structure with too many layers, the lightweight design mode is adopted. While ensuring the detection accuracy, the computational load of the network is significantly reduced, and the detection efficiency is improved. In particular, in practical applications such as military and disaster situations where real-time detection results are required, the timeliness is significantly improved.

[0042] (2) The present invention designs a multi-level feature reconstruction module and a spatiotemporal difference information transmission module, which enhances the expression of effective information in the network and improves the network's ability to extract effective spatial features of input images, thereby enhancing the robustness of the overall network framework; the technical solution of the present invention promotes the development of remote sensing image change detection in the direction of automation, intelligence, and wisdom, and provides strong support for practical application fields such as environmental monitoring, disaster warning, and agricultural and forestry surveys. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention will be further described below in conjunction with the accompanying drawings.

[0044] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] See also Figure 1 As shown, the present invention is a multimodal remote sensing image change detection method based on multi-level feature fusion, comprising the following steps:

[0047] S1: Preprocess the multimodal remote sensing image change detection data and divide it into training set and test set;

[0048] S2, collect two image blocks X at different times in the test set t , where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model with the same structure to perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0049] S3, yes and Similarly, the convolutional block attention model with the same structure is used to extract spatial-channel joint features, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0050] S4. Yes and Similarly, the convolutional block attention model with the same structure is used to extract spatial-channel joint features, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ;

[0051] S5. Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X1, denoted as F 12 ; Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X2, denoted as F 21 ;

[0052] S6. Calculate F using the mean square error loss function 21 The distance from X2 is marked as l 21 , use the mean square error loss function to calculate F 12 The distance from X1 is marked as l 12 ;

[0053] S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Use the spatiotemporal difference information transfer module to analyze the effective change information, marked as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred;

[0054] S8. Use the mean square error loss function to calculate the distance between Pred and the input human-labeled true value map GT, marked as l0; calculate the total network loss L=l 21 +l 12 +l0, and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model;

[0055] S9. Use the saved network model to predict the test set and obtain the final detection result graph.

[0056] The present invention aims to use artificial intelligence technology to analyze multimodal remote sensing images of any area acquired at different times by sensors mounted on different remote sensing platforms, such as multispectral, hyperspectral, radar, drones, etc., to interpret, analyze and identify changes in surface cover and utilization from the perspective of high robustness and timeliness, and promote the development of this field towards automation, intelligence and wisdom.

[0057] The present invention designs a new multimodal remote sensing image change detection framework based on multi-level feature fusion. The framework adopts a twin network structure, through multi-level feature reconstruction and spatiotemporal difference information transmission modules, it can obtain competitive detection results under lightweight conditions.

[0058] (1) By designing a multi-level feature reconstruction module, the heterogeneous domain features are fully integrated at multiple levels while fully capturing the image features, so as to reconstruct the spatial information of the image, enhance the expression of effective information in the network, and improve the robustness of the network;

[0059] (2) By designing a spatiotemporal difference information transmission module, the spatiotemporal joint method is used to analyze the difference information from the extracted multi-temporal same-level reconstruction features, and then this difference information is transmitted step by step, so that the network can fuse effective change information from shallow to deep levels and reduce the impact of input heterogeneity.

[0060] In a preferred embodiment of the present invention, in S1, the pre-processing step is:

[0061] S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images;

[0062] S1.2: Calculate the mean mu1 and standard deviation std1 of each band of T1, and calculate the result t1 after standardization of T1 according to the following formula, where n is the pixel value in each band;

[0063] ;

[0064] S1.3: Calculate the average value for each band and standard deviation , the result after normalization of T2 is calculated according to the following formula ;

[0065] ;

[0066] S1.4: Cut t1 into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value;

[0067] S1.5: Cut t2 into m×m blocks with a step size of 1×1, and the result is recorded as ;

[0068] S1.6: Randomly select the same position of X1 and X2 The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

[0069] The present invention performs standardization processing by calculating the mean and standard deviation of each band to reduce the differences between remote sensing images acquired by different sensors and the same sensor at different times, thereby improving the efficiency and accuracy of subsequent model training.

[0070] The remote sensing image is cut into blocks according to a certain step size, and image blocks at the same position are randomly selected as training sets, and the rest are used as test sets, so that the change information can be better learned during model training and the effect of the model can be verified in the test set.

[0071] In another preferred embodiment of the present invention, in S2, X t The formula for space-channel joint feature extraction is:

[0072] ;

[0073] in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

[0074] Through the attention mechanism, the model of the present invention can pay more attention to the significant features in the input image, reduce the interference of irrelevant information, thereby improving the quality of feature expression and the accuracy of subsequent change detection. It has better adaptability and robustness to noise and heterogeneous images acquired by different sensors.

[0075] The model learns which areas are important in the spatial dimension, which helps the network focus on areas that may change rather than all parts of the entire image. At the same time, the model also learns which feature channels are important in the channel dimension, that is, determining which spectral bands or image features are more critical for change detection.

[0076] The joint work of the attention mechanisms in the spatial and channel dimensions allows the network to optimize feature representations from two dimensions at the same time, thereby more effectively extracting information useful for change detection tasks. Therefore, the use of the convolutional block attention model for spatial-channel joint feature extraction is to enable the network to better understand and utilize the key information in the image data, thereby improving the performance of change detection.

[0077] In another preferred embodiment of the present invention, in S6, and The process of analyzing effective change information is:

[0078] Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

[0079] In a preferred case of this embodiment, in S6, the operation formula for convolution, activation and separation of the splicing result is:

[0080] .

[0081] In another preferred embodiment of the present invention, in S6, effective change characteristics are calculated for the preceding and following phases respectively. The formula is:

[0082] .

[0083] In another preferred embodiment of the present invention, in S6, and Convolution is performed on the concatenated result of and activation The formula is:

[0084] ;

[0085] in, Represents the features after convolution.

[0086] By analyzing the differences between images at different time points, the module can identify changes in surface coverage and utilization, which is crucial for change detection tasks. The spatiotemporal difference information transmission module combines the change information in the time dimension with the feature information in the space dimension, allowing the network to simultaneously consider dynamic changes in the time series and static features in space. By focusing on change information, the module helps reduce the interference of irrelevant information, thereby improving the accuracy of change detection.

[0087] Since remote sensing images may come from different sensors and have different imaging characteristics, the spatiotemporal difference information transmission module helps to reduce the impact of this heterogeneity on change detection results. By analyzing the spatiotemporal difference information, the model can better adapt to different application scenarios and data types and enhance its generalization ability.

[0088] Compared with the traditional network structure with too many layers and large amount of calculation, the spatiotemporal difference information transmission module can improve the processing speed while maintaining high detection accuracy to meet real-time or near real-time application requirements.

[0089] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multimodal remote sensing image change detection method based on multi-level feature fusion, characterized in that: The following steps are involved: S1. Preprocess the multimodal remote sensing image change data and divide it into a training set and a test set; S2, collect two image blocks X in the test set t , where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model to X t Perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S3, yes and Perform spatial-channel joint feature extraction respectively, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S4. Yes and Similarly, space-channel joint feature extraction is performed separately, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S5. Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X1, denoted as F 12 ; Splicing , and , and the dimension is also reduced to the same channel dimension as X2, marked as F 21 ; S6. Calculate F using the mean square error loss function 21 The distance from X2 is marked as l 21 , use the mean square error loss function to calculate F 12 The distance from X1 is marked as l 12 ; S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Analyze the effective change information and mark the results as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred; S8. Use the mean square error loss function to calculate the distance between Pred and the input human-labeled true value map GT, marked as l0; calculate the total network loss L=l 21 +l 12 +l0, and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model; S9. Use the saved network model to predict the test set and obtain the final detection result graph.

2. According to claim 1, a multimodal remote sensing image change detection method based on multi-level feature fusion is characterized in that: In S1, the preprocessing steps are: S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images; S1.2: Calculate the mean mu1 and standard deviation std1 of each band of T1, and calculate the result t1 after standardization of T1 according to the following formula, where n is the pixel value in each band; ； S1.3: Calculate the average value for each band and standard deviation , the result after normalization of T2 is calculated according to the following formula ; ； S1.4: Cut t1 into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value; S1.5: Cut t2 into m×m blocks with a step size of 1×1, and the result is recorded as ; S1.6: Randomly select the same position of X1 and X2 The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

3. The method for detecting changes in multimodal remote sensing images based on multi-level feature fusion according to claim 1, characterized in that: In S2, for X t The formula for space-channel joint feature extraction is: ； in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

4. The method for detecting changes in multimodal remote sensing images based on multi-level feature fusion according to claim 1, characterized in that: In S6, and The process of analyzing effective change information is: Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

5. The method for detecting changes in multimodal remote sensing images based on multi-level feature fusion according to claim 4 is characterized in that: In S6, the operation formula for convolution, activation and separation of the splicing result is: 。 6. The method for detecting changes in multimodal remote sensing images based on multi-level feature fusion according to claim 4, characterized in that: In S6, effective change features are calculated for the previous and next phases respectively. The formula is: 。 7. The method for detecting changes in multimodal remote sensing images based on multi-level feature fusion according to claim 4, characterized in that: In S6, and Convolution is performed on the concatenated result of and activation The formula is: ； in, Represents the features after convolution.

Images