Multimodal remote sensing image change detection method based on multistage feature fusion

Through the design of multi-level feature fusion and space-time difference information transmission module, the detection accuracy and timeliness in remote sensing image change detection are solved, and lightweight and efficient change detection is realized. It is suitable for real-time applications and promotes remote sensing image change detection. Automation and intelligence.

CN118941910AActive Publication Date: 2024-11-12ANHUI SAIDA TECH

Patent Information

Application Number
CN202411421285.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12
Estimated Expiration
2044-10-12

AI Technical Summary

Technical Problem

The prior art has problems such as noise-affected detection accuracy, poor timeliness, and large network computing volume in remote sensing image change detection, especially in real-time detection applications for military and disasters.

Method used

Using multimodal remote sensing image change detection method based on multi-level feature fusion, space-channel joint feature extraction is performed through convolutional block attention model, space-time difference information transmission module is designed, combined with twin network structure, lightweight and efficient change detection is achieved .

Benefits of technology

It improves the robustness and timeliness of the detection results, significantly reduces the amount of network computing, enhances the ability to extract effective information, is suitable for real-time application scenarios, and promotes the automation and intelligence of remote sensing image change detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941910A_ABST
    Figure CN118941910A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal remote sensing image change detection method based on multistage feature fusion, which belongs to the technical field of remote sensing image analysis, and specifically comprises the following steps: firstly, preprocessing multi-modal remote sensing image change detection data, and dividing into a training set and a test set; performing space-channel joint feature extraction on images of different time phases by adopting a convolution block attention model, and performing multi-level feature splicing and dimension reduction processing; then, calculating the distance between the input image and the dimension-reduced feature by using a mean square error loss function, and analyzing effective change information through a space-time difference information transfer module; calculating the total loss of the network, updating network parameters by using a stochastic gradient descent algorithm, and storing the network model until the number of iterations is reached; and finally, predicting the test set by using the stored network model, and generating a detection result graph. According to the invention, through the multi-level feature reconstruction and space-time difference information transmission module, a competitive detection result can be obtained under the condition of light weight.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image analysis, and in particular to a multimodal remote sensing image change detection method based on multi-level feature fusion. Background Art

[0002] With the development of sensing and communication technology, the integrated space-air-ground remote sensing monitoring platform has made significant progress, realizing the efficient collection of multimodal data from the ground, air and space. However, the limitations brought by the inability of traditional single data sources to provide surface change information in a short period of time, as well as the impact of harsh environmental detection conditions (such as rain, snow and long-term image acquisition) on remote sensing images, make how to achieve real-time and high-precision remote sensing image change detection a key issue in this field.

[0003] In the field of remote sensing image change detection, existing technologies mainly focus on using deep learning networks to process and analyze the difference information between synthetic aperture radar (SAR) and optical remote sensing images. Specifically, researchers have developed a model based on conditional generative adversarial networks, which converts optical images into SAR styles through a translation network and reduces the difference between the two. At the same time, a discriminator is used to verify the consistency of image features and extract change information through a differential method. In addition, there are studies that use deep translation networks to achieve the conversion of optical images to SAR subspaces through a cyclic structure, and then identify changed pixels through a change detection network. These methods aim to improve the accuracy and efficiency of change detection, especially in the processing of heterogeneous images.

[0004] However, due to the unique inherent imaging principle of SAR, coherent speckle noise is generated in the image. Obviously, this noise information seriously affects the network's extraction of effective information and the improvement of detection accuracy; optical images are easily blocked by rain, snow and clouds, resulting in the loss of spatial information. Therefore, how to extract as much effective information as possible from the image has become one of the key issues in solving this field.

[0005] In addition, existing detection schemes usually design two translation and detection networks with too many layers to achieve heterogeneous change detection. Although this scheme can achieve high detection accuracy, it is not timely, especially in practical applications such as military and disasters that require real-time detection results. Therefore, how to improve detection accuracy while constructing a lighter network is also a key and difficult problem that needs to be solved in this field. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal remote sensing image change detection method based on multi-level feature fusion to solve the following technical problems: From the perspective of high robustness and high timeliness, the multimodal remote sensing images of any area acquired by sensors mounted on different remote sensing platforms at different times are analyzed.

[0007] The purpose of the present invention can be achieved through the following technical solutions: A multimodal remote sensing image change detection method based on multi-level feature fusion includes the following steps: S1. Preprocess the multimodal remote sensing image change data and divide it into a training set and a test set; S2, collect two image blocks X in the test set t , where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model to X t Perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S3, yes and Perform spatial-channel joint feature extraction respectively, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S4. Yes and Similarly, space-channel joint feature extraction is performed separately, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S5. Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X1, denoted as F 12 ; Splicing , and , and the dimension is also reduced to the same channel dimension as X2, marked as F 21 ; S6. Calculate F using the mean square error loss function 21The distance from X2 is marked as l 21 , use the mean square error loss function to calculate F 12 The distance from X1 is marked as l 12 ; S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Analyze the effective change information and mark the results as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred; S8. Use the mean square error loss function to calculate the distance between Pred and the input human-labeled true value map GT, marked as l0; calculate the total network loss L=l 21 +l 12 +l0, and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model; S9. Use the saved network model to predict the test set and obtain the final detection result graph.

[0008] As a further solution of the present invention: in S1, the pretreatment step is: S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images; S1.2: Calculate the mean mu1 and standard deviation std1 of each band of T1, and calculate the result t1 after standardization of T1 according to the following formula, where n is the pixel value in each band; ; S1.3: Calculate the average value for each band and standard deviation , the result after normalization of T2 is calculated according to the following formula ; ; S1.4: Cut t1 into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value; S1.5: Cut t2 into m×m blocks with a step size of 1×1, and the result is recorded as ; S1.6: Randomly select the same position of X1 and X2 The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

[0009] As a further solution of the present invention: in S2, X t The formula for space-channel joint feature extraction is: ; in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

[0010] As a further solution of the present invention: in S6, and The process of analyzing effective change information is: Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

[0011] As a further solution of the present invention: in S6, the operation formula for convolution, activation and separation of the splicing result is: .

[0012] As a further solution of the present invention: In S6, the effective change characteristics are calculated for the previous and next phases respectively. The formula is: .

[0013] As a further solution of the present invention: in S6, and Convolution is performed on the concatenated result of and activation The formula is: ; in, Represents the features after convolution.

[0014] Beneficial effects of the present invention: (1) The present invention can fully extract the spatiotemporal information of dual-phase images through multi-level fusion and reconstruction of heterogeneous domain features, overcome the shortcomings of existing technologies in detecting heterogeneous images, and make the detection results more reliable. Compared with the traditional complex network structure with too many layers, the lightweight design mode is adopted. While ensuring the detection accuracy, the computational load of the network is significantly reduced, and the detection efficiency is improved. In particular, in practical applications such as military and disaster situations where real-time detection results are required, the timeliness is significantly improved. (2) The present invention designs a multi-level feature reconstruction module and a spatiotemporal difference information transmission module, which enhances the expression of effective information in the network and improves the network's ability to extract effective spatial features of input images, thereby enhancing the robustness of the overall network framework; the technical solution of the present invention promotes the development of remote sensing image change detection in the direction of automation, intelligence, and wisdom, and provides strong support for practical application fields such as environmental monitoring, disaster warning, and agricultural and forestry surveys. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings.

[0016] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] See also Figure 1 As shown, the present invention is a multimodal remote sensing image change detection method based on multi-level feature fusion, comprising the following steps: S1: Preprocess the multimodal remote sensing image change detection data and divide it into training set and test set; S2, collect two image blocks X at different times in the test set t, where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model with the same structure to perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S3, yes and Similarly, the convolutional block attention model with the same structure is used to extract spatial-channel joint features, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S4. Yes and Similarly, the convolutional block attention model with the same structure is used to extract spatial-channel joint features, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S5. Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X1, denoted as F 12 ; Splicing , and , and use a linear layer to reduce the dimension to the same channel dimension as X2, denoted as F 21 ; S6. Calculate F using the mean square error loss function 21 The distance from X2 is marked as l 21 , use the mean square error loss function to calculate F 12 The distance from X1 is marked as l 12 ; S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Use the spatiotemporal difference information transfer module to analyze the effective change information, marked as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred; S8. Use the mean square error loss function to calculate the distance between Pred and the input human-labeled true value map GT, marked as l0; calculate the total network loss L=l 21 +l 12 +l0, and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model; S9. Use the saved network model to predict the test set and obtain the final detection result graph.

[0019] The present invention aims to use artificial intelligence technology to analyze multimodal remote sensing images of any area acquired at different times by sensors mounted on different remote sensing platforms, such as multispectral, hyperspectral, radar, drones, etc., to interpret, analyze and identify changes in surface cover and utilization from the perspective of high robustness and timeliness, and promote the development of this field towards automation, intelligence and wisdom.

[0020] The present invention designs a new multimodal remote sensing image change detection framework based on multi-level feature fusion. The framework adopts a twin network structure, through multi-level feature reconstruction and spatiotemporal difference information transmission modules, it can obtain competitive detection results under lightweight conditions. (1) By designing a multi-level feature reconstruction module, the heterogeneous domain features are fully integrated at multiple levels while fully capturing the image features, so as to reconstruct the spatial information of the image, enhance the expression of effective information in the network, and improve the robustness of the network; (2) By designing a spatiotemporal difference information transmission module, the spatiotemporal joint method is used to analyze the difference information from the extracted multi-temporal same-level reconstruction features, and then this difference information is transmitted step by step, so that the network can fuse effective change information from shallow to deep levels and reduce the impact of input heterogeneity.

[0021] In a preferred embodiment of the present invention, in S1, the pre-processing step is: S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images; S1.2: Calculate the mean mu1 and standard deviation std1 of each band of T1, and calculate the result t1 after standardization of T1 according to the following formula, where n is the pixel value in each band; ; S1.3: Calculate the average value for each band and standard deviation , the result after normalization of T2 is calculated according to the following formula ; ; S1.4: Cut t1 into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value; S1.5: Cut t2 into m×m blocks with a step size of 1×1, and the result is recorded as ; S1.6: Randomly select the same position of X1 and X2 The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

[0022] The present invention performs standardization processing by calculating the mean and standard deviation of each band to reduce the differences between remote sensing images acquired by different sensors and the same sensor at different times, thereby improving the efficiency and accuracy of subsequent model training.

[0023] The remote sensing image is cut into blocks according to a certain step size, and image blocks at the same position are randomly selected as training sets, and the rest are used as test sets, so that the change information can be better learned during model training and the effect of the model can be verified in the test set.

[0024] In another preferred embodiment of the present invention, in S2, X t The formula for space-channel joint feature extraction is: ; in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

[0025] Through the attention mechanism, the model of the present invention can pay more attention to the significant features in the input image, reduce the interference of irrelevant information, thereby improving the quality of feature expression and the accuracy of subsequent change detection. It has better adaptability and robustness to noise and heterogeneous images acquired by different sensors.

[0026] The model learns which areas are important in the spatial dimension, which helps the network focus on areas that may change rather than all parts of the entire image. At the same time, the model also learns which feature channels are important in the channel dimension, that is, determining which spectral bands or image features are more critical for change detection.

[0027] The joint work of the attention mechanisms in the spatial and channel dimensions allows the network to optimize feature representations from two dimensions at the same time, thereby more effectively extracting information useful for change detection tasks. Therefore, the use of the convolutional block attention model for spatial-channel joint feature extraction is to enable the network to better understand and utilize the key information in the image data, thereby improving the performance of change detection.

[0028] In another preferred embodiment of the present invention, in S6, and The process of analyzing effective change information is: Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

[0029] In a preferred case of this embodiment, in S6, the operation formula for convolution, activation and separation of the splicing result is: .

[0030] In another preferred embodiment of the present invention, in S6, effective change characteristics are calculated for the preceding and following phases respectively. The formula is: .

[0031] In another preferred embodiment of the present invention, in S6, and Convolution is performed on the concatenated result of and activation The formula is: ; in, Represents the features after convolution.

[0032] By analyzing the differences between images at different time points, the module can identify changes in surface coverage and utilization, which is crucial for change detection tasks. The spatiotemporal difference information transmission module combines the change information in the time dimension with the feature information in the space dimension, allowing the network to simultaneously consider dynamic changes in the time series and static features in space. By focusing on change information, the module helps reduce the interference of irrelevant information, thereby improving the accuracy of change detection.

[0033] Since remote sensing images may come from different sensors and have different imaging characteristics, the spatiotemporal difference information transmission module helps to reduce the impact of this heterogeneity on change detection results. By analyzing the spatiotemporal difference information, the model can better adapt to different application scenarios and data types and enhance its generalization ability.

[0034] Compared with the traditional network structure with too many layers and large amount of calculation, the spatiotemporal difference information transmission module can improve the processing speed while maintaining high detection accuracy to meet real-time or near real-time application requirements.

[0035] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that The following steps are involved: S1. Preprocess the multimodal remote sensing image change data and divide it into a training set and a test set; S2, collect two image blocks X in the test set t , where t=1,2, represents the two phases before and after, respectively using the convolutional block attention model to X t Perform space-channel joint feature extraction, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S3, yes and Perform spatial-channel joint feature extraction respectively, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S4. Yes and Similarly, space-channel joint feature extraction is performed separately, denoted as ;Will Pass to , the concatenated feature is marked as ,Will Pass to , the concatenated feature is marked as ; S5. Splicing , and , and use a linear layer to reduce the dimension to the same value as X 1 The same channel dimension, denoted as F 12 ; Splicing , and , and the same dimension reduction is achieved as X 2 The same channel dimension, denoted as F 21 ; S6. Calculate F using the mean square error loss function 21 With X 2 The distance between them is marked as l 21 , use the mean square error loss function to calculate F 12 With X 1 The distance between them is marked as l 12 ; S7, yes and Use the spatiotemporal difference information transfer module to analyze the effective change information and mark the result as C 1 , and similarly and , and Analyze the effective change information and mark the results as C 2 and C 3 , splicing C 1 , C 2 and C 3 , convolve the concatenated result and mark it as Pred; S8. Use the mean square error loss function to calculate the distance between Pred and the input human-annotated true value map GT, marked as l 0 ; Calculate the total network loss L=l 21 +l 12 +l 0 , and use the stochastic gradient descent algorithm to reversely update the network parameters until the network reaches the total number of iterations Ep, and save the network model; S9. Use the saved network model to predict the test set and obtain the final detection result graph.

2. According to claim 1, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S1, the preprocessing steps are: S1.1: Mark the remote sensing image of phase 1 as , marking the remote sensing image of phase 2 as ,in, Indicates the height, width and number of bands of dual-temporal remote sensing images; S1.2: Calculate T 1 The mean value mu1 and standard deviation std1 of each band are calculated according to the following formula to standardize T 1 The result after 1 , where n is the pixel value in each band; ; S1.3: Calculate the average value for each band and standard deviation , the standardized T is calculated according to the following formula 2 The result after ; ; S1.4: For t 1 Cut into m×m blocks with a step size of 1×1, and the result is recorded as , m is the preset value; S1.5: For t 2 Cut into m×m blocks with a step size of 1×1, and the result is recorded as ; S1.6: For X 1 and X 2 Randomly select the same position The correct image blocks are used as training sets, and the remaining image blocks are used as test sets.

3. According to claim 1, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S2, for X t The formula for space-channel joint feature extraction is: ; in, Represents the channel characteristics of the image block, Represents the spatial characteristics of the image block, is the maximum pooling function, is the average pooling function, It is a multi-layer perceptron, consisting of three linear layers. It is a channel-dimensional splicing operation. is the Sigmoid function.

4. According to claim 1, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S6, and The process of analyzing effective change information is: Splicing and , and convolve the concatenated result ,activation and separation Operation, marking the result as and ; Calculate the effective change characteristics for the previous and next phases respectively ;get and ; Splicing and , and and Convolution is performed on the concatenated result of and activation , denoted as C 1 .

5. According to claim 4, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S6, the operation formula for convolution, activation and separation of the splicing result is: 。 6. According to claim 4, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S6, effective change features are calculated for the previous and next phases respectively. The formula is: 。 7. According to claim 4, a multimodal remote sensing image change detection method based on multi-level feature fusion, It is characterized in that In S6, and Convolution is performed on the concatenated result of and activation The formula is: ; in, Represents the features after convolution.

Citation Information

Patent Citations

  • Multi-modal remote sensing data change detection method

    CN117523401A

  • Multi-modal remote sensing image real-time target detection method based on channel and space exchange

    CN117788992A

  • Remote sensing image target detection method based on visual selective state space network

    CN118691990A

Cited By

  • Remote sensing change detection method and system based on multi-modal space-time difference feature enhancement

    CN119580119A