A change detection method and system based on feature interaction and spatio-temporal correlation

Abstract

The application provides a change detection method and system based on feature interaction and space-time correlation, and relates to the technical field of remote sensing image change detection. The specific steps are as follows: first, a double-time change detection data set is obtained and preprocessed. Second, a change detection model based on a double-encoding double-decoding structure is constructed. The model includes a mixed effective channel module, which mainly enables the double-time features to maintain spatial information similarity and retain more effective channel information when interacting; a spatial denoising attention module, which mainly strengthens position information such as details and edges on the shallow layer features; and a feature fusion up-sampling module, which is mainly used for model training and data transmission in the up-sampling stage. Finally, the effect of the trained model is verified, and the result is saved for use. The application can effectively solve the challenges faced in remote sensing application tasks due to the limitations of remote sensing images themselves, such as fog, shadow, complex targets, etc., and improve the accuracy of the change detection task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image change detection, and in particular, to a change detection method and system based on feature interaction and spatio-temporal correlation. BACKGROUND

[0002] Remote sensing image change detection is a research topic that has attracted much attention in the field of remote sensing image processing. The core goal of change detection is to extract the location information of changes in an area by analyzing two or more remote sensing images taken at different times of the same area. These change information plays a crucial role in disaster assessment, urban expansion, land change, and forest deforestation monitoring applications. In the early stage of remote sensing image change detection, the method of manual interpretation was mainly used. However, with the continuous development of deep learning technology, more and more methods have been introduced into remote sensing image change detection. The use of deep learning to solve the problem of remote sensing image change detection has made significant progress, which is due to the increasing number of high-resolution remote sensing images and aerial images currently available.

[0003] However, change detection based on high-resolution remote sensing images still faces some challenges. First, the change target in the change scene is complex, which will affect the true change target to some extent. Second, due to the light and shadow factors at different positions, some interference noise will be introduced, which will cause more key information to be lost in the process of image feature extraction. In order to cope with the above challenges, some studies mention using feature interaction. Since the input of change detection is double-time images, their features contain different semantic information, and the interaction between their features is crucial to the entire change detection process. At the same time, researchers have mentioned that usually shallow features contain more location and detail information, but their semanticity is lower and there is more noise. In contrast, deep features usually have stronger semantic information, but their ability to perceive details is poor.

[0004] Therefore, how to make full use of the dependency relationship between the deep and shallow features of the dual temporal images to reduce the loss of spatial detail information is a problem to be solved. In other words, in the change detection scene, it is crucial to model the spatio-temporal dependency relationship between the two images. In order to complete the modeling of the spatio-temporal dependency relationship, some researchers introduce the attention mechanism into the change detection method, and the introduction of the attention mechanism helps to alleviate the influence of the noise in the remote sensing image to a certain extent. Common methods using attention mechanism, for example, DASNet captures long-range dependency relationships through a double attention mechanism to obtain more discriminative features. IFN reconstructs the change map by fusing the multi-level deep features of the original image and the image difference features through the attention module. MSPSNet effectively integrates multi-scale features using a parallel convolution structure and a self-attention module. SNUNet-CD alleviates the problem of loss of deep position detail information by integrating a channel attention module. Some researchers also propose to use multi-scale feature fusion to make up for the loss of spatial information in deep features and the lack of rich semantic information in shallow features, thereby completing this task. Among them, the feature pyramid network idea is widely used in the change detection task, for example, the hierarchical refusion mode in MFPF-Net improves the effective communication number of feature maps and the sufficient fusion of feature information in change detection. Although the above methods alleviate the false detection and boundary blur problems in the detection results to a certain extent, they still have some limitations when facing complex change scenes, especially in the feature loss encountered in the feature extraction process.

[0005] Therefore, it is necessary to design a change detection model that can fully utilize the spatio-temporal dependency relationship between the dual temporal remote sensing images.

[0006] Noun explanation: DASNet, IFN, MSPSNet, SNUNet-CD, MFPF-Net are methods proposed by some researchers for the change detection task. SUMMARY

[0007] The purpose of the present application is to provide a change detection method that can solve the above problems and achieve high precision. In order to achieve the above purpose, the technical solution adopted by the present application is: a change detection method and system based on feature interaction and spatio-temporal association, comprising the following steps:

[0008] S1, obtaining a remote sensing change detection dataset, the dataset should include two or more images of the same place and contain change detection labels, and the dataset is divided into a training set, a validation set and a test set.

[0009] S2, constructing a network model based on a dual-encoder dual-decoder structure, the model utilizes feature interaction and spatio-temporal correlation of dual-time images, proposes a mixed effective channel module (MECM) and a spatial denoising attention module (SDAM), and uses a feature fusion up-sampling module (FFUM) to fuse the outputs from the two, and then passes them down for training, the construction ideas and design details of the modules include steps S21-S23:

[0010] S21, first, the mixed effective channel module is mainly used for processing feature interaction. Second, the spatial denoising attention module is mainly used in the process of modeling spatio-temporal dependence. Finally, the above outputs are fused and up-sampled. The implementation process can be represented by the formula:

[0011] ,

[0012] wherein subscript 1 represents the previous time phase, subscript 2 represents the later time phase, skip represents a skip connection, H represents a prediction head, P represents a prediction result;

[0013] S22, based on the design of the spatial attention module, we propose a spatial denoising attention module, and use the module as a feature optimization processing operation before the skip connection. We divide the input into horizontal and vertical branches. The main role of the vertical branch is to retain the original feature information, so as to be spliced and element product operation with the horizontal branch. The horizontal branch first passes through the spatial attention module, which first splices the feature maximum pooling and average pooling, then performs feature fusion and channel number adjustment through 7x7 convolution, and obtains the attention mask through Sigmoid activation function. Then the obtained Mask is element product operation with the original feature, and then spliced with the original feature map of the vertical branch. Finally, 3x3 convolution is used for feature extraction, and a feature map with the same size as the original feature is obtained. The implementation process can be represented by the formula:

[0014] ,

[0015] wherein M is the feature mask generated by the spatial attention module, X' is the last output of the module. In the formula, σ represents the Sigmoid activation function, Ci represents the i×i convolution operation, cat represents the splicing operation, and represents the element product operation, max represents the maximum pooling operation, avg represents the average pooling operation, and the following also follows this rule;

[0016] S23, in order to contact the context information and retain more effective channel information when interacting with double time characteristics, we introduce a mixed effective channel module, and add the module to the completion of the encoding and decoding stages respectively. In order to make full use of the feature information and channel related connection between the two groups of features, we use 3x3 group convolution to mix the features obtained in the last step, and then pass through a channel attention to weight the important channels, here we use kxk convolution to enhance the channel information. The implementation process can be expressed by the formula:

[0017] ,

[0018] wherein represents the nearest odd number to t, c is the number of channels, r and b are 2 and 1 by default, is the feature after superposition of double time images, is the weighted feature.

[0019] S3, the feature fusion upsampling module receives the optimized features from SDAM and the effective channel features from MECM. The process first upsamples the latter to the same resolution as the former, here all 2x upsampling is used. Then the feature fusion is performed, and finally the feature information is enhanced through several convolution layers to obtain the final result. The implementation process can be expressed by the formula:

[0020] ,

[0021] wherein Up2 represents 2x upsampling operation, F z from MECM, X' from SDAM.

[0022] S4, after the model training is completed, the model with the best performance is obtained, and the double time images are input into the model to obtain the changed area and save it as a binary image.

[0023] Preferably, in the acquisition of change detection data set, the registered multi-temporal change detection data set is used, and the images with greater detection difficulty can be selected.

[0024] Preferably, the model architecture and each module of the change detection method based on feature interaction and space-time correlation should be strictly combined according to the described process.

[0025] Preferably, in the spatial attention denoising module, since the shallow feature map mainly contains local details and edge information of the image, it has relatively rich positioning information, but the shallow feature usually contains more noise information, which will interfere with the extracted features and cause the loss of context information. In order to maintain the space-time dependent relationship on the feature, we use jump connection to optimize.

[0026] Preferably, the mixed effective channel module uses a channel attention mechanism that helps to reveal the correlation between different channels. The importance of each feature channel is automatically obtained through network learning, and different weight coefficients are assigned to each channel to enhance key features and suppress less important features, thereby improving the ability of feature representation. In this method, this module is mainly applied after feature extraction is completed. At this stage, the features extracted by the backbone are usually more, and by suppressing features that have little effect, the expression of complex targets can be enhanced, which helps to retain complex targets and provides accurate data for subsequent decoding operations; Preferably, the feature mixing process in the mixed effective channel module, since the two inputs are extracted by a twin network with shared weights, which means they have the same semantics, therefore, the following rules are used to connect and mix x and y to get:

[0027]

[0028] where c represents the number of channels of the feature map, and x and y represent the feature maps of the previous and subsequent time phases, respectively.

[0029] Preferably, during the upsampling process on the model, the upsampling operation has two main advantages. On the one hand, it can improve the resolution of the image, making the image clearer and smoother, thereby improving the image quality; on the other hand, it can improve the classification accuracy and detail expression at the pixel level, which helps to enhance the perception ability of the neural network to image details. Using the skip connection can directly connect the low-level features with the high-level features, so that the detailed information can be transmitted to the decoder, thereby improving the accuracy and detail retention ability of the segmentation result; on the other hand, the skip connection can also alleviate the problem of gradient propagation, making the network easier to optimize and the training effect better.

[0030] Preferably, the model training and testing process should be carried out on multiple data sets, so as to highlight the generalization ability of the method.

[0031] A change detection system based on feature interaction and spatio-temporal correlation, comprising:

[0032] Data processing module: after obtaining the necessary change detection data set, it is processed to improve the generalization ability of the model, and the processing process includes random flipping, translation, scaling, and cropping operations;

[0033] Model training module: the training process includes the setting of hyperparameters, the selection of data sets, and the selection and saving of the optimal model. The hyperparameters include learning rate, batch size, training times, optimizer, and loss function selection. The selection and saving of the optimal model means retaining the model with the best result during the training process for use;

[0034] Model using module: first, the optimal model needs to be loaded, then the data to be processed is selected, at least two remote sensing images are needed. Then the predicted result is generated, and the result is saved to the local;

[0035] Application module: with the prediction result of the previous step, the result can be selected to be imported to the cloud, and a data loading link is provided for direct application in actual application, so that the calling process becomes convenient;

[0036] The system is divided into several modules according to functions, so that the system shows a simpler use mode and clearer use rules in the overall scheduling process.

[0037] Compared with the prior art, the advantages of the present application are:

[0038] 1. A spatial denoising attention module is proposed, which effectively utilizes the spatial information in the feature map, enhances the change target and reduces the background noise, and reduces the influence of noise on effective features;

[0039] 2. A mixed effective channel module is proposed, which can maintain a low computational complexity when interacting with double-time-phase features, pay more attention to channel features effective for change detection tasks, and reduce channel information redundancy;

[0040] 3. Feature fusion upsampling strategy and jump connection method are used, which can associate space-time features and fully utilize the space-time dependence relationship between double-time-phase images.

[0041] In summary, the present application is especially suitable for the case that the change detection effect is not good due to the fact that there is more image feature noise information caused by the limitations of the image itself, such as fog, shadow and complex target. BRIEF DESCRIPTION OF DRAWINGS

[0042] Fig. 1 The flowchart of the present application;

[0043] Fig. 2 The example diagram of the limitations of the remote sensing image itself;

[0044] Fig. 3 The change detection model based on feature fusion and space-time association;

[0045] Fig. 4 The comparison diagram before and after using the spatial denoising attention module;

[0046] Fig. 5 The prediction result comparison diagram of the model (green represents TP, yellow represents FN, red represents FP, and black represents TN). DETAILED DESCRIPTION

[0047] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] Embodiment 1

[0049] The embodiment provides a change detection method based on feature interaction and spatio-temporal correlation, referring to Figs. 1 to 4 , comprising the following steps:

[0050] S1, obtaining a remote sensing change detection data set, the data set should include two or more images of the same place and contain change detection labels, and the data set is divided into a training set, a validation set and a test set. A registered multi-temporal change detection data set is used, and images with greater detection difficulty can be selected;

[0051] S11, the registration method can be completed by using an open source tool or algorithm, and the images with greater detection difficulty include remote sensing images with heavy fog, shadow and complex change targets;

[0052] S12, the data set division method can divide the training set, the validation set and the test set according to a ratio of 8:1:1.

[0053] S2, constructing a network model based on a double-encoder double-decoder structure, the model uses feature interaction and spatio-temporal correlation of double-time images, proposes a mixed effective channel module (MECM) and a spatial denoising attention module (SDAM), and uses a feature fusion up-sampling module (FFUM) to fuse the outputs from the two, and then transmits downward and trains, the construction idea and design details of the module include steps S21-S23:

[0054] S21, first, the mixed effective channel module is mainly used for processing feature interaction. Second, the spatial denoising attention module is mainly used in the process of modeling the spatio-temporal dependence relationship. Finally, the above outputs are fused and up-sampled. The implementation process can be represented by the formula:

[0055] ,

[0056] Wherein, subscript 1 represents a previous time phase, subscript 2 represents a later time phase, skip represents a skip connection, H represents a prediction head, P represents a prediction result;

[0057] Specifically:

[0058] The model overall architecture and each module should be strictly combined according to the described process.

[0059] In order to maintain the spatio-temporal dependence on the characteristics, we use the jump connection to optimize. Based on the design of the spatial attention module, we propose a spatial denoising attention module, and use this module as the feature optimization processing operation before the jump connection;

[0060] Specifically:

[0061] We divide the input into horizontal and vertical branches. The main role of the vertical branch is to retain the original feature information, so as to splice and element product operation with the horizontal branch. The horizontal branch first passes through the spatial attention module. The module first splices the feature maximum pooling and average pooling, and then adjusts the number of channels through 7x7 convolution for feature fusion. And through the Sigmoid activation function to get the attention mask. Then the mask obtained is multiplied with the original feature, and then concatenated with the original feature map of the vertical branch. Finally, 3x3 convolution is used for feature extraction, and the feature map with the same size as the original feature is obtained. The implementation process can be expressed as:

[0062] ,

[0063] Among them M is the feature mask generated by the spatial attention module, X' is the output of the module. In the formula, σ represents the Sigmoid activation function, Ci represents the i x i convolution operation, cat represents the splicing operation, and represents the element product operation, max represents the maximum pooling operation, avg represents the average pooling operation, and the following also follows this rule;

[0064] S23, in order to contact the context information and retain more effective channel information when interacting with double time characteristics, we introduce a mixed effective channel module, which uses a channel attention mechanism that helps to reveal the correlation between different channels. The importance of each feature channel is automatically obtained through network learning, and different weight coefficients are assigned to each channel to enhance key features and suppress less important features, thereby improving the ability of feature representation. In this method, this module is mainly applied after feature extraction is completed. In this stage, the features extracted by the backbone are usually more, and by suppressing the features that have little effect, the expression of complex targets can be enhanced, which helps to retain complex targets and provides accurate data for subsequent decoding operations. And add this module to the end of the encoding and decoding stages respectively. In order to make full use of the feature information and channel correlation between the two groups of features, we use 3x3 group convolution to mix the features obtained in the last step, and then pass through a channel attention to weight the important channels. Here we use kxk convolution to enhance channel information. The implementation process can be represented by the formula:

[0065] ,

[0066] wherein represents the nearest odd number to t, c is the number of channels, r and b are 2 and 1 by default, is the feature after the superposition of the double time phase image, is the weighted feature.

[0067] S3, the feature fusion upsampling module receives the optimized features from SDAM and the effective channel features from MECM. On the one hand, it can improve the resolution of the image, making the image clearer and smoother, thereby improving the image quality; on the other hand, it can improve the pixel-level classification accuracy and detail expression, which helps to enhance the perception ability of the neural network to image details. Using skip connection can directly connect the low-level features with the high-level features, so that the detailed information can be transmitted to the decoder, thereby improving the accuracy and detail preservation ability of the segmentation result; on the other hand, the skip connection can also alleviate the problem of gradient propagation, making the network easier to optimize and the training effect better;

[0068] Specifically:

[0069] The process first upsamples the latter to the same resolution as the former, and here all 2x upsampling is used. Then the feature fusion is performed, and finally the feature information is enhanced through several convolution layers to obtain the final result. The implementation process can be represented by the formula:

[0070] ,

[0071] wherein Up2 represents a 2x upsampling operation,F z from MECM, X' from SDAM.

[0072] S4, after the model training is completed, the best-performing model is obtained, the dual-phase image is input into the model, the changed area is obtained and saved as a binary image.

[0073] On multiple data sets, compared with the prior art:

[0074] Table 1 Quantitative results of comparative experiments on LEVIR-CD and S2Looking

[0075]

[0076] The results show that our method shows the best F1 and IoU indicators on the LEVIR-CD and S2Looking data sets, which is better than some SOTA methods.

[0077] Embodiment 2:

[0078] A change detection system based on feature interaction and spatio-temporal correlation, comprising:

[0079] The data processing module: after obtaining the necessary change detection data set, it is processed to improve the generalization ability of the model, and the processing process includes random flipping, translation, scaling, and cropping operation;

[0080] The model training module: the training process includes the setting of hyperparameters, the selection of data sets, and the selection and saving of the optimal model. The hyperparameters include learning rate, batch size, training times, optimizer, and loss function selection. The selection and saving of the optimal model means that the model with the best result in the training process is retained for use;

[0081] The model using module: first, the optimal model needs to be loaded, then the data to be processed needs to be selected, and at least two remote sensing images are needed. Then the predicted result is generated, and the result is saved to the local;

[0082] The application module: with the prediction result of the previous step, the result can be selected to be imported to the cloud, and a data loading link is provided for direct application in actual application, so that the calling process becomes convenient;

[0083] The system is divided into several modules according to the function, so that the system shows a simpler use mode and clearer use rules in the overall planning and scheduling process.

[0084] The above merely describes preferred examples of the present application, and is not intended to limit the present application. Those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or replace some of the technical features. Any modification, replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A change detection method based on feature interaction and spatio-temporal correlation, characterized in that, Comprising the following steps: S1, obtaining a remote sensing change detection data set, the data set should include two or more images of the same place and contain change detection labels, and dividing the data set into a training set, a validation set and a test set; S2, constructing a network model based on a double-encoder double-decoder structure, the model utilizes feature interaction and spatio-temporal correlation of double-time images, proposes a hybrid effective channel module and a spatial denoising attention module, and uses a feature fusion upsampling module to fuse the outputs from the two, and then passes them down for training, the construction idea and design details of the modules include steps S21-S23: S21, first, the hybrid effective channel module is used for processing feature interaction; second, the spatial denoising attention module is used in the process of modeling spatio-temporal dependence; finally, the outputs of the hybrid effective channel module and the spatial denoising attention module are fused and upsampled to realize jump connection; The implementation process can be represented by the formula: mix=MECM(x1,x2) S i = skip(SDAM(x i ), mix), i = 1,2 P=H(MECM(FFUM(S1),FFUM(S2))) wherein x1 represents the feature of the previous phase image, x2 represents the feature of the subsequent phase image; MECM represents the processing of the mixed effective channel module; mix represents the result after the processing of MECM; skip represents the skip connection; SDAM represents the processing of the spatial denoising attention module; i represents the phase sequence; S i represents the result after the processing of SDAM; FFUM represents the processing of the feature fusion up-sampling module; H represents the prediction head, and P represents the prediction result; S22, the spatial denoising attention module is designed based on the spatial attention module and used as a feature optimization processing operation before jump connection; the input is divided into horizontal and vertical branches; the vertical branch is used to retain the original feature information for splicing and element product operation with the horizontal branch; the horizontal branch first passes through the spatial attention module, which first splices the feature maximum pooling and average pooling, then performs feature fusion and channel number adjustment through 7x7 convolution, and obtains the attention mask through the Sigmoid activation function; then the obtained mask is subjected to element product operation with the original feature, and then subjected to concat operation with the original feature map of the vertical branch; finally, 3x3 convolution is used for feature extraction, and a feature map with the same size as the original feature is obtained; S23, a hybrid effective channel module is proposed, and the module is added to the completion of the encoding and decoding stages respectively, which contacts the context information and retains more effective channel information during double-time feature interaction; in order to fully utilize the feature information and channel correlation between the two groups of features, 3x3 group convolution is used to mix the features obtained in the last step, and then a channel attention is used to weight the important channels and enhance the channel information; S3, the feature fusion upsampling module receives the optimized features from the spatial denoising attention module and the effective channel features from the hybrid effective channel module; the process first upsamples the latter to the same resolution as the former, which uses 2x upsampling here; then the features are fused, and finally a number of convolution layers are used to enhance the feature information, thereby obtaining the final result; S4, after the model training is completed, the best-performing model is obtained, the double-time images are input into the model, the changed area is obtained and saved as a binary image.

2. The change detection method based on feature interaction and spatio-temporal correlation according to claim 1, characterized in that: In step S1, a registered multi-temporal change detection data set is used, and images with high detection difficulty are selected; the images with high detection difficulty include remote sensing images with heavy fog, shadow and complex change targets.

3. The method of claim 1, wherein: In step S21, the overall architecture of the model and each module are combined strictly according to the described process.

4. The method of claim 1, wherein: In step S22, the shallow features usually contain more noise information, which will interfere with the extracted features, cause the loss of context information, and cannot identify useful information in the subsequent detection process. Therefore, the shallow features need to be denoised. The shallow denoising attention module helps to link the context features and locate the complex targets effective for change detection tasks.

5. The method of claim 1, wherein: In step S23, the channel attention mechanism is used, which helps to show the correlation between different channels. The importance of each feature channel is automatically obtained through network learning, and different weight coefficients are assigned to each channel to enhance key features and suppress less important features, thereby improving the ability of feature representation.

6. The method and system for change detection based on feature interaction and spatio-temporal correlation according to claim 1, characterized in that: In step S23, since the two inputs are extracted by a twin network with shared weights, it means that they have the same semantics.

7. The method of claim 1, wherein: In step S3, the upsampling operation has two advantages. On the one hand, it can improve the resolution of the image, making the image clearer and smoother, thereby improving the image quality. On the other hand, it improves the classification accuracy and detail expression at the pixel level, which helps to enhance the perception ability of the neural network to image details. The use of skip connection can directly connect the low-level features with the high-level features, so that the detail information can be transmitted to the decoder, thereby improving the accuracy and detail preservation ability of the segmentation result. On the other hand, the skip connection also alleviates the problem of gradient propagation, making the network easier to optimize and the training effect better.

8. The method of claim 1, wherein: The model training and testing process should be carried out on multiple datasets to highlight the generalization ability of the method.

Images