A non-aligned remote sensing image change detection method based on residual offset feature correction

Abstract

The present application relates to a kind of non-alignment remote sensing image change detection method based on residual offset feature correction, belong to the field of remote sensing data processing.The present application is realized by designing image vertex offset prediction module, from coarse granularity to fine granularity, and the prediction accuracy of weak vertex offset of model is effectively improved by stage offset estimation.Through the design feature correction module, the spatial features of ground cover are comprehensively captured, and the model detection performance is enhanced.The present application realizes the efficient spatial feature alignment of double-phase non-alignment remote sensing image, and significantly improves the change detection performance.

Description

Technical Field

[0001] This invention relates to a method for detecting changes in unaligned remote sensing images based on residual offset feature correction, belonging to the field of remote sensing data processing. Background Technology

[0002] Remote sensing change detection tasks identify land cover change characteristics by comparing and analyzing dual-temporal remote sensing image data of the same geographic area. This technology has been widely applied in various fields such as dynamic monitoring of land resources, monitoring of urbanization processes, and emergency assessment of disaster impacts, becoming one of the core means of remote sensing data interpretation and land cover research.

[0003] Artificial intelligence technologies, especially deep learning methods, have been widely applied in remote sensing change detection tasks. However, existing algorithms are generally based on the assumption that accurate spatial registration has been achieved for bi-temporal images. In practical applications, remote sensing images acquired at different times usually originate from different satellite platforms. Due to factors such as differences in satellite orbital altitude and observation angle deviations, significant spatial misalignment exists between images. The traditional processing flow of performing image registration before change detection not only suffers from low computational efficiency but also incurs high processing costs.

[0004] Explicitly fusing image alignment and change detection within a deep learning framework has become an effective technical approach. Recently, Jing et al. proposed the ChangeRD method, which provides an innovative solution for change detection in unaligned remote sensing images. This method learns to predict vertex offsets during the optimization process and performs perspective transformation on multi-scale feature maps, thereby achieving alignment of the feature spaces of bi-temporal images. This end-to-end integrated design significantly reduces the dependence of change detection accuracy on the pre-registration step, overcoming the procedural shortcomings of traditional methods that require independent registration before detection.

[0005] However, the aforementioned methods still have limitations in addressing minute-scale land cover changes in unaligned images. In unaligned remote sensing imagery, existing methods struggle to effectively identify changes such as the addition / disappearance of smaller features like vehicles. This is because existing methods use high-level semantic features to predict image vertex offsets, but these features have low sensitivity to minute spatial displacements, resulting in spatial misalignment between the corrected image and the reference image. Furthermore, since the feature extraction process directly operates on the original unaligned image, the generated feature maps cannot accurately represent the true spatial distribution of land cover, thus limiting the effective perception of subtle changes. Therefore, existing methods still have shortcomings in processing minute-scale land cover changes in unaligned bi-temporal images. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of existing technologies, namely, the insufficient ability of current change detection algorithms to predict vertex offsets and the distortion of spatial relationship representation of ground features in non-aligned image scenarios. This invention proposes a change detection method for non-aligned remote sensing images based on residual offset feature correction. In this method, a vertex offset prediction module is designed to achieve progressive offset estimation from coarse to fine granularity, effectively improving the model's prediction accuracy for subtle vertex offsets. A feature correction module is designed to more comprehensively capture the spatial features of land cover, enhancing the model's change detection performance in non-aligned image scenarios. This patent proposes a change detection method for non-aligned remote sensing images based on residual offset feature correction. This method constructs a multi-level offset prediction mechanism from coarse to fine and combines joint perspective correction of images and multi-scale features to achieve efficient spatial alignment of dual-temporal remote sensing images under non-aligned conditions, thereby significantly improving change detection performance. This method consists of two core modules: a vertex offset prediction module and a feature correction module. The vertex offset prediction module effectively addresses the prediction difficulties caused by the dilution of image offset information in high-level semantic features. By constructing a progressively refined prediction mechanism at the feature level, it achieves step-by-step offset estimation from coarse to fine granularity, effectively improving the model's prediction accuracy for subtle vertex offsets. Simultaneously, to address the problem of distorted representation of land cover spatial relationships in misaligned image features, this patent proposes a feature correction module. This module uses parallel design of image correction and feature correction branches to perform fine-grained correction on the original distorted image and its multi-scale feature maps, respectively. The fusion of the output features from the two branches can more comprehensively capture the spatial features of land cover, thereby enhancing the model's change detection performance in misaligned image scenarios.

[0007] The technical solution of this invention is: A method for detecting changes in unaligned remote sensing images based on residual offset feature correction, the method comprising the following steps: The first step is to label the changed regions and vertex offsets of the two-phase non-aligned remote sensing images acquired by remote sensing satellites, and then use the two-phase non-aligned remote sensing images, changed region labels, and vertex offsets to form a data sample for training. The second step is to construct a deep learning network architecture, which includes a feature extraction module, a vertex offset prediction module, a feature correction module, and a change detection module. The third step is to feed the data samples generated in the first step into the deep learning network architecture designed in the second step for training. During training, non-aligned dual-temporal remote sensing images are fed into the feature extraction module to obtain dual-temporal image features; The obtained dual-temporal image features are fed into the vertex offset prediction module to obtain the vertex offset prediction value; The obtained vertex offset prediction value and the dual-temporal image features are sent to the feature correction module to correct the dual-temporal image features. The corrected features are then sent to the change detection module for detection to obtain the detection results. Training is supervised based on the labeled change regions and image vertex offsets. During training, the network parameters are iteratively optimized, and the network parameters in the deep learning network architecture are updated during the training process. After training, the trained deep learning network architecture is obtained. The fourth step involves inputting the dual-temporal unaligned remote sensing images to be detected into the deep learning network architecture trained in the third step to obtain the detection results of the change regions in the unaligned remote sensing images.

[0008] In the first step, the dual-temporal non-aligned remote sensing images are acquired by optical remote sensing satellites. Each image contains three RGB channels and has a size of 256×256 pixels. The labels for the changed areas are represented by binary values: 0 indicates no change, and 1 indicates change. The labeled image vertex offset consists of the offsets of the four vertices of the image in the horizontal and vertical directions, totaling 8 values.

[0009] In the second step, the constructed deep learning network architecture includes a feature extraction module. This module uses a three-stage Transformer as a twin feature extractor. By feeding the unaligned dual-temporal remote sensing images into the feature extraction module, dual-temporal image features can be obtained, which can be expressed by the formula:

[0010] in, Represents twin networks, An image showing that phase T1 is spatially misaligned with phase T2. The image at time T2 and This represents the features of the generated bi-temporal image.

[0011] In the second step, the constructed deep learning network architecture includes a vertex offset prediction module. This module employs three sets of independently designed prediction heads with identical structures to predict offsets at different granularities. The prediction head first convolves the features of the two temporal images and then performs channel concatenation, followed by global average pooling. Finally, it obtains the prediction result through two subsequent convolution operations.

[0012] For deep features and The prediction head generates the basic offset:

[0013] in, This represents the convolution operation. Indicates global average pooling. This indicates concatenation along the channel dimension. This is the predicted basic offset.

[0014] For mid-layer features and The prediction head generates a residual offset once:

[0015] in, It is the predicted residual offset, which can achieve fine calibration of the basic offset prediction.

[0016] For shallow features and The prediction head generates a second-order residual offset:

[0017] in, It is the predicted secondary residual offset, which can enable further fine calibration of the offset.

[0018] The final predicted offset is the sum of the above three factors, which can be expressed as: . In the second step, the constructed deep learning network architecture includes a feature correction module, which consists of two correction branches: image correction and feature correction. The features output from the two correction branches are then processed by a correction fusion module to obtain the final output features. For the feature correction branch, the image vertex offsets predicted in the second step are used... Spatial perspective transformation of the T1 phase characteristics:

[0019] in, Represents the perspective transformation matrix. This represents the function for calculating the transformation matrix. This indicates the feature correction function performed using the perspective transformation matrix.

[0020] For the image correction branch, the spatial perspective transformation is first performed on the distorted image:

[0021] Then, the corrected image The data is fed into the feature extractor for feature extraction.

[0022] Then, the output features of the feature correction branch Output features of the image correction branch Fusion is performed using a corrective fusion method:

[0023] The corrected T1 phase characteristics are obtained.

[0024] In the second step, the constructed deep learning network architecture includes a change detection module, which detects changes based on the corrected T1 time-phase features. T2 phase characteristics Through convolution, upsampling, channel splicing, and fully connected operations, the change detection results are finally obtained.

[0025] In the third step, supervision is performed based on the labeled change region tags and image vertex offsets. The supervision is divided into two parts: residual offset prediction loss and change detection loss. The residual offset prediction loss supervises the basic offset, first-order residual offset, and second-order residual offset output by the vertex offset prediction module, thereby achieving accurate prediction of image vertex offsets. Specifically, the basic offset prediction loss is:

[0026] in, The predicted basic offset, This is the true value of the offset.

[0027] The loss of a single residual offset prediction can be expressed as:

[0028] in, This is the predicted residual offset.

[0029] The second-order residual migration prediction loss can be expressed as:

[0030] in, This is the predicted offset of the second-order residual.

[0031] The residual offset prediction loss is the sum of the above three factors, and can be expressed as:

[0032] The change detection loss supervises the detection results output by the change detection module based on the labeled change regions, and adopts cross-entropy loss, which can be specifically expressed as:

[0033] in, Represents the cross-entropy loss function. This refers to the final change detection result output by the change detection module. This indicates the true value of the labeled area of ​​change.

[0034] The multi-task loss function is a weighted sum of the residual offset prediction loss and the change detection loss:

[0035] Under the supervision and guidance of the multi-task loss function, the model gradually completes parameter updates.

[0036] In the fourth step, the detection rate of the trained deep learning network architecture obtained in the third step is recorded on the test samples, and the average detection rate on all samples is calculated as the final detection result of the deep learning network architecture.

[0037] Beneficial effects 1. To address the limitation of existing remote sensing change detection technologies, which largely rely on spatially aligned images and are unsuitable for unaligned images, this invention proposes a change detection method for unaligned remote sensing images based on residual offset feature correction. By designing a vertex offset prediction module and a feature correction module in a deep learning network, efficient spatial alignment of features from dual-temporal remote sensing images under unaligned conditions is achieved, thereby significantly improving change detection performance. This effectively overcomes the shortcomings of existing technologies that rely on spatial alignment conditions, enabling change detection in satellite remote sensing images across a wider range of application scenarios.

[0038] 2. In the vertex offset prediction module, unlike the method that only uses the deepest feature for prediction, this invention uses three sets of independent prediction heads with the same structural design to predict vertex offsets from coarse to fine for bi-temporal features at different levels, which can more effectively and accurately predict the vertex offset of the image.

[0039] 3. In the feature correction module, unlike methods that only correct the perspective of intermediate layer features, this invention designs a dual-branch feature correction method, which can comprehensively capture the spatial feature information of land cover and enhance the model's change detection performance in non-aligned image scenes.

[0040] The method proposed in this invention represents a significant step forward in detecting changes in misaligned remote sensing images and provides an effective technical means for intelligent interpretation of remote sensing images. Attached Figure Description

[0041] Figure 1 This is a diagram illustrating the overall framework of the method of the present invention; Figure 2 Here is a block diagram of the vertex offset prediction module; Figure 3 This is a schematic diagram illustrating the effect of image correction using vertex offset prediction values. Figure 4 Here is a flowchart of perspective transformation calculation; Figure 5 This is a block diagram of the correction fusion method in the feature correction module. Detailed Implementation

[0042] The present invention will be further described below with reference to the embodiments.

[0043] Example A method for detecting changes in unaligned remote sensing images based on residual offset feature correction is presented. Taking the practical application scenario of detecting changes in unaligned remote sensing images as an example, the overall framework of this method is shown in the figure below. Figure 1 As shown, the steps of this method include: The first step is to label the changed regions and vertex offsets of the two-phase non-aligned remote sensing images acquired by remote sensing satellites, and then use the two-phase non-aligned remote sensing images, changed region labels, and vertex offsets to form a data sample for training. The second step is to construct a deep learning network architecture, which includes a feature extraction module, a vertex offset prediction module, a feature correction module, and a change detection module. The third step is to feed the data samples generated in the first step into the deep learning network architecture designed in the second step for training. During training, non-aligned dual-temporal remote sensing images are fed into the feature extraction module to obtain dual-temporal image features; The obtained dual-temporal image features are fed into the vertex offset prediction module to obtain the vertex offset prediction value; The obtained vertex offset prediction value and the dual-temporal image features are sent to the feature correction module to correct the dual-temporal image features. The corrected features are then sent to the change detection module for detection to obtain the detection results. Training is supervised based on the labeled change regions and image vertex offsets. During training, the network parameters are iteratively optimized, and the network parameters in the deep learning network architecture are updated during the training process. After training, the trained deep learning network architecture is obtained. The fourth step involves inputting the dual-temporal unaligned remote sensing images to be detected into the deep learning network architecture trained in the third step to obtain the detection results of the change regions in the unaligned remote sensing images.

[0044] In the first step, the dual-temporal unaligned remote sensing images were captured from optical remote sensing satellite images in Google Earth. Each image contains three RGB channels and has a size of 256×256 pixels, totaling 1000 images. The labels for the changed areas are represented by binary values: 0 indicates no change, and 1 indicates change. The labeled image vertex offset consists of the offsets of the four vertices of the image in the horizontal and vertical directions, totaling 8 values.

[0045] In the second step, the constructed deep learning network architecture includes a feature extraction module. This module uses a three-stage Transformer as a twin feature extractor. By feeding the unaligned dual-temporal remote sensing images into the feature extraction module, dual-temporal image features can be obtained, which can be expressed by the formula:

[0046] in, Represents twin networks, An image showing that phase T1 is spatially misaligned with phase T2. The image at time T2 and This represents the features of the generated bi-temporal image.

[0047] In the second step, the constructed deep learning network architecture includes a vertex offset prediction module. This module employs three sets of independently designed prediction heads with identical structures to predict offsets at different granularities. The prediction head first convolves the features of the two temporal images and then performs channel concatenation, followed by global average pooling. Finally, it obtains the prediction result through two subsequent convolution operations.

[0048] For deep features and The prediction head generates the basic offset:

[0049] in, This represents the convolution operation. Indicates global average pooling. This indicates concatenation along the channel dimension. This is the predicted basic offset.

[0050] For mid-layer features and The prediction head generates a residual offset once:

[0051] in, It is the predicted residual offset, which can achieve fine calibration of the basic offset prediction.

[0052] For shallow features and The prediction head generates a second-order residual offset:

[0053] in, It is the predicted secondary residual offset, which can enable further fine calibration of the offset.

[0054] The final predicted offset is the sum of the above three factors, which can be expressed as: . In the second step, the constructed deep learning network architecture includes a feature correction module, which consists of two correction branches: image correction and feature correction. The features output from the two correction branches are then processed by a correction fusion module to obtain the final output features. For the feature correction branch, the image vertex offsets predicted in the second step are used... Spatial perspective transformation of the T1 phase characteristics:

[0055] in, Represents the perspective transformation matrix. This represents the function for calculating the transformation matrix. This indicates the feature correction function performed using the perspective transformation matrix.

[0056] For the image correction branch, the spatial perspective transformation is first performed on the distorted image:

[0057] Then, the corrected image The data is fed into the feature extractor for feature extraction.

[0058] Then, the output features of the feature correction branch Output features of the image correction branch Fusion is performed using a corrective fusion method:

[0059] The corrected T1 phase characteristics are obtained.

[0060] In the second step, the constructed deep learning network architecture includes a change detection module, which detects changes based on the corrected T1 time-phase features. T2 phase characteristics Through convolution, upsampling, channel splicing, and fully connected operations, the change detection results are finally obtained.

[0061] In the third step, supervision is performed based on the labeled change region tags and image vertex offsets. The supervision is divided into two parts: residual offset prediction loss and change detection loss. The residual offset prediction loss supervises the basic offset, first-order residual offset, and second-order residual offset output by the vertex offset prediction module, thereby achieving accurate prediction of image vertex offsets. Specifically, the basic offset prediction loss is:

[0062] in, The predicted basic offset, This is the true value of the offset.

[0063] The loss of a single residual offset prediction can be expressed as:

[0064] in, This is the predicted residual offset.

[0065] The second-order residual migration prediction loss can be expressed as:

[0066] in, This is the predicted offset of the second-order residual.

[0067] The residual offset prediction loss is the sum of the above three factors, and can be expressed as:

[0068] The change detection loss supervises the detection results output by the change detection module based on the labeled change regions, and adopts cross-entropy loss, which can be specifically expressed as:

[0069] in, Represents the cross-entropy loss function. This refers to the final change detection result output by the change detection module. This indicates the true value of the labeled area of ​​change.

[0070] The multi-task loss function is a weighted sum of the residual offset prediction loss and the change detection loss:

[0071] Under the supervision and guidance of the multi-task loss function, the model gradually completes parameter updates.

[0072] In the fourth step, the detection rate of the trained deep learning network architecture obtained in the third step is recorded on the test samples, and the average detection rate on all samples is calculated as the final detection result of the deep learning network architecture.

[0073] The training framework used was the deep learning framework PyTorch. Training was performed on a computer equipped with an NVIDIA RTX8000 GPU, with AdamW as the training optimizer, 200 training epochs, a learning rate of 0.0001, and a batch size of 16.

[0074] The comparison results between the method proposed in this invention and existing methods are shown in Table 1. It can be seen that the method proposed in the embodiments has better accuracy.

[0075] Table 1. Comparison of classification accuracy of various method implementations on the CDD dataset.

[0076] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting changes in unaligned remote sensing images based on residual offset feature correction, characterized by the following steps: include: The first step is to label the changed regions and vertex offsets of the two-phase non-aligned remote sensing images acquired by remote sensing satellites, and then use the two-phase non-aligned remote sensing images, changed region labels, and vertex offsets to form a data sample for training. The second step is to construct a deep learning network architecture, which includes a feature extraction module, a vertex offset prediction module, a feature correction module, and a change detection module. The third step is to feed the data samples generated in the first step into the deep learning network architecture designed in the second step for training. During training, non-aligned dual-temporal remote sensing images are fed into the feature extraction module to obtain dual-temporal image features; The obtained dual-temporal image features are fed into the vertex offset prediction module to obtain the vertex offset prediction value; The obtained vertex offset prediction value and the dual-temporal image features are sent to the feature correction module to correct the dual-temporal image features. The corrected features are then sent to the change detection module for detection to obtain the detection results. Training is supervised based on the labeled change regions and image vertex offsets. During training, the network parameters are iteratively optimized, and the network parameters in the deep learning network architecture are updated during the training process. After training, the trained deep learning network architecture is obtained. The fourth step involves inputting the dual-temporal unaligned remote sensing images to be detected into the deep learning network architecture trained in the third step to obtain the detection results of the change regions in the unaligned remote sensing images. In the second step, the vertex offset prediction module in the constructed deep learning network architecture adopts three sets of independent prediction heads with the same structural design to achieve offset prediction at different granularities. The prediction head first convolves the features of the two temporal images and then performs channel concatenation, then performs global average pooling, and then obtains the prediction result through two subsequent convolution operations. For deep features and The prediction head generates the basic offset: in, This represents the convolution operation. Indicates global average pooling. This indicates concatenation along the channel dimension. This is the predicted basic offset; For mid-layer features and The prediction head generates a residual offset once: in, It is the predicted residual offset, which enables fine calibration of the basic offset prediction; For shallow features and The prediction head generates a second-order residual offset: in, It is the predicted secondary residual offset, which enables further fine calibration of the offset; The final predicted offset is the sum of the above three factors, expressed as: In the second step, the feature correction module in the constructed deep learning network architecture consists of two correction branches: image correction and feature correction. The features output by the two correction branches are fused by the correction fusion module to obtain the final output features. For the feature correction branch, the image vertex offset predicted in the second step is used. Spatial perspective transformation of the T1 phase characteristics: in, Represents the perspective transformation matrix. This represents the function for calculating the transformation matrix. This indicates the feature correction function performed using the perspective transformation matrix; For the image correction branch, the spatial perspective transformation is first performed on the distorted image: Then, the corrected image The data is fed into the feature extractor for feature extraction. Then, the output features of the feature correction branch Output features of the image correction branch Fusion is performed using a corrective fusion method: The corrected T1 phase characteristics are obtained.

2. The method for detecting changes in unaligned remote sensing images based on residual offset feature correction according to claim 1, characterized in that: In the first step, the dual-temporal non-aligned remote sensing images are acquired by optical remote sensing satellites. Each image contains three RGB channels and has a size of 256×256 pixels. The labels for the changed areas are represented by binary values: 0 indicates no change, and 1 indicates change. The labeled image vertex offset consists of the offsets of the four vertices of the image in the horizontal and vertical directions, totaling 8 values.

3. The method for detecting changes in unaligned remote sensing images based on residual offset feature correction according to claim 2, characterized in that: In the second step, the feature extraction module in the constructed deep learning network architecture uses a three-stage Transformer as a twin feature extractor. The unaligned dual-temporal remote sensing images are fed into the feature extraction module to obtain dual-temporal image features, expressed by the formula: in, Represents twin networks, An image showing that phase T1 is spatially misaligned with phase T2. The image at time T2 and This represents the features of the generated bi-temporal image.

4. The method for detecting changes in unaligned remote sensing images based on residual offset feature correction according to claim 1, characterized in that: In the second step, the change detection module in the constructed deep learning network architecture is based on the corrected T1 time-phase features. T2 phase characteristics Through convolution, upsampling, channel splicing, and fully connected operations, the change detection results are finally obtained.

5. The method for detecting changes in unaligned remote sensing images based on residual offset feature correction according to claim 4, characterized in that: In the third step, supervision is performed based on the labeled change region tags and image vertex offsets. The supervision is divided into two parts: residual offset prediction loss and change detection loss. The residual offset prediction loss supervises the basic offset, first-order residual offset, and second-order residual offset output by the vertex offset prediction module, thereby achieving accurate prediction of image vertex offsets. Specifically, the basic offset prediction loss is: in, The predicted basic offset, This is the true value of the offset; The loss for predicting a single residual offset is expressed as: in, This is the predicted residual offset. The second-order residual offset prediction loss is expressed as: in, This is the predicted second-order residual offset; The residual offset prediction loss is the sum of the above three factors, expressed as: The change detection loss supervises the detection results output by the change detection module based on the labeled change regions, and adopts cross-entropy loss, specifically expressed as follows: in, Represents the cross-entropy loss function. This refers to the final change detection result output by the change detection module. Indicates the true value of the labeled area of ​​change; The multi-task loss function is a weighted sum of the residual offset prediction loss and the change detection loss: Under the supervision and guidance of the multi-task loss function, the model gradually completes parameter updates.

6. The method for detecting changes in unaligned remote sensing images based on residual offset feature correction according to claim 5, characterized in that: In the fourth step, the detection rate of the trained deep learning network architecture obtained in the third step is recorded on the test samples, and the average detection rate on all samples is calculated as the final detection result of the deep learning network architecture.

Images