A remote sensing image change detection method based on axial shift feature interaction and prototype guidance penalty constraint

Abstract

The application discloses a remote sensing image change detection method based on axial shift feature interaction and prototype guided penalty constraint, which effectively extracts change targets in complex remote sensing scenes through the design of network architecture and loss function. By introducing continuous axial shift operations in the width and height directions, effective interaction of adjacent features can be achieved, and deep semantic representations of change targets can be extracted without reducing spatial resolution and introducing a large number of additional calculations. On this basis, by designing a prototype guided penalty constraint loss function, the model is guided to generate separable change / invariant features to obtain accurate change detection results.

Description

Technical Field

[0001] This invention relates to a method for detecting changes in remote sensing images based on axial displacement feature interaction and prototype-guided penalty constraints, belonging to the field of remote sensing image processing. Background Technology

[0002] Remote sensing change detection technology can acquire information on changes in ground features using remote sensing images taken at different times, making it a research hotspot in the field of remote sensing. Change detection technology is playing an increasingly important role in areas such as urban planning, environmental monitoring, and disaster emergency response.

[0003] Currently, deep learning technology, due to its powerful feature representation capabilities, has been widely applied to remote sensing change detection tasks. Among these, Siamese networks are a representative method, capable of extracting features from bi-temporal remote sensing images using a weight-sharing network. Based on different feature extraction methods, existing deep learning-based change detection methods can be divided into convolutional neural network (CNN)-based methods and Transformer-based methods. CNN-based methods utilize continuous convolution and pooling operations to extract multi-scale features. For example, Daudt et al. proposed three fully convolutional change detection networks: FC-EF, FC-Siam-conc, and FC-Siam-diff. Zhang et al. proposed a deep supervised image fusion network (IFN), utilizing a pre-trained VGG16 network to extract bi-temporal features. Transformer-based methods can model long-range dependencies in remote sensing images. For example, Chen et al.'s bi-temporal image Transformer model (BIT) effectively correlates long-range features in the spatiotemporal domain.

[0004] The aforementioned deep learning-based methods have significantly advanced change detection technology. Even so, change detection in complex remote sensing scenes still faces numerous challenges. For example, due to factors such as season and illumination, the radiometric differences between different features within the same remote sensing scene are small, making it difficult to separate changing targets from the background. Utilizing the interaction of local features to extract deep semantic representations is an effective means of identifying changing targets, but convolution and pooling operations in convolutional neural networks reduce spatial resolution, and the self-attention mechanism in Transformers has low computational efficiency, thus existing methods struggle to achieve satisfactory results. Furthermore, the difficulty in distinguishing changing targets from the background often stems from the high degree of confusion between changed and unchanged features in the feature space, posing a significant challenge to subsequent classifiers in generating accurate change detection results. Summary of the Invention

[0005] To address the challenges of effective local feature interaction and confusion between changed and invariant features in existing remote sensing change detection methods, this invention proposes a remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraints. This method introduces continuous axial displacement operations in the width and height directions, enabling effective interaction of neighboring features. It extracts deep semantic representations of changed targets without reducing spatial resolution or introducing significant additional computation. Furthermore, a prototype-guided penalty constraint loss function guides the model to generate separable changed / invariant features, thereby obtaining accurate change detection results.

[0006] The technical solution of this invention is:

[0007] A remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint, the method comprising the following steps:

[0008] Step 1: Acquire dual-temporal remote sensing images and input them into the twin encoder M. S Obtain dual-temporal multi-level feature maps {f 11 ,f 12 ,f 13 ,...,f1n} and {f 21 ,f 22 ,f 23 ,...,f2n}, and set {f 11 ,f 12 ,f 13 ,...,f1n} and {f 21 ,f 22 ,f 23 The features are fed into the cross-temporal feature enhancement module to enhance the features and obtain the enhanced feature map. and

[0009] Step two: The enhanced features are fed into the decoder. The decoder then processes the last layer of bi-temporal feature maps. and Channel-dimensional concatenation is performed, followed by convolution, axis-shifted feature interaction module, and upsampling to obtain the interacted features. For all biphase feature maps except the last layer, they are concatenated with the interactive features obtained from the next layer, then processed through convolution, axis-shifted feature interaction module, and upsampling, and this process is repeated until the interactive features corresponding to the second layer feature map are obtained. Calculation complete. With the first layer of dual-temporal features and The channels are concatenated together, then convolved to obtain the decoded feature fd output by the decoder.

[0010] Step 3: The decoded feature file descriptors (fd) obtained in Step 2 are fed into the prototype-guided penalty constraint function. Easily confused pixel feature samples in the fd are adaptively weighted. The prototype-guided penalty constraint function and the cross-entropy loss function are then used simultaneously to optimize both the encoder and decoder, completing the remote sensing change detection task. The overall loss function of the model is:

[0011] L = L CE +L PGPC

[0012] Among them, L CE and L PGPC These represent the cross-entropy loss function and the prototype-guided penalty constraint loss function, respectively.

[0013] In step one, a twin network refers to two convolutional neural networks that share parameters;

[0014] In step one, the cross-temporal feature enhancement module first extracts the channel and spatial weights of the two temporal features using mean pooling. Then, it concatenates the channel and spatial weights obtained from the two temporal phases and extracts the channel and spatial weights of common interest in both phases using fully connected layers and convolutional layers. Finally, the enhanced features are obtained through channel and spatial fusion.

[0015] In step two, the axis-shift feature interaction module first divides the data into m groups through a block operation along the channel dimension:

[0016]

[0017] in, It refers to the features after grouping (where m is an odd number), and C(·) refers to the block operation on the channel dimension;

[0018] The grouped features undergo axial displacement operations of varying degrees in the width direction:

[0019]

[0020] Where S(·) represents the axial displacement operation, and i-(m+1) / 2 represents the distance and direction of the axial displacement (its value represents the distance of displacement, and its sign represents the direction of displacement).

[0021] Similarly, axial displacement operations of different degrees in the height direction can be represented as:

[0022]

[0023] Where j-(m+1) / 2 also represents the distance and direction of axial displacement.

[0024] Therefore, the final output of the axis-shift feature interaction module can be written as:

[0025]

[0026] In step three, the prototype-guided penalty constraint loss is prototype-oriented, measuring the similarity between each feature and its corresponding prototype, and adaptively constraining features that differ significantly from the prototype. This is first achieved by decoding feature f. d The mean of the changing / invariant features is used to calculate the changing / invariant prototype:

[0027]

[0028] Among them, Ω c and Ω n ...

[0029] For each pixel feature in the feature space, the corresponding prototype is:

[0030]

[0031] Based on this, the average similarity between the changing / invariant features and their corresponding prototypes is:

[0032]

[0033] Where sim{·} represents the cosine similarity function.

[0034] Therefore, the positive average similarity for each pixel is:

[0035]

[0036] Based on the relative relationship between the actual similarity and the average similarity between each pixel feature and the prototype, adaptive weighting can be applied to easily confused feature samples that differ significantly from the prototype:

[0037]

[0038] Finally, the prototype-guided penalty constraint loss function can be expressed as:

[0039]

[0040] Where Ω represents the set of all pixel features.

[0041] Beneficial effects

[0042] Compared with existing technologies, this invention proposes a remote sensing image change detection method based on axial shift feature interaction and prototype-guided penalty constraint for the task of remote sensing image change detection. This method consists of an axial shift feature interaction method and a prototype-guided penalty constraint loss function to improve the performance of the change detection model. The proposed axial shift feature interaction method learns the intrinsic relationship between adjacent pixel features through interaction between adjacent features. The proposed prototype-guided penalty constraint loss function can identify confused pixel features and adaptively weight them to obtain accurate change detection results. The method proposed in this invention represents a significant step forward in achieving a high-precision change detection model and provides an effective technical means for intelligent interpretation of remote sensing images. Attached Figure Description

[0043] Figure 1 Overall schematic diagram of the method;

[0044] Figure 2 Schematic diagram of the cross-temporal feature enhancement module;

[0045] Figure 3 A schematic diagram of the axis-shift feature interaction module;

[0046] Figure 4 Schematic diagram of prototype-guided penalty constraint loss. Detailed Implementation

[0047] The present invention will be further described below with reference to the embodiments.

[0048] Example

[0049] A remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint, taking remote sensing change detection as a practical application scenario, includes the following steps:

[0050] Step 1: Acquire dual-temporal remote sensing images and input them into the twin encoder M. S In the middle, the twin encoder M S Using a VGG16 network pre-trained on the ImageNet dataset, four levels of biphase feature maps {f 11 ,f 12 ,f 13 ,f 14} and {f 21 ,f 22 ,f 23 ,f 24}, and {f 11 ,f 12 ,f 13 ,f 14} and {f 21 ,f22 ,f 23 ,f 24 The data is fed into the cross-temporal feature enhancement module for feature enhancement, resulting in an enhanced feature map. and

[0051] Step two: The enhanced features are fed into the decoder. The decoder then processes the last layer of bi-temporal feature maps. and Channel-dimensional concatenation is performed, followed by convolution, axis-shifted feature interaction module, and upsampling to obtain the interacted features. For other biphase feature maps except the last layer (including f) 12 f 22 and f 13 f 23 This process involves concatenating the features obtained from the previous layer with the interacted features obtained from the next layer, then performing convolution, axis-shifted feature interaction module, and upsampling, and so on, until the interacted features corresponding to the second layer feature map are obtained. Calculation complete. With the first layer of dual-temporal features and The channels are concatenated together, then convolved to obtain the decoded feature fd output by the decoder.

[0052] Step 3: The decoded feature file descriptors (fd) obtained in Step 2 are fed into the prototype-guided penalty constraint function. Easily confused pixel feature samples in the fd are adaptively weighted. The prototype-guided penalty constraint function and the cross-entropy loss function are then used simultaneously to optimize both the encoder and decoder, completing the remote sensing change detection task. The overall loss function of the model is:

[0053] L = L CE +L PGPC

[0054] Among them, L CE and L PGPC These represent the cross-entropy loss function and the prototype-guided penalty constraint loss function, respectively.

[0055] Figure 1 This is a schematic diagram of a remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraints.

[0056] In step one, a twin network refers to two convolutional neural networks that share parameters;

[0057] In step one, the cross-temporal feature enhancement module first extracts the channel and spatial weights of the two temporal features using mean pooling. Then, it concatenates the channel and spatial weights obtained from the two temporal phases and extracts the channel and spatial weights of common interest in both phases using fully connected layers and convolutional layers. Finally, the enhanced features are obtained through channel and spatial fusion.

[0058] In step two, the axis displacement feature interaction module is as follows: Figure 2 As shown, the data is first divided into 5 groups through a block operation along the channel dimension:

[0059]

[0060] in, These are the features after grouping, where C(·) refers to the block operation on the channel dimension;

[0061] The grouped features undergo axial displacement operations of varying degrees in the width direction:

[0062]

[0063] Where S(·) represents the axial shift operation, and i-3 represents the distance and direction of the axial shift (its value represents the distance of the shift, and its sign represents the direction of the shift).

[0064] Similarly, axial displacement operations of different degrees in the height direction can be represented as:

[0065]

[0066] Here, j-3 also represents the distance and direction of axial displacement.

[0067] Therefore, the final output of the axis-shift feature interaction module can be written as:

[0068]

[0069] In step three, the prototype-guided penalty constraint loss is as follows: Figure 3 As shown, this approach is prototype-oriented, measuring the similarity between each feature and its corresponding prototype, and adaptively constraining features that differ significantly from the prototype. This is first achieved by decoding feature f. d The mean of the changing / invariant features is used to calculate the changing / invariant prototype:

[0070]

[0071] Among them, Ω c and Ω n ...

[0072] For each pixel feature in the feature space, the corresponding prototype is:

[0073]

[0074] Based on this, the average similarity between the changing / invariant features and their corresponding prototypes is:

[0075]

[0076] Where sim{·} represents the cosine similarity function.

[0077] Therefore, the positive average similarity for each pixel is:

[0078]

[0079] Based on the relative relationship between the actual similarity and the average similarity between each pixel feature and the prototype, adaptive weighting can be applied to easily confused feature samples that differ significantly from the prototype:

[0080]

[0081] Finally, the prototype-guided penalty constraint loss function can be expressed as:

[0082]

[0083] Where Ω represents the set of all pixel features.

[0084] Taking remote sensing change detection as an example, a comparison of various existing remote sensing change detection methods on multiple public remote sensing datasets is shown in Table 1. It can be seen that the method proposed in the embodiments has better accuracy.

[0085]

[0086]

[0087] Table 1 compares the classification accuracy of various existing change detection methods and implementations on the WHU-CD and LEVIR-CD datasets.

[0088] In summary, the above are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint, characterized in that, The method includes the following steps: Step 1: Acquire dual-temporal remote sensing images. Input the dual-temporal remote sensing images into a twin encoder to obtain dual-temporal multi-level feature maps. Then, send the feature maps to the cross-temporal feature enhancement module for feature enhancement to obtain the enhanced feature maps. The cross-temporal feature enhancement module performs feature enhancement using the following methods: First, channel and spatial weights of the features from the two time phases are extracted separately using mean pooling. Second, the channel and spatial weights obtained from the two time phases are concatenated, and fully connected layers and convolutional layers are used to extract the channel and spatial weights of common interest in both time phases. Finally, channel and spatial weights are fused to obtain the final enhanced features. Step 2: The enhanced feature map is fed into the decoder. In the decoder, the last layer of dual-temporal feature maps is concatenated along the channel dimension, then processed through convolution, the axis-shift feature interaction module, and upsampling to obtain the interacted features. For other dual-temporal feature maps except the last layer, they are concatenated along with the interacted features obtained from the next layer, then processed through convolution, the axis-shift feature interaction module, and upsampling, and so on, until the interacted features corresponding to the second layer feature map are calculated. These are then concatenated along with the first layer of dual-temporal features, and then processed through convolution to finally obtain the decoded features output by the decoder. The axis-shift feature interaction module performs feature interaction using the following method: First, the features are divided into several groups through block operations in the channel dimension; second, the grouped features are axially shifted to different degrees in the width direction; third, the grouped features are axially shifted to different degrees in the height direction; finally, the output of the axis-shifted feature interaction module is the feature set after axial shifting operations in the width and height directions. Step 3: Feed the decoded features into the prototype-guided penalty constraint function, adaptively weight easily confused pixel feature samples, and simultaneously optimize the encoder and decoder using the prototype-guided penalty constraint function and cross-entropy loss function to complete the remote sensing change detection task; the prototype-guided penalty constraint function is calculated using the following method: First, the variable / invariant prototype is calculated by the mean of the variable / invariant features in the decoded features; second, the similarity between each feature and its corresponding prototype is measured, and features that differ greatly from the prototype are adaptively constrained; finally, based on the relative relationship between the actual similarity and the average similarity between each pixel feature and the prototype, easily confused feature samples that differ greatly from the prototype are adaptively weighted.

2. The remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint as described in claim 1, characterized in that, The twin encoder consists of two convolutional neural networks that share parameters.

3. The remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint as described in claim 1, characterized in that, The axial shift operation includes translating the feature by a specified distance and direction, where the distance value represents the distance of the shift and the sign represents the direction of the shift.

4. The remote sensing image change detection method based on axial displacement feature interaction and prototype-guided penalty constraint as described in claim 1, characterized in that, The prototype-guided penalty constraint loss function is: ； in, A set representing all pixel features.

Images