Remote sensing image change detection method and system guided by hierarchical supervision signal

Abstract

The invention discloses a remote sensing image change detection method and system guided by a hierarchical supervision signal. The method comprises the following steps: respectively inputting a first image and a second image into a conjoined learning branch to obtain first local representation information and second local representation information; inputting the spliced image into a residual learning branch to obtain refined local features; inputting the difference image into a self-attention learning branch to obtain a global feature; fusing the first local representation information, the second local representation information, the residual local feature and the global feature to obtain a fused feature; mapping the fusion features, inputting the mapped fusion features into a high-order attention module, and carrying out feature refinement on the fusion features as hierarchical supervision data to obtain refined fusion features; and carrying out feature mapping and nonlinear mapping on the refined fusion features to obtain a two-channel feature map, and classifying the two-channel feature map to obtain a binary change detection result. The accuracy of remote sensing image change detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of high-resolution remote sensing image change detection technology, and in particular to a method and system for remote sensing image change detection guided by hierarchical supervision signals. Background Technology

[0002] Change detection (CD) is a primary means of timely and accurately identifying changes in Earth's surface features, understanding the interaction between humans and nature, and a crucial scientific tool for supporting decision-making. Image-based CD uses remote sensing imagery as its data source and employs mathematical, computer, or engineering techniques to study temporal differences in regions of interest. Its applications are wide-ranging, including disaster information extraction and assessment, urban land use change surveys, arable land resource management, and environmental monitoring. It also holds significant importance for global ecological research and the study of the dynamic evolution of human society.

[0003] Driven by artificial intelligence, computer technology, and aerospace technology, the theoretical methods in the field of remote sensing image processing have been continuously iterated and upgraded. In particular, the successful application of deep learning (DL) technology in this field has led to the emergence of a large number of DL-based models in scene classification, object detection, semantic segmentation, and CD tasks. DL-based models have powerful feature extraction and semantic information representation capabilities, and can effectively cope with the complexity of HRSIs, thus becoming the mainstream method for CD tasks.

[0004] Siamese-based, ResNet-based, and Transformer-based networks have each achieved considerable CD (Conceptual Data Encoding) results with their unique data modeling methods. However, the diverse high-resolution remote sensing images acquired by different satellite sensors place higher demands on CD models. Meanwhile, network models with brain-like cognitive patterns and multi-head, multi-level, and high-order abstraction capabilities, along with better generalization performance, are scarce in the CD field. Furthermore, remote sensing images differ from ordinary pictures; they possess unique properties, such as more complex spectral, textural, geometric, contextual, and semantic features, as well as domain-specific land cover association attributes. These necessitate a comprehensive consideration of the source data from multiple dimensions. However, most current methods tend to use simple early and late fusion methods, extracting image features through single or dual encoding paths, lacking a multi-faceted understanding and modeling of the source data. Summary of the Invention

[0005] This invention provides a hierarchical supervision signal-guided remote sensing image change detection method and system, which can solve the problems of single data modeling method and low feature fusion degree, resulting in poor applicability to different remote sensing data, poor recognition of different semantic targets, and limited accuracy.

[0006] This invention provides a method for detecting changes in remote sensing images guided by hierarchical supervision signals, comprising: Acquire dual-temporal remote sensing images: a first image and a second image; stitch the first image and the second image together to obtain a stitched image; perform differential processing on the first image and the second image to obtain a differential image. The first image and the second image are respectively input into the conjoint learning branch to obtain the first local representation information and the second local representation information. The stitched image is input into the residual learning branch to obtain refined local features; The difference image is input into the self-attention learning branch to obtain global features; The first local representation information, the second local representation information, the residual local features, and the global features are fused to obtain the fused features; The fused features are mapped and then input into a higher-order attention module as hierarchical supervision data to refine the fused features, resulting in refined fused features. The refined fused features are processed through feature mapping and nonlinear mapping to obtain a two-channel feature map. The two-channel feature map is then classified to obtain the binary change detection result.

[0007] Furthermore, according to the above-mentioned hierarchical supervision signal-guided remote sensing image change detection method, the conjoined learning branch includes two sub-paths with shared weights, which are used to process the first image and the second image respectively; each sub-path includes a conjoined module and a lightweight convolutional attention module; The processing of the sub-path includes: The first image or the second image is input into the conjoined module to obtain the first convolutional feature or the second convolutional feature; The first or second convolutional feature is input into a lightweight convolutional attention module for spatial and channel attention refinement operations to obtain the first or second local representation information.

[0008] Furthermore, according to the above-mentioned hierarchical supervision signal-guided remote sensing image change detection method, the residual branch includes a feature mapping module and a residual block, the residual block includes multiple processing layers, and each processing layer is sequentially connected to a resampling module and a lightweight convolutional attention module; The stitched image is input into the residual learning branch to obtain refined local features, which are expressed by the following formula:

[0009] in, This represents the three-channel feature map obtained through the feature mapping module. Indicates the residual block at the th Layer operations, This represents a 1×1 convolution operation. Indicates the first The feature map of the layer is resampled to a preset size. To refine local features, This is the input to the (j-1)th layer residual block, which is also the input to the jth layer residual block.

[0010] Furthermore, according to the above-mentioned hierarchical supervision signal-guided remote sensing image change detection method, the self-attention learning branch includes a self-attention learning module, the self-attention learning module includes multiple processing layers, and each processing layer is also connected to a resampling module; The difference image is input into the self-attention learning branch to obtain global features, which are expressed by the following formula:

[0011] in, For the input features of the j-th self-attention learning module, This indicates that the self-attention learning branch is in the first... Layer operations, Indicates the first The feature map of the layer is resampled to a preset size.

[0012] Furthermore, according to the above-mentioned hierarchical supervision signal-guided remote sensing image change detection method, the connected module is based on the 5 layers before the last pooling layer of the pre-trained VGG16 model, the residual block is based on all layers before the average pooling layer of the pre-trained ResNet18 model, each layer corresponds to a processing layer of the residual block, and the self-attention learning module is based on the pre-trained PVTv2 model, and a processing layer is added at the front end of the PVTv2 model, the structure of which is the same as the first layer of the PVTv2 model. The connected module, the residual block, and the self-attention learning module have the same number of layers.

[0013] Furthermore, according to the above-mentioned hierarchical supervision signal-guided remote sensing image change detection method, the first local representation information, the second local representation information, the residual local features, and the global features are fused to obtain fused features, which are expressed by the following formula:

[0014] in, This is the first local representation information output by the j-th layer of the conjoined module. This refers to the second local representation information output by the j-th layer of the conjoined module. This represents the refined local features output from the j-th layer of the residual block. The global features output by the j-th layer of the self-attention learning module. For splicing operations, This is a feature of fusion.

[0015] Furthermore, according to the aforementioned hierarchical supervision signal-guided remote sensing image change detection method, the fused features are mapped and input into a higher-order attention module as hierarchical supervision data to refine the fused features, resulting in refined fused features, expressed by the following formula:

[0016] in, This is the fusion feature of layer 5. For a 3×3 convolution operation, This is the high-level semantic feature map of layer j, which is derived from the refined fused feature map of the previous layer. Obtained through upsampling, For the fusion features of the j-th layer, This indicates a high-order attention feature refinement operation. This represents the refined fusion feature of the j-th layer.

[0017] Furthermore, according to the aforementioned hierarchical supervision signal-guided remote sensing image change detection method, the method is applied to a remote sensing image change detection model, which is trained based on a total loss function, wherein the total loss function is:

[0018] in, For the total loss function, All are weighted parameters. Let be the cross-entropy loss function of the j-th layer. For the domain loss function, This is the sieve loss function.

[0019] Furthermore, according to the aforementioned hierarchical supervision signal-guided remote sensing image change detection method, the cross-entropy loss function is:

[0020] in, This represents the probability that the prediction is positive. Indicates the labels of real samples; The domain loss function is:

[0021] in, This is the adjustment coefficient for the positive and negative sample weights. To focus parameters, To predict the probability that a sample is a positive class, and Modulation factor; The loss function of the sieve is:

[0022] in, This is a two-channel feature map. For the corresponding truth value label, This represents the intersection operation. This indicates the union operation.

[0023] The present invention also provides a hierarchical supervision signal-guided remote sensing image change detection system, comprising: The acquisition and preprocessing module is used to acquire dual-temporal remote sensing images: a first image and a second image, stitch the first image and the second image together to obtain a stitched image, and perform differential processing on the first image and the second image to obtain a differential image. The conjoined learning branch is used to input the first image and the second image into the conjoined learning branch respectively to obtain the first local representation information and the second local representation information; The residual learning branch is used to input the stitched image into the residual learning branch to obtain refined local features; The self-attention learning branch is used to input the difference image into the self-attention learning branch to obtain global features; The fusion module is used to fuse the first local representation information, the second local representation information, the residual local features, and the global features to obtain fused features; A higher-order attention module is used to input the mapped fused features into the higher-order attention module as hierarchical supervision data to refine the fused features and obtain the refined fused features. The change detection module is used to obtain a two-channel feature map by performing feature mapping on the refined fused features, and to classify the two-channel feature map to obtain a binary change detection result.

[0024] This invention provides a hierarchical supervised signal-guided remote sensing image change detection method and system. The invention offers four encoding branches: a conjoint learning branch, a residual learning branch, and a self-attention learning branch. Different encoding methods are equivalent to different processing ends; their integration enhances the network's feature representation capabilities, forming a local-global-cross-domain data modeling approach, thus enabling the network to possess powerful data perception and mining capabilities. The decoding end aggregates semantic information from each layer and integrates it into a unified linearized feature mapping module, achieving full modeling of the separability of target domain information. This invention proposes a novel high-order attention mechanism to adaptively refine the features of each layer at each encoding end, selectively guiding the network to focus on the target region, ensuring the boundary integrity and internal compactness of the detection results. The hierarchical network supervision mode designed in this invention adds supervised signals at different feature abstraction levels to apply differentiated soft constraints to each layer of the network, ensuring high semantic consistency of features at each layer and promoting rapid network fitting. Compared with existing technologies, this invention has the following advantages: (1) The proposed novel remote sensing image change detection network uses three learning paths (Siamese-learning path, Residual-learning path, and Transformer-learning path) to perform multi-dimensional representation of images through multi-end data modeling, which can fully explore the feature information of remote sensing images from low to high levels. Features of multiple abstract levels from multiple paths converge in the composite decoding path, constructing a stacked data form describing the image information domain, thereby improving the accuracy of CD tasks and robustness to small samples; (2) A high-order attention module, namely the High-Order Attention Module (HOAM), was designed to refine features from low to high dimensions. HOAM is a novel attention mechanism that can refine features from low to high dimensions, perform multiple enhancements on features related to changes, and enhance the network's sensitivity to target regions. Furthermore, HOAM is a flexible module that can be easily embedded into any other end-to-end network. (3) The proposed Hierarchical Supervised Training Model (HSM) is a novel model that uses a specific loss function to effectively supervise different abstraction layers of the network. HSM not only focuses on the boundary accuracy and compactness of the output prediction results, but also on the general accuracy of the deep prediction probabilities of the network, which can provide targeted guidance to the network and achieve rapid fitting and accuracy improvement. Moreover, this model can also be applied to end-to-end networks. Attached Figure Description

[0025] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0026] Figure 1A flowchart of a hierarchical supervision signal-guided remote sensing image change detection method provided in an embodiment of the present invention.

[0027] Figure 2 The flowchart illustrates the processing of the conjoined learning branch provided in this embodiment of the invention.

[0028] Figure 3 This is a flowchart illustrating the processing of residual learning branches provided in an embodiment of the present invention.

[0029] Figure 4 The flowchart illustrates the processing of the self-attention learning branch provided in this embodiment of the invention.

[0030] Figure 5 This is a flowchart illustrating the processing of a high-order attention module provided in an embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram of the structure of a hierarchical supervision signal-guided remote sensing image change detection system provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] While numerous convolutional and transformer-based methods have achieved significant success in high-resolution remote sensing image change detection, the inherent diversity and complexity of high-resolution remote sensing images pose ongoing challenges to CD methods. These challenges include non-ceiling-angle imaging, seasonal changes, and simultaneous changes in multiple ground cover types. Ordinary convolutional or transformer-based encoder-decoder networks suffer from limited applicability to different remote sensing data due to their simplistic data modeling methods and low feature fusion levels. Furthermore, they exhibit poor recognition of different semantic targets, resulting in limited accuracy. This invention discloses a hierarchical supervised signal-guided remote sensing image change detection method. (See also...) Figure 1 , Figure 1 The flowchart illustrates a hierarchical supervision signal-guided remote sensing image change detection method provided in this embodiment of the invention. This method includes the following steps: S1, acquire dual-temporal remote sensing images: first image and second image, stitch the first image and second image together to obtain a stitched image, and perform differential processing on the first image and second image to obtain a differential image.

[0034] S2, input the first image and the second image into the conjoined learning branch respectively to obtain the first local representation information and the second local representation information.

[0035] Figure 2 The flowchart of the processing of the conjoined learning branch provided in the embodiment of the present invention is as follows: Figure 2 As shown, the Siamese-learning branch includes two sub-paths with shared weights, which are used to process the first image and the second image respectively; each sub-path includes a Siamese module (Siamese_Block) and a lightweight convolutional attention module (CBAM).

[0036] The subpath processing includes: S21, input the first image or the second image into the conjoined module to obtain the first convolutional feature or the second convolutional feature.

[0037] The Siamese Block is based on the five layers preceding the last pooling layer of the pre-trained VGG16 model. Images from two different time phases (the first and second images) are input into the sub-path, then first pass through the Siamese Block, which converts them from RGB color space data format. Data form mapped to a high-level feature space (( , It's worth noting that the basic structure of Siamese_Block, VGG16, is a CNN network. Its 13 convolutional layers and 5 pooling layers are used for feature extraction, capable of extracting general features of the three-channel HRSI. Furthermore, many of its internal convolutional layers use small kernels, allowing for more non-linear mapping and extracting richer local details from the image, thus increasing the network's expressive power. To maintain consistent feature map dimensions across branches, control the number of network parameters, and improve convergence speed, a feature mapping layer is added after the output of each Siamese_Block layer.

[0038] S22, input the first convolutional feature or the second convolutional feature into the lightweight convolutional attention module to perform spatial and channel attention refinement operations to obtain the first local representation information or the second local representation information.

[0039] Furthermore, to enhance important information and weaken secondary information in the feature maps, a lightweight convolutional attention module (CBAM) is introduced. This module calculates layer-level and pixel-level attention coefficients at both the channel and spatial levels, enabling the network to distinguish "which" and "where" is best when selecting output features from each layer. The Siamese-learning path can extract representative features from two temporal high-resolution remote sensing images without information loss, learning an image-to-image consistency representation for the CD task, thus providing rich local detail information for the final CD task.

[0040] Specifically, the five-layer structure of the connected module is as follows: First layer: Convolutional layer-Convolutional layer-Pooling layer-Feature mapping layer-CBAM; Second layer: Convolutional layer-Convolutional layer-Pooling layer-Feature mapping layer-CBAM; Third layer: Convolutional layer-Convolutional layer-Convolutional layer-Pooling layer-Feature mapping layer-CBAM; Fourth layer: Convolutional layer-Convolutional layer-Convolutional layer-Pooling layer-Feature mapping layer-CBAM; Fifth layer: Convolutional layer-Convolutional layer-Convolutional layer-Pooling layer-Feature mapping layer-CBAM.

[0041] Output local representation information of each layer of the conjoined module (i.e., first local representation information) Second local characterization information The result requires three steps of calculation, as shown in the following formula:

[0042] in, This represents the first or second image input to the j-th layer; This indicates that Siamese_Block is in the... The layer operations include different numbers of convolution operations, ReLU non-linear activation operations, and max pooling operations; Indicates the kernel size as Feature mapping operations; This indicates the finer-grained operation of spatial and channel attention.

[0043] Specifically, The formulas for refining spatial and channel attention are as follows:

[0044]

[0045] in, This represents the feature output after channel attention refinement; This represents the final output of CBAM; Indicates the process The output after the feature mapping module; This indicates that the output feature map size is max pooling operation, This indicates an average pooling operation; Multi-layer sensing operations representing weight sharing; Indicates the kernel size as Feature mapping operations; This represents the Sigmoid activation function.

[0046] S3, input the stitched image into the residual learning branch to obtain refined local features.

[0047] Figure 3 The flowchart of the residual learning branch provided in the embodiment of the present invention is as follows: Figure 3 As shown, the residual branch (Residual-learning) includes a feature mapping module and a residual block (Residual_Block). The residual block includes multiple processing layers, each of which is connected to the resampling module and the lightweight convolutional attention module in sequence.

[0048] This path is responsible for extracting variation features from a 6-channel image (stitched image) resulting from the overlay of two bi-temporal remote sensing images. Specifically, this is a paradigm that treats two high-resolution remote sensing images as a whole, extracts features, and then performs a classification task. While a complex data distribution can be approximated gradually through multiple non-linear layers, the accuracy saturation problem caused by the increasing depth of the network is not negligible. Therefore, to fully extract low, medium, and high-level features from the overlay image using a deep network without increasing the computational burden on the network, and to ensure the value of these features for the final CD task, the overlay of two high-resolution remote sensing images results in a 6-channel color space data format. To adapt to the input of the pre-trained ResNet18, go through After processing by the feature mapping layer, a 3-channel feature map is obtained. After that, Residual_Block will... Data form mapped to a high-level feature space The feature information extracted by the residual-learning path can characterize the overall differences between two temporal images. Furthermore, this learning path is also an effective way to mine valuable coupled variation information for the CD task. In the residual-learning path, besides the output features of the first layer... In addition, the feature maps output by other layers are obtained by processing the two residual modules of that layer. The high-level semantic information and low-level spatial information are fused and then processed by CBAM to obtain the final output.

[0049] The residual blocks are structured based on all layers preceding the average pooling layer of the pre-trained ResNet18, extracting features at each level from the overlaid high-resolution remote sensing images. Furthermore, a resampling module is used to restore the feature maps output by each layer of the Residual_Block to a preset size. A lightweight convolutional attention module (CBAM) is also configured at the end of each layer to enhance the features.

[0050] Specifically, the residual module includes 5 processing layers, with the following structures: First processing layer: convolutional layer - pooling layer - feature mapping layer - resampling module - CBAM; Second processing layer: four convolutional layers - resampling module - CBAM; Third processing layer: four convolutional layers - resampling module - CBAM; Fourth processing layer: three convolutional layers - resampling module - CBAM; Fifth processing layer: four convolutional layers - resampling module - CBAM.

[0051] In this layer, the four convolutional layers of each processing layer are grouped in pairs (the first and second convolutional layers of the third processing layer form the first group, and the last convolutional layer forms the second group). Each processing layer includes two skip connection modules. In each processing layer, the output of the first group is used as the input of the second group, and skip connections are used to input the input of the first group into the second group as well. Then, the output of the second group is used as the input of the first group of the next layer, and skip connections are used to input the input of the second group into the first group of the next layer as well.

[0052] It is worth noting that in each processing layer, the features following the second skip connection in the output contain spatial details from earlier layers, as well as more abstract and less localized high-level features from later layers. These features are fed into CBAM, allowing the fused features at different depths to be further refined at the pixel and dimensional levels.

[0053] Obtain refined local features (Right now The calculation is as follows:

[0054] in, This represents the three-channel feature map obtained through the feature mapping module. Indicates the residual block at the th Layer operations, This represents a 1×1 convolution operation in the feature mapping module. Indicates the first The feature map of the layer is resampled to a preset size. To refine local features, This is the input to the (j-1)th layer residual block, which is also the input to the jth layer residual block.

[0055] S4. Input the difference image into the self-attention learning branch to obtain global features.

[0056] Figure 4 The flowchart of the self-attention learning branch provided in the embodiments of the present invention is as follows: Figure 4 As shown, the self-attention learning branch (Transformer-learning) includes a self-attention learning module (Transformer_Block), which comprises multiple processing layers, each of which is also connected to a resampling module. The self-attention learning module is based on a pre-trained PVTv2 model and adds a processing layer at the very front of the PVTv2 model, the structure of which is the same as the first layer of the PVTv2 model.

[0057] Specifically, the self-attention learning module includes five processing layers, with the following structures: First processing layer: overlapping block - multiple PVTv2 Blocks - resampling module; Second processing layer: overlapping block - multiple PVTv2 Blocks - resampling module; Third processing layer: overlapping block - multiple PVTv2 Blocks - resampling module; Fourth processing layer: overlapping block - multiple PVTv2 Blocks - resampling module; Fifth processing layer: overlapping block - multiple PVTv2 Blocks - resampling module.

[0058] This path is responsible for extracting change information from the differential images. Generally, bi-temporal images used in CD programs must undergo rigorous radiometric correction and registration. However, errors are unavoidable; in addition to residual radiometric errors, the corresponding pixels between the two bi-temporal images used in CD tasks are not always perfectly aligned. Even commonly used public datasets have this problem. Furthermore, some datasets only provide images from both temporal phases without calibrating pixel brightness values ​​(brightness differences caused by variations in the solar zenith angle). High-resolution remote sensing images, with their even higher resolution, are more sensitive to these issues. Therefore, to reduce the impact of pre-processing on subsequent CD tasks, it is necessary to model the asymmetry between bi-temporal images from the outset.

[0059] In this approach, differential images are used as input. The global modeling capabilities of the Transformer are leveraged to simulate the pixel correspondences between two temporal images and the global dependencies between pixels. Furthermore, the Transformer possesses strong representational capabilities; it can mine global semantic relationships within the differential images, representing changes of interest and identifying changed regions while excluding irrelevant ones, all while allowing for errors. The basic modules of the pre-trained PVTv2 model are redesigned and incorporated as fundamental components of the Transformer_Block, responsible for extracting change information from the differential images. The HRSI values ​​of the two color spaces are subtracted and their absolute values ​​are taken to obtain the output of the differential feature space. This refers to the input of `Transformer_Block`. Because of the patch embedding operation used by `Transformer` when processing visual tasks, each layer of it abstracts the image at a deeper level than a CNN. That is, starting from the self-attention mechanism, each layer's output... Both have smaller dimensions, and the information contained in the feature maps is more globally representative. To retain more low-level features of the difference images and to adapt to the structures of the other two learning paths, an additional layer was added at the very front of the PVTv2 feature extraction structure. The feature map dimension of this layer was set to 32, and other configurations were the same as the first layer of PVTv2. Therefore, Transformer_Block contains a total of 5 layers, with the first two layers simultaneously receiving... The Transformer-learning path is used to model general global information. Subsequent layers receive the output features from the previous layer to extract deeper global representations. Similar to the Residual-learning path, each feature map in the Transformer-learning path needs to be processed by a resampling module to obtain the final output.

[0060] Global features (Right now The calculation formula for ) is as follows:

[0061] in, For the input features of the j-th self-attention learning module, This indicates that the self-attention learning branch is in the first... Layer operations, Indicates the first The feature map of the layer is resampled to a preset size.

[0062] Modeling the data features inherent in differencing images using a Transformer-learning path is highly innovative. Existing research, considering that differencing bi-temporal images destroys the original data structure—fundamental features such as spectral and textural characteristics are lost—means that differencing images are either directly used for threshold-based change detection (CD) or not used at all. However, by using a Transformer-learning path to model the data features of differencing images… As input, the goal is to further capture the variation information contained in the image, extract global features with stronger local continuity, model the semantic relationships between pixels, and discover the variation mechanisms between images. This is made possible by the combined effect of overlapping patch embedding and linear spatial restoration attention mechanism in PVTv2. The linear spatial restoration attention mechanism can model local continuity information, effectively strengthen the connection between patches, and make the Transformer-based global features more capable of representing details.

[0063] The LSRA mechanism is a highly efficient multi-head self-attention mechanism that can model long-range dependencies between pixels and capture global contextual information of images. Its core unit, the multi-head self-attention module, extends the model's focus on different locations, projects the input into different representation subspaces, and updates the state of each overlapping patch by enhancing the global information of each layer's features. It is an effective means of handling the complexity of HRSIs (High-Resolution Satellite Images) and also ensures the accuracy of dense prediction results. The computation of each self-attention module is performed by transforming the input into a set of q, k, and v, and then performing matrix operations. The calculation formula is as follows:

[0064] In the formula, q, k, and v represent the query, key, and value, respectively; This represents the balance coefficient, used to make the gradient more stable.

[0065] S5, the first local representation information, the second local representation information, the residual local features and the global features are fused to obtain the fused features.

[0066] The feature maps of layer 5 in the network contain the highest level of semantic information. To supplement the differential information content of the higher-level feature maps, the feature maps of the two sub-paths of Siamese-learning are subtracted and then concatenated with the feature maps of the Residual-learning path and the Transformer-learning path to obtain the fused features of layer 5. The fused features of the remaining layers are obtained by directly concatenating the feature map outputs of the corresponding layers of the four branches. (Fused Features) It can be represented as:

[0067] in, This is the first local representation information output by the j-th layer of the conjoined module. This refers to the second local representation information output by the j-th layer of the conjoined module. This represents the refined local features output from the j-th layer of the residual block. The global features output by the j-th layer of the self-attention learning module. For splicing operations, This is a feature of fusion.

[0068] S6, after mapping the fused features, input them into the higher-order attention module as hierarchical supervision data to refine the fused features and obtain the refined fused features.

[0069] The composite decoding path aggregates multi-level abstract information from different paths, fusing features from different domains within a unified feature mapping module, and performing higher-level feature abstraction and image reconstruction. Simultaneously, the High-Order Attention Module (HOAM) is embedded into each layer of the decoding path, aligning low-level features with high-level features, correcting the network's attention direction, and guiding each path towards effective perception of the CD task. Furthermore, each layer independently maps binary inference results, which are supervised by corresponding labels, thereby further refining the network. Ultimately, the entire composite decoding path further simulates the higher functions of the human brain's visual cortex, giving the network stronger reasoning and judgment abilities.

[0070] fusion features of layer 5 After performing multi-path feature fusion through the linear feature mapping module, a feature map with richer semantic information is output. Starting from the decoding path, upsampling is performed to restore the image and classify and locate pixels. The operations on the feature maps are the same from the first to the fourth layer. First, the refined features from the previous layer... The input high-level semantic feature map of this layer is obtained after upsampling. . As a supervisory signal source for the higher-order attention module HOAM, it participates in the feature fusion of this layer. Feature refinement, guidance Maintain semantic consistency with the high-level feature map. Then, it undergoes HOAM processing. and The feature maps of this layer are concatenated and then processed through feature mapping to obtain the output feature map. Refined fusion features The calculation formula is as follows:

[0071] in, This is the fusion feature of layer 5. For a 3×3 convolution operation, This is the high-level semantic feature map of layer j, which is derived from the refined fused feature map of the previous layer. Obtained through upsampling, For the fusion features of the j-th layer, This indicates a high-order attention feature refinement operation. This represents the refined fusion feature of the j-th layer.

[0072] Figure 5 The processing flowchart of the high-order attention module provided in the embodiments of the present invention is as follows: Figure 5 As shown, HOAM (Higher-Order Attention Mechanism) is embedded into the decoding path to further process the information, refining it before it is uniformly mapped. Features from each encoding branch are used in each layer. (i.e., first local representation information) Second local representation information Refine local features and global features The high-level semantic feature maps (layer j) contain rich spatial information, which converge in the decoding path and are used to supplement the spatial details lacking in high-level features. From the fourth layer onwards, each layer has a higher abstraction representation of the high-level semantic feature map of layer j. It is responsible for providing activation signals for the low-dimensional features of each branch. The high-level semantic feature map of layer j. Refined fusion feature map from the previous layer It is obtained through upsampling, and although it has a lower resolution, it captures the semantic contextual information of the image, including high-dimensional perceptual information that is sensitive to CD tasks.

[0073] Using the high-level semantic feature map of layer j For reference, attention guidance can be applied to the features of each branch, making low-dimensional features more likely to represent image regions valuable for the CD task. In the HOAM computation process, the high-level semantic feature map of the j-th layer... As a source of supervisory signals, the characteristics of each branch are the objects to be adjusted, and the objects to be adjusted receive... The provided dual-feature response signal refines low-dimensional features at both the global and subset domain levels.

[0074] First, the high-level semantic feature map of layer j. Based on the characteristics of each branch To adjust the target, a targeted first feature refinement attention coefficient matrix is ​​calculated. (Right now , , and This set of attention coefficients can provide supervisory signals for different data modeling channels, guiding each branch to selectively enhance features in subset domains. Subsequently, the high-level semantic feature map of layer j... The characteristics after merging the branches are called the fusion characteristics. To adjust the object, a second feature refinement attention coefficient matrix with a higher level of abstraction is calculated. The second feature refines the attention coefficient matrix. It is more sensitive to holistic features, guiding the network to represent overall features consistently and strengthening important features to suppress secondary features. Finally, the initial high-order attention features are obtained by multiplying the features of each branch with their corresponding coefficient matrices. (Right now , , and ), merge and then with Performing a dot product operation yields the final higher-order attention features. In this way, the enhanced low-dimensional features not only maintain and It improves semantic consistency and enhances sensitivity to foreground pixels, making the feature information in the composite coding path more capable of representing the image. The calculation process of HOAM is as follows:

[0075]

[0076]

[0077] in, This represents a linear transformation operation, including convolutional kernels with a size of 1. 1. Feature mapping operation, BatchNorm operation, and ReLU nonlinear activation operation; This indicates the splicing operation performed on the feature maps of each branch. Refine the attention coefficient matrix for the first feature. To refine the attention coefficient matrix for the second feature, This is the final high-order attention feature output by the high-order attention module.

[0078] S7. The refined fused features are processed through feature mapping and nonlinear mapping to obtain a two-channel feature map. The two-channel feature map is then classified to obtain the binary change detection result.

[0079] The above method can be applied to a remote sensing image change detection model. This model is based on a hierarchical supervision mechanism and is trained using a total loss function, which is:

[0080] in, For the total loss function, All are weighted parameters. Let be the cross-entropy loss function of the j-th layer. For the domain loss function, This is the sieve loss function.

[0081] The hierarchical supervision mechanism is a combination of hard and soft constraints, comprising three parts. The first part is the soft constraint model, responsible for semantic consistency supervision of the prediction results at each layer. The main loss function uses the more universal cross-entropy loss function, denoted as... The refined fused features of each layer's output. The first output is obtained after one feature mapping operation and one nonlinear mapping operation. It will shrink from a standard-sized binary label image. The result is twice The values ​​are fed into the loss function along with the ground truth values. It's worth noting that in each iteration, the ratio of the loss value of each layer to the total loss value of all five layers is used as the coefficient of that layer's loss function. The purpose of this is to dynamically assign intensity to the supervisory signal for each layer, allowing layers with larger loss values ​​to be prioritized for adjustment, thereby guiding each layer of the network to better adapt to extracting feature information at its corresponding depth.

[0082] The cross-entropy loss function is widely used because it effectively measures the difference between two probability distributions. Because... Since it is obtained through resampling, its accuracy will be weakened. Therefore, only supervising the accuracy of the probability distribution of its intermediate layers can gently guide the network to approach the true value, rather than seeking absolute alignment of the prediction results. It can be represented as:

[0083] in, This represents the probability that the prediction is positive. This represents the label of the actual sample.

[0084] The second part is the hard constraint mode, which is responsible for supervising the edge accuracy of the final output. The main loss function uses the domain loss function, denoted as... The refined fusion features of the first layer are output at the end of the network. The edge output is obtained after one feature mapping operation and one nonlinear mapping operation. Edge label map extracted from label map using Canny algorithm It is treated as a truth value. It is worth noting that, due to... The number of positive and negative samples is severely imbalanced. To overcome the difficulty of supervision caused by the fact that the number of negative samples is much greater than the number of positive samples, we used focal loss.

[0085] Domain loss functions can effectively adjust for severe imbalances between positive and negative samples, and provide more targeted supervision of prediction results for positive samples. Not only are the number of positive samples far fewer than the number of negative samples, but factors such as satellite imagery angle and Earth's rotation also result in weak edge consistency of ground features in bi-temporal HRSIs, making the positive samples at the edges difficult to learn. Domain loss functions are also suitable for solving this type of problem because they allow the learning process to focus on mining these difficult samples. It can be represented as:

[0086] in, This is the adjustment coefficient for the positive and negative sample weights. To focus parameters, To predict the probability that a sample is a positive class, and is the modulation factor.

[0087] The third part is also a hard constraint mode, responsible for supervising the internal compactness of the output. The main loss function uses the sieve loss function, denoted as... The refined fusion features of the first layer are output at the end of the network. The second output is obtained after one feature mapping operation and one nonlinear mapping operation. A standard-sized binary label image As the true value, the sieve loss function is more sensitive to the completeness of the prediction results, and can effectively supervise whether the similarity between the prediction results and the true value is the highest, thereby continuously guiding the network to approach the overall accuracy.

[0088] The sieve loss function guides the network to focus more on mining the foreground region during training, and it directly calculates the similarity between the predicted result and the ground truth. This is the final output of the network in the true sense, so its accuracy and completeness should be comprehensively monitored. The sieve loss function uses precision and recall as evaluation metrics, and can effectively monitor... Supervision. It can be represented as:

[0089] in, This is a two-channel feature map. For the corresponding truth value label, This represents the intersection operation. This indicates the union operation.

[0090] Based on the methods described in the above embodiments, this embodiment will further describe the system from the perspective of a hierarchical supervision signal-guided remote sensing image change detection system.

[0091] Please see Figure 6 , Figure 6 This invention specifically describes a hierarchical supervision signal-guided remote sensing image change detection system provided in an embodiment of the invention. This hierarchical supervision signal-guided remote sensing image change detection system may include: The acquisition and preprocessing module is used to acquire dual-temporal remote sensing images: a first image and a second image, stitch the first image and the second image together to obtain a stitched image, and perform differential processing on the first image and the second image to obtain a differential image. The conjoined learning branch is used to input the first image and the second image into the conjoined learning branch respectively to obtain the first local representation information and the second local representation information; The residual learning branch is used to input the stitched image into the residual learning branch to obtain refined local features; The self-attention learning branch is used to input the difference image into the self-attention learning branch to obtain global features; The fusion module is used to fuse the first local representation information, the second local representation information, the residual local features, and the global features to obtain fused features; A higher-order attention module is used to input the mapped fused features into the higher-order attention module as hierarchical supervision data to refine the fused features and obtain the refined fused features. The change detection module is used to obtain a two-channel feature map by performing feature mapping on the refined fused features, and to classify the two-channel feature map to obtain a binary change detection result.

[0092] In practice, the above modules and / or units can be implemented as independent entities or can be arbitrarily combined as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0093] The above provides a detailed description of a hierarchical supervision signal-guided remote sensing image change detection method and system provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for detecting changes in remote sensing images guided by hierarchical supervision signals, characterized in that, The method includes: Acquire dual-temporal remote sensing images: a first image and a second image; stitch the first image and the second image together to obtain a stitched image; perform differential processing on the first image and the second image to obtain a differential image. The first image and the second image are respectively input into the conjoint learning branch to obtain the first local representation information and the second local representation information. The stitched image is input into the residual learning branch to obtain refined local features; The difference image is input into the self-attention learning branch to obtain global features; The first local representation information, the second local representation information, the residual local features, and the global features are fused to obtain the fused features; The fused features are mapped and then input into a higher-order attention module as hierarchical supervision data to refine the fused features, resulting in refined fused features. The refined fused features are processed through feature mapping and nonlinear mapping to obtain a two-channel feature map. The two-channel feature map is then classified to obtain the binary change detection result.

2. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 1, characterized in that, The conjoined learning branch includes two sub-paths with shared weights, which are used to process the first image and the second image respectively; each sub-path includes a conjoined module and a lightweight convolutional attention module. The processing of the sub-path includes: The first image or the second image is input into the conjoined module to obtain the first convolutional feature or the second convolutional feature; The first or second convolutional feature is input into a lightweight convolutional attention module for spatial and channel attention refinement operations to obtain the first or second local representation information.

3. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 1, characterized in that, The residual branch includes a feature mapping module and a residual block. The residual block includes multiple processing layers, and each processing layer is connected to a resampling module and a lightweight convolutional attention module in sequence. The stitched image is input into the residual learning branch to obtain refined local features, which are expressed by the following formula: in, This represents the three-channel feature map obtained through the feature mapping module. Indicates the residual block at the th Layer operations, This represents a 1×1 convolution operation. Indicates the first The feature map of the layer is resampled to a preset size. To refine local features, This is the input to the (j-1)th layer residual block, which is also the input to the jth layer residual block.

4. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 1, characterized in that, The self-attention learning branch includes a self-attention learning module, which includes multiple processing layers, and each processing layer is also connected to a resampling module. The difference image is input into the self-attention learning branch to obtain global features, which are expressed by the following formula: in, For the input features of the j-th self-attention learning module, This indicates that the self-attention learning branch is in the first... Layer operations, Indicates the first The feature map of the layer is resampled to a preset size.

5. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to any one of claims 1-4, characterized in that, The conjoined module is based on the five layers before the last pooling layer of the pre-trained VGG16 model. The residual block is based on all layers before the average pooling layer of the pre-trained ResNet18 model. Each layer corresponds to a processing layer of the residual block. The self-attention learning module is based on the pre-trained PVTv2 model and adds a processing layer at the very front of the PVTv2 model. The structure of the processing layer is the same as the first layer of the PVTv2 model. The connected module, the residual block, and the self-attention learning module have the same number of layers.

6. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 5, characterized in that, The first local representation information, the second local representation information, the residual local features, and the global features are fused to obtain the fused feature, which is expressed by the following formula: in, This is the first local representation information output by the j-th layer of the conjoined module. This refers to the second local representation information output by the j-th layer of the conjoined module. This represents the refined local features output from the j-th layer of the residual block. The global features output by the j-th layer of the self-attention learning module. For splicing operations, This is a feature of fusion.

7. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 6, characterized in that, The fused features are mapped and then input into a higher-order attention module as hierarchical supervision data to refine the fused features, resulting in refined fused features, expressed by the following formula: in, This is the fusion feature of layer 5. For a 3×3 convolution operation, This is the high-level semantic feature map of layer j, which is derived from the refined fused feature map of the previous layer. Obtained through upsampling, For the fusion features of the j-th layer, This indicates a high-order attention feature refinement operation. This represents the refined fusion feature of the j-th layer.

8. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 1, characterized in that, The method is applied to a remote sensing image change detection model, which is trained based on a total loss function, namely: in, For the total loss function, All are weighted parameters. Let be the cross-entropy loss function of the j-th layer. For the domain loss function, This is the sieve loss function.

9. The method for detecting changes in remote sensing images guided by hierarchical supervision signals according to claim 8, characterized in that, The cross-entropy loss function is: in, This represents the probability that the prediction is positive. Indicates the labels of real samples; The domain loss function is: in, This is the adjustment coefficient for the positive and negative sample weights. To focus parameters, To predict the probability that a sample is a positive class, and Modulation factor; The loss function of the sieve is: in, This is a two-channel feature map. For the corresponding truth value label, This represents the intersection operation. This indicates the union operation.

10. A hierarchical monitoring signal-guided remote sensing image change detection system, characterized in that, include: The acquisition and preprocessing module is used to acquire dual-temporal remote sensing images: a first image and a second image, stitch the first image and the second image together to obtain a stitched image, and perform differential processing on the first image and the second image to obtain a differential image. The conjoined learning branch is used to input the first image and the second image into the conjoined learning branch respectively to obtain the first local representation information and the second local representation information; The residual learning branch is used to input the stitched image into the residual learning branch to obtain refined local features; The self-attention learning branch is used to input the difference image into the self-attention learning branch to obtain global features; The fusion module is used to fuse the first local representation information, the second local representation information, the residual local features, and the global features to obtain fused features; A higher-order attention module is used to input the mapped fused features into the higher-order attention module as hierarchical supervision data to refine the fused features and obtain the refined fused features. The change detection module is used to obtain a two-channel feature map by performing feature mapping on the refined fused features, and to classify the two-channel feature map to obtain a binary change detection result.

Images