Remote sensing image semantic change detection method based on cross-sample semantic contrast learning

Abstract

The application discloses a remote sensing image semantic change detection method based on cross-sample semantic contrast learning, comprising the following steps: constructing a shared weight twin residual network; cutting the original double-time remote sensing image pair into blocks and inputting the blocks into corresponding branches of the twin residual network respectively; constructing a feature extraction module and embedding the feature extraction module after the first stage of the twin residual network; constructing a change detection module and inputting the intermediate features generated in the third stage of the twin residual network into the change detection module; constructing a multi-scale feature pyramid fusion module, inputting auxiliary fusion feature maps, abstract fusion feature maps and change result feature maps into the multi-scale feature pyramid fusion module, and obtaining the final result of semantic change detection. The application embeds a cross-sample semantic consistency feature extraction module, so that the ground object consistency information required for semantic change detection can be better learned, and the accuracy and reliability of the semantic change detection result can be improved.

Description

Technical Field

[0001] This invention relates to image information processing technology, and in particular to a method for detecting semantic changes in remote sensing images based on cross-sample semantic contrast learning. Background Technology

[0002] The main purpose of semantic change detection in remote sensing images is to identify changed targets at the pixel level by comparing remote sensing images of the same area at different time periods, and to accurately label the actual semantic categories of these targets before and after the changes. As one of the important Earth observation tools in remote sensing technology, it plays a crucial role in many application fields such as disaster assessment, land use statistics, and urban and rural planning. Therefore, designing a change detection method with high accuracy and strong generalization is of great significance.

[0003] In recent years, with the continuous advancement of deep learning technology, deep learning methods have been widely applied in the field of semantic change detection in remote sensing images and have achieved significant success. However, these methods still have certain shortcomings. Existing deep learning-based semantic change detection methods for remote sensing images typically only learn semantic change information between sample pairs within the same region. Furthermore, mainstream semantic change detection datasets suffer from limitations such as a small total number of semantic label samples and imbalanced sample types. This makes it difficult for existing methods to fully learn the semantic consistency information in remote sensing images, resulting in inaccurate semantic category labeling. For example, in a certain dataset, there are few water body samples, and due to the influence of lighting conditions at different times of shooting, some water bodies may appear similar to building roofs or roads, easily leading to incorrect classification of water bodies as buildings or roads, thus resulting in misidentification of land cover categories.

[0004] Therefore, how to design a method to enhance the model's ability to resist interference when there are insufficient samples, thereby improving its ability to learn semantic consistency information, has become an urgent problem to be solved in the field of semantic change detection of remote sensing images. Summary of the Invention

[0005] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a method for detecting semantic changes in remote sensing images based on cross-sample semantic contrastive learning.

[0006] Technical solution: The present invention provides a method for detecting semantic changes in remote sensing images based on cross-sample semantic contrastive learning, comprising:

[0007] Construct a twin residual network with shared weights;

[0008] The original dual-time remote sensing images are cropped into corresponding blocks and then input into the corresponding branches of the Siamese residual network.

[0009] A feature extraction module is constructed and embedded after the first stage of the Siamese residual network to generate auxiliary fusion feature maps and abstract fusion feature maps;

[0010] A change detection module is constructed, and the intermediate features generated in the third stage of the twin residual network are input into the change detection module to generate a change result feature map;

[0011] A multi-scale feature pyramid fusion module is constructed. The auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

[0012] Furthermore, a feature extraction module is embedded after the first stage of the Siamese residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, including:

[0013] In a certain temporal branch of the twin residual network, the feature map Input is fed into the first stage to generate feature maps.

[0014] feature map After passing through a 7×7 convolutional layer, a feature map is generated. The formula is:

[0015]

[0016] feature map Feature extraction is performed using three branches, employing 1×1, 3×3, and 7×7 convolutional layers respectively, to obtain feature maps. as well as The formula is:

[0017]

[0018] In the first branch, a max-pooling strategy is used for downsampling to obtain an auxiliary fusion feature map. The formula is:

[0019]

[0020] In the formula, MaxPool2D represents the maximum pooling function;

[0021] The feature maps generated by the second and third branches are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out.

[0022] Directly embed the multi-scale feature map Out into the feature map. Obtain feature map The feature maps are then fused with an auxiliary layer after passing through a 7×7 convolutional layer. Low-rank attention fusion is performed again to obtain an abstract fused feature map. The formula is:

[0023]

[0024] In the formula, LowRankAttn represents low-rank attention fusion.

[0025] Furthermore, the feature maps generated by the second and third branches are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out, including:

[0026] feature map Using a 1×1 convolutional layer for feature mapping, we obtain the query feature q, the matching feature k, and the value feature v, as shown in the following formulas:

[0027]

[0028] For the matching feature k and the value feature v, a 4×4 max pooling function and a 16×16 average pooling function are used to obtain the feature representations of individual salient information and local average information, respectively. At the same time, the scale compression of the original feature map is performed. Then, the salient information and the average information are flattened along the spatial dimension and concatenated to obtain the corresponding feature map. and The formulas are as follows:

[0029]

[0030] In the formula, Concat represents a connection, and AvgPool2D represents the average pooling function;

[0031] feature map Perform a matrix multiplication with the query feature q, and obtain the association weights between the input features using the Softmax function, then sum... Perform multiplication to generate a multi-scale feature map Out, using the following formula:

[0032]

[0033] Furthermore, the intermediate features generated in the third stage of the Siamese residual network are input into the change detection module to generate a change result feature map, including:

[0034] The intermediate feature maps generated in the third stage of the Siamese residual network are denoted as follows: and Obtained through cosine similarity calculation The similarity weight Sim is calculated using the following formula:

[0035]

[0036] In the formula, Cosine represents the cosine similarity function;

[0037] The difference weights Dif are obtained by subtracting the all-one matrices, using the following formula:

[0038] Dif = 1 - Sim

[0039] Using 1×1 convolutional layers Perform feature mapping, then calculate the difference and take the absolute value, and finally multiply by the similarity weight Sim to obtain the hard-to-distinguish feature map A. t The formula is:

[0040]

[0041] Cascaded feature maps along the channel direction Then multiply by the difference weight Dif, and finally fuse them at the channel level through a 1×1 convolutional layer to obtain the significant difference feature map D. t The formula is:

[0042]

[0043] Significant difference feature map D t Using 3×3 and 5×5 convolution kernels respectively, saliency information was extracted. And used as a global reference and difficult-to-distinguish feature map A t Perform attention interaction to obtain a map of the changed regions. The formula is:

[0044]

[0045] Map of changing areas and Channel fusion is performed using cascaded 1×1 convolutional layers, and significant difference feature maps D are embedded in the form of residual connections. t The feature map O of the fine-grained change results is obtained. t The formula is:

[0046]

[0047] Furthermore, the auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including:

[0048] Assist fusion feature map pair The features are concatenated along the temporal dimension and extracted using 3D convolution. Then, a cross-attention mechanism is used to capture the spatial relationship between the two temporal fusion features, resulting in a more comprehensive auxiliary feature map F. tThe formula is:

[0049]

[0050] In the formula, CrissCrossAttn represents the cross-attention mechanism;

[0051] Abstract fusion feature map pairs Concatenate along the temporal dimension and use 3D convolution for temporal feature extraction. Use a low-rank attention mechanism to transform the resulting feature map O. t By integrating it into the global semantic information, an abstract feature map H is obtained. t The formula is:

[0052]

[0053] Feature map of the change result O t The channel-level scaling factor β and the spatial-level offset matrix γ are generated through two branches, respectively, where β is based on the feature map O of the transformation result. t The average variation information at each channel level is obtained by projecting from the global average pooling layer and the linear layer, as shown in the formula:

[0054] β = Linear(GlobalAvgPool2D(O t ))

[0055] In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation;

[0056] γ is obtained through upsampling 1×1 convolution mapping and Tanh gate function, as shown in the formula:

[0057] γ=Tanh(Conv 1×1 (Upsample 2× (O t )))

[0058] In the formula, Upsample represents the upsampling operation;

[0059] By explicitly constructing a linear transformation process using the scaling factor β and the offset matrix γ, F t Projecting the changes into the feature space yields a semantically consistent change detection feature map C. t The formula is:

[0060] C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t ))

[0061] The auxiliary fusion feature maps are respectively and abstract feature map H t The features obtained after upsampling are channel-concatenated and then aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is:

[0062]

[0063] The generated bi-temporal semantic segmentation results and change detection feature map C t These are fed into the prediction head to generate the final result of semantic change detection.

[0064] Furthermore, the generated bi-temporal semantic segmentation results and change detection feature map C t The results of semantic change detection are fed into the prediction head and include:

[0065] Change detection feature map C t The feature map C is obtained by bilinear interpolation upsampling and convolution, followed by the application of the Sigmoid function, as shown in the formula:

[0066] C = Sigmoid(Conv) 1×1 (BN(Conv 1×1 (Upsample(C t )))))

[0067] In the formula, BN represents the batch normalization operation;

[0068] right Upsampling is performed separately, and then the Softmax function is used to obtain the pixel-level semantic labels [S1, S2], as shown in the formula:

[0069]

[0070] The semantic change detection result is obtained by multiplying the feature map C and the semantic label [S1,S2].

[0071] Furthermore, the loss function is composed of the contrastive loss L CON Cross-entropy loss L CD and combination loss L SS The composition, and the formula are:

[0072] L = L CD +L SS +L CON

[0073] in:

[0074]

[0075] L CON =g·L Pair +(1-g)·L Cross ,

[0076] In the formula, y represents the actual label. For the predicted sequence, |·| denotes the magnitude of the vector, L Pair L represents the loss between the same pair. Cross This represents the contrast loss between sample pairs, where m and n are the subscripts of the batch dimension.

[0077] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0078] 1. This invention incorporates a cross-sample semantic consistency feature extraction module, which enables better learning of the ground feature consistency information required for semantic change detection and improves the accuracy and reliability of semantic change detection results. This invention verifies the accuracy and reliability of the invention from both qualitative and quantitative analysis perspectives.

[0079] 2. In terms of accuracy: Through quantitative analysis, the accuracy metrics of this invention on the public dataset SECOND are significantly improved compared to current state-of-the-art methods. Through visual comparative analysis, the semantic change detection results generated by this invention show a significant improvement in overall detection performance compared to other methods.

[0080] 3. From the perspective of reliability: Through ablation experiments, it was verified that after removing the proposed semantic consistency extraction module and the corresponding consistency loss function, the detection index of the model decreased significantly, and the performance on ground features that are greatly affected by environmental changes deteriorated significantly, indicating that the cross-sample semantic consistency feature extraction module and consistency loss function designed in this invention play an important role. Attached Figure Description

[0081] Figure 1 This is a flowchart of a remote sensing image semantic change detection method based on cross-sample semantic contrast learning.

[0082] Figure 2 This is a structural diagram of a remote sensing image semantic change detection model;

[0083] Figure 3 This is a structural diagram of the low-rank attention mechanism fusion module;

[0084] Figure 4 This is a structural diagram of the change detection module;

[0085] Figure 5 This is a structural diagram of the multi-scale feature pyramid fusion module;

[0086] Figure 6 This is a comparison chart of the detection results of this example and other models;

[0087] Figure 7 This is a graph showing the ablation experiment results for this example. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0089] The flowchart of the remote sensing image semantic change detection method based on cross-sample semantic contrast learning described in this embodiment is as follows: Figure 1 As shown, it includes at least steps 1 to 5.

[0090] Step 1: Construct a twin residual network with shared weights;

[0091] The twin residual network is ResNet-18.

[0092] Step 2: Crop the original dual-time remote sensing images into corresponding blocks and input them into the corresponding branches of the Siamese residual network.

[0093] The original high-resolution dual-time remote sensing image is cropped into image pairs of uniform size [t1,t2], where t1 represents the first time phase, i.e. the image before the change, and t2 represents the second time phase, i.e. the image after the change. The image pairs [t1,t2] are then input into a Siamese residual network ResNet-18 with shared weights.

[0094] Step 3: Construct a feature extraction module and embed it after the first stage of the Siamese residual network to generate auxiliary fusion feature maps and abstract fusion feature maps.

[0095] The Siamese Residual Network ResNet-18 consists of four stages. A feature extraction module is embedded after the first stage of the Siamese Residual Network, so that the intermediate features output by the first stage are fed into the feature extraction module for cross-sample semantic consistency learning. The feature extraction module outputs the auxiliary fusion feature map and abstract fusion feature map of the first time phase, as well as the auxiliary fusion feature map and abstract fusion feature map of the second time phase.

[0096] Step 4: Construct a change detection module and input the intermediate features generated in the third stage of the twin residual network into the change detection module to generate a change result feature map.

[0097] After the third stage of the Siamese residual network, intermediate feature maps for the first and second time phases are generated respectively. The intermediate features of the two time phases are then combined. Figure 1 The data is input into the change detection module to obtain preliminary detection results for the changed areas.

[0098] Step 5: Construct a multi-scale feature pyramid fusion module. Input the auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

[0099] To ensure consistency between the segmented semantic labels and the change detection results, and to apply semantic consistency information to improve the accuracy of the change detection part, a multi-scale feature pyramid fusion module based on 3D convolution is constructed. By fusing semantic features and change detection features at different scales, the synergistic consistency between the two is enhanced, and weighted fusion is performed. Finally, the semantic change detection results are obtained through a pixel-level classifier.

[0100] In one example, due to the large size of the high-resolution remote sensing image, and considering computational complexity and resources, it is necessary to crop the original two-time images t1 and t2 to obtain the two-time input. and Where t1 represents the image before the change, and t2 represents the image after the change. H represents the height of the image, W represents the width of the image, and 3 represents that the image is an RGB three-channel image.

[0101] Furthermore, a feature extraction module is embedded after the first stage of the Siamese residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, such as... Figure 2 As shown, it includes:

[0102] Taking the first temporal branch as an example, in the first temporal branch of the Siamese residual network, the feature map is... The input is fed into a Siamese residual network, and intermediate feature maps are generated after the first stage. After the third stage, intermediate feature maps are generated. T represents the baseline number of feature map channels.

[0103] feature map A 7×7 convolutional layer is used for sampling at the same scale to generate a feature map with a larger receptive field. The formula is:

[0104]

[0105] feature map The system employs three branches, each using convolutional kernels of different sizes for feature extraction to capture ground feature information of varying volumes. Specifically, 1×1, 3×3, and 7×7 convolutional layers are used for feature extraction, yielding feature maps respectively. as well as The formula is:

[0106]

[0107] In the first branch, a max-pooling strategy is used for downsampling to obtain an auxiliary fusion feature map. The formula is:

[0108]

[0109] In the formula, MaxPool2D represents the maximum pooling function;

[0110] The feature maps generated by the second and third branches are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out.

[0111] Directly embed the multi-scale feature map Out into the feature map. Obtain feature map The feature maps are then fused with an auxiliary layer after passing through a 7×7 convolutional layer. Low-rank attention fusion is performed again to obtain an abstract fused feature map. The formula is:

[0112]

[0113] In the formula, LowRankAttn represents low-rank attention fusion.

[0114] Similarly, in the second time-phase branch, the feature map The input is fed into a Siamese residual network, and intermediate feature maps are generated after the first stage. After the third stage, intermediate feature maps are generated. Finally, the auxiliary fusion feature map is obtained through the feature extraction module. and abstract fusion feature maps

[0115] To aggregate feature maps with smaller receptive fields and feature maps with larger receptive fields To fully capture ground features at various scales, this application designs a low-rank attention mechanism as a feature fusion strategy. This strategy downsamples features with large receptive fields by using MaxPool2D with a small mask size and AvgPool2D with a large mask size, and uses them as alternative features to input into the attention mechanism process. This can greatly reduce computational costs while preserving important information in the feature map as much as possible, thereby achieving the goal of low-overhead and fast computation.

[0116] like Figure 3 As shown, further, the feature maps generated by the second and third branches are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out, including:

[0117] feature map Using a 1×1 convolutional layer for feature mapping, we obtain the query feature q, the matching feature k, and the value feature v, as shown in the following formulas:

[0118]

[0119] For the matching feature k and the value feature v, a 4×4 max pooling function and a 16×16 average pooling function are used to obtain the feature representations of individual salient information and local average information, respectively. At the same time, the scale compression of the original feature map is performed. Then, the salient information and the average information are flattened along the spatial dimension and concatenated to obtain the corresponding feature map. and The formulas are as follows:

[0120]

[0121] In the formula, Concat represents a connection, and AvgPool2D represents the average pooling function; relative to the equivalent replacement of k and v, and It can reduce the amount of computation to the original amount.

[0122] feature map Perform a matrix multiplication with the query feature q, and obtain the association weights between the input features using the Softmax function, then sum... Perform multiplication to generate a multi-scale feature map Out, using the following formula:

[0123]

[0124] To initially complete the change detection task, a change detection module was constructed, employing a "dual-end suppression" strategy. This strategy simultaneously suppresses both unchanged and significantly changed regions, thereby increasing the focus on areas with difficult-to-distinguish changes. This leads to more accurate initial change detection and the acquisition of a change feature map. t .

[0125] like Figure 4 As shown, specifically, the intermediate features generated in the third stage of the Siamese residual network are input into the change detection module to generate a change result feature map, including:

[0126] The intermediate feature maps generated in the third stage of the Siamese residual network are denoted as follows: and Obtained through cosine similarity calculation The similarity weight Sim is calculated using the following formula:

[0127]

[0128] In the formula, Cosine represents the cosine similarity function;

[0129] The more similar the channel features, the greater the position weight, and vice versa, with a value range of [0,1]. The difference weight Dif, where the difference features have larger weights, is obtained by subtracting matrices of all ones. The formula is as follows:

[0130] Dif = 1 - Sim

[0131] For difficult-to-separate regions, since their similarity and differences are not prominent, a 1×1 convolutional layer is used. Feature mapping is performed, then the difference is calculated and the absolute value is taken to directly subtract the features with high similarity. Afterwards, a dot product is applied to the similarity weight Sim, thereby indirectly amplifying those difficult-to-distinguish variation regions, resulting in a difficult-to-distinguish feature map A. t The formula is:

[0132]

[0133] on the other hand, The semantic and saliency information contained within these features is also needed for change detection tasks. For these features, the cascaded feature mapping along the channel direction... Then multiply by the difference weight Dif, and finally fuse them at the channel level through a 1×1 convolutional layer to obtain the significant difference feature map D. t The formula is:

[0134]

[0135] In order to efficiently utilize D t To identify indistinguishable changes at different scales, based on the significant difference feature map D t Using 3×3 and 5×5 convolution kernels respectively, saliency information was extracted. And used as a global reference and difficult-to-distinguish feature map A t Engage with attention to better extract A t The true regions of change are identified within indistinguishable changes at different scales, resulting in a change region map. The formula is:

[0136]

[0137] Map of changing areas and Channel fusion is performed using cascaded 1×1 convolutional layers, and significant difference feature maps D are embedded in the form of residual connections. t The feature map O of the fine-grained change results is obtained. t The formula is:

[0138]

[0139] In the change detection module, cosine similarity is used to model the changes at each pixel position of the dual-temporal features. Positions with lower cosine similarity have greater differences in feature representation, while vectors with similar feature representations have relatively smaller differences. Therefore, in calculating the difficult-to-distinguish feature map A... t First, by using difference, similar expressions in the two-phase features are suppressed. Then, the similarity weight Sim is multiplied to suppress significantly different regions through a lower cosine similarity weight, thus indirectly amplifying the difficult-to-distinguish change regions. On the other hand, considering that significantly changed regions are also an important target of the change detection task, those significantly changed regions are preserved in parallel by calculating the difference weight Dif, and convolution is used to learn them. Finally, an attention mechanism is used to guide the detection of the real change positions in the difficult-to-distinguish change regions through the features of significantly changed features, ultimately obtaining a fine-grained change result feature map O. t .

[0140] To fuse auxiliary fusion feature maps Abstract fusion feature map and the characteristic diagram of the change results O t To obtain semantic change detection results with correct change locations in the bi-temporal semantic labels, this application constructs a multi-scale feature pyramid fusion module to aggregate semantically related features and change features, improving their synergy, and upsampling the features to a scale suitable for prediction by the prediction head. This assists in fusion feature maps... The scale is relatively large, preserving more intuitive details, while the abstract fusion feature map... The scale is relatively small, representing an abstract feature that aggregates information from multiple receptive fields. Therefore, different strategies should be adopted when interacting with semantic information and change detection information.

[0141] like Figure 5 As shown, specifically, the auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including:

[0142] Assist fusion feature map pair The features are concatenated along the temporal dimension and extracted using 3D convolution. Then, a cross-attention mechanism is used to capture the spatial relationship between the two temporal fusion features, resulting in a more comprehensive auxiliary feature map F. t The formula is:

[0143]

[0144] In the formula, CrissCrossAttn represents the cross-attention mechanism;

[0145] Abstract fusion feature map pairs Concatenate along the temporal dimension and use 3D convolution for temporal feature extraction. Use a low-rank attention mechanism to transform the resulting feature map O. t By integrating it into the global semantic information, an abstract feature map H is obtained. t The formula is:

[0146]

[0147] Feature map of the change result O t The channel-level scaling factor β and the spatial-level offset matrix γ are generated through two branches, respectively, where β is based on the feature map O of the transformation result. t The average change information at each channel level is obtained by projecting from the global average pooling layer and the linear layer, as shown in the formula:

[0148] β = Linear(GlobalAvgPool2D(O t ))

[0149] In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation;

[0150] γ is obtained through upsampling 1×1 convolution mapping and Tanh gate function, as shown in the formula:

[0151] γ=Tanh(Conv 1×1 (Upsample 2× (O t )))

[0152] In the formula, Upsample represents the upsampling operation;

[0153] By explicitly constructing a linear transformation process using the scaling factor β and the offset matrix γ, F t Projecting the changes into the feature space yields a semantically consistent change detection feature map C. t The formula is:

[0154] C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t ))

[0155] The auxiliary fusion feature maps are respectively and abstract feature map H t The features obtained after upsampling are channel-concatenated and then aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is:

[0156]

[0157] The generated bi-temporal semantic segmentation results and change detection feature map C t These are fed into the prediction head to generate the final result of semantic change detection.

[0158] Considering The receptive field is smaller but the detailed information is richer, making it necessary to extract the global correlation of the features contained within it. In this example, a cross-attention mechanism is used to achieve this goal efficiently and cost-effectively. Additionally, in the fusion of F... t and O t At that time, a 2D scaling transformation strategy was used, that is, for F... t For each position in the O, construct a set based on O t The affine transformation coefficients β and γ are obtained, where β is obtained through global average pooling (GlobalAvgPool2D), focusing on channel-level scaling, while γ is obtained through convolution and the Tanh() function, focusing on spatial-level transformation. In the fusion of H... t and O t In this case, a low-rank attention mechanism is used to efficiently establish a global association between representative abstract features and changing features. Ultimately, the change detection result C... t F after 2D scaling transformation t Same as H t The result is obtained by adding and then upsampling, while the semantic segmentation result is... By convolution fusion F t and H t get.

[0159] Furthermore, the generated bi-temporal semantic segmentation results and change detection feature map C t The results of semantic change detection are fed into the prediction head and include:

[0160] Change detection feature map C t The feature map C is obtained by bilinear interpolation upsampling and convolution, followed by the application of the Sigmoid function, as shown in the formula:

[0161] C = Sigmoid(Conv) 1×1 (BN(Conv 1×1 (Upsample(C t )))))

[0162] In the example, each pixel can be considered to have a threshold of 0.5. Pixels with a threshold greater than 0.5 are considered to be invariant, while those with a threshold less than 0.5 are considered to be invariant.

[0163] right Upsampling is performed separately, but the number of output channels in the second convolution is the actual number of label categories (not 1). Then, the Softmax function is used to obtain the pixel-level semantic labels [S1, S2], as shown in the formula:

[0164]

[0165] Since all unchanged regions in the semantic change detection results are defaulted to the invariant class (class 0), the feature map C and the semantic label [S1,S2] are multiplied by a dot to obtain the semantic change detection result, so as to ensure the consistency of the final semantic change detection result.

[0166] Based on the steps described above, a shared-weight twin residual network, a feature extraction module, a change detection module, and a multi-scale feature pyramid fusion module are constructed to form the remote sensing image semantic change detection model in this application. This model is used to realize change detection of remote sensing images at different time phases.

[0167] The loss function of this remote sensing image semantic change detection model consists of a contrastive loss L that optimizes semantic consistency information. CON Optimize the cross-entropy loss L of change detection results CD The combined loss L with the optimized semantic segmentation results SS The composition, and the formula are:

[0168] L = L CD +L SS +L CON

[0169] in:

[0170]

[0171] In the formula, y represents the actual label. For the predicted sequence, ||·|| represents the magnitude of the vector.

[0172] Comparison loss L CON The calculation of loss across sample pairs is explained here. First, for For any pair of pixels [p1, p2] that meets the requirements, cosine similarity is used as the consistency metric between them, i.e.:

[0173]

[0174] The following criteria govern the selection of pixel pairs [p1, p2]. Any sample from each batch Where m and n are the subscripts of the batch dimension, if m = n, that is, when the two-phase samples are the same pair, [p1, p2] should have the same pixel position, then:

[0175]

[0176] At this point, only the similarity of the unchanged regions between the two time phases is compared.

[0177] If m ≠ n, meaning that when performing cross-sample semantic consistency comparison, [p1, p2] does not need to be strictly corresponding pixel positions, then:

[0178]

[0179] L Cross =f·∑Cosine(p1,p2)+(1-f)·∑1-Cosine(p1,p2)

[0180] Comprehensive L Pair and L Cross The loss function L for semantic consistency CON It can be represented as:

[0181] L CON =g·L Pair +(1-g)·L Cross ,

[0182] To verify the effectiveness of this invention, the remote sensing image semantic change detection model described herein was trained and tested on the public dataset SECOND. The detection results were compared with current state-of-the-art semantic change detection networks, including HRSCD-4, SSCDL, Bi-SRNet, and DEFO. Furthermore, both quantitative and qualitative analyses were performed in comparative and ablation experiments.

[0183] The SECOND dataset is a widely used open-source semantic change detection dataset. It collects 4662 pairs of aerial images distributed in Hangzhou, Chengdu, and Shanghai from multiple platforms and sensors. Due to commercial copyright restrictions, 3561 image pairs are currently publicly available. Each image is 512×512 pixels in size and is labeled at the pixel level, covering six main land cover categories: non-vegetated land, trees, low vegetation, water, buildings, and land. There is also an invariant category, which includes all land features that have not changed between two temporal phases. Following current mainstream methods, these images are not cropped and are divided into 2375 training pairs, 593 validation pairs, and 593 test pairs. The model weights that perform best on the validation set are selected for testing.

[0184] To verify the effectiveness of the method of this invention from the perspective of accuracy, this invention is compared with the HRSCD-4 network based on fully convolutional semantic recognition, the SSCDL network based on fully convolutional and aggregated feature change detection, the Bi-SRNet spatiotemporal attention network based on similarity information and global features, and the DEFO fully convolutional network focusing on decoder feature restoration on the SECOND dataset.

[0185] From a qualitative analysis perspective, the test results are visualized, and the results are as follows: Figure 6 As shown in the diagram, white represents unchanged areas, dark green represents trees, light green represents low vegetation, dark blue represents buildings, gray represents non-vegetated surfaces, and light blue represents soil. Figure 6 In the figure, (1) to (4) represent different sample pairs, where the first row of each sample pair represents the image before the change and the second row represents the image after the change. As can be seen from the figure, the semantic change detection effect of the present invention is significantly better than other models.

[0186] From a quantitative analysis perspective, four commonly used metrics are calculated on the test set to evaluate detection performance: average F1 score (F1-Score), accuracy, mean intersection-over-union (mIoU), and discrete Kappa coefficient (SeK). The calculation formulas are as follows:

[0187]

[0188] In the formula, TP represents the region where the model correctly detected changes, TN represents the region where the model correctly detected no changes, FP represents the region where the model incorrectly detected no changes as changes, FN represents the region where the model failed to detect changes, and η represents the ratio of the number of predicted samples to the number of actual samples. The evaluation results are shown in Table 1.

[0189] Table 1

[0190] F1-Score (%) Accuracy (%) mIoU (%) SeK (%) HRSCD-4 59.17 86.32 72.01 20.15 SSCDL 61.84 87.54 72.63 22.31 Bi-SRNet 61.95 87.48 72.73 22.35 DEFO 62.61 87.60 73.64 23.55 Method of the present invention 63.23 88.64 75.30 25.02

[0191] As can be seen, the method proposed in this invention significantly outperforms the other four methods in all metrics. Compared to the HRSCD-4 network based on fully convolution and difference, the SSCDL network based on fully convolution and aggregated feature change detection, the Bi-SRNet spatiotemporal attention network based on similarity information and global features, and the DEFO fully convolutional network focusing on decoder feature reconstruction, this invention has significant advantages in semantic change detection performance and accuracy.

[0192] To demonstrate that introducing a cross-sample semantic consistency information extraction module and a corresponding semantic consistency loss function can enhance the accuracy and reliability of this invention, an ablation experiment was conducted on this branch, and the positive effect of introducing cross-sample semantic consistency information on this invention was verified from both quantitative and qualitative analysis perspectives. Furthermore, an ablation experiment was also designed to verify the effectiveness of the cross-attention mechanism pyramid structure, which can better reconstruct strip-shaped and small-volume features.

[0193] From a quantitative analysis perspective, the semantic consistency loss function, the entire cross-sample semantic consistency extraction process, and the pyramid structure of the cross attention mechanism were removed. The detection results were evaluated by calculating four commonly used metrics—F1-Score, Accuracy, mIoU, and SeK—on the test set. The detection results are shown in Table 2.

[0194] Table 2

[0195]

[0196]

[0197] From a qualitative analysis perspective, the visualization results are as follows: Figure 7 As shown in the figure, (1) to (2) represent different sample pairs, where the first row of each sample pair represents the image before the change and the second row represents the image after the change. GT represents the true annotation result of the image. In the figure, (a) to (d) represent the test results without semantic feature extraction branch, without semantic consistency function, without cross attention mechanism pyramid structure, and with the complete model, respectively. It can be seen from the two examples that without using semantic feature extraction branch, the model cannot accurately complete the detection in terms of both semantics and shape. After adding semantic feature extractor and corresponding loss function, this situation is improved to a certain extent, but more accurate and reliable recognition still requires the assistance of cross attention mechanism pyramid structure.

[0198] In summary, through quantitative and qualitative analysis, this invention has achieved good results in terms of accuracy and reliability.

Claims

1. A method for detecting semantic changes in remote sensing images based on cross-sample semantic contrastive learning, characterized in that, include: Construct a twin residual network with shared weights; The original dual-time remote sensing images are cropped into corresponding blocks and then input into the corresponding branches of the Siamese residual network. A feature extraction module is constructed and embedded after the first stage of the Siamese residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, including: In a certain temporal branch of the twin residual network, the feature map Input is fed into the first stage to generate feature maps. feature map After passing through a 7×7 convolutional layer, a feature map is generated. The formula is: feature map Feature extraction is performed using three branches, employing 1×1, 3×3, and 7×7 convolutional layers respectively, to obtain feature maps. as well as The formula is: In the first branch, a max-pooling strategy is used for downsampling to obtain an auxiliary fusion feature map. The formula is: In the formula, MaxPool2D represents the maximum pooling function; The feature maps generated by the second and third branches are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out. Directly embed the multi-scale feature map Out into the feature map. Obtain feature map The feature maps are then fused with an auxiliary layer after passing through a 7×7 convolutional layer. Low-rank attention fusion is performed again to obtain an abstract fused feature map. The formula is: In the formula, LowRankAttn represents low-rank attention fusion; A change detection module is constructed, and the intermediate features generated in the third stage of the twin residual network are input into the change detection module to generate a change result feature map; A multi-scale feature pyramid fusion module is constructed. The auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

2. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 1, characterized in that, The feature maps generated by the second and third branches are fused using a low-rank attention mechanism to generate a multi-scale feature map Out, including: feature map Using a 1×1 convolutional layer for feature mapping, we obtain the query feature q, the matching feature k, and the value feature v, as shown in the following formulas: For the matching feature k and the value feature v, a 4×4 max pooling function and a 16×16 average pooling function are used to obtain the feature representations of individual salient information and local average information, respectively. At the same time, the scale compression of the original feature map is performed. Then, the salient information and the average information are flattened along the spatial dimension and concatenated to obtain the corresponding feature map. and The formulas are as follows: In the formula, Concat represents a connection, and AvgPool2D represents the average pooling function; feature map Perform a matrix multiplication with the query feature q, and obtain the association weights between the input features using the Softmax function, then sum... Perform multiplication to generate a multi-scale feature map Out, using the following formula:

3. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 2, characterized in that, The intermediate features generated in the third stage of the Siamese residual network are then input into the change detection module to generate a change result feature map, including: The intermediate feature maps generated in the third stage of the Siamese residual network are denoted as follows: and Obtained through cosine similarity calculation The similarity weight Sim is calculated using the following formula: In the formula, Cosine represents the cosine similarity function; The difference weights Dif are obtained by subtracting the all-one matrices, using the following formula: Dif = 1 - Sim Using 1×1 convolutional layers Perform feature mapping, then calculate the difference and take the absolute value, and finally multiply by the similarity weight Sim to obtain the hard-to-distinguish feature map A. t The formula is: Cascaded feature maps along the channel direction Then multiply by the difference weight Dif, and finally fuse them at the channel level through a 1×1 convolutional layer to obtain the significant difference feature map D. t The formula is: Significant difference feature map D t Using 3×3 and 5×5 convolution kernels respectively, saliency information was extracted. And used as a global reference and difficult-to-distinguish feature map A t Perform attention interaction to obtain a map of the changed regions. The formula is: Map of changing areas and Channel fusion is performed using cascaded 1×1 convolutional layers, and significant difference feature maps D are embedded in the form of residual connections. t The feature map O of the fine-grained change results is obtained. t The formula is:

4. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 3, characterized in that, The auxiliary fusion feature map, the abstract fusion feature map, and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including: Assist fusion feature map pair The features are concatenated along the temporal dimension and extracted using 3D convolution. Then, a cross-attention mechanism is used to capture the spatial relationship between the two temporal fusion features, resulting in a more comprehensive auxiliary feature map F. t The formula is: In the formula, CrissCrossAttn represents the cross-attention mechanism; Abstract fusion feature map pairs Concatenate along the temporal dimension and use 3D convolution for temporal feature extraction. Use a low-rank attention mechanism to transform the resulting feature map O. t By integrating it into the global semantic information, an abstract feature map H is obtained. t The formula is: Feature map of the change result O t The channel-level scaling factor β and the spatial-level offset matrix γ are generated through two branches, respectively, where β is based on the feature map O of the transformation result. t The average change information at each channel level is obtained by projecting from the global average pooling layer and the linear layer, as shown in the formula: β=Linear(GlobalAvgPool2D(O t )) In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation; γ is obtained through upsampling 1×1 convolution mapping and Tanh gate function, as shown in the formula: γ=Tanh(Conv 1×1 (Upsample 2× (O t ))) In the formula, Upsample represents the upsampling operation; By explicitly constructing a linear transformation process using the scaling factor β and the offset matrix γ, F t Projecting the changes into the feature space yields a semantically consistent change detection feature map C. t The formula is: C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t )) The auxiliary fusion feature maps are respectively and abstract feature map H t The features obtained after upsampling are channel-concatenated and then aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is: The generated bi-temporal semantic segmentation results and change detection feature map C t These are fed into the prediction head to generate the final result of semantic change detection.

5. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 4, characterized in that, The generated bi-temporal semantic segmentation results and change detection feature map C t The results of semantic change detection are fed into the prediction head and include: Change detection feature map C t The feature map C is obtained by bilinear interpolation upsampling and convolution, followed by the application of the Sigmoid function, as shown in the formula: C=Sigmoid(Conv 1×1 (BN(Conv 1×1 (Upsample(C t ))))) In the formula, BN represents the batch normalization operation; right Upsampling is performed separately, and then the Softmax function is used to obtain the pixel-level semantic labels [S1, S2], as shown in the formula: The semantic change detection result is obtained by multiplying the feature map C and the semantic label [S1,S2].

6. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 1, characterized in that, The loss function is composed of the contrastive loss L CON Cross-entropy loss L CD and combination loss L SS The composition, and the formula are: L=L CD +L SS +L CON in: L CON =g·L Pair +(1-g)·L Cross , In the formula, y represents the actual label. For the predicted sequence, |·| denotes the magnitude of the vector, L Pair L represents the loss between the same pair. Cross This represents the contrast loss between sample pairs, where m and n are the subscripts of the batch dimension.

Images