Remote sensing image semantic change detection method based on cross-sample semantic contrast learning

Abstract

The invention discloses a remote sensing image semantic change detection method based on cross-sample semantic contrast learning. The method comprises the following steps: constructing a twinborn residual network sharing weight; correspondingly cutting an original dual-time remote sensing image into blocks, and respectively inputting the blocks into corresponding branches of the twin residual network; constructing a feature extraction module, and embedding the feature extraction module after the first stage of the twin residual network; constructing a change detection module, and inputting an intermediate feature generated in the third stage of the twin residual network into the change detection module; and constructing a multi-scale feature pyramid fusion module, and inputting the auxiliary fusion feature map, the abstract fusion feature map and the change result feature map into the multi-scale feature pyramid fusion module to obtain a final result of semantic change detection. According to the method, a cross-sample semantic consistency feature extraction module is embedded, so that ground feature consistency information required by semantic change detection can be better learned, and the accuracy and reliability of a semantic change detection result can be improved.

Description

Technical Field

[0001] The present invention relates to image information processing technology, and in particular to a remote sensing image semantic change detection method based on cross-sample semantic contrast learning. Background Art

[0002] The main purpose of semantic change detection in remote sensing images is to identify the changed objects at the pixel level by comparing remote sensing images of the same area at different time periods, and accurately mark the actual semantic categories of these objects before and after the change. As one of the important earth observation methods in remote sensing technology, it plays a key role in many application fields such as disaster assessment, surface utilization statistics, urban and rural planning, etc. Therefore, it is of great significance to design a change detection method with high accuracy and strong generalization.

[0003] In recent years, with the continuous advancement of deep learning technology, deep learning methods have been widely used in the field of remote sensing image semantic change detection and have achieved remarkable success, but these methods still have certain shortcomings. Since the existing remote sensing image semantic change detection methods based on deep learning usually only learn the semantic change information between pairs of samples in the same area, and the mainstream semantic change detection datasets have the limitations of a small total number of semantic label samples and imbalanced samples of various types, it is difficult for existing methods to fully learn the semantic consistency information in remote sensing images, making the final annotated semantic categories inaccurate. For example, in a certain dataset, there are fewer water samples, and affected by the lighting conditions when shooting at different times, some water bodies look similar to building roofs or roads in appearance, which can easily lead to the water body being mistakenly identified as a building or road, resulting in the misidentification of the ground object category.

[0004] Therefore, how to design a method to enhance the model's anti-interference ability when samples are insufficient, thereby improving its ability to learn semantic consistency information, has become an urgent problem to be solved in the field of semantic change detection in remote sensing images. Summary of the invention

[0005] Purpose of the invention: In view of the above problems, the purpose of the present invention is to provide a remote sensing image semantic change detection method based on cross-sample semantic contrast learning.

[0006] Technical solution: A remote sensing image semantic change detection method based on cross-sample semantic contrast learning of the present invention comprises:

[0007] Construct a twin residual network with shared weights;

[0008] The original dual-time remote sensing images are correspondingly cropped into blocks and input into the corresponding branches of the twin residual network respectively;

[0009] Construct a feature extraction module and embed it after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps;

[0010] Construct a change detection module and input the intermediate features generated in the third stage of the twin residual network into the change detection module to generate a change result feature map;

[0011] A multi-scale feature pyramid fusion module is constructed, and the auxiliary fusion feature map, abstract fusion feature map and change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

[0012] Furthermore, a feature extraction module is embedded after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, including:

[0013] In a certain phase branch of the twin residual network, the feature map Input to the first stage to generate feature maps

[0014] The feature map After a 7×7 convolution layer, a feature map is generated The formula is:

[0015]

[0016] The feature map Through three branches, 1×1 convolution layer, 3×3 convolution layer and 7×7 convolution layer are used for feature extraction, and feature maps are obtained respectively. as well as The formula is:

[0017]

[0018] Use the maximum pooling strategy to downsample in the first branch to obtain the auxiliary fusion feature map The formula is:

[0019]

[0020] In the formula, MaxPool2D represents the maximum pooling function;

[0021] The feature maps generated by the second branch and the third branch are fused using the low-rank attention mechanism fusion module to generate a multi-scale feature map Out;

[0022] Embed the multi-scale feature map Out directly into the feature map Get feature map After passing through the 7×7 convolution layer, it is fused with the auxiliary feature map Perform low-rank attention fusion again to obtain the abstract fusion feature map The formula is:

[0023]

[0024] Where LowRankAttn represents low-rank attention fusion.

[0025] Furthermore, the feature maps generated by the second branch and the third branch are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out, including:

[0026] The feature map Use a 1×1 convolutional layer for feature mapping to obtain query feature q, matching feature k, and value feature v. The formulas are:

[0027]

[0028] For the matching feature k and value feature v, a 4×4 maximum pooling function and a 16×16 average pooling function are used to obtain the feature expressions of individual salient information and local average information, respectively, and the scale compression of the original feature map is completed at the same time. Then, the salient information and the average information are flattened and connected along the spatial dimension to obtain the corresponding feature map. and The formulas are:

[0029]

[0030] In the formula, Concat means connection, AvgPool2D means average pooling function;

[0031] The feature map Perform matrix multiplication with the query feature q, and obtain the association weights between the input features through the Softmax function, and then Multiply them to generate a multi-scale feature map Out. The formula is:

[0032]

[0033] Furthermore, the intermediate features generated in the third stage of the twin residual network are input into the change detection module to generate a change result feature map, including:

[0034] The intermediate feature maps generated in the third stage of the twin residual network are recorded as and Calculated by cosine similarity The similarity weight Sim is as follows:

[0035]

[0036] In the formula, Cosine represents the cosine similarity function;

[0037] Use all-1 matrix subtraction to obtain the difference weight Dif, the formula is:

[0038] Dif=1-Sim

[0039] Use a 1×1 convolutional layer pair Perform feature mapping, then make a difference and take the absolute value, and then multiply the similarity weight Sim to obtain the difficult feature map A t , the formula is:

[0040]

[0041] After concatenating feature maps along the channel direction Then multiply it by the difference weight Dif, and finally fuse it from the channel level through a 1×1 convolution layer to obtain the significant difference feature map D t , the formula is:

[0042]

[0043] For the significantly different feature maps D t Use 3×3 convolution kernel and 5×5 convolution kernel to extract saliency information respectively. And serve as a global reference and hard-to-distinguish feature map A t Perform attention interaction to obtain the change area map The formula is:

[0044]

[0045] Change area map and Cascade and use 1×1 convolutional layers for channel fusion, and embed the significant difference feature map D in the form of residual connection t , and obtain the fine-grained change result feature map O t , the formula is:

[0046]

[0047] Furthermore, the auxiliary fusion feature map, the abstract fusion feature map and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including:

[0048] The auxiliary fusion feature map Cascade along the time dimension, use 3D convolution for feature extraction, and then use the cross attention mechanism to capture the relationship between the dual-time fusion features in the spatial dimension to obtain an auxiliary feature map F with more comprehensive spatiotemporal features. t, the formula is:

[0049]

[0050] In the formula, CrissCrossAttn represents the cross attention mechanism;

[0051] Abstract fusion feature map Cascade along the time dimension, use 3D convolution to extract time features, and use the low-rank attention mechanism to convert the change result feature map O t Integrate into the global semantic information to obtain the abstract feature map H t , the formula is:

[0052]

[0053] The change result feature map O t The two branches are used to generate the channel-level scaling factor β and the spatial-level offset matrix γ, where β is based on the feature map O of the change result. t The average change information of each channel level is obtained through the global average pooling layer and the linear layer projection, and the formula is:

[0054] β=Linear(GlobalAvgPool2D(O t ))

[0055] In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation;

[0056] γ is obtained by upsampling the 1×1 convolution map and the Tanh gating function, and the formula is:

[0057] γ=Tanh(Conv 1×1 (Upsample 2× (O t )))

[0058] In the formula, Upsample represents the upsampling operation;

[0059] By using the scaling factor β and the offset matrix γ, a linear transformation process is explicitly constructed to transform F t Projected into the change feature space, we get a change detection feature map C with semantic consistency t , the formula is:

[0060] C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t ))

[0061] The auxiliary fusion feature maps are and the abstract feature map H t The features obtained after upsampling are channel-concatenated and channel-aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is:

[0062]

[0063] The generated bi-temporal semantic segmentation results And the change detection feature map C t They are sent to the prediction head respectively to generate the final result of semantic change detection.

[0064] Furthermore, the generated bi-temporal semantic segmentation results And the change detection feature map C t They are sent to the prediction head respectively to generate the final results of semantic change detection, including:

[0065] The change detection feature map C t Through bilinear interpolation upsampling and convolution, and then applying the Sigmoid function, the feature map C is obtained, and the formula is:

[0066] C = Sigmoid (Conv 1×1 (BN(Conv 1×1 (Upsample(C t )))))

[0067] In the formula, BN represents batch normalization operation;

[0068] right Upsample them respectively, and then use the Softmax function to obtain pixel-level semantic labels [S 1 ,S 2 ], the formula is:

[0069]

[0070] The feature map C and the semantic label [S 1 ,S 2 ] to perform dot multiplication and obtain the semantic change detection result.

[0071] Furthermore, the loss function is composed of the contrast loss L CON , cross entropy loss L CD and the combined loss L SS Composition, formula:

[0072] L=L CD +L SS +L CON

[0073] in:

[0074]

[0075] L CON = g·L Pair +(1-g)·L Cross ,

[0076] In the formula, y is the true label, is the prediction sequence, ‖·‖ represents the modulus of the vector, L Pair Indicates the loss between the same pair, L Cross represents the contrast loss between pairs of samples, and m and n are subscripts of the batch dimension.

[0077] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0078] 1. The present invention incorporates a cross-sample semantic consistency feature extraction module, so that it can better learn the object consistency information required for semantic change detection and improve the accuracy and reliability of the semantic change detection results. The present invention verifies the accuracy and reliability of the invention from the perspectives of qualitative and quantitative analysis.

[0079] 2. In terms of accuracy: Through quantitative analysis, the accuracy indicators of the present invention on the public dataset SECOND are significantly improved compared with the current advanced methods. Through visual comparative analysis, the semantic change detection results produced by the present invention are significantly improved in overall detection effect compared with other methods.

[0080] 3. From the perspective of reliability: verified by ablation experiments, after removing the proposed semantic consistency extraction module and the corresponding consistency loss function, the detection index of the model dropped significantly, and the performance was significantly worse on objects that are greatly affected by environmental changes, indicating that the cross-sample semantic consistency feature extraction module and consistency loss function designed by the present invention play an important role. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flow chart of the remote sensing image semantic change detection method based on cross-sample semantic contrast learning;

[0082] Figure 2 This is the structural diagram of the remote sensing image semantic change detection model;

[0083] Figure 3 This is the structural diagram of the low-rank attention mechanism fusion module;

[0084] Figure 4 It is the structural diagram of the change detection module;

[0085] Figure 5 This is the structural diagram of the multi-scale feature pyramid fusion module;

[0086] Figure 6 This is a comparison chart of the detection results of this example and other models;

[0087] Figure 7 This is the ablation experiment result of this example. DETAILED DESCRIPTION

[0088] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0089] The flowchart of the remote sensing image semantic change detection method based on cross-sample semantic contrast learning described in this embodiment is as follows: Figure 1 As shown, it at least includes the following steps 1 to 5.

[0090] Step 1: Build a twin residual network with shared weights;

[0091] Among them, the twin residual network is ResNet-18.

[0092] Step 2: Crop the original dual-time remote sensing image into blocks and input them into the corresponding branches of the twin residual network respectively.

[0093] The original high-resolution dual-time remote sensing images are cropped into image pairs of uniform size [t 1 ,t 2 ], t 1 represents the first phase, i.e. the image before the change, t 2 Represents the second phase, that is, the changed image, and the image pair [t 1 ,t 2 ] is input into the shared weight twin residual network ResNet-18.

[0094] Step 3: construct a feature extraction module and embed it after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps.

[0095] The twin residual network ResNet-18 is divided into four stages. A feature extraction module is embedded after the first stage of the twin residual network, so that the intermediate features output by the first stage are sent to the feature extraction module for learning cross-sample semantic consistency. The feature extraction module outputs the auxiliary fusion feature map and abstract fusion feature map of the first phase, as well as the auxiliary fusion feature map and abstract fusion feature map of the second phase.

[0096] Step 4: Build a change detection module and input the intermediate features generated in the third stage of the twin residual network into the change detection module to generate a change result feature map.

[0097] After the third stage of the twin residual network, the intermediate feature map of the first phase and the intermediate feature map of the second phase are generated respectively, and the intermediate feature maps of the two phases are Figure 1 The same is input into the change detection module to obtain the preliminary detection results of the changed area.

[0098] Step 5: construct a multi-scale feature pyramid fusion module, input the auxiliary fusion feature map, the abstract fusion feature map and the change result feature map into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

[0099] In order to ensure the consistency between the segmented semantic labels and the change detection results, and to apply the semantic consistency information to improve the accuracy of the change detection part, a multi-scale feature pyramid fusion module based on 3D convolution is constructed. By fusing semantic features and change detection features of different scales, the collaborative consistency of the two is enhanced, and weighted fusion is performed. Finally, the semantic change detection results are obtained through a pixel-level classifier.

[0100] In one example, since the high-resolution remote sensing image itself is large in size, in consideration of computational complexity and computing resources, it is necessary to convert the original dual-time image t 1 ,t 2 Cut and get dual time input after cutting and Among them, t 1 represents the image before the change, t 2 Represents the changed image. H represents the height of the image, W represents the width of the image, and 3 represents that the image is an RGB three-channel image.

[0101] Furthermore, a feature extraction module is embedded after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, such as Figure 2 As shown, including:

[0102] Taking the first phase branch as an example, in the first phase branch of the twin residual network, the feature map Input into the twin residual network, generating an intermediate feature map after the first stage After the third stage, the intermediate feature map is generated T represents the base number of feature map channels.

[0103] The feature map After the 7×7 convolution layer, the same scale sampling is performed to generate a feature map with a larger receptive field. The formula is:

[0104]

[0105] The feature map Through three branches, convolution kernels of different sizes are used in each branch to extract features to capture the information of objects with different volumes. Specifically, 1×1 convolution layer, 3×3 convolution layer and 7×7 convolution layer are used for feature extraction to obtain feature maps respectively. as well as The formula is:

[0106]

[0107] Use the maximum pooling strategy to downsample in the first branch to obtain the auxiliary fusion feature map The formula is:

[0108]

[0109] In the formula, MaxPool2D represents the maximum pooling function;

[0110] The feature maps generated by the second branch and the third branch are fused using the low-rank attention mechanism fusion module to generate a multi-scale feature map Out;

[0111] Embed the multi-scale feature map Out directly into the feature map Get feature map After passing through the 7×7 convolution layer, it is fused with the auxiliary feature map Perform low-rank attention fusion again to obtain the abstract fusion feature map The formula is:

[0112]

[0113] Where LowRankAttn represents low-rank attention fusion.

[0114] Similarly, in the second phase branch, the feature map Input into the twin residual network, generating an intermediate feature map after the first stage After the third stage, the intermediate feature map is generated Finally, the auxiliary fusion feature map is obtained through the feature extraction module And abstract fusion feature map

[0115] To aggregate feature maps with smaller receptive fields And feature maps with larger receptive fields In order to fully capture the features of objects of various scales, this application designs a low-rank attention mechanism as a feature fusion strategy. This strategy downsamples features with larger receptive fields by using the maximum pooling MaxPool2D with a smaller mask size and the average pooling AvgPool2D with a larger mask size, and inputs them into the attention mechanism process as alternative features. This can greatly reduce the computational cost while ensuring the important information of the feature map as much as possible, thereby achieving the purpose of low-overhead and fast calculation.

[0116] like Figure 3 As shown, further, the feature maps generated by the second branch and the third branch are fused using a low-rank attention mechanism fusion module to generate a multi-scale feature map Out, including:

[0117] The feature map Use a 1×1 convolutional layer for feature mapping to obtain query feature q, matching feature k, and value feature v. The formulas are:

[0118]

[0119] For the matching feature k and value feature v, a 4×4 maximum pooling function and a 16×16 average pooling function are used to obtain the feature expressions of individual salient information and local average information, respectively, and the scale compression of the original feature map is completed at the same time. Then, the salient information and the average information are flattened and connected along the spatial dimension to obtain the corresponding feature map. and The formulas are:

[0120]

[0121] In the formula, Concat represents connection, AvgPool2D represents the average pooling function; relative to k and v before equivalent substitution, and Can reduce the amount of calculation to the original

[0122] The feature map Perform matrix multiplication with the query feature q, and obtain the association weights between the input features through the Softmax function, and then Multiply them to generate a multi-scale feature map Out. The formula is:

[0123]

[0124] In order to preliminarily complete the change detection task, a change detection module was constructed, using the "double-end suppression" strategy, that is, suppressing both unchanged and significantly changed areas at the same time, thereby increasing the attention paid to the areas with difficult changes, so as to more accurately complete the preliminary change detection and obtain the change result feature map Ot .

[0125] like Figure 4 As shown, specifically, the intermediate features generated in the third stage of the twin residual network are input into the change detection module to generate a change result feature map, including:

[0126] The intermediate feature maps generated in the third stage of the twin residual network are recorded as and Calculated by cosine similarity The similarity weight Sim is as follows:

[0127]

[0128] In the formula, Cosine represents the cosine similarity function;

[0129] The more similar the channel features are, the greater the position weight is, and vice versa, the smaller it is, and the value range is [0,1]. Use the all-1 matrix subtraction to obtain the difference weight Dif with a larger difference feature weight. The formula is:

[0130] Dif=1-Sim

[0131] For the difficult-to-separate regions, since their similarities and differences are not prominent, a 1×1 convolutional layer is used. Perform feature mapping, then make a difference and take the absolute value to directly subtract the features with greater similarity, and then multiply them by the similarity weight Sim, thereby enlarging those difficult-to-distinguish change areas in disguise and obtaining the difficult-to-distinguish feature map A. t , the formula is:

[0132]

[0133] on the other hand, The semantic information and significant difference information contained in it are also required for the change detection task. For this part of the features, the cascade feature mapping along the channel direction Then multiply it by the difference weight Dif, and finally fuse it from the channel level through a 1×1 convolution layer to obtain the significant difference feature map D t , the formula is:

[0134]

[0135] In order to effectively utilize D t To identify indistinguishable changes at different scales, the D t Use 3×3 convolution kernel and 5×5 convolution kernel to extract saliency information respectively. And serve as a global reference and hard-to-distinguish feature map A tPerform attention interaction to better extract A t The real change area in the indistinguishable changes of different scales is obtained, and the change area map is obtained The formula is:

[0136]

[0137] Change area map and Cascade and use 1×1 convolutional layers for channel fusion, and embed the significant difference feature map D in the form of residual connection t , and obtain the fine-grained change result feature map O t , the formula is:

[0138]

[0139] In the change detection module, cosine similarity is used to model the change of each pixel position of the bi-phase feature. The lower the cosine similarity, the greater the difference in feature expression, and the difference between vectors with similar feature expressions is also smaller. t When , we first suppress the similar expressions in the dual-phase features by means of differences, and then multiply the similarity weight Sim to suppress the areas with significant differences through lower cosine similarity weights, thereby enlarging the difficult-to-distinguish change areas in disguise. On the other hand, considering that the significant change area is also an important goal of the change detection task, we retain those significant change areas by calculating the difference weight Dif in parallel, and use convolution to learn them. Finally, through the attention mechanism, the features of significant changes are used to guide the detection of the real change position in the difficult-to-distinguish change area, and finally obtain a fine-grained change result feature map O t .

[0140] In order to fuse the auxiliary fusion feature map Abstract fusion feature map And the change result feature map O t , thereby obtaining a bi-temporal semantic label with the correct change position, that is, a semantic change detection result. This application constructs a multi-scale feature pyramid fusion module to aggregate semantically related features and change features, improve the synergy between the two, and upsample the features to a scale suitable for prediction by the prediction head. The scale of is larger, retaining more intuitive detail features, while the abstract fusion feature map The scale of is small, which is an abstract feature that aggregates multi-scale receptive field information. Therefore, different strategies should be adopted when interacting semantic information and change detection information.

[0141] like Figure 5Specifically, the auxiliary fusion feature map, the abstract fusion feature map and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including:

[0142] The auxiliary fusion feature map Cascade along the time dimension, use 3D convolution for feature extraction, and then use the cross attention mechanism to capture the relationship between the dual-time fusion features in the spatial dimension to obtain an auxiliary feature map F with more comprehensive spatiotemporal features. t , the formula is:

[0143]

[0144] In the formula, CrissCrossAttn represents the cross attention mechanism;

[0145] Abstract fusion feature map Cascade along the time dimension, use 3D convolution to extract time features, and use the low-rank attention mechanism to convert the change result feature map O t Integrate into the global semantic information to obtain the abstract feature map H t , the formula is:

[0146]

[0147] The change result feature map O t The two branches are used to generate the channel-level scaling factor β and the spatial-level offset matrix γ, where β is based on the feature map O of the change result. t The average change information of each channel level is obtained through the global average pooling layer and the linear layer projection, and the formula is:

[0148] β=Linear(GlobalAvgPool2D(O t ))

[0149] In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation;

[0150] γ is obtained by upsampling the 1×1 convolution map and the Tanh gating function, and the formula is:

[0151] γ=Tanh(Conv 1×1 (Upsample 2× (O t )))

[0152] In the formula, Upsample represents the upsampling operation;

[0153] By using the scaling factor β and the offset matrix γ, a linear transformation process is explicitly constructed to transform Ft Projected into the change feature space, we get a change detection feature map C with semantic consistency t , the formula is:

[0154] C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t ))

[0155] The auxiliary fusion feature maps are and the abstract feature map H t The features obtained after upsampling are channel-concatenated and channel-aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is:

[0156]

[0157] The generated bi-temporal semantic segmentation results And the change detection feature map C t They are sent to the prediction head respectively to generate the final result of semantic change detection.

[0158] Considering The receptive field of is smaller and the detailed information is richer. It is necessary to extract the global correlation of the features of the objects it contains. In this example, the cross attention mechanism is used to achieve this goal with low cost and high efficiency. t and O t When , the 2D scaling transformation strategy is adopted, that is, F t For each position in O, construct a set of t The affine transformation coefficients β and γ of , where β is obtained by global average pooling (GlobalAvgPool2D), focusing on channel-level scaling, and γ is obtained by convolution and Tanh() function, focusing on spatial-level transformation. t and O t When , a low-rank attention mechanism is used to efficiently establish a global association between representative abstract features and change features. Finally, the change detection result C t By F after 2D scaling transformation t Same as H t After adding, upsampling is obtained, and the semantic segmentation result is By convolution fusion F t and H t get.

[0159] Furthermore, the generated bi-temporal semantic segmentation results And the change detection feature map C tThey are sent to the prediction head respectively to generate the final results of semantic change detection, including:

[0160] The change detection feature map C t Through bilinear interpolation upsampling and convolution, and then applying the Sigmoid function, the feature map C is obtained, and the formula is:

[0161] C = Sigmoid (Conv 1×1 (BN(Conv 1×1 (Upsample(C t )))))

[0162] In the example, each pixel can be thresholded at 0.5. Pixels with a value greater than the threshold are considered to be in the change class, and pixels with a value less than 0.5 are considered to be in the unchanged class.

[0163] right The upsampling is performed separately, but the number of output channels in the second convolution is the actual number of label categories (not 1), and then the Softmax function is used to obtain the pixel-level semantic label [S 1 ,S 2 ], the formula is:

[0164]

[0165] Since all unchanged regions in the results of semantic change detection are defaulted to the unchanged class (class 0), the feature map C and the semantic label [S 1 ,S 2 ] to perform dot multiplication to obtain the semantic change detection result to ensure the consistency of the final semantic change detection result.

[0166] The shared weighted twin residual network, feature extraction module, change detection module and multi-scale feature pyramid fusion module constructed based on the above steps constitute the remote sensing image semantic change detection model in this application, and this model is used to realize change detection of remote sensing images of different phases.

[0167] The loss function of the remote sensing image semantic change detection model is composed of the contrast loss L that optimizes the semantic consistency information. CON , optimize the cross entropy loss L of the change detection results CD and the combined loss L that optimizes the semantic segmentation results SS Composition, formula:

[0168] L=L CD +L SS +L CON

[0169] in:

[0170]

[0171] In the formula, y is the true label, To predict the sequence, ‖·‖ represents the modulus of the vector.

[0172] Contrastive loss L CON This involves the loss calculation between cross-sample pairs, which is explained here. First, for Any set of pixels that meet the requirements [p 1 ,p 2 ], using cosine similarity as the consistency metric between the two, namely:

[0173]

[0174] About pixel pairs [p 1 ,p 2 The selection criteria for ] are as follows: Any sample in each batch Where m and n are the subscripts of the batch dimension. If m = n, that is, the two-phase samples are the same pair, [p 1 ,p 2 ] should have the same pixel positions, with:

[0175]

[0176] At this time, only the similarity of the unchanged regions between the two phases is compared.

[0177] If m≠n, that is, when comparing semantic consistency across samples, [p 1 ,p 2 ] does not need to be strictly corresponding to the position pixels, but:

[0178]

[0179] L Cross =f·∑Cosine(p 1 ,p 2 )+(1-f)·∑1-Cosine(p 1 ,p 2 )

[0180] Comprehensive L Pair and L Cross , the semantic consistency loss function L CON It can be expressed as:

[0181] L CON = g·L Pair +(1-g)·L Cross ,

[0182] To verify the effectiveness of the present invention, the remote sensing image semantic change detection model described in the present invention was trained and tested on the public dataset SECOND. The detection results were compared with the current more advanced semantic change detection networks, including HRSCD-4, SSCDL, Bi-SRNet and DEFO. At the same time, the present invention conducted quantitative and qualitative analysis in both comparative experiments and ablation experiments.

[0183] The SECOND dataset is a widely used open source semantic change detection dataset. It collects 4662 pairs of aerial images distributed in Hangzhou, Chengdu and Shanghai from multiple platforms and sensors. Due to commercial copyright restrictions, the number of publicly available image pairs is currently 3561 pairs. Each image is 512×512 in size and annotated at the pixel level, covering six major land cover categories, namely non-vegetated surface, trees, low vegetation, water, buildings and land, and an invariant class, namely all objects that have not changed between the two time phases. Following the existing mainstream approach, these images will not be cropped and will be divided into 2375 pairs of training samples, 593 pairs of validation samples and 593 pairs of test samples. The model weights that perform best on the validation set are selected for testing.

[0184] In order to verify the effectiveness of the method of the present invention from the perspective of accuracy, the present invention is compared with the HRSCD-4 network based on fully convolutional semantic recognition, the SSCDL network based on full convolution and aggregated feature change detection, the spatiotemporal attention network Bi-SRNet based on similarity information and global features, and the fully convolutional network DEFO focusing on decoder feature restoration on the SECOND dataset.

[0185] From the perspective of qualitative analysis, the test results are visualized. Figure 6 In the figure, white represents unchanged areas, dark green represents trees, light green represents low vegetation, dark blue represents buildings, gray represents non-vegetated surfaces, and light blue represents land. Figure 6 (1) to (4) represent different sample pairs, where the first row of each pair of samples represents the image before the change, and the second row represents the image after the change. It can be seen from the figure that the semantic change detection effect of the present invention is significantly better than other models.

[0186] From the perspective of quantitative analysis, the detection results are calculated on the test set to evaluate the detection performance by calculating the four common indicators: the average F1 score (F1-Score), accuracy (Accuracy), mean intersection over union (mIoU) and discrete Kappa coefficient (SeK). The calculation formulas are as follows:

[0187]

[0188] In the formula, TP represents the area where the model correctly detects changes, TN represents the area where the model correctly detects no changes, FP represents the area where the model mistakenly detects no changes as changes, FN represents the area where the model fails to detect real changes, η represents the ratio of the number of predicted samples to the number of real samples, and the evaluation results are shown in Table 1.

[0189] Table 1

[0190] F1-Score(%) Accuracy(%) mIoU(%) SeK(%) HRSCD-4 59.17 86.32 72.01 20.15 SSCDL 61.84 87.54 72.63 22.31 Bi-SRNet 61.95 87.48 72.73 22.35 DEFO 62.61 87.60 73.64 23.55 Method of the present invention 63.23 88.64 75.30 25.02

[0191] It can be seen that the method proposed in this invention is significantly higher than the other four methods in all indicators. Compared with the HRSCD-4 network based on full convolution and differential, the SSCDL network based on full convolution and aggregated feature change detection, the spatiotemporal attention network Bi-SRNet based on similarity information and global features, and the full convolution network DEFO focusing on decoder feature restoration, this invention has great advantages in semantic change detection performance and accuracy.

[0192] In order to prove that the introduction of the cross-sample semantic consistency information extraction module and the corresponding semantic consistency loss function can enhance the accuracy and reliability of the present invention, the present invention conducted an ablation experiment on this branch and verified the positive effect of the introduction of cross-sample semantic consistency information on the present invention from the perspective of quantitative and qualitative analysis. In addition, for the cross-attention mechanism pyramid structure that can better restore strip-shaped and small-volume objects, an ablation experiment was also designed to verify its effectiveness.

[0193] From the perspective of quantitative analysis, we remove the semantic consistency loss function, the entire cross-sample semantic consistency extraction process, and the cross-attention mechanism pyramid structure. The four common indicators of F1-Score, Accuracy, mIoU and SeK are calculated on the test set to evaluate the detection performance. The detection results are shown in Table 2.

[0194] Table 2

[0195]

[0196]

[0197] From the perspective of qualitative analysis, the visualization results are as follows: Figure 7As shown in the figure, (1) to (2) represent different sample pairs, where the first row of each pair of samples represents the image before the change, and the second row represents the image after the change. GT represents the true annotation result of the image. (a) to (d) in the figure represent the test results without semantic feature extraction branch, without semantic consistency function, without cross attention mechanism pyramid structure and complete model, respectively. It can be seen from the two examples that without using the semantic feature extraction branch, the model cannot accurately complete the detection in terms of semantics and shape. After adding the semantic feature extractor and the corresponding loss function, this situation has been improved to a certain extent, but more accurate and reliable recognition still requires the assistance of the cross attention mechanism pyramid structure.

[0198] In summary, through quantitative and qualitative analysis, the present invention has achieved good results in accuracy and reliability.

Claims

1. A remote sensing image semantic change detection method based on cross-sample semantic contrast learning, characterized in that: include: Construct a twin residual network with shared weights; The original dual-time remote sensing images are correspondingly cropped into blocks and input into the corresponding branches of the twin residual network respectively; Construct a feature extraction module and embed it after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps; Construct a change detection module, and input the intermediate features generated in the third stage of the twin residual network into the change detection module to generate a change result feature map; A multi-scale feature pyramid fusion module is constructed, and the auxiliary fusion feature map, abstract fusion feature map and change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection.

2. The remote sensing image semantic change detection method based on cross-sample semantic contrastive learning according to claim 1 is characterized in that: And embed the feature extraction module after the first stage of the twin residual network to generate auxiliary fusion feature maps and abstract fusion feature maps, including: In a certain phase branch of the twin residual network, the feature map Input to the first stage to generate feature maps The feature map After a 7×7 convolution layer, a feature map is generated The formula is: The feature map Through three branches, 1×1 convolution layer, 3×3 convolution layer and 7×7 convolution layer are used for feature extraction, and feature maps are obtained respectively. as well as The formula is: Use the maximum pooling strategy to downsample in the first branch to obtain the auxiliary fusion feature map The formula is: In the formula, MaxPool2D represents the maximum pooling function; The feature maps generated by the second branch and the third branch are fused using the low-rank attention mechanism fusion module to generate a multi-scale feature map Out; Embed the multi-scale feature map Out directly into the feature map Get feature map After passing through the 7×7 convolutional layer, it is fused with the auxiliary feature map Perform low-rank attention fusion again to obtain the abstract fusion feature map The formula is: Where LowRankAttn represents low-rank attention fusion.

3. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 2 is characterized in that: The feature maps generated by the second branch and the third branch are fused using the low-rank attention mechanism fusion module to generate a multi-scale feature map Out, including: The feature map Use a 1×1 convolutional layer for feature mapping to obtain query feature q, matching feature k, and value feature v. The formulas are: For the matching feature k and value feature v, a 4×4 maximum pooling function and a 16×16 average pooling function are used to obtain the feature expressions of individual salient information and local average information, respectively, and the scale compression of the original feature map is completed at the same time. Then, the salient information and the average information are flattened and connected along the spatial dimension to obtain the corresponding feature map. and The formulas are: In the formula, Concat means connection, AvgPool2D means average pooling function; The feature map Perform matrix multiplication with the query feature q, and obtain the association weights between the input features through the Softmax function, and then Multiply them to generate a multi-scale feature map Out. The formula is:

4. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 3 is characterized in that: The intermediate features generated in the third stage of the twin residual network are input into the change detection module to generate a change result feature map, including: The intermediate feature maps generated in the third stage of the twin residual network are recorded as and Calculated by cosine similarity The similarity weight Sim is as follows: In the formula, Cosine represents the cosine similarity function; Use all-1 matrix subtraction to obtain the difference weight Dif, the formula is: Dif=1-Sim Use a 1×1 convolutional layer pair Perform feature mapping, then make a difference and take the absolute value, and then multiply the similarity weight Sim to obtain the difficult feature map A t , the formula is: After concatenating feature maps along the channel direction Then multiply it by the difference weight Dif, and finally fuse it from the channel level through a 1×1 convolution layer to obtain the significant difference feature map D t , the formula is: For the significantly different feature maps D t Use 3×3 convolution kernel and 5×5 convolution kernel to extract saliency information respectively. And serve as a global reference and hard-to-distinguish feature map A t Perform attention interaction to obtain the change area map The formula is: Change area map and Cascade and use 1×1 convolutional layers for channel fusion, and embed the significant difference feature map D in the form of residual connection t , and obtain the fine-grained change result feature map O t , the formula is:

5. The remote sensing image semantic change detection method based on cross-sample semantic contrast learning according to claim 4 is characterized in that: The auxiliary fusion feature map, the abstract fusion feature map and the change result feature map are input into the multi-scale feature pyramid fusion module to obtain the final result of semantic change detection, including: The auxiliary fusion feature map Cascade along the time dimension, use 3D convolution for feature extraction, and then use the cross attention mechanism to capture the relationship between the dual-time fusion features in the spatial dimension to obtain an auxiliary feature map F with more comprehensive spatiotemporal features. t , the formula is: In the formula, CrissCrossAttn represents the cross attention mechanism; Abstract fusion feature map Cascade along the time dimension, use 3D convolution to extract time features, and use the low-rank attention mechanism to convert the change result feature map O t Integrate into the global semantic information to obtain the abstract feature map H t , the formula is: The change result feature map O t The two branches are used to generate the channel-level scaling factor β and the spatial-level offset matrix γ, where β is based on the feature map O of the change result. t The average change information of each channel level is obtained through the global average pooling layer and the linear layer projection, and the formula is: β=Linear(GlobalAvgPool2D(O t )) In the formula, GlobalAvgPool2D represents the global average pooling operation, and Linear represents the linear operation; γ is obtained by upsampling the 1×1 convolution map and the Tanh gating function, and the formula is: γ=Tanh(Conv 1×1 (Upsample 2× (O t ))) In the formula, Upsample represents the upsampling operation; By using the scaling factor β and the offset matrix γ, a linear transformation process is explicitly constructed to transform F t Projected into the change feature space, we get a semantically consistent change detection feature map C t , the formula is: C t =Upsample(Conv 1×1 (β⊙(1+γ)⊙F t +H t )) The auxiliary fusion feature maps are and the abstract feature map H t The features obtained after upsampling are channel-concatenated and channel-aggregated through a 1×1 convolutional layer to obtain the semantic segmentation result. and The formula is: The generated bi-temporal semantic segmentation results And the change detection feature map C t They are sent to the prediction head respectively to generate the final result of semantic change detection.

6. The remote sensing image semantic change detection method based on cross-sample semantic contrastive learning according to claim 5, characterized in that: The generated bi-temporal semantic segmentation results And the change detection feature map C t They are sent to the prediction head respectively to generate the final results of semantic change detection, including: The change detection feature map C t Through bilinear interpolation upsampling and convolution, and then applying the Sigmoid function, the feature map C is obtained, and the formula is: C=Sigmoid(Conv 1×1 (BN(Conv 1×1 (Upsample(C t ))))) In the formula, BN represents batch normalization operation; right Upsample them separately, and then use the Softmax function to obtain the pixel-level semantic label [S1, S2], the formula is: Perform dot multiplication on the feature map C and the semantic label [S1, S2] to obtain the semantic change detection result.

7. The remote sensing image semantic change detection method based on cross-sample semantic contrastive learning according to claim 1, characterized in that: The loss function is composed of contrast loss L CON , cross entropy loss L CD and the combined loss L SS Composition, formula: L=L CD +L SS +L CON in: In the formula, y is the true label, is the prediction sequence, ‖·‖ represents the modulus of the vector, L Pair Indicates the loss between the same pair, L Cross represents the contrast loss between pairs of samples, and m and n are subscripts of the batch dimension.

Images