Interactive change detection method based on confidence calculation and interaction loss accumulation

Abstract

The invention discloses an interactive change detection method based on confidence calculation and interaction loss accumulation. The method comprises the following steps: preprocessing data, generating a sampling label for distinguishing first interaction, building an interactive change detection model based on three-aspect fusion, and training the interaction capability of the model. And the data preprocessing step comprises the step of performing communication marking on the change labels so as to perform corresponding purification on the data set according to the obtained change. The step of generating the sampling labels for distinguishing the first interaction comprises the steps of calculating first-time corrosion labels and second-time corrosion labels, calculating FN areas, FP areas and corresponding corrosion labels, and calculating variable edge area labels, unchanged area labels and extreme point labels. The step of building the interactive change detection model based on the three-aspect fusion comprises the step of adding a coding module in front of an existing change detection model and an additional information fusion module based on the three-aspect fusion and fused with space channel attention. The step of training the interaction ability of the model comprises the steps of accumulating interaction loss of each batch of data of a single epoch and calculating the confidence coefficient of stopping iteration.

Description

Technical Field

[0001] This paper proposes an interactive change detection method based on confidence calculation and interaction loss accumulation. During training, the losses in the interaction process are accumulated to provide more refined interaction information guidance for the interaction model. This invention relates to the field of image processing technology, and more particularly to interactive change detection technology. Background Art

[0002] Change detection technology is a method widely used to monitor changes in images or maps in fields such as urban planning, environmental monitoring, and military reconnaissance. Traditional change detection methods based on pixel-level comparisons or feature extraction perform poorly in complex scenarios. Deep learning-based change detection methods, by learning image features and patterns, can achieve more accurate change detection.

[0003] Specifically, because deep learning-based change detection technology can leverage deep neural networks to perform end-to-end learning and feature extraction on images to accurately identify changed areas, this approach can effectively handle complex scenes and large amounts of data, and has achieved remarkable results in change detection tasks. However, this technology still faces some challenges and limitations, such as the need for large amounts of labeled data for training change detection models, the limited generalization capabilities of existing models, and the high requirements for data quality and annotation accuracy.

[0004] Furthermore, traditional change detection methods that rely on manually outlining changes often require extensive human intervention, are time-consuming and labor-intensive, and prone to subjective errors. For large-scale data and complex scenarios, manual outlining is both inefficient and inaccurate.

[0005] In summary, the interactive change detection technology proposed in this paper fully combines human intelligence with the efficient processing power of computers, improving the accuracy and reliability of change detection through user interaction and feedback. This technology can, to a certain extent, overcome the limitations of deep learning methods while also improving the efficiency and accuracy of traditional manual delineation methods. Therefore, interactive change detection technology has broad prospects and important significance in practical applications. Summary of the Invention

[0006] Purpose of the Invention: The purpose of the present invention is to provide an interactive change detection method based on confidence calculation and interaction loss accumulation. This method can firstly address the problem of continuing iterations of a single epoch's data even when the prediction is good. Secondly, it can superimpose the losses calculated from historical iterations, thereby providing the model with more reliable interaction information. Finally, this interactive change detection method can overcome the limitations of existing change detection methods when dealing with complex scenarios and large-scale data, thereby improving the accuracy, efficiency, and reliability of change detection.

[0007] Technical solution: An interactive change detection method based on confidence calculation and interaction loss accumulation, including the following steps:

[0008] (1.1) Data preprocessing: Read each set of image data in the existing public dataset, extract the change information in the label and delete the corresponding data from the dataset accordingly;

[0009] (1.2) Generating sampling labels to distinguish the first interaction: The steps to generate sampling labels to distinguish the first interaction include calculating the 1st and 2nd corrosion labels, calculating the FN and FP areas and their corresponding corrosion labels, calculating the labels of the changed edge areas, the unchanged areas, and the extreme point labels;

[0010] (1.3) Build an interactive change detection model based on three-aspect fusion: Add an encoding module and an additional information fusion module based on three-aspect fusion that integrates spatial channel attention to the existing change detection model DSAMNet;

[0011] (1.4) Training model interaction capabilities: Accumulate the interaction loss of each batch of data in a single epoch and calculate the confidence level for stopping iteration.

[0012] 2. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, wherein step (1.1) comprises the following steps:

[0013] (2.1) Sequentially detect each label in the training dataset, use the eight-connected labeling method to calculate the change information of each location distinguished by different numbers, and then calculate the area size of each change;

[0014] (2.2) Remove the image groups with little or no changes in the current dataset with a certain probability;

[0015] (2.3) Remove the image groups with very small area changes in the current dataset with a certain probability.

[0016] 3. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, wherein generating sampling labels for distinguishing first interactions in step (1.2) comprises the following steps:

[0017] (3.1) Read the label of the current image group and erode each changed area twice in succession to obtain the 1st erosion change label and the 2nd erosion change label;

[0018] (3.2) Read the label of the current image group and the predicted label of the previous iteration, calculate the FP area and FN area of ​​the error prediction between the two, and then erode the error part of the error area to obtain the corresponding FP erode label and FN erode label;

[0019] (3.3) Read the label of the current image group. If there is a change, calculate the labels of all the changed edge patches and the labels of all unchanged areas based on the current label.

[0020] (3.4) Read the label of the current image group. If the change in the current label is a large-scale change, calculate the labels of the positions of the four extreme points of each large-scale change.

[0021] 4. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, wherein the step (1.3) of building an interactive change detection model based on the fusion of three aspects comprises the following steps:

[0022] (4.1) Add an encoding module before the existing change detection model DSAMNet to encode the interactive clicks generated according to the sampling labels in steps (3.1) to (3.4) into distance images, and obtain additional channels for positive and negative clicks;

[0023] (4.2) Add an additional information fusion module based on three-way fusion after the encoding module. Superimpose the two channels obtained in step (4.2) with the prediction output of the previous model in the channel dimension to obtain 3-channel additional input information, and superimpose it with the dual-time image data in the channel dimension to obtain two 5-channel input data;

[0024] (4.3) For the 3-channel input data, it is upgraded to 8 channels through 1*1 convolution, and then passes through BN and LeakyRelu. At the same time, the same operation is performed on the two 5-channel input data to obtain two 8-channel data;

[0025] (4.4) The feature map obtained by superimposing and upgrading the last predicted label is superimposed on the feature map obtained by upgrading the dual-temporal image on the channel to obtain two 16-channel feature map data;

[0026] (4.5) The two 16-channel feature map data are passed through the CBAM module without changing the size and then subjected to 1*1 convolution for dimensionality reduction to obtain two 3-channel input data. These two 3-channel data are further subjected to BN and LeakyRelu to obtain the final input data of the existing change detection model and sent to the model for predictive learning.

[0027] 5. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, wherein the training model interaction capability in step (1.4) comprises the following steps:

[0028] (5.1) For a batch of data in a training epoch, randomly train it 1-4 times;

[0029] (5.2) No additional interactive messages are provided during the first iteration;

[0030] (5.2) During the second iteration, the two erosion labels in step (3.1) and the extreme point labels in step (3.4) are generated, and the interactive click guidance model training is generated based on them;

[0031] (5.3) In the subsequent iterative process, the 1-time corrosion label in step (3.1) and the sampling labels in steps (3.2) and (3.3) are generated, and the interactive click guidance model training is generated based on them;

[0032] (5.4) Starting from the second iteration, the confidence level is calculated to determine whether the next iterative training process needs to continue. The confidence level is mIoU, which is used to measure the degree of overlap between the prediction outputs of two iterations. If the mIoU is greater than a certain threshold, the above iteration process is stopped.

[0033] (5.5) During a certain iteration of a batch of data, the BCL loss value obtained by the model prediction calculation for each iteration is retained, and different weights are assigned to the loss values ​​of different iteration times. Since there is a deep supervision error in the interaction model selected by this patent, when the error is back-propagated for the current batch of data, what is back-propagated is the sum of the accumulated loss including the interaction loss and the deep supervision loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the structure of the interactive change detection model in the present invention;

[0035] Figure 2 Schematic diagram of the structure of the existing change detection model DSAMNet used in the present invention;

[0036] Figure 3 Schematic diagram of a method for preprocessing data labels in the present invention;

[0037] Figure 4 Schematic diagram of the sampling strategy for distinguishing first interactions adopted in the present invention;

[0038] Figure 5 Schematic diagram of the encoding effect of encoding the sampled interactive clicks in the experiment of the present invention;

[0039] Figure 6 Schematic diagram of the model interaction effect in the experiment of the present invention; DETAILED DESCRIPTION

[0040] The specific implementation of the interactive method proposed in the present invention is described in detail below with reference to six accompanying drawings.

[0041] First, we need to build an interactive change detection dataset. Since the labels of traditional change detection datasets only have two values, 0 and 1, this label format cannot help us distinguish the specific change information in the current label. Therefore, we need to use the 8-connected component labeling method to purify the dataset, marking each change in a label with a different number, so as to obtain the number of changes and the area of ​​each change. Figure 3 As shown in the figure, the areas with the original label of 1 are marked as 1, 2, 3, and 4 respectively after labeling. Based on this, we can further remove data with small area changes and data with a small number of changes in the existing change detection dataset.

[0042] Secondly, if Figure 4 As shown, when getting the current Figure 4 Connected component labels of (a) Figure 4 (b) After that, we can combine the predicted labels of the previous iteration of the model Figure 4 (c) Generate corresponding sampling labels. Figure 4 (d) with Figure 4 (e) is to Figure 4 The process of sampling at the correct point after the change area in (a) is corroded. Figure 4 (f) with Figure 4 (g) is to Figure 4 The process of sampling at the correct point after further corrosion of the changed area in (a). Figure 4 (h) with Figure 4 (i) is based on Figure 4 (a) with Figure 4 (c) The gap between the calculated FN region labels and the sampling process after the FN region is corroded. Figure 4 (j) with Figure 4 (k) is based on Figure 4 (a) with Figure 4 (c) The gap between the calculated FP region labels and the process of sampling negative points after corroding the FP region. Figure 4 (1) with Figure 4 (m) is based on Figure 4 (a) The process of sampling negative points based on the calculated edge labels of all changes and unchanged labels. Figure 4 (n) is calculated Figure 4 The process of sampling the four extreme points of the large-area change in (a). In the actual training process, the sampled labels generated at different iterations are different.

[0043] Secondly, it is necessary to build an interactive change detection model based on the integration of three aspects, such as Figure 1 As shown, Figure 2 It is the network structure of the DSAMNet model. Figure 1 The interaction model structure in Figure 4 Specifically, we first add an encoding module, which is used to encode the positive and negative clicks generated according to the sampling labels, such as Figure 5 As shown in the figure. The left side shows the three click locations, and the right side shows the generated single-channel encoding map of the same size as the original image. The obtained encoding channels of positive and negative clicks are then superimposed on the channel dimension with the dual-time image and the predicted labels obtained by the previous model prediction, respectively, to obtain three feature maps with 5, 3, and 5 channels, respectively. The three feature maps are then upscaled to 8 channels through 1*1 convolution, and then activated through BN and LeakyRelu. At this point, the 8-channel feature maps obtained by the last predicted label upscaling are superimposed on the 8-channel feature maps obtained from the dual-time image to generate two 16-channel feature maps, which are then passed through the spatial channel attention CBAM module. They are then reduced to 3 dimensions through 1*1 convolution, and the dual-time input of the DSAMNet model is obtained through BN and LeakyRelu.

[0044] Finally, the interactive model's interactive capabilities are trained. During the training of a single batch of data in a single epoch, random iterations are performed 1-4 times. Starting from the second iteration, the confidence level is calculated, i.e., the mIoU value between the predicted labels of the previous iteration and the predicted labels of the current iteration, as shown in Formula (1). The stopping threshold is set to 99.5%. If the confidence level is greater than this threshold, subsequent iterations are not performed.

[0045]

[0046] In addition, no Figure 4 The sampling labels in , that is, no interactive click guidance model training is provided. In the second iteration, according to Figure 4 (g) with Figure 4 (n) The generated sample labels generate simulated clicks to guide model training. Since the second iteration is the first interaction, it needs to be distinguished from subsequent iterations. Starting from the third iteration, Figure 4 (e) Figure 4 (i) Figure 4 (k) Figure 4 (1) with Figure 4(m) Generate interactive clicks to guide model training.

[0047] In terms of loss functions, traditional change detection loss functions fail to account for losses during iteration, potentially leading to the loss of critical interaction information. The interaction model consists of two components: deep supervision loss and prediction output loss. The deep supervision module calculates the sum of the Dice loss of the encoder's two stages (Formula (2)). The predicted output loss is calculated using the BCL loss, which is used to separate the true label from the predicted output. The overall model loss can be expressed as Formula (3).

[0048] L DICE =λ(diceLoss1+diceLoss2)(2)

[0049] loss = λ * DsLoss + L BCL (3)

[0050] The present invention improves the above loss function. That is, since there will be several iterations in the training process, the loss generated by each iteration is accumulated and balanced by weights. The weight coefficients are set to [1, 2, 3...]. Then the loss between the predicted output and the true label can be expressed as formula (4), and the overall model loss is expressed as formula (5).

[0051]

[0052] loss = λ * DsLoss + L ICL (5)

[0053] In terms of evaluation indicators, the F1 value (Formula (6)) and the mIoU value (Formula (1)) are used to measure the interactive ability of the interactive model. Among them, TP represents the number of correctly predicted changed pixels; FP represents the number of pixels that are actually unchanged but predicted to be changed; TN represents the number of correctly predicted unchanged pixels; and FN represents the number of pixels that are actually changed but predicted to be unchanged. Figure 5 The interactive effects of the proposed method for a specific scenario are shown for the first five clicks. The F1 values ​​are 0.79989, 0.84194, 0.86046, 0.87353, 0.88339, and 0.88868, respectively, and the mIoU values ​​are 0.66652, 0.72703, 0.75510, 0.77546, 0.79114, and 0.79966, respectively. It can be seen that the proposed method can quickly and precisely correct mispredicted areas.

[0054]

Claims

1. An interactive change detection method based on confidence calculation and interaction loss accumulation, characterized in that: The steps include: (1.1) Data preprocessing: Read each set of image data in the existing public dataset, extract the change information in the label and delete the corresponding data from the dataset accordingly; (1.2) Generating sampling labels to distinguish the first interaction: The steps to generate sampling labels to distinguish the first interaction include calculating the 1st and 2nd corrosion labels, calculating the FN and FP areas and their corresponding corrosion labels, calculating the labels of the changed edge areas, the unchanged areas, and the extreme point labels; (1.3) Build an interactive change detection model based on three-aspect fusion: Add an encoding module and an additional information fusion module based on three-aspect fusion that integrates spatial channel attention to the existing change detection model DSAMNet; (1.4) Training model interaction capabilities: Accumulate the interaction loss of each batch of data in a single epoch and calculate the confidence level for stopping iteration.

2. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1 is characterized in that The step (1.1) includes the following steps: (2.1) Sequentially detect each label in the training dataset, use the eight-connected labeling method to calculate the change information of each location distinguished by different numbers, and then calculate the area size of each change; (2.2) Remove the image groups with little or no changes in the current dataset with a certain probability; (2.3) Remove the image groups with very small area changes in the current dataset with a certain probability.

3. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, characterized in that Generating sampling labels for distinguishing first interactions in step (1.2) includes the following steps: (3.1) Read the label of the current image group and erode each changed area twice in succession to obtain the 1st erosion change label and the 2nd erosion change label; (3.2) Read the label of the current image group and the predicted label of the previous iteration, calculate the FP area and FN area of ​​the error prediction between the two, and then erode the error part of the error area to obtain the corresponding FP erode label and FN erode label; (3.3) Read the label of the current image group. If there is a change, calculate the labels of all the changed edge patches and the labels of all unchanged areas based on the current label. (3.4) Read the label of the current image group. If the change in the current label is a large-scale change, calculate the labels of the positions of the four extreme points of each large-scale change.

4. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1, characterized in that The step (1.3) of building an interactive change detection model based on the integration of three aspects includes the following steps: (4.1) Add an encoding module before the existing change detection model DSAMNet to encode the interactive clicks generated according to the sampling labels in steps (3.1) to (3.4) into distance images, and obtain additional channels for positive and negative clicks; (4.2) Add an additional information fusion module based on three-way fusion after the encoding module. Superimpose the two channels obtained in step (4.2) with the prediction output of the previous model in the channel dimension to obtain 3-channel additional input information, and superimpose it with the dual-time image data in the channel dimension to obtain two 5-channel input data; (4.3) For the 3-channel input data, it is upgraded to 8 channels through 1*1 convolution, and then passes through BN and LeakyRelu. At the same time, the same operation is performed on the two 5-channel input data to obtain two 8-channel data; (4.4) The feature map obtained by superimposing and upgrading the last predicted label is superimposed on the feature map obtained by upgrading the dual-temporal image on the channel to obtain two 16-channel feature map data; (4.5) The two 16-channel feature map data are passed through the CBAM module without changing the size and then subjected to 1*1 convolution for dimensionality reduction to obtain two 3-channel input data. These two 3-channel data are further subjected to BN and LeakyRelu to obtain the final input data of the existing change detection model and sent to the model for predictive learning.

5. The interactive change detection method based on confidence calculation and interaction loss accumulation according to claim 1 is characterized in that The training model interaction capability in step (1.4) includes the following steps: (5.1) For a batch of data in a training epoch, randomly train it 1-4 times; (5.2) No additional interactive messages are provided during the first iteration; (5.2) During the second iteration, the two erosion labels in step (3.1) and the extreme point labels in step (3.4) are generated, and the interactive click guidance model training is generated based on them; (5.3) In the subsequent iterative process, the 1-time corrosion label in step (3.1) and the sampling labels in steps (3.2) and (3.3) are generated, and the interactive click guidance model training is generated based on them; (5.4) Starting from the second iteration, the confidence level is calculated to determine whether the next iterative training process needs to continue. The confidence level is mIoU, which is used to measure the degree of overlap between the prediction outputs of two iterations. If the mIoU is greater than a certain threshold, the above iteration process is stopped. (5.5) During a certain iteration of a batch of data, the BCL loss value obtained by the model prediction calculation for each iteration is retained, and different weights are assigned to the loss values ​​of different iteration times. Since there is a deep supervision error in the interaction model selected by this patent, when the error is back-propagated for the current batch of data, what is back-propagated is the sum of the accumulated loss including the interaction loss and the deep supervision loss.

Images