A deep learning-based image change detection method

Abstract

The application discloses a kind of based on deep learning's image change detection method, specifically including the following steps: first, original image pair is subtracted and absolute value operation is obtained difference image;Second, generate three groups of appearance change image pairs by cyclic generation network;Third, the feature map generated by the three groups of image pairs is consistency training, by reducing consistency loss, to reduce the influence of appearance change on change detection result, and improve generalization and adaptability;Fourth, the feature map generated by original image pair is unsupervised training and supervised training;Fifth, weighted consistency loss, unsupervised loss and supervised loss, obtain total loss function;Sixth, threshold segmentation is carried out to the feature map to obtain the final binary change map.The application can detect the change area in image more accurately, and has higher generalization and adaptability.

Description

Technical Field

[0001] This invention belongs to the field of image change detection and relates to an image change detection method based on deep learning. Background Technology

[0002] In the field of image processing, change detection is a crucial task, aiming to identify significant changes occurring in images under different times or conditions. Research in this area has wide applications in monitoring urban development, environmental change, and geological evolution, and is of significant practical importance. With technological advancements, traditional pixel-difference-based methods have gradually evolved into advanced change detection techniques based on deep learning.

[0003] Change detection technology is constantly evolving. Early, simple methods involved aligning and subtracting two input images to obtain a difference image. A suitable threshold was then applied to split this difference image into a binary image: black areas corresponded to regions with low grayscale values, representing areas without change, while white areas corresponded to regions with high grayscale values, representing areas of change. However, this method has drawbacks: subtle changes in brightness caused by natural variations such as lighting and seasons could be misidentified as changes, leading to false positives; furthermore, this method only considers differences between individual pixels, neglecting contextual information between pixels. With the development of deep learning technology and the increasing demands for accuracy and efficiency in change detection research, deep learning-based change detection methods have been proposed. The papers "Fully Convolutional Siamese Networks for Change Detection" and "Building Change Detection for Remote Sensing Images Using a Dual Task Constrained Deep Siamese Convolutional Network Mode" both propose deep learning-based change detection methods. While these methods fully consider the relationships between pixel contexts, they do not address the impact of significant appearance changes in the original image pair on change detection, such as variations in lighting and season. The paper "An End-to-end Supervised Domain Adaptation Framework for CrossDomain Change Detection" considers appearance changes in the original image pair and performs appearance style transformation. However, this paper directly fuses the feature maps obtained after inputting the three transformed images into the network. When the input images have significant appearance differences, this may lead to information loss and has limited mitigation effect on appearance differences. Therefore, in practice, how to better apply deep learning to change detection technology remains a challenge to be solved. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to provide an image change detection method based on deep learning. This invention effectively utilizes the information from the difference image, focusing more intently on learning and capturing changes in the image, and reduces the impact of appearance changes in the original image pair on the change detection results through consistency training, thereby improving generalization and adaptability.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A deep learning-based image change detection method includes the following steps:

[0007] Step 1: Subtract each pixel from the two original images acquired at the two time points, and then take the absolute value of the new image to obtain the difference image; the formula is as follows:

[0008] |AB|=DI

[0009] Where A is the image acquired at an earlier time point, B is the image acquired at a later time point, and DI is the difference image;

[0010] Step 2: Generate three sets of image pairs with altered appearances using a recurrent generative network.

[0011] Let image B, which has the appearance style of image A, be denoted as image B′. The formula for generating B′ is as follows:

[0012] B′=GenA(B)

[0013] GenA(·) is a generator for the appearance of image A, used to generate an image with the appearance of image A.

[0014] Let image A, which has the appearance style of image B, be denoted as image A′. The formula for generating A′ is as follows:

[0015] A′=GenB(A)

[0016] GenB(·) is a generator of the appearance of image B, used to generate an image with the appearance of image B;

[0017] The original image pair A-image B is used to generate two additional image pairs through a recurrent generative network, resulting in the AB image pair, the AB′ image pair, and the A′B image pair.

[0018] Step 3: Generate feature maps from the three sets of images in Step 2 and perform consistency training to reduce consistency loss. The specific process is as follows:

[0019] Step 3.1: Each image pair is input into the Siamese network to generate three feature maps, as follows:

[0020] Siamese(AB)=fea1

[0021] Siamese(AB′)=fea2

[0022] Siamese(A′B=fea3

[0023] Where Siamese(·) is the operation of generating feature maps by the Siamese network, fea1 represents the feature map obtained by the Siamese network for the AB image pair, fea2 represents the feature map obtained by the Siamese network for the AB′ image pair, and fea3 represents the feature map obtained by the Siamese network for the A′B image pair.

[0024] Step 3.2: Minimize the loss among these three feature maps, i.e., reduce their consistency loss:

[0025]

[0026] Among them, fea1 i,j fea2 i,j fea3 i,j Let L represent the pixel value at position (i, j) of the feature map, respectively. consistency This indicates a loss of consistency.

[0027] Step 4: Perform unsupervised and supervised training on the feature map fea1 generated in step 3.1. The specific process is as follows:

[0028] Step 4.1: Unsupervised training is performed on the feature map fea1 and the difference image DI from step 1. The process is as follows:

[0029] L unsupervise =MSE(Res(Conv(fea1)), Res(Conv(DI)))

[0030] Where Conv(·) represents a 1*1 convolution kernel, Res(·) represents the ResNet network, MSE(·) represents the MSE loss function, and L unsupervise This indicates unsupervised loss;

[0031] Step 4.2, supervised training is performed on feature map fea1 and the ground truth labels, as follows:

[0032] L supervise =FocalLoss(fea1, GroundTruth)

[0033] Where GroundTruth represents the true label, FocalLoss(·) represents the Focal loss function, and L supervise This indicates a loss due to oversight;

[0034] Step 5, weighting L from step 3.2 consistency Step 4.1 L unsupervise And L in step 4.2 supervise The final total loss function is obtained, as shown in the following formula:

[0035] L total =αL consistency +βL supervise +ηL unsupervise

[0036] Where α, β, and η are hyperparameters, which can be set by the user; L totalIt is the total loss function;

[0037] Step 6, minimize L from step 5. total Then, the feature map fea1 is adjusted to its final state, and a binary transformation map is obtained through threshold segmentation. The process is as follows:

[0038] changeMap = Seg(fea1)

[0039] Here, Seg(·) represents the threshold segmentation process, and changeMap represents the final binary change map, which is the change detection result map.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention provides a complete deep learning-based image change detection method, which provides corresponding deep learning-based methods and implements each step of the process from image preprocessing to image change detection. It fully utilizes the advantages of deep learning methods in image feature extraction and image processing to improve the accuracy and efficiency of change detection.

[0042] This invention helps the network learn robustness to appearance changes through consistent training, making the network easier to generalize to different scenarios and datasets, and also helps improve the model's adaptability and performance in real-world change detection applications.

[0043] This invention improves sensitivity and accuracy in detecting changes by combining difference images, making them as similar as possible to feature maps. Difference images reflect the differences between images, and using them for supervision helps the model to learn and capture changes within images more effectively. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention;

[0045] Figure 2 This is a comparison of the difference plot and the true label in the Tsunami dataset.

[0046] Figure 3 The change detection maps obtained using the method of this invention for the Tsunami dataset are compared with those obtained using other methods. Detailed Implementation

[0047] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] Example

[0049] See Figure 1 The present invention provides a deep learning-based image change detection method, comprising the following specific steps:

[0050] Step 1: Obtain the raw data for the change detection task from the public dataset Tsunami. This dataset includes 100 sets of images, each set containing images collected at earlier and later time points, and is divided into training, validation, and test sets in a ratio of 7:1.5:1.5.

[0051] Step 2, taking 00000070_pre.png and 00000070_post.png from the training set as examples, subtract and take the absolute value of these two original images to obtain the difference image. The specific process is as follows: Assume the initial two images are: the image 00000070_pre.png acquired at the earlier time point is denoted as image A, and the image 00000070_post.png acquired at the later time point is denoted as image B. Subtract each pixel from the two original images acquired at the two time points, and then take the absolute value of the new image to finally obtain the difference image DI. The formula is as follows:

[0052] |AB|=DI

[0053] Step 3: Generate three pairs of images with altered appearances using a recurrent generative network. Image B, which carries the appearance style of image A, is denoted as image B′. The formula for generating B′ is as follows:

[0054] B′=GenA(B)

[0055] GenA(·) is a generator for the appearance of image A, used to generate an image with the appearance of image A.

[0056] Let image A, which has the appearance style of image B, be denoted as image A′. The formula for generating A′ is as follows:

[0057] A′=GenB(A)

[0058] GenB(·) is a generator for the appearance of image V, used to generate images with the appearance of image V.

[0059] The original image pair A and image B are used to generate two additional image pairs through a recurrent generative network, resulting in the AB image pair, the AB′ image pair, and the A′B image pair.

[0060] Step 4 involves performing consistency training on the feature maps generated from the three sets of images in Step 3 to reduce consistency loss. The specific process is as follows:

[0061] Step 4.1: Each image pair is input into the Siamese network to generate three feature maps. The process is as follows:

[0062] Siamese(AB)=fea1

[0063] Siamese(AB′)=fea2

[0064] Siamese(A′B=fea3

[0065] Where Siamese(·) is the operation of generating feature maps by the Siamese network, fea1 represents the feature map of image pair AB obtained by the Siamese network, fea2 represents the feature map of image pair AB′ obtained by the Siamese network, and fea3 represents the feature map of image pair A′B obtained by the Siamese network.

[0066] Step 4.2: Minimize the loss among these three feature maps, i.e., reduce their consistency loss:

[0067]

[0068] Among them, fea1 i,j fea2 i，j fea3 i，j Let L represent the pixel value at position (i, j) of the feature map, respectively. consistency This indicates a loss of consistency.

[0069] Step 5 involves performing unsupervised and supervised training on the feature map fea1 generated in step 4.1. The specific process is as follows:

[0070] Step 5.1, as follows Figure 2 As shown, the brighter the region in the difference image, i.e., the larger the value, the more likely the corresponding true label region is white, indicating the change. The difference image highlights the change information in the image, and using the difference image for supervision helps the model to more focusedly learn and capture changes in the image. Therefore, the feature map fea1 and the difference image DI from step 1 are used for unsupervised training, as follows:

[0071] L unsupervise =MSE(Res(Conv(fea1)), Res(Conv(DI)))

[0072] Where Conv(·) represents a 1*1 convolution kernel, Res(·) represents the ResNet network, MSE(·) represents the MSE loss function, and L unsupervise This indicates unsupervised loss;

[0073] Step 5.2: Supervised training is performed using fea1 and the ground truth labels. The process is as follows:

[0074] L supervise =FocalLoss(fea1, GroundTruth)

[0075] Where GroundTruth represents the true label, FocalLoss(·) represents the Focal loss function, and L supervise This indicates a loss of oversight.

[0076] Step 6, weighting L from step 4.2 consistency Step 5.1 L unsupervise And L in step 5.2 supervise The final total loss function is obtained. The formula is as follows:

[0077] L total =αL consistency +βL supervise +ηL unsupervise

[0078] Where L total This is the total loss function. α, β, and η are hyperparameters, set to 0.3, 0.5, and 200 in this experiment. The formula is as follows:

[0079] L total =0.3*L consistency +0.5*L supervise +200*L unsupervise

[0080] Step 7, minimize L from step 6. total Then, fea1 is adjusted to its final state. A binary transformation map is obtained through threshold segmentation. The process is as follows:

[0081] changeMap = Seg(fea1)

[0082] Here, Seg(·) represents the threshold segmentation process, and changeMap represents the final binary change map, which is the change detection result map.

[0083] The effects of this invention can be specifically demonstrated through experiments:

[0084] 1. Experimental conditions

[0085] GeForce RTX 3090 graphics card, Python 3.8, PyTorch 1.10, dataset is Tsunami dataset.

[0086] 2. Experiment Content

[0087] First, the network is fed with an image captured at an earlier time point and two images captured at a later time point, generating three image pairs with different appearance styles. Based on these three image pairs, three corresponding feature maps are generated. Unsupervised and supervised training is performed on the first image, and consistency training is conducted on the three feature maps. Finally, thresholding is applied to the feature maps generated from the original image pairs to obtain the final binary transformation map.

[0088] 3. Experimental Results

[0089] Figure 3 The images shown are change detection maps obtained using the deep learning-based image change detection method of this invention and other methods for the Tsunami dataset. (a) represents images acquired at earlier time points, (b) represents images acquired at later time points, (c) represents the ground truth label, (i) is the change detection map obtained using the deep learning-based image change detection method of this invention, and (d), (e), (f), (g), and (h) are the change detection maps obtained using other methods. The boxes in the figures demonstrate that the deep learning-based image change detection method of this invention can clearly detect changes in trees within the original image, a capability that other methods lack.

Claims

1. A deep learning-based image change detection method, characterized in that, The method specifically includes the following steps: Step 1: Subtract each pixel from the two original images acquired at the two time points, and then take the absolute value of the new image to obtain the difference image; the formula is as follows: |AB|=DI Where A is the image acquired at an earlier time point, B is the image acquired at a later time point, and DI is the difference image; Step 2: Generate three sets of image pairs with altered appearances using a recurrent generative network. Let image B, which has the appearance style of image A, be denoted as image B′. The formula for generating B′ is as follows: B′=GenA(B) Where GenA(·) is a generator of the appearance of image A, used to generate an image with the appearance of image A; Let image A, which has the appearance style of image B, be denoted as image A′. The formula for generating A′ is as follows: A′=GenB(A) GenB(·) is a generator of the appearance of image B, used to generate an image with the appearance of image B; The original image pair A-image B is used to generate two additional image pairs through a recurrent generative network, resulting in the AB image pair, the AB′ image pair, and the A′B image pair. Step 3: Generate feature maps from the three sets of images in Step 2 and perform consistency training to reduce consistency loss. The specific process is as follows: Step 3.1: Each image pair is input into the Siamese network to generate three feature maps, as follows: Siamese(AB)=fea1 Siamese(AB′)=fea2 Siamese(A′B=fea3 Where Siamese(·) is the operation of generating feature maps by the Siamese network, fea1 represents the feature map obtained by the Siamese network for the AB image pair, fea2 represents the feature map obtained by the Siamese network for the AB′ image pair, and fea3 represents the feature map obtained by the Siamese network for the A′B image pair. Step 3.2: Minimize the loss among these three feature maps, i.e., reduce their consistency loss: Among them, fea1 i,j fea2 i,j fea3 i,j Let L represent the pixel value at position (i, j) of the feature map, respectively. consistency This indicates a loss of consistency. Step 4: Perform unsupervised and supervised training on the feature map fea1 generated in step 3.

1. The specific process is as follows: Step 4.1: Unsupervised training is performed on the feature map fea1 and the difference image DI from step 1. The process is as follows: L unsupervise =MSE(Res(Conv(fea1)),Res(Conv(DI))) Where Conv(·) represents a 1*1 convolution kernel, Res(·) represents the ResNet network, MSE(·) represents the MSE loss function, and L unsupervise This indicates unsupervised loss; Step 4.2, supervised training is performed on feature map fea1 and the ground truth labels, as follows: L supervise =FocalLoss(fea1,GroundTruth) Where GroundTruth represents the true label, FocalLoss(·) represents the Focal loss function, and L supervise This indicates a loss due to oversight; Step 5, weighting L from step 3.2 consistency Step 4.1 L unsupervise And L in step 4.2 supervise The final total loss function is obtained, as shown in the following formula: L total =αL consistency +βL supervise +ηL unsupervise Where α, β, and η are hyperparameters, which can be set by the user; L total It is the total loss function; Step 6, minimize L from step 5. total Then, the feature map fea1 is adjusted to its final state, and a binary transformation map is obtained through threshold segmentation. The process is as follows: changeMap = Seg(fea1) Here, Seg(·) represents the threshold segmentation process, and changeMap represents the final binary change map, which is the change detection result map.

Images