Weakly supervised change detection method and device based on background mixed data expansion technology

Abstract

The application discloses a weakly supervised change detection method and device based on background mixed data expansion technology, and the method comprises the following steps: constructing a weakly supervised training set, a background guide set and an augmented data set; using a mask and a background to enhance an input image pair through a background-aware enhancement operation, so as to obtain a new image pair, and then obtaining an expanded final output through an enhancement strategy; constantly updating change detection model parameters by using a learning algorithm, and finally obtaining an optimal model; and finally testing the trained model. The device comprises a processor and a memory. The application uses a background guide set to enrich potential background changes in training examples; through a background-aware enhancement operation, the application helps a change detection model to see different background changes; and through a consistency loss function, the generalization ability is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data expansion, and in particular to a weakly supervised change detection method and device based on background mixed data expansion technology. BACKGROUND

[0002] Change detection is a technology that can detect the change area of a pair of dual-time images. Early change detection techniques are based on algebra, such as gradient, change vector analysis (CVA) and some other methods. These traditional change detection models cannot handle background noise, and some complex models have been gradually proposed. For example, CNN (convolutional neural network), GAN (generative adversarial network) and other models have also been gradually applied to change detection. In recent research, transformers have also been used for change detection. The change detection method mentioned in document [1] combines a hierarchical structure of the Transformer encoder with a multi-layer perception (MLP) decoder in a twin network structure, effectively presenting the multi-scale long-range details required for accurate CD. The above methods have achieved good results, but since the change detection dataset cannot contain all environmental transformations, a data expansion suitable for change detection is necessary.

[0003] Data expansion is a method that can effectively improve the generalization ability of deep models. Current data expansion techniques are mainly applied to classification and detection tasks. Document [2] introduces the CutOut (cutting out images) data expansion method, which cuts out a rectangular area of the input image and fills the pixels in the area with 0; document [3] introduces the random erasing algorithm, which randomly selects a rectangular box in the input image and randomly replaces the pixel values in the rectangular box, thereby achieving the effect of data expansion; document [4] introduces the CutMix method, which cuts out a part of the area of the image, and then fills the cut-out area with pixel values of other data in the training set to improve the robustness of the model; document [5] introduces the Mixup image mixing enhancement method, which adds two sample data according to a certain proportion to obtain new sample data. Correctly using some data expansion methods can solve the problem of insufficient data and quickly improve the generalization ability and robustness of the model.

[0004] The above-mentioned data expansion methods are only applicable to classification or detection tasks, but cannot be applied to change detection. Therefore, it is necessary to propose a method for change detection.

[0005] REFERENCES

[0006] [1] Bandara, W. G. C.; and Patel, V. M. 2022. A Transformer-Based Siamese Network for Change Detection. arXiv preprint arXiv:2201.01293.

[0007] [2] De Vries, T.; and Taylor, G. W. 2017. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552.

[0008] [3] Zhong Z, Zheng L, Kang G, et al. Random Erasing Data Augmentation [C] / / AAAI. 2020: 13001-13008.

[0009] [4] Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE / CVF international conference on computer vision, 6023-6032.

[0010] [5] Zhang H, Cisse M, Dauphin Y N, et al. mixup: Beyond Empirical Risk Minimization [C] / / International Conference on Learning Representations. 2018. SUMMARY

[0011] The application provides a weakly supervised change detection method and device based on background mixed data expansion technology (BGMix), provides a new consistency loss function (Augmented&Real Data Consistency Loss), and constructs a background guide set; the background mixed technology is used to train the deep model of the change detection in a weakly supervised manner; the background guide set is used to enrich the potential background changes in the training examples; the background-aware enhancement operation helps the change detection model to see different background changes; the generalization ability is enhanced through the consistency loss function, and details are described below:

[0012] A weakly supervised change detection method based on background mixed data expansion technology, the method comprises:

[0013] A weakly supervised training set, a background guide set and an augmented data set are constructed.

[0014] A new image pair is obtained through a background-aware enhancement operation using a mask and a background to enhance the input image pair An expanded final output is obtained through an enhancement strategy.

[0015] The change detection model parameters are continuously updated using a learning algorithm, and finally the optimal model is obtained; finally, the trained model is tested.

[0016] A new image pair is obtained through a background-aware enhancement operation using a mask and a background to enhance the input image pair Specifically,

[0017] A background pair <B1, B2> is randomly sampled from the background guide set, and the input image pair <I1, I2> is processed using the augmented operation in the augmented data set.

[0018] Further, the expanded final output obtained through the enhancement strategy is specifically:

[0019] For each augmented path, three operations are sampled from the augmented data set and stacked to construct a new operation; a sampling operation is randomly sampled from the newly constructed operation, and the sampling operation is used for mixing the mask C, the background B and the image pair I1, I2 to obtain a new image pair;

[0020] The above steps are repeated, and when the enhancement is performed K times, K new image pairs are obtained, and the K new image pairs are mixed with the randomly sampled weight.

[0021] The learning algorithm is used to continuously update the change detection model parameters, and finally the optimal model is obtained.

[0022] The final loss function is:

[0023] wherein,

[0024]

[0025] wherein, [·] is a connection operation, ψ(·) is a pre-trained VGG16 network for perceptual feature extraction, Cos(·) is a trigonometric function, augmented image pairs;

[0026]

[0027] A weakly supervised change detection device based on background mixed data expansion technology, the device comprises: a processor and a memory, the memory has program instructions stored therein, the processor calls the program instructions stored in the memory to make the device execute any item described method steps.

[0028] The technical scheme provided by the application has the beneficial effects that:

[0029] 1. The application provides a data expansion method for improving the performance of a change detection model, and a background guided set B is constructed to enrich the potential background of an input image.

[0030] 2. The application uses a background-aware enhancement operation to help the change detection model see different background changes; the application converts an original image pair <I1, I2> into a new image pair by constructing an augmented operation set

[0031] 3. The application uses Augmented&Real Data Consistency Loss to evaluate the similarity between the enhanced image pair with different backgrounds and the original image pair <I1, I2>, so as to reduce errors caused by image background changes and enhance the generalization ability of the change detection model. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flowchart of the weakly supervised change detection method based on background mixed data expansion technology;

[0033] Figure 2 Flowchart of the BGMix function proposed by the application;

[0034] Figure 3 Structural schematic diagram of BGMix proposed by the application;

[0035] Figure 4 Schematic diagram of the generation result of the method proposed by the application.​ DETAILED DESCRIPTION

[0036] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below.

[0037] The embodiment of the present application proposes a background mixed enhancement technique specially designed for change detection to solve the problem of performance deficiency of change detection of remote sensing images. The technique can mix the original input image pairs, so that the change detection model can well detect the change region.

[0038] A background mixed enhancement technique specially designed for change detection, which includes the following steps:

[0039] I. Constructing weakly supervised training set T, background guide set B and augmentation operation set O

[0040] This step is specifically: prepare images with image-level labels to construct the weakly supervised training set T, for example: AICD (Change Detection Benchmark in Aerial Images) and BCD (Building Change Detection) data sets; collect N image pairs to construct the background guide set B, each image pair only includes background change but no target change; set the augmentation operation set O including M enhancement operations, for example: posterize, rotate, equalize, etc.

[0041] II. Perform BGMix enhancement operation

[0042] The BGMix enhancement operation is subdivided into two operations: Background-aware augmentation operation and Augmentation strategy.

[0043] Background-aware augmentation operation: first, randomly sample a background pair <B1, B2> from the background guide set. Then process the input image pair <I1, I2> by using the augmentation operation in the augmentation data set.

[0044] The operation function is specifically:

[0045]

[0046] Wherein, o j is the operation of transforming the image pair <I1, I2>, α j is the operation of changing the background of the original image pair, C is the change mask, is the image pair obtained after o j operation.

[0047] The formula can be further described as:

[0048]

[0049]

[0050] Where, is the product of the corresponding elements of two matrices, and Rep is the replacement of the original background image function.

[0051] The operation steps of the augmentation strategy are as follows: setting K augmented paths, for each augmented path, first sampling three operations from the augmented operation set O, and stacking them to construct a new operation. Then, a random operation is sampled from the newly constructed operation, and the sampling operation is used to mix the mask C, the background B and the image pair I1, I2 to obtain a new image pair, so as to achieve the purpose of enhancement. Repeat the above steps, when K times of enhancement are performed, obtain K new image pairs, mix the K new image pairs with the randomly sampled weight. Finally, mix the mixed image pair with the original input image to obtain the final output image.

[0052] III. Execution of learning algorithm

[0053] In the tth iteration during training, an image pair is sampled from the training data set T train , and the change detection is performed through φ θ (·). Then, the enhanced image pair is obtained through the BGMix function. After that, the defined loss function is calculated and the model parameters are updated.

[0054] Where, the loss function includes 5 parts, the first part takes the perceptual similarity between and as the first loss function, which can be described as:

[0055]

[0056] Where, [·] is the connection operation, ψ(·) is a pre-trained VGG16 network for perceptual feature extraction, Cos(·) is a trigonometric function, is the enhanced image pair.

[0057] The second part randomly samples a background pair from the background guide set B, then replaces the background of <I1, I2> with this background pair and obtains a new image pair, and the specific function is:

[0058] I′1=Rep(I1,B1,C),I′2=Rep(I2,B2,C), (5)

[0059] where I'1, I'2 are the images obtained by replacing the background of the original image pair <I1, I2> with Rep function.

[0060] In addition, a new background pair can be synthesized by pasting the background regions in <I1, I2> to the corresponding positions in <B1, B2>. Then a new background pair can be obtained, and the specific function is:

[0061] B'1 = Rep(B1, I1, C), B'2 = Rep(B2, I2, C), (6)

[0062] where B'1, B'2 are the background pair obtained by Rep function.

[0063] Then a new loss function is defined to evaluate the similarity of <I1, I2> and <I'1, I'2>, <B1, B2> and <B'1, B'2>. The loss function can be described as:

[0064]

[0065] The third part embeds more background context from the background pair, which will be beneficial to the recognition of small area regions. The loss function is described as:

[0066]

[0067] where SSIM(·) is the structural similarity loss.

[0068] In the fourth part, two discriminators are used as adversarial loss to make the convolutional neural network φ θ (·) produce accurate change results. The loss function is described as the following function:

[0069]

[0070] where D1 and D2 are two discriminators.

[0071] Finally, in order to control the potential prediction error caused by background change, when the background pair is used as input, a loss function is set to make φ θ (·) predict the all-zero mask. The loss function is described as the following function:

[0072]

[0073] where the number 0 is the all-zero tensor. Through all the above loss functions, the final loss function can be obtained, and the specific function is as follows:

[0074]

[0075] where, is defined as the following function:

[0076]

[0077] where, {lambda i} is used to balance each term in the equation.

[0078] Four, test of network

[0079] After the third step, the trained model can be obtained, and then the images of the test set are read to test the trained change detection model (FCD, AFA and WCD).

[0080] In summary, the data expansion method of BGMix is used in the embodiment of the application, the background guide set B and the change mask C are used through the background-aware augmentation operation to enhance the input image pair I1, I2, so as to obtain a new image pair The final output after expansion is obtained through the augmentation strategy, then the change detection model parameters are continuously updated using the learning algorithm, and finally the optimal model is obtained; and finally, the trained model is tested.

[0081] A weakly supervised change detection device based on a background mixed data expansion technology, the device comprises a processor and a memory, the memory stores program instructions, and the processor calls the program instructions stored in the memory to make the device execute any method steps:

[0082] Constructing a weakly supervised training set, a background guide set and an augmented data set;

[0083] The input image pair is enhanced through the background-aware enhancement operation using the mask and the background, so as to obtain a new image pair The final output after expansion is obtained through the augmentation strategy;

[0084] The change detection model parameters are continuously updated using the learning algorithm, and finally the optimal model is obtained; and finally, the trained model is tested.

[0085] The input image pair is enhanced through the background-aware enhancement operation using the mask and the background, so as to obtain a new image pair Specifically,

[0086] A background pair <B1, B2> is randomly sampled from the background guide set, and the input image pair <I1, I2> is processed using the augmentation operation in the augmented data set.

[0087] Further, the extended final output obtained by further enhancing strategy is specifically:

[0088] For each augmented path, three operations are sampled from the augmented data set and stacked to construct a new operation; a new operation is randomly sampled from the newly constructed operation, and the sampling operation is used to mix the mask C, the background B and the image pair I1, I2 to obtain a new image pair;

[0089] The above steps are repeated, and when K times of enhancement are performed, K new image pairs are obtained, and the K new image pairs are mixed with the randomly sampled weight.

[0090] Wherein, the change detection model parameters are constantly updated using a learning algorithm, and finally an optimal model is obtained, which is specifically:

[0091] The final loss function is:

[0092] Wherein,

[0093]

[0094] Wherein, [·] is a connection operation, ψ(·) is a pre-trained VGG16 network for perceptual feature extraction, and Cos(·) is a trigonometric function, For the enhanced image pair;

[0095]

[0096] The model of each device in the embodiments of the present application is not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0097] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above embodiment numbers are only for description, not representing the advantages and disadvantages of the embodiments.

[0098] The above is only a preferred embodiment of the present application, and does not limit the present application, any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A weakly supervised change detection method based on background mixture data augmentation technique characterized in that, The method comprises: constructing a weakly supervised training set, a background guided set and an augmented data set; enhancing the input image pair using a mask, a background, by a background-aware augmentation operation to obtain a new image pair ; and obtaining an extended final output by an augmentation strategy using a learning algorithm to continuously update the change detection model parameters, and finally obtaining an optimal model; and finally testing using the trained model; The background-aware enhancement operation uses a mask, a background to enhance the input image pair to obtain a new image pair Specifically: sampling a background pair at random from the background guiding set processing the input image pair using an augmentation operation in the augmented data set ; wherein the final output obtained through the enhancement strategy is specifically: For each augmented path, three operations are sampled from the augmented dataset and stacked to build a new operation; one operation is randomly sampled from the newly constructed operations, which is used to perform a mix of the three on the mask C, the background B and the image pair to obtain a new image pair; repeating the above steps, obtaining K new image pairs when the enhancement is performed K times, and mixing the K new image pairs with the randomly sampled weights.

2. The weakly supervised change detection method based on background mixture data expansion technique of claim 1, wherein, The use of a learning algorithm to continuously update the change detection model parameters, and finally obtaining an optimal model is specifically: The final loss function is: ; wherein ; ； ； wherein, is a concatenation operation, refers to a pre-trained VGG16 network for perceptual feature extraction, , is an enhanced image pair; ； ； ； ； ； wherein, is a structural similarity loss; and are two discriminators; is a convolutional neural network.

3. A device for weakly supervised change detection based on background mixture data augmentation technique, characterized in that, The device comprises a processor and a memory, the memory storing program instructions, and the processor invoking the program instructions stored in the memory to enable the device to perform the method steps of any one of claims 1-2.

Images