Multi-scale image change detection method based on DCGAN and storage medium

Abstract

The application discloses a multi-scale image change detection method based on DCGAN and a storage medium, a multi-scale change detection network is built, and a semi-supervised method based on DCGAN is adopted to expand sample data to meet network training requirements; on the other hand, a multi-scale network structure is adopted, and multi-scale targets can be better detected, especially in some irregular and uncertain change scenes or multi-target and multi-scale change detection scenes, and excellent performance can be achieved. A large number of change detection samples similar to a source domain are generated by training the DCGAN firstly, and are used for training a discriminator, so that the discriminator has good feature extraction function and image authenticity distinguishing ability, the accuracy and stability of the network are enhanced, and the discriminator is regarded as a double-channel feature extractor of the detection network. A multi-scale feature module is further added to extract convolution features at different stages, and the detection ability for different scale targets is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a multi-scale image change detection method based on DCGAN and a storage medium. BACKGROUND

[0002] Deep learning is an important research direction in the field of machine learning, which is to learn the internal rules and representation levels of sample data. The information obtained in this learning process is very helpful for the interpretation of data such as text, images and sound. Change detection refers to the process of identifying changes in images taken at different times in the same geographic location. Since images often contain changes in targets of multiple scales, such as buildings, lawns, lakes, etc., the detection is more difficult. For this problem, using deep learning technology to optimize the feature extraction and selection process of multi-scale images is a good solution. In recent years, deep learning technology, especially convolutional neural network (CNN), has achieved very good results in various computer vision tasks. CNN automatically learns image features from low to high levels through training, without the need for manual feature selection, and has wide applicability in scene classification, pixel classification, feature detection and image change detection. Compared with traditional supervised methods, CNN-based classification and supervised change detection methods can achieve satisfactory results, but more training samples are needed to learn the deep features of images. However, in practical applications, it is extremely difficult to obtain a large number of training samples. Therefore, we are faced with the problem of lack of data to support the training of deep learning models. In view of this, Zhang Yi et al. of Shandong Zhuoyuan Data Technology Co., Ltd. proposed a "generative adversarial network remote sensing image sample expansion method under spatial constraint conditions for feature change detection" (patent number 202010254751.1, publication number CN111428678A) in 2020, which can to some extent alleviate the problem of lack of training samples, but the patent does not propose a method for change detection of remote sensing images.

[0003] The patent "SAR image change detection based on unsupervised deep neural network" (patent application number: 201310733194.1, publication number: CN103810699A) of Xi'an University of Electronic Science and Technology proposes a SAR image change detection based on unsupervised DBN neural network. This method is mainly aimed at SAR images, and the selection of pseudo-label samples is based on the FCM algorithm, which is an unsupervised algorithm and is not very reliable. Moreover, the deep neural network used is a deep belief network composed of multiple RBMs, which belongs to a shallow network and has insufficient feature extraction capability.

[0004] Gong Maoguo of Xi'an University of Electronic Science and Technology proposes a generative adversarial network multispectral image change detection method in the paper AGenerative Discriminatory Classified Network for Change Detection in Multispectral Imagery. (IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12 (1): 321-333.) and the patent "Multispectral image change detection method based on generative adversarial network" (application number: 201810340804.4, publication number CN108492298A). However, this method is only composed of two adversarial networks, and the training is easy to fall into mode collapse, and the first two dimensions output by the discriminator are used as the change detection result, and the detection of different scales is not fully considered, so there may be a problem of low recognition accuracy for small changes. SUMMARY

[0005] The multiscale image change detection method based on DCGAN proposed by the present application can solve the above technical problems.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A multiscale image change detection method based on DCGAN, comprising the following steps,

[0008] S1, collect registered images and their change region annotation data of the same position in two different periods, and crop them to a set size as real sample data sets R1 and R2, and change map ground truth GT;

[0009] S2, input a batch of real image sample sets (R1, R2) in step S1 into the discriminator D of the deep convolutional generative adversarial network DCGAN, and preliminarily train the feature extraction and discrimination ability of the discriminator D, then input the batch of real samples and random noise z into the initialized DCGAN network generator G, and train to make the generator G generate a synthetic image of mixed true and false images;

[0010] S3, input the synthetic image sample generated in step S2 and the real image sample into the discriminator D of the DCGAN network, judge the image authenticity, and make the discriminator D have the functions of judging the image authenticity and image feature extraction;

[0011] S4, iteratively train the generator G and the discriminator D in steps S2 and S3 until the DCGAN network is trained, and use the better synthetic images generated by the final DCGAN network as the dataset G(z);

[0012] S5, first take a batch of images from the real image sample set (R1, R2) and a batch of images from the synthetic image set G(z) to form pairs of images at different time phases, and input them into the multi-scale change detection network MCDN for preliminary training, then take pairs of images at different time phases from the real image sample set (R1, R2) to fine-tune the network, and when the loss function of the network converges, the network is trained;

[0013] S6, for a pair of images to be tested, input them into the MCDN network trained in S5, and the output is the change prediction result.

[0014] Further, the deep convolutional generative adversarial network DCGAN in step S2 is composed of a generator G and a discriminator D. The generator G can generate synthetic images by inputting real samples and random noise, and the discriminator D has the functions of feature extraction and discrimination of input images.

[0015] Further, the structure of the MCDN network in step S5 is as follows: the MCDN network includes a dual-channel multi-scale feature extraction network Net1, a feature difference network Net2, and a feature fusion network Net3, wherein the dual-channel multi-scale feature extraction network Net1 is composed of the convolutional layers and multi-scale feature modules of the discriminator D trained in step S4.

[0016] Further, the specific steps for training the MCDN network in step S5 are as follows:

[0017] S51, in the preliminary training stage, first take a batch of images from the real image set and a batch of images from the synthetic image set to form pairs of images at different time phases, and input them into the multi-scale change detection network MCDN. For each pair of images, the predicted change result CMM2 is obtained, and the change amplitude map M of the input image is calculated, then normalized to obtain CMM1, and the distance between CMM1 and CMM2 is calculated as the loss function, wherein the calculation method of CMM1 is as follows:

[0018]

[0019]

[0020] wherein X 1j and X 2j respectively represent the pixel value matrix of the jth channel of X1 and X2, X1 represents a real image, X2 represents a synthetic image, j takes a value from 1 to c, and c is the total number of channels; wherein The value of constant a is determined by different data sets, CMM1 is a binary matrix with the same row and column number as the input image, and 1 and 0 represent whether to change or not, wherein

[0021] S52, fine-tuning network phase, a batch of paired different time change images is taken from the real image set, input into the MCDN network for prediction, and the network parameters are optimized through back propagation until the loss function converges;

[0022] S53, for the process of MCDN network prediction change result in S51, S52, specifically includes, using a dual-channel multi-scale feature extraction network Net1 to respectively perform convolution operation on a pair of input images, feature extraction, and then taking the features of the last 6 layers as Into the feature difference network Net2 to better extract multi-scale features; the specific process is as follows:

[0023]

[0024]

[0025] Wherein Upsample(*) represents the up-sampling operation;

[0026] S54, the multi-scale features are calculated by Net2, and then the results are sent into the feature fusion network Net3, and the feature difference calculation formula is as follows:

[0027]

[0028] Wherein FD i represents the feature difference image;

[0029] S55, the operation process of Net3 is as follows, first, the five feature difference images output by Net2 are directly enlarged to the same size as the input image by using the up-sampling method, and then added element by element; then use the Sigmoid function to get the probability value of each pixel change, take the threshold value T to produce a binary image, which is a binary matrix with the same row and column number as the input image, that is, the prediction result CMM2, and its calculation formula is as follows:

[0030]

[0031] S56, based on CMM2, the loss function of MCDN network is defined as:

[0032]

[0033] Wherein, L is the deformation of cross-entropy loss function, and β -Beta represents the ratio of the number of negative samples, i.e. unchanged pixels, to the total number + Beta represents the ratio of the number of positive samples, i.e. changed pixels, to the total number - + Beta + = 1; W is a weight matrix generated according to the change amplitude information of each pixel, ranging from 0 to 1; y n and respectively represent the true value and network prediction value of the nth pixel, wherein the preliminary training stage y n from CMM1, the fine-tuning network stage from GT, from CMM2, n is from 1 to N, N is the total number of image pixels; wherein W is represented as:

[0034]

[0035] In still another aspect, the application discloses a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the steps of the above method.

[0036] From the above technical solution, the multi-scale image change detection method based on DCGAN has the following beneficial effects:

[0037] The purpose of the application is to overcome the deficiencies in the above background art, solve the model construction problem in the prior art, and improve the accuracy and robustness of the model. In general, a multi-scale change detection network is built by combining GAN network and CNN. On the one hand, a semi-supervised method based on DCGAN is used to expand the sample data to meet the network training needs. On the other hand, a multi-scale network structure is used to better detect multi-scale targets, especially in some irregular, uncertain change scenes, or multi-target, multi-scale change detection scenes, which can have excellent performance.

[0038] Specifically, in the first stage, a large number of change detection samples similar to the source domain are generated by training the DCGAN, which is used to train the discriminator to have good feature extraction function and image authenticity discrimination ability, and to enhance the accuracy and stability of the network, which is used as a double-channel feature extractor of the detection network. In order to solve the problem of different scale target change detection, a multi-scale feature extraction network is added to extract convolutional features at different stages, which correspond to different scale features and enhance the detection ability of different scale targets. In the training of the main network stage, considering that most of the regions of the image to be detected have no change or small change amplitude, resulting in low learning rate. For unchanged pixels, there is no learning rate. If a very low weight is used, the training speed will decrease, so a weight matrix W and a cut-off point The changed pixels can be optimized (highlighted) to improve the training speed. The loss function adopts a deformation of the binary cross entropy function, which can also improve the training speed. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a flowchart of a multi-scale change detection method based on DCGAN of the present application;

[0040] Figure 2 is a structural diagram of a multi-scale change detection method based on DCGAN of the present application;

[0041] Figure 3 is a flowchart of DCGAN training of the present application. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments of the present application. Multi-spectral remote sensing images are used as the embodiment data, but the present application is not limited to this kind of data set.

[0043] As shown in Figure 1 , the multi-scale image change detection method based on DCGAN described in the present embodiment includes:

[0044] S1, collect registered images of the same position in two different periods and their change region annotation data, and crop them to a set size as real sample data sets R1 and R2 and change map ground truth GT;

[0045] S2, input the real image sample sets (R1, R2) in step S1 into the discriminator D of the deep convolutional generative adversarial network DCGAN network, and preliminarily train the feature extraction and discrimination ability of the discriminator D, then input the real sample and random noise z into the initialized generator G of the DCGAN network together, and train to make the generator G generate a synthetic image of a confused true and false image;

[0046] S3, input the synthetic image sample generated in step S2 and the real image sample into the discriminator D of the DCGAN network together, judge the image true and false, and make the discriminator D have the functions of discriminating the image true and false and extracting the image features;

[0047] S4, iteratively train the generator G and the discriminator D in steps S2 and S3 until the DCGAN network training is completed, and use the better synthetic image generated by the final DCGAN network as the data set G(z);

[0048] S5, first take a batch of images from the real image sample set (R1, R2) and a batch of images from the synthetic image set G(z) to form a pair of images at different times, input into the multi-scale change detection network MCDN for preliminary training, and then take a pair of images at different times from the real image sample set (R1, R2) to fine-tune the network, when the loss function of the network converges, the network is trained well;

[0049] S6, for a pair of images to be tested, input the network MCDN trained in step S5, and the output is the change prediction result.

[0050] The following is an example:

[0051] 1, collect multi-spectral remote sensing registered images and their change area annotation data at the same position in two different periods, here we use the public dataset WV3, WorldView-3 (WV3) is a commercial satellite sensor, which can provide very high resolution remote sensing images, WV3 site1 and site2 are two different areas in Shenzhen, China, with a size of 1431*1431 pixels and a spatial resolution of 2m. We only use 3 of the 8 multi-spectral bands (R, G and B), and crop them to 256*256 pixels as real sample data sets, denoted as R1, R2 and change graph GT respectively.

[0052] 2, input the real sample set (R1, R2) in step 1 into the discriminator D of the DCGAN network for simple training, then input the real sample and 100-dimensional random noise z into the initialized generator G of the DCGAN network, and train it to make the generator G generate synthetic images G(z) that can confuse real and fake images. Specifically, the generator G expands a 100-dimensional noise vector into a 256*256*3 matrix output through deconvolution.

[0053] 3, input the synthetic image samples generated in step 2 and the real image samples into the discriminator D of the DCGAN network, extract image features through a series of convolution and pooling operations, and judge the image to be true or false, so that the discriminator D has the functions of distinguishing the image to be true or false and extracting the image features.

[0054] 4, the cycle iteration training generator G and discriminator D, until the DCGAN network training is completed, and the good synthetic image generated by the DCGAN network is used as the data set G(z), the specific training details of DCGAN can refer to this paper, Radford A, Metz L, Chintala S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[J]. Computer ence, 2015.

[0055] 5, first from the real image set (R1, R2) a batch of images and synthetic image set G(z) a batch of images are taken to form different time phase pairs of images input to the multi-scale change detection network MCDN for preliminary training, and then from the real image (R1, R2) different time phase pairs of images are taken to fine tune the network, when the loss function of the network converges, the network is trained.

[0056] 6, for the MCDN network in step 5, the convolutional layer of the discriminator D trained in step 4 and the multi-scale feature module are used to form a double-channel multi-scale feature network Net1, and the MCDN network mainly includes a double-channel multi-scale feature network Net1, a feature difference network Net2 and a feature fusion network Net3.

[0057] 7, for the network MCDN trained in step 6, input a pair of images to be tested, and the output is the change prediction result.

[0058] The specific steps of training the MCDN network are as follows:

[0059] S51, in the preliminary training stage, a batch of images are taken from the real image set, and a batch of images are taken from the synthetic image, which are input to the multi-scale change detection network MCDN to form different time phase pairs of images, and the change result is obtained for each pair of images CMM2, and the change amplitude map M of the input image is calculated, then normalized to obtain CMM1, and the distance between CMM1 and CMM2 is calculated as the loss function, wherein the calculation method of CMM1 is as follows:

[0060]

[0061]

[0062] Wherein X 1j and X 2j respectively represent the pixel value matrix of the jth channel of X1 and X2, X1 represents the real image, X2 represents the synthetic image, j is from 1 to c, and c is the total number of channels; wherein The value of constant a is determined by different data sets, CMM1 is a binary matrix with the same row and column number as the input image, and 1 and 0 represent whether to change or not, wherein

[0063] S52, fine-tuning network phase, a batch of paired different time change images is taken from the real image set, input into the MCDN network for prediction, and the network parameters are optimized through back propagation until the loss function converges;

[0064] S53, for the process of MCDN network prediction change result in S51 and S52, specifically, a pair of images are respectively subjected to convolution operation, feature extraction by using a dual-channel multi-scale feature extraction network Net1, and then the features of the last 6 layers are taken as Input into the feature difference network Net2 to better extract multi-scale features; the specific process is as follows:

[0065]

[0066]

[0067] Wherein Upsample(*) represents the up-sampling operation.

[0068] S54, the multi-scale features are calculated by using Net2, and then the results are input into the feature fusion network Net3, and the feature difference calculation formula is as follows:

[0069]

[0070] Wherein FD i represents the feature difference image;

[0071] S55, the operation process of Net3 is as follows, first, the five feature difference images output by Net2 are directly enlarged to the same size as the input image by using the up-sampling method, and then added element by element; then the probability value of each pixel change is obtained by using the Sigmoid function, and the threshold T is taken to generate a binary image, which is a binary matrix with the same row and column number as the input image, that is, the prediction result CMM2, and its calculation formula is as follows:

[0072]

[0073] S56, based on CMM2, the loss function of MCDN network is defined as:

[0074]

[0075] Wherein, L is the deformation of cross-entropy loss function, β - represents the ratio of the number of negative samples, i.e. no change pixels, to the total number, and β+ represents the ratio of the number of positive samples, i.e., changed pixels, to the total number, β - + β + = 1; W is a weight matrix generated according to the change amplitude information of each pixel, ranging from 0 to 1; y n and respectively represent the true value and the network prediction value of the nth pixel, wherein the preliminary training stage y n from CMM1, the fine-tuning network stage from GT, from CMM2, n is from 1 to N, and N is the total number of image pixels; wherein W is represented as:

[0076]

[0077] To sum up, the multi-scale image change detection method based on DCGAN provided by the embodiment of the application adopts a method combining GAN network and CNN, and builds a multi-scale change detection network. On the one hand, a semi-supervised method based on DCGAN is used to expand sample data to meet the network training requirements. On the other hand, a multi-scale network structure is used, which can better detect multi-scale targets, especially some irregular and uncertain change scenes, or multi-target and multi-scale change detection scenes, and can have excellent performance.

[0078] In another aspect, the application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of any of the above methods.

[0079] In another aspect, the application further discloses a computer device, which comprises a memory and a processor, and the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of any of the above methods.

[0080] In another embodiment provided in the application, a computer program product containing instructions is also provided, which makes a computer execute the steps of any of the above methods when the computer program product is run on the computer.

[0081] It can be understood that the system provided by the embodiment of the application corresponds to the method provided by the embodiment of the application, and the explanation, examples and beneficial effects of the related content can refer to the corresponding part in the above method.

[0082] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0084] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-scale image change detection method based on DCGAN, characterized in that, Comprise the following steps, S1, collect the registration image and its change region annotation data of the same position in two different periods, and cut to a set size as a real sample data set respectively denoted as R1, R2 and change map ground truth GT; S2, input a batch of real image sample sets R1, R2 in step S1 into the discriminator D of the deep convolutional generative adversarial network DCGAN network, preliminarily train the feature extraction and discrimination ability of the discriminator D, then input these real samples and random noise z into the generator G of the initialized DCGAN network, and train to make the generator G generate synthetic images of mixed true and false images; S3, input the synthetic image samples generated in step S2 and the real image samples into the discriminator D of the DCGAN network, judge the image authenticity, and make the discriminator D have the functions of judging the image authenticity and image feature extraction; S4, iteratively train the generator G and the discriminator D in steps S2 and S3 until the DCGAN network training is completed, and the synthetic image generated by the final DCGAN network is used as a data set G(z); S5, first select a batch of images from the real image sample sets R1, R2 and a batch of images from the synthetic image set G(z) to input into the multi-scale change detection network MCDN for preliminary training, then select a pair of images in different periods from the real image sample sets R1, R2 to fine-tune the MCDN network, and when the loss function of the MCDN network converges, the MCDN network is trained; S6, input a pair of images to be tested into the MCDN network trained in S5, and the output is the change prediction result.

2. The DCGAN-based multi-scale image change detection method of claim 1, wherein, The deep convolutional generative adversarial network DCGAN in step S2 is composed of a generator G and a discriminator D, the generator G generates a synthetic image by inputting a real sample and random noise z, and the discriminator D has the functions of feature extraction and discrimination of the input image.

3. The DCGAN-based multi-scale image change detection method of claim 1, wherein, The structure of the MCDN network in step S5 is: the MCDN network comprises a double-channel multi-scale feature extraction network Net1, a feature difference network Net2 and a feature fusion network Net3, wherein the double-channel multi-scale feature extraction network Net1 is composed of the convolutional layer of the discriminator D trained in step S4 and a multi-scale feature module.

4. The DCGAN-based multi-scale image change detection method of claim 1, wherein, The specific steps for training the MCDN network in step S5 are: S51, in the preliminary training stage, first take a batch of images from the real image set, take a batch of images in the synthetic image, form the paired images of different time phases, input into the multi-scale change detection network MCDN, and obtain the change result for each pair of images , and calculate the change amplitude map M of the input image, then normalize to obtain , according to and , the loss is calculated, wherein The calculation method is as follows: wherein and respectively represent and a pixel value matrix of the jth channel of the image, represents a real image, represents a synthetic image, j is from 1 to c, c is the total number of channels; wherein , constant is determined by different data sets, is a binary matrix with the same row and column number as the input image, 1 and 0 represent whether to change or not; wherein ; S52, fine-tuning network stage, input a batch of paired images in different periods from the real image set into the MCDN network for prediction, optimize the network parameters through back propagation until the loss function converges; S53、for S51, S52, the process of predicting changes in MCDN network result, specifically includes, with double channel multi-scale feature extraction network Net1 respectively on a pair of images input convolution operation, feature extraction, and then take the last 6 layer feature mapping respectively represented as 、 , into the feature difference network Net2, in order to better extract multi-scale features; The specific process is: wherein denotes an up-sampling operation; S54, calculate the feature difference of the multi-scale features with Net2, and then send the result to the feature fusion network Net3, and the feature difference calculation formula is represented as: wherein denotes a feature difference image; S55, the operation process of Net3 is as follows: firstly, five feature difference images output by Net2 are directly enlarged to the same size as the input image by upsampling, and finally added element by element; then the probability value of each pixel change is obtained by using the Sigmoid function, and a threshold T is taken to generate a binary image, which is a binary matrix with the same row and column number as the input image, that is, the prediction result The calculation formula is as follows: S56, based on The loss function of the MCDN network is defined as: where L is a variant of the cross-entropy loss function, represents the ratio of the number of negative samples, i.e. pixels without changes, to the total number, represents the ratio of the number of positive samples, i.e. pixels with changes, to the total number, is a weight matrix generated according to the change amplitude information of each pixel, ranging from 0 to 1; respectively represent the true value and the network prediction value of the nthpixel, where the preliminary training stage from , the fine-tuning network stage comes from GT, from , the value of n is from 1 to N, N is the total number of image pixels; wherein is represented as: 。 5. A computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to make the processor execute the steps of the method according to any one of claims 1 to 4.

Images