Unsupervised change detection method based on content encoding

Abstract

This invention discloses an unsupervised change detection method based on content encoding. The method comprises the following steps: constructing a content encoding network structure; establishing a probability function model for change detection; defining a mask loss function to measure the deviation between content encodings; constructing a content constraint function to constrain the image difference results based on the object; and combining the mask loss function and the content constraint function to define an energy function for training to obtain change detection results. This method effectively avoids the reliance of existing change detection methods on precise registration and is applicable to most change detection problems, including unsupervised change detection of multi-view images, which cannot be solved by existing methods.

Description

Technical Field

[0001] The present invention belongs to the field of change detection, and in particular relates to an unsupervised change detection method based on content coding. Background Art

[0002] Change detection aims to detect changed regions in images of the same scene taken at different times. It is of great significance in many applications, including video surveillance, medical diagnosis and treatment, and especially in remote sensing monitoring and land use analysis. Depending on whether the change detection method requires manually annotated samples to learn, it can be divided into supervised, semi-supervised and unsupervised detection methods. Supervised methods can adapt to different complex scenes with the help of training samples. However, due to the diversity of images in practical problems, their application is usually limited. For example, models trained on natural images are difficult to apply directly to remote sensing images. Unsupervised methods are widely used, but their accuracy is highly dependent on the effectiveness of preprocessing methods such as geometric adjustment (co-registration) and radiometric correction (denoising, atmospheric correction, normalization).

[0003] By comparison, change detection methods can be roughly divided into pixel-based, feature-based, and object-based methods. The pixel-based change detection method is intuitive. It compares the preprocessed multi-temporal images pixel by pixel to generate the degree of difference for each pixel. Then, the image segmentation method is used to determine the changed pixels. Different pixel-based methods are designed for different types of data. For example, due to the influence of speckle noise, the logarithmic ratio operator and the average ratio operator are widely used in the change detection task of synthetic aperture radar (SAR) images. For multi / hyperspectral images, change vector analysis (CVA) is a classic method that analyzes the change vector by comparing the pixel and channel aspects, thereby generating multiple changes. Due to its intuitive mechanism, pixel-based methods are usually designed as unsupervised methods. But it also makes it difficult to handle complex scenes such as multi-source images and misaligned images.

[0004] Feature-based change detection methods focus more on the comparison process to produce significant changes in complex scenes. Some methods learn comparable features from multiple source images. Prendes et al. proposed a physical model based on a mixture of multidimensional distributions with available invariant samples. Since change detection can be cast as a classification problem, many supervised methods have been developed based on deep learning methods, which learn to compare multi-temporal images with trainable hierarchical features. However, in order to train a deep network, sufficient labeled training samples are required, which limits their widespread application. Feature-based change detection methods can avoid many unimportant changes caused by sensor noise, illumination changes, non-uniform attenuation, atmospheric absorption, and even heterogeneous sensors. However, for unsupervised methods, since they usually compare local features, they require high alignment accuracy, i.e., co-registration methods.

[0005] Given two images, most co-registration methods treat the captured scene as a flat surface and transform the images using a fixed transformation template, such as shift, rotation, and affine transformation. Therefore, it is difficult to fully align high-resolution images captured from different angles. Such images are common in many scenarios, such as very high-resolution (VHR) optical remote sensing images and those captured by unmanned aerial vehicles (UAVs). Therefore, in many change detection scenarios, methods that are robust to co-registration errors are needed. Object-based change detection methods first classify objects in the images and then compare them, which is robust to co-registration errors. To generate accurate change regions, a dedicated classification method should be designed. An intuitive approach is to classify multi-temporal images separately and then compare the corresponding classes to generate change regions. The accuracy of these methods depends on the accuracy of the classification method, and error propagation is common. Even with high self-alignment and robustness to co-registration errors, object-based methods are often supervised to learn an accurate classifier. Summary of the Invention

[0006] The purpose of the present invention is to provide an unsupervised change detection method based on content coding.

[0007] The technical solution for achieving the purpose of the present invention is: an unsupervised change detection method based on content coding, comprising the following steps:

[0008] The first step is to build a content extraction network based on convolutional neural networks to encode the input image and output a feature vector without reference labels;

[0009] In the second step, we assume that each element in the network output vector represents a certain content in the input image. By inputting two images, we obtain the encoding of the two images and define the content alignment loss function based on the encoding.

[0010] The third step is to optimize the energy model and define the content constraint function of the encoding to meet the content hypothesis;

[0011] The fourth step is to combine the alignment loss function and the content constraint function to establish an energy-based probability model;

[0012] Step 5: After optimization, compare the feature vectors of the two images and solve the probability model;

[0013] In the sixth step, the feature vectors of the two images are compared, and the change regions are generated by optimizing the change masks in the change regions. Finally, the image clustering algorithm of FLICM is used to generate the difference map.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned unsupervised change detection method based on content coding is implemented.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned unsupervised change detection method based on content coding.

[0016] Compared with the prior art, the present invention has the following significant features: (1) a content extraction network based on a convolutional neural network is defined; (2) a probabilistic model based on an energy function is established to learn network parameters in an unsupervised manner using two input images; (3) the energy model is optimized, a mask loss function is defined to measure the deviation between the encodings of the two images, and a content constraint function is constructed to implement the content hypothesis, constraining the differential result of the encoding to the input image according to the content object.

[0017] The present invention is further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of the unsupervised change detection method based on content coding of the present invention.

[0019] Figure 2 This is a schematic diagram of the content coding network structure proposed by the present invention.

[0020] Figure 3(a) and Figure 3(b) show the ROC and PR curves of different methods on the drone dataset.

[0021] Figure 4(a) and Figure 4(b) show the ROC and PR curves of different methods on the remote sensing dataset U2.

[0022] Figure 5 Schematic diagram of the entire change detection process given two images I1 and I2. DETAILED DESCRIPTION

[0023] Combine Figure 1 、 Figure 2 This paper proposes an unsupervised change detection method based on content coding. By constructing a content coding network based on a convolutional neural network, a probabilistic model driven by an energy function that combines a mask loss and a content constraint function is established. The network parameters are learned in an unsupervised manner, ultimately obtaining change detection results. This method is suitable for the change detection task of multi-view images, which has been a common but difficult task in recent years. The specific steps of this method are as follows:

[0024] In the first step, a content extraction network based on convolutional neural network (CNN) is constructed to encode the input image and output a feature vector without reference labels.

[0025] In the second step, in order to learn the network, we assume that each element in the network output vector represents a certain content in the input image. By inputting two images, we obtain the encoding of the two images. Based on the encoding, we define the content alignment loss function and learn the distribution of the input images. The specific process is as follows:

[0026] 1) The probability model is defined as:

[0027]

[0028] Where Z is the partition function and u is a pseudo-probability vector representing the invariant probability of each content object. The probability model is optimized by maximizing the log-likelihood of the probabilities, which means increasing the energy of the input data I1 and I2 while reducing the energy of all other data. The network can then specifically learn the relationship between the two images.

[0029] 2) In order to realize change detection, the energy function E(I1,I2,u,θ) is designed by the definition of change detection. This energy function aligns the same content in two images, which means that the feature vector v1=f θ (I1) and v2 = f θ The components of the invariant content in (I2) should be similar, so the alignment loss of the two images is defined as follows:

[0030]

[0031] Among them, i is the component index of the feature vector, u is a pseudo probability vector representing the invariant probability of each content object, which is a trainable parameter and will be trained together with the network parameter set θ.

[0032] The third step is to define the content constraint function of the encoding to meet the content assumption and define the content constraint. Through the constraint, each encoded element can represent a content. The specific process is as follows:

[0033]

[0034] Where (i, j, k) represents the pixel (i, j) in the kth channel, Ω (i,j) Represents the square neighborhood of pixel (i, j); δI represents the differential coefficient of the output feature vector v relative to the input image I, which is expressed as follows:

[0035]

[0036] Where ω represents the weight matrix within each pixel neighborhood, which is calculated by the difference between adjacent pixels and the central pixel and is expressed as follows:

[0037]

[0038] Where σ represents the standard deviation of the pixels in the neighborhood. This weight matrix is ​​derived from the superpixel segmentation method. A superpixel should contain pixels that are similar in color, texture, etc., and therefore are likely to belong to the same physical world object; therefore, after superpixel segmentation, we use a weight matrix to define the boundaries of different contents; the larger the weight value, the more likely the corresponding pixel belongs to the same content as the center pixel; for a content extractor, if two pixels belong to the same content, the corresponding output code relative to the difference of these two pixels should also be similar, and the change in the feature vector represents the change of the entire content, rather than other randomly learned semantic regions. Therefore, similar pixels within a region can be encoded by the constraint of content.

[0039] The fourth step is to combine the alignment loss function and the content constraint function to establish an energy-based probability model. The energy function is as follows:

[0040] E(I1,I2,θ,u)=L(I1,I2,θ,u)+λ[C(I1,θ)+C(I2,θ)]

[0041] Among them, λ is a user-defined parameter that controls the weights of these two items. Then the energy function is set to the probability model of change detection. Then we calculate the optimization framework of the probability model based on the log-likelihood gradient of the probability model, which is expressed as follows:

[0042]

[0043] Where I′1, I′2 represent all possible data in the data space, but this brings difficulty to calculation. Increase the probability relative to the input data The contrast divergence algorithm is used to ensure efficiency; the parameter update gradient is as follows:

[0044]

[0045] From the above derivatives, we can see that the basic operator of optimization is gradient. In the whole optimization process, two types of gradients must be derived, including the trainable parameters and input data There are two terms in the energy function, and we derive their gradients separately. Through the back-propagation algorithm, the gradient of the alignment loss is obtained as follows:

[0046]

[0047]

[0048] For content gradient, a similar gradient can also be derived, the formula is as follows:

[0049]

[0050] Corresponding The gradient of can also be calculated as follows:

[0051]

[0052] where l k represents the output of the kth layer of the network. Using the above gradient, we update the model because we define u as a probability vector with each component in the range [0, 1]. However, u is updated based on the gradient without any constraints. Therefore, the optimized u may not be within this range. Therefore, we define u as a sigmoid function u = sigmoid(t). Then, t can be updated using the following formula:

[0053]

[0054] After optimization, the feature vectors u1 and u2 can represent the contents of the two images I1 and I2; mark the changed pixels of the input image and highlight the changed content of the input image.

[0055] The fifth step is to compare the feature vectors of the two images after optimization and solve the probability model. First, define the constant loss function, the formula is as follows:

[0056]

[0057] Among them L c Represents the unchanged loss, which is used here as the energy function of the probability model; ⊙ represents the dot product, M1 and M2 are its change masks in the range of [0, 1]; this loss function cannot be minimized because the optimal value is reached when M1 and M2 are both 0. Therefore, the probability model is established as follows:

[0058]

[0059] P c (I1, I2; M1, M2) is the probability model to be solved, which is similar to the above model P(I1, I2; u, θ), and Z is the partition function Although different from P(I1, I2; u, θ), the optimization parameters M1 and M2 here can be optimized according to the above description; similar to u, define M k =sigmoid(S k ), k = 1, 2; the optimization process includes sampling and updating parameters. During the sampling process, the sampled data I′1I′2 is obtained by the gradient Then update the parameters S1 and S2 using the following formula:

[0060]

[0061] In the sixth step, the feature vectors of the two images are compared, and the change region is generated by optimizing the change mask in the change region. Finally, the image clustering algorithm of FLICM is used to generate the difference map, as follows:

[0062] After optimization, in order to highlight the changed areas, the difference image is: D k =1-M k , k = 1, 2; then the FLICM image segmentation method is used to divide the pixels into changed pixels and unchanged pixels, and the final change map is generated through the image clustering algorithm.

[0063] Figure 5 The entire change detection process is given for two images I1 and I2.

[0064] This method uses two input images to learn network parameters in an unsupervised manner. It also satisfies content assumptions and can be used for image content comparison. This method effectively avoids false positives for misaligned objects in existing change detection methods and improves the robustness to co-registration errors.

[0065] The effect of the present invention can be further illustrated by the following simulation experiments:

[0066] Simulation conditions

[0067] The simulation experiments use two datasets: a drone dataset and a remote sensing dataset. Their size is 512×512 pixels. Since the drone is constantly moving, even if it captures the same scene, the position and angle of view are different, which makes registration difficult. Since satellites or camera drones cannot capture the same scene in exactly the same position at different times, we compare the proposed method with several unsupervised change detection methods. The simulation experiments are all configured under the Windows operating system with an Intel i7-8700K CPU (3.7GHz) and an NVIDART TX3090 GPU. The program is written in C++ and Visual Studio 2017.

[0068] The evaluation indicators used in the present invention are the calculation method of clustering accuracy (ACC), precision recall (PR), receiver operating characteristic curve (ROC), average precision (AP), and kappa coefficient.

[0069] Simulation content

[0070] This paper uses real drone and remote sensing datasets to test the performance of the proposed algorithm. To test the performance of the proposed unsupervised change detection method based on content coding, we compared it with currently popular change detection algorithms. The comparison methods include CVA, DCVA, SFA, and DCCN.

[0071] Analysis of simulation experiment results

[0072] Table 1 shows the comparison results of different evaluation indicators under different change detection algorithms in two data sets. It can be seen from Table 1 that in the drone data set, the change detection method based on content coding proposed by the present invention can well constrain background objects, highlight changed areas, and avoid the influence of unchanged buildings by virtue of its robustness to local positions. Compared with CVA, DCVA, SFA, and DCCN, the accuracy in different evaluation indicators is significantly improved. Table 2 shows the running time under this method. The effect diagrams of the method of the present invention on different data sets are shown in Figures 3(a), 3(b), 4(a), and 4(b). The simulation experimental results of the above two groups of real data sets demonstrate the effectiveness of the method of the present invention.

[0073] Table 1 Quantitative evaluation of different algorithms on drone dataset (AP, AuR, ACC, Kappa)

[0074]

[0075]

[0076] Table 2 Time cost of each dataset

[0077] Dataset Drone Dataset Remote sensing datasets Time(s) 23.9 190.5

[0078] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An unsupervised change detection method based on content coding, characterized in that: The following steps are involved: The first step is to build a content extraction network based on convolutional neural networks to encode the input image and output a feature vector without reference labels; In the second step, we assume that each element in the network output vector represents a certain content in the input image. By inputting two images, we obtain the encoding of the two images and define the content alignment loss function based on the encoding. In the second step, in order to learn the network, we assume that each element in the network output vector represents a certain content in the input image. By inputting two images, we obtain the encoding of the two images. Based on the encoding, we define the content alignment loss function and learn the distribution of the input images. The specific process is as follows: (1) The probability model is defined as: Where Z is the partition function and u is a pseudo-probability vector representing the invariant probability of each content object. By maximizing the log-likelihood of the probabilities to optimize the probability model, the energy of the input data I1 and I2 is increased while the energy of all other data is reduced. The network can learn the relationship between the two images. (2) The energy function E(I1, I2, u, θ) is designed by defining the change detection. This energy function aligns the same content in two images. The feature vector v1 = f θ (I1) and v2 = f θ The components of the invariant content in (I2) are similar, so the alignment loss of the two images is defined as follows: Among them, i is the component index of the feature vector, u is a pseudo probability vector, which represents the invariant probability of each content object. It is a trainable parameter and will be trained together with the network parameter set θ; The third step is to optimize the energy model and define the content constraint function of the encoding to meet the content assumption. The content constraint is defined to ensure that each encoded element represents a content through the constraint. The specific process is as follows: Where (i, j, k) represents the pixel (i, j) in the kth channel, Ω (i,j) Represents the square neighborhood of pixel (i, j); δI represents the differential coefficient of the output feature vector v relative to the input image I, which is expressed as follows: Where ω represents the weight matrix within each pixel neighborhood, which is calculated by the difference between adjacent pixels and the central pixel and is expressed as follows: Where σ represents the standard deviation of pixels in the neighborhood; In the fourth step, the alignment loss function and the content constraint function are combined to establish an energy-based probabilistic model. The energy model is defined as the difference between two feature vectors with content constraints. The following energy function is obtained by combining them: E(I1,I2,θ,u)=L(I1,I2,θ,u)+λ[C(I1,θ)+C(I2,θ)] Among them, λ is a user-defined parameter that controls the weights of these two items. Then the energy function is set to the probability model of change detection. The optimization framework of the probability model is calculated according to the log-likelihood gradient of the probability model, which is expressed as follows: Among them I ' 1,I ' 2 represents all possible data in the data space; increasing the probability relative to the input data The contrast divergence algorithm is used to ensure efficiency; the parameter update gradient is as follows: From the above derivatives, we can see that the basic operator of optimization is the gradient; during the entire optimization process, two types of gradients must be derived, including the trainable parameters and input data There are two items in the energy function, and their gradients are derived respectively. Through the back propagation algorithm, the gradient of the alignment loss is obtained as follows: For content gradient, a similar gradient can also be derived, the formula is as follows: corresponding The gradient of can also be calculated as follows: where l k Represents the output of the kth layer of the network; Using the above gradient, update the model; treat u as a sigmoid function u = sigmoid(t), and then t can be updated by the following formula: After optimization, the feature vectors v1 and v2 can represent the contents of the two images I1 and I2; mark the changed pixels of the input image and highlight the changed content of the input image; In the fifth step, after optimization, the feature vectors of the two images are compared to solve the probability model. First, the constant loss function is defined, which is as follows: Among them L c Represents the unchanged loss, which is used as the energy function of the probability model; ⊙ represents the dot product, M1 and M2 are its change masks in the range of [0,1]; the probability model is established as follows: P c (I1, I2; M1, M2) is the probability model to be solved, and Z is the partition function Although different from P(I1, I2; u, θ), the optimization parameters M1 and M2 here can be optimized according to the above description; define M k =sigmoid(S k ),k=1,2; the optimization process includes sampling and updating parameters. During the sampling process, the sampling data I'1, I'2 are obtained by gradient Then update the parameters S1 and S2 using the following formula: In the sixth step, the feature vectors of the two images are compared, and the change regions are generated by optimizing the change masks in the change regions. Finally, the image clustering algorithm of FLICM is used to generate the difference map.

2. The unsupervised change detection method based on content coding according to claim 1, characterized in that In the sixth step, the FLICM image segmentation method is used to divide the pixels into changed pixels and unchanged pixels, and the final change map is generated through the image clustering algorithm.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the unsupervised change detection method based on content coding is implemented as described in any one of claims 1-2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the unsupervised change detection method based on content coding as described in any one of claims 1-2 is implemented.

Images