Remote sensing image change detection method based on coupled discriminant feature self-learning network

Abstract

The present invention proposes a remote sensing image change detection method based on a coupled discriminant feature self-learning network. The method mainly designs a coupled discriminant feature self-learning network, which extracts features of the image to be detected through two sub-networks. The established coupled training model can obtain a coupled feature space, and the discriminant ability of the features in the space will be enhanced, which is helpful to generate a clearer difference map and obtain accurate change detection results. The method does not require label data and is not limited by data types, and has a broad application space. The effectiveness of the method has been verified on a public dataset.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image processing, and in particular relates to a remote sensing image change detection method based on a coupled discriminant feature self-learning network. Background Art

[0002] Remote sensing images are now being used more and more widely, among which change detection using remote sensing images is a very important application field, such as land and resources survey, urban development monitoring, post-disaster situation analysis, natural resource statistical analysis, etc. The main difficulty in remote sensing image change detection is the difference in remote sensing image representation caused by different times, different sensors, and different data types, which will seriously interfere with the discovery of important change areas in remote sensing images.

[0003] The key issue in change detection is how to make appropriate feature representations for different remote sensing images. In this feature space, the difference between the changed areas can be larger and the non-changed areas can be smaller, so that a difference map reflecting the change of real objects can be obtained. Simple operations can be performed on the basis of the difference map to obtain the final change detection results. At present, the main research work of change detection revolves around solving the feature space. In recent years, deep learning methods have also been gradually applied to change detection, but most of them are based on labeled change detection data. This type of method regards the change detection problem as a binary classification problem, uses training data to obtain the mapping between the class label, and then applies it to other data. There are some problems with this approach. First, the labeling of training samples. Remote sensing images cover large scenes and complex situations. Special personnel are required to label them to obtain more reliable labeled data, which is very costly. Secondly, the various data of remote sensing images vary greatly. Even the data captured by the same sensor at different times are very different. Therefore, the network structure trained on known data may not be suitable for new data, and it is difficult to guarantee good results. To address the above issues, a new remote sensing image change detection method needs to be proposed to reduce the requirements for label data and find a suitable feature representation space only based on the data to be detected. Summary of the invention

[0004] The purpose of the present invention is to propose a remote sensing image change detection method based on a coupled discriminant feature self-learning network, which is used to solve the technical problems that the above-mentioned prior art has high requirements for label data and large differences in remote sensing image change detection accuracy.

[0005] Technical solution of the present invention:

[0006] The remote sensing image change detection method based on coupled discriminant feature self-learning network includes the following steps:

[0007] Step 1: Establish a network structure model, which includes two sub-networks: G network and F network; the number of layers of G network is l G , the number of neurons in the i-th layer is The number of network layers is l F , the number of neurons in the i-th layer is Satisfy the conditions

[0008] Step 2: Design the objective function of the coupled discriminant feature self-learning network to realize the self-learning process of the discriminant feature;

[0009] Step 3: The training process of the coupled discriminant feature self-learning network;

[0010] Step 3.1: Data preprocessing;

[0011] Step 3.2: Network initialization;

[0012] Step 3.3: Initialize the matrix S;

[0013] Step 3.4: Alternately optimize network parameters;

[0014] Step 3.5: Update the matrix S;

[0015] Step 3.6: When the objective function converges or the maximum number of iterations is reached, the optimization process ends;

[0016] Step 4: Calculate the difference map;

[0017] Step 5: Generate change detection result graph.

[0018] In the step 1: establishing a network structure model, the following specific steps are also included:

[0019] The structure of the G network and the F network is a typical fully connected network. The input of the network is the original features of each pixel of the remote sensing image, and the pixels in the local area around the pixel are used as the original features p, p∈R s×1 , this feature contains the local neighborhood information of the central pixel; the G network output and the F network output are respectively expressed by the following equations (1) and (2):

[0020]

[0021] Among them, I1 and I2 are two remote sensing images to be detected, and the size of each remote sensing image is

[0022] It is,M×N, where,M is the width of the remote sensing image and N is the height of the remote sensing image;

[0023] g(·) and f(·) represent the mapping functions of G network and F network, and the parameters they contain are (W G ,θG ) and (W F ,θ F ), where W represents the weight of the network, and θ represents the network bias; the number of network layers and the number of nodes in each layer of the G network and the F network can be adjusted, but the dimensions of the network output Out_G and the network output Out_F must be consistent. By defining the relationship between the network output Out_G and the network Out_F, the coupled self-learning of the two networks is realized.

[0024] The step 2 designs the objective function of the coupled discriminant feature self-learning network to realize the self-learning process of the discriminant feature, and also includes the following specific steps: using an objective function as the final objective function of the coupled learning of the two networks, and the objective function is as follows (3) (4):

[0025]

[0026] 0≤S x,y ≤1……………………(4)

[0027] in, and are the feature images after preprocessing of input images I1 and I2; S x,y It is used to indicate the probability of unchanged pixels, and a value close to 1 indicates and The probability that it has not changed is relatively high, and a value close to 0 indicates that the probability that the pixel has changed is relatively high; x and y represent the coordinates of the pixel point on the feature image respectively.

[0028] The step 3.1 data preprocessing also includes: extracting features at each pixel position of the image to be detected using a 3×3, 5×5 or 7×7 sliding window, converting the image block in the window into a column vector as a feature vector of the center pixel;

[0029] Step 3.2 network initialization includes: using the autoencoder network training method to perform network pre-training to complete the initialization process; using 50% of the data randomly extracted from the image to be tested I1 to train the G network, and using 50% of the data randomly extracted from the image to be tested I2 to train the F network;

[0030] Step 3.3 initializes the matrix S, including: using random initialization to generate an initialized pixel unchanged matrix S, where the initialization matrix elements are in [0, 1] and satisfy the Gaussian distribution;

[0031] Step 3.4 alternately optimizes network parameters, including: training the coupled discriminant feature learning network through alternate optimization; alternately optimizing the coupled discriminant feature learning network training is divided into three steps:

[0032] Step 3.4.1: The mapping function f(·) of the F network remains unchanged, the mapping function g(·) of the G network is trained, and the parameters in the G network are updated using the back propagation method to obtain the trained network mapping function g(·). The objective function is as follows (5):

[0033]

[0034] Step 3.4.2: The mapping function g(·) of the G network remains unchanged, the mapping function f(·) of the F network is trained, and the parameters in the network F are updated using the back propagation method to obtain the trained network mapping function f(·). The objective function is as follows (6):

[0035]

[0036] The network training method uses the back propagation method, and the probability S of the pixel not changing x,y Under the control of , error back propagation is performed to update parameters and minimize Formula 5 and Formula 6;

[0037] Step 3.5: Given g(·) and f(·), updating the unchanged pixel matrix S includes: calculating each element in S according to equation (7);

[0038] When g(·) and f(·) are given, the feature representation of the image to be detected is obtained, S x,y The probability that a pixel at the (x, y) position is a non-changing pixel should be inversely proportional to the difference in the pixel features at that position;

[0039] On this basis, the pixel unchanged matrix S is updated by formula 7:

[0040]

[0041] Step 3.6: When the objective function converges or the maximum number of iterations is reached, the optimization process ends. The judgment conditions are as follows:

[0042] Judgment condition 1: The change of the objective function value does not exceed 0.001 for 20 consecutive iterations;

[0043] Judgment condition 2: The number of iterations is greater than the maximum number of iterations MaxIter, MaxIter = 2000;

[0044] If one of the above stopping conditions is met, proceed to step 4. If the stopping condition is not met, return to step 3.4 and continue the training process.

[0045] The step 4 also includes: using the Euclidean distance to calculate the difference in pixel features between the outputs of the networks G and F to obtain a difference map; calculating the difference map Dif∈R M×N ; Use Euclidean distance calculation, the calculation formula is as follows (8):

[0046]

[0047] Step 5: Generate a change detection result graph V∈R M×N ; Use the threshold segmentation method to obtain the change detection result map V. The calculation formula of the change detection result map is as follows (9):

[0048]

[0049] Among them, λ is the threshold determined by the threshold segmentation method.

[0050] Technical effects of the present invention:

[0051] In order to overcome the differences in the data itself due to the imaging process, the present invention designs a coupled discriminant feature learning network to find a feature space suitable for remote sensing image change detection. Since the remote sensing image change information labels are difficult to obtain, the present invention adopts an unsupervised self-learning method to perform feature learning, avoiding the need for labeled data.

[0052] The present invention introduces neural network technology to extract features from the image to be detected, and uses a coupled training method to associate the outputs of the two sub-networks, which can achieve discrimination enhancement of the final feature representation, that is, the greater the feature difference of the changed pixel, the smaller the feature difference of the non-changed pixel. The difference in the difference map obtained in this way is more obvious, which is conducive to the generation of the final change detection map.

[0053] The present invention starts from the data to be detected, utilizes the autonomous learning ability of the deep learning network, adaptively finds a suitable feature representation space, calculates a suitable difference map, and finally obtains accurate change detection results through simple threshold segmentation.

[0054] The present invention designs a coupled discriminant feature self-learning network, which extracts features from the image to be detected through two sub-networks respectively. The established coupled training model can obtain a coupled feature space, and the discriminant ability of the features in the space will be enhanced, which will help to generate a clearer difference map and obtain accurate change detection results. The method does not require label data and is not limited by the data type. It meets the needs of different types of data for change detection, and does not require label data, which reduces the cost of labor, has strong practical application value, and has a broad application space. The effectiveness of the method has been verified on a public data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flow chart of the remote sensing image change detection method based on coupled discriminant feature self-learning network designed for the present invention

[0056] Figure 2Figure a is the surface change diagram of Ottawa area affected by the rainy season in the embodiment;

[0057] Figure 3 Figure b is the surface change diagram of Ottawa area affected by the rainy season in the embodiment;

[0058] Figure 4 It is a reference map of the surface change detection data in the Ottawa area affected by the rainy season in the embodiment;

[0059] Figure 5 Figure a is the surface change of vegetation destruction after the forest fire in a Mexican city in the embodiment;

[0060] Figure 6 Figure b is the surface change of vegetation destruction after the forest fire in the Mexican city in the embodiment;

[0061] Figure 7 It is a reference map of surface change detection data of vegetation destruction after forest fire in a Mexican city in the embodiment;

[0062] Figure 8 This is an optical remote sensing image of the Yellow River Bay caused by flooding in the embodiment;

[0063] Fig. 9 This is a SAR remote sensing image of the Yellow River Bay caused by flooding in the embodiment;

[0064] Fig.10 It is a reference diagram of the river channel changes of the Yellow River Bay in the embodiment;

[0065] Fig.11 This is a difference diagram of the change detection results in the Ottawa area in the embodiment;

[0066] Fig.12 This is a graph of change detection results in the Ottawa area in the embodiment;

[0067] Fig.13 It is a reference map of change detection data in the Ottawa area in the embodiment;

[0068] Fig.14 This is a difference diagram of the change detection results in Mexico in the embodiment;

[0069] Fig.15 This is a diagram of the change detection results for the Mexico region in the embodiment;

[0070] Fig.16 It is a reference diagram of change detection data in Mexico in the embodiment;

[0071] Fig.17 This is a difference diagram of the detection results of the Yellow River Bay changes in the embodiment;

[0072] Fig.18 This is a diagram showing the detection results of the Yellow River Bay changes in the embodiment;

[0073] Fig.19 It is a reference diagram of the Yellow River Bay change detection data in the embodiment; DETAILED DESCRIPTION

[0074] The method designed by the present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0075] This paper proposes a coupled discriminant feature self-learning network, which can be divided into two parts, denoted as G and F. Its structure is a typical fully connected network. The input end of the network is the original feature of each pixel of the remote sensing image. Here we use the pixels in the local area around the pixel as its original feature (p, p∈R s×1 ), this feature contains the local neighborhood information of the central pixel. The output of the network is:

[0076]

[0077] Among them, I1 and I2 are two remote sensing images to be detected, and the size of each remote sensing image is M×N, where M is the width of the remote sensing image and N is the height of the remote sensing image. g(·) and f(·) represent the mapping functions of the networks G and F, and the parameters included are (W G ,θ G ) and (W F ,θ F ), the number of network layers and the number of nodes in each layer of the G network and the F network can be adjusted, but the dimensions of the outputs Out_G and Out_F must be consistent. By defining the relationship between Out_G and Out_F, we achieve coupled self-learning of the two networks.

[0078] Step 1. Establish a network structure model, including two sub-networks (G, F), G network layer number is l G , the number of neurons in the i-th layer is The number of network layers is l F , the number of neurons in the i-th layer is l G and l F Can be equal or unequal, the number of neurons in each layer can be equal or unequal, but the condition must be met

[0079] Step 2. Design the objective function of the coupled discriminant feature self-learning network to realize the self-learning process of the discriminant feature. In order to couple the two sub-networks, we use an objective function as the final objective function of the coupled learning of the two networks. The objective function is formula 3 and 4.

[0080] Step 3. The learning process of the coupled discriminant feature self-learning network is a multiple iterative optimization process, which mainly includes the following sub-steps:

[0081] Step 3.1 Data preprocessing: Extract features from each pixel position of the image to be detected. You can use a 3×3, 5×5, or 7×7 window sliding window to convert the image block in the window into a column vector as the feature vector of the center pixel.

[0082] Step 3.2 Network initialization. Generally, the network is initialized randomly, that is, the weight bias is randomly assigned. In order to speed up the convergence of the later network training, the autoencoder network training method is used here to pre-train the network and complete the initialization process. The G network is trained using 50% of the data randomly extracted from the image to be tested I1, and the F network is trained using 50% of the data randomly extracted from the image to be tested I2.

[0083] Step 3.3 Initialize the matrix S. Generate S using random initialization, with elements in [0,1] satisfying Gaussian distribution.

[0084] Step 3.4 Given the matrix S, train the coupled discriminant feature learning network by optimizing formula 4 and formula 5. This process can be divided into two steps:

[0085] Step 3.4.1f(·) remains unchanged, train g(·), use the back-propagation method to update the parameters in the network G, and obtain the trained network mapping function g(·);

[0086] Step 3.4.2: Keep g(·) unchanged, train f(·), use the back-propagation method to update the parameters in the network G, and obtain the trained network mapping function f(·).

[0087] Step 3.5 Given g(·) and f(·), update the matrix S. Calculate each element in S according to formula 6.

[0088] Step 3.6 determines whether the training stop condition is met. Condition 1: The change of the objective function value does not exceed 0.001 for 20 consecutive iterations; Condition 2: The number of iterations is greater than the maximum number of iterations MaxIter (MaxIter = 2000). If the stop condition is met, proceed to step 4. If not, return to step 3.4 and continue the training process.

[0089] Step 4 Calculate the difference map (Dif∈R M×N ). Use Euclidean distance calculation, the formula is as follows:

[0090]

[0091] After the difference map is generated, in order to verify whether the features learned by this method are more suitable for the change detection problem, we use the simplest threshold segmentation method (OTSU method) to convert the difference map into a binary change detection map.

[0092] Step 5 Generate the change detection result graph (V∈R M×N ). Use threshold segmentation (OTSU method) to obtain the change detection result map V. The formula is as follows:

[0093]

[0094] Here, λ is the threshold determined by the OTSU method.

[0095] The present invention introduces neural network technology to extract features from the image to be detected, and uses a coupled training method to associate the outputs of the two sub-networks, which can achieve discrimination enhancement of the final feature representation, that is, the greater the feature difference of the changed pixel, the smaller the feature difference of the non-changed pixel. The difference in the difference map obtained in this way is more obvious, which is conducive to the generation of the final change detection map.

[0096] To verify the effectiveness of this method, we conducted example experiments on multiple sets of public experimental data. The specific experimental settings and experimental results are given below.

[0097] 1. Simulation conditions

[0098] (1) Software Environment

[0099] This method was developed under Windows 10 system, the compiler was VS2015 (C#), and the Python code called in the code was written and debugged on VScode. The deep learning platform used was PyTorch 1.0.1. The Python code was written based on Python 3.6, and the library functions called included: pytorch 1.0.1, torchvision 0.1.8 numpy1.15.4, matplotlib 3.1.1, scipy 1.0.0, scikit-learn 0.20.1, opencv 3.4.2, opencv-python 4.1.1.26, easydict 1.9, cffi 1.13.2, tensorboardX 1.9, pyyaml0.1.7.

[0100] (2) Hardware environment

[0101] HP workstation, hardware configuration: dual-core CPU (Intel Xeon E5-2630 v4), two GPUs (NVIDIATITAN X 12G).

[0102] 2. Introduction of experimental data

[0103] (1) Change detection data in the Ottawa area

[0104] Figure 2-4 The image was taken by Radarsat SAR satellite, with an image size of 290*350. It reflects the surface changes in Ottawa, Canada, affected by the rainy season. The images were taken in May and August 1997. The image data mainly consists of land and water, and the change information mainly comes from the surface changes caused by floods. Its spatial resolution is 12m. The white area in the change reference image is the area that has changed.

[0105] (2) Mexican change detection data

[0106] Figure 5-7 This dataset consists of two optical images acquired by the US satellite Landsat-7 in the urban area of ​​Mexico in April 2000 and May 2002; these two images are extracted from band 4 of the ETM+ image, and the size of both images is 512 × 512. This image dataset shows the vegetation damage after the forest fire in the Mexican city.

[0107] (3) Yellow River Bay change detection data

[0108] Figure 9-10 This set of data includes SAR images and optical remote sensing images. The SAR images were collected by Radarsat-2 at the mouth of the Yellow River in June 2008, and the optical images were obtained from Google Earth in September 2010. The images cover the same area as the SAR images. Google Earth provides a large amount of VHR image data of the earth's surface. The data provided by Google Earth integrates satellite and aerial photography images. Figure 9-10 The changes in image data are mainly due to the changes in the Yellow River channel caused by flooding.

[0109] 3. Experimental results

[0110] (1) Change detection results in the Ottawa area

[0111] When testing this set of data, we set the G network and F network structures to the same structure, with 4 layers, the number of neurons in each layer is [20, 20, 20, 10], and the neighborhood window is 3*3. The detection results are compared with the reference image, and the detection accuracy is 98.11%. See the experimental results. Figure 11-13 .

[0112] (2) Results of change detection data in Mexico

[0113] When testing this set of data, we set the G network and F network structures to the same structure, with 3 layers, the number of neurons in each layer is [50, 20, 10], and the neighborhood window is 5*5. The detection results are compared with the reference image, and the detection accuracy is 96.49%. See the experimental results. Figure 14-16 .

[0114] (3) Yellow River Bay change detection data detection results

[0115] When testing this set of data, we set the G network and F network structures to the same structure, with 2 layers, the number of neurons in each layer is [20, 20], and the neighborhood window is 3*3. The detection results are compared with the reference image, and the detection accuracy is 96.75%. See the experimental results. Figure 17-19 .

[0116] The experimental results of the above data can prove the effectiveness of the method proposed in the present invention. This method can break through the limitation of data format, meet the needs of change detection of different types of data, and does not require label data, reducing the cost of labor, and has strong practical application value.

Claims

1. A remote sensing image change detection method based on a coupled discriminant feature self-learning network, characterized in that: The following steps are involved: Step 1: Establish a network structure model, which includes two sub-networks: G network and F network; the number of layers of G network is l G , the number of neurons in the i-th layer is The number of network layers is l F , the number of neurons in the i-th layer is Satisfy the conditions In the step 1: establishing a network structure model, the following specific steps are also included: The structure of the G network and the F network is a typical fully connected network. The input of the network is the original features of each pixel of the remote sensing image, and the pixels in the local area around the pixel are used as the original features p, p∈R s×1 , this feature contains the local neighborhood information of the central pixel; the G network output and the F network output are respectively expressed by the following equations (1) and (2): Among them, I1 and I2 are two remote sensing images to be detected, and the size of each remote sensing image is M×N, where M is the width of the remote sensing image and N is the height of the remote sensing image; g(·) and f(·) represent the mapping functions of G network and F network, and the parameters they contain are (W G ,θ G ) and (W F ,θ F ), where W represents the weight of the network, and θ represents the network bias; the number of network layers and the number of nodes in each layer of the G network and the F network can be adjusted, but the dimensions of the network output Out_G and the network output Out_F must be consistent. By defining the relationship between the network output Out_G and the network Out_F, the coupled self-learning of the two networks is realized; Step 2: Design the objective function of the coupled discriminant feature self-learning network to realize the self-learning process of the discriminant feature; the step 2 designs the objective function of the coupled discriminant feature self-learning network to realize the self-learning process of the discriminant feature, and also includes the following specific steps: use an objective function as the final objective function of the coupled learning of the two networks, and the objective function is as follows (3) (4): 0≤S x,y ≤1…………………(4) in, and are the feature images after preprocessing of input images I1 and I2; S x,y It is used to indicate the probability of unchanged pixels, and a value close to 1 indicates and The probability that it has not changed is relatively high, and a value close to 0 indicates that the probability that the pixel has changed is relatively high; x and y represent the coordinates of the pixel point on the feature image respectively; Step 3: The training process of the coupled discriminant feature self-learning network; Step 3.1: Data preprocessing; Step 3.2: Network initialization; Step 3.3: Initialize the matrix S; Step 3.4: Alternately optimize network parameters; Step 3.5: Update the matrix S; Step 3.6: When the objective function converges or the maximum number of iterations is reached, the optimization process ends; The step 3.1 data preprocessing also includes: extracting features at each pixel position of the image to be detected using a 3×3, 5×5 or 7×7 sliding window, converting the image block in the window into a column vector as a feature vector of the center pixel; Step 3.2 network initialization includes: using the autoencoder network training method to perform network pre-training to complete the initialization process; using 50% of the data randomly extracted from the image to be tested I1 to train the G network, and using 50% of the data randomly extracted from the image to be tested I2 to train the F network; Step 3.3 initializes the matrix S, including: using random initialization to generate an initialized pixel unchanged matrix S, where the initialization matrix elements are in [0, 1] and satisfy the Gaussian distribution; Step 3.4 alternately optimizes network parameters, including: training the coupled discriminant feature learning network through alternate optimization; alternately optimizing the coupled discriminant feature learning network training is divided into three steps: Step 3.4.1: The mapping function f(·) of the F network remains unchanged, the mapping function g(·) of the G network is trained, and the parameters in the G network are updated using the back propagation method to obtain the trained network mapping function g(·). The objective function is as follows (5): Step 3.4.2: The mapping function g(·) of the G network remains unchanged, the mapping function f(·) of the F network is trained, and the parameters in the network F are updated using the back propagation method to obtain the trained network mapping function f(·). The objective function is as follows (6): The network training method uses the back propagation method, and the probability S of the pixel not changing x,y Under the control of , error back propagation is performed to update parameters and minimize Formula 5 and Formula 6; Step 3.5: Given g(·) and f(·), updating the unchanged pixel matrix S includes: calculating each element in S according to equation (7); When g(·) and f(·) are given, the feature representation of the image to be detected is obtained, S x,y The probability that a pixel at the (x, y) position is a non-changing pixel should be inversely proportional to the difference in the pixel features at that position; On this basis, the pixel unchanged matrix S is updated by formula 7: Step 3.6: When the objective function converges or the maximum number of iterations is reached, the optimization process ends. The judgment conditions are as follows: Judgment condition 1: The change of the objective function value for 20 consecutive iterations does not exceed 0.001; Judgment condition 2: The number of iterations is greater than the maximum number of iterations MaxIter, MaxIter = 2000; If one of the above stopping conditions is met, proceed to step 4. If the stopping condition is not met, return to step 3.4 and continue the training process; Step 4: Calculate the difference map; Step 5: Generate change detection result graph.

2. The remote sensing image change detection method based on coupled discriminant feature self-learning network according to claim 1 is characterized in that: The step 4 also includes: using the Euclidean distance to calculate the difference in pixel features between the outputs of the networks G and F to obtain a difference map; calculating the difference map Dif∈R M×N ; Use Euclidean distance calculation, the calculation formula is as follows (8):

3. The remote sensing image change detection method based on coupled discriminant feature self-learning network according to claim 2 is characterized in that: Step 5: Generate a change detection result graph V∈R M×N ; Use the threshold segmentation method to obtain the change detection result map V. The calculation formula of the change detection result map is as follows (9): Among them, λ is the threshold determined by the threshold segmentation method.

Images