SAR image change detection method based on global dynamic convolutional neural network

Abstract

The application provides a SAR image change detection method based on a global dynamic convolutional neural network, difference analysis is performed on two multi-temporal SAR images captured in the same geographical area to obtain a difference image; the difference image is pre-classified to construct a training data set and a test data set; the global dynamic convolutional neural network model is trained by using the training data set, and the test data set is used for testing, so that the multi-temporal SAR image change detection result of the whole same geographical area is obtained. In the scheme, a two-stage mixed sample data enhancement method is used to increase the number of data sets when the training data set is constructed, so that the sample diversity and reliability are ensured; meanwhile, the global dynamic convolution method is used to improve the performance of the convolutional neural network, so that the robustness to noise and the generalization ability of the SAR image change detection are improved; in combination with the digital image processing and the deep learning technology, the method has important significance in the fields of national defense and military reconnaissance, natural environment monitoring, natural disaster monitoring and urban land planning.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and utilizes SAR (Synthetic Aperture Radar) images to detect changes in ground objects in the same geographical area, and specifically relates to a SAR image change detection method based on a global dynamic convolutional neural network. BACKGROUND

[0002] Change detection technology aims to detect changes in the same geographical area at different times. Image change detection technology mainly relies on texture changes in images to detect changes in ground objects. Generally, texture changes in images are caused by actual changes in ground objects in the geographical area, or by differences in external environmental conditions and hardware facilities when images are taken, such as shooting angles, atmospheric environment, and sensor accuracy.

[0003] The basic premise of image change detection technology is that changes caused by these external environmental conditions and hardware facility differences and actual changes in ground objects can be distinguished. At present, due to the high development of remote sensing technology, ground object change detection technology based on remote sensing images is a key technology in many ground observation applications, and is widely used in many real-world scenarios, such as national defense and military reconnaissance, natural environment monitoring, natural disaster monitoring, and urban land planning. In particular, synthetic aperture radar (SAR) can obtain large-area high-resolution remote sensing images using pulse compression technology, and can accurately obtain ground target information. More importantly, the remote sensing images obtained by this synthetic aperture radar are not affected by weather conditions, and its sensor can capture images at night and observe through clouds and smoke, which is an all-weather technology. Therefore, SAR images are widely considered to be an ideal source of remote sensing image change detection.

[0004] According to whether there is an artificial true label as a priori, SAR image change detection methods are mainly divided into supervised and unsupervised two kinds. Most of the existing work focuses on unsupervised change detection, because it is difficult to obtain ground true labels in many practical applications. Generally speaking, the current mainstream method based on deep learning to study unsupervised SAR image change detection mainly includes three steps: (1) difference map generation, (2) unsupervised pre-classification, (3) neural network model training and classification. In the difference map generation step, the ratio-based method is widely used, and the ratio operator is used to analyze the difference between two registered and radiometric corrected SAR images to obtain a difference map; in the unsupervised pre-classification step, the unsupervised clustering method is widely used, and the unsupervised clustering method is used to divide each pixel in the difference map obtained in step one into an unchanged class, a changed class and an uncertain class to obtain the pseudo label of each pixel; in the neural network model training and classification step, the pixels determined as unchanged class and changed class in step two are used as the training data set of the neural network to train the neural network model, and then the trained neural network model is used to reclassify each pixel in the difference map, mainly the pixels in the uncertain class in step two, which are classified into unchanged class or changed class, and finally the class of all pixels is obtained to obtain the change result map of the whole SAR image.

[0005] Although most of the current deep learning-based unsupervised SAR image change detection methods perform well, there are still some unresolved problems, which are summarized as follows: (1) global context and local feature interaction problem. Traditional convolutional layers mainly extract local features, and context information is difficult to be utilized in neural networks. Therefore, it is crucial to explore the interaction between global context and local features in the change detection framework. (2) Instability of data augmentation. In order to collect enough high-quality training samples for the SAR image change detection task, hybrid sample data augmentation techniques are usually considered, which effectively enrich the representation space, but also bring instability in the training process. Therefore, how to effectively eliminate the side effects of data augmentation is an important task that cannot be ignored. SUMMARY

[0006] The present application is aimed at the global context and local feature interaction problem and the data augmentation instability problem existing in the current SAR image change detection method based on SAR image change detection method, and proposes a SAR image change detection method based on global dynamic convolutional neural network to improve the accuracy and performance of SAR image change detection.

[0007] The present application is realized by adopting the following scheme: a SAR image change detection method based on global dynamic convolutional neural network, comprising the following steps:

[0008] Step 1, difference analysis is performed on two multi-temporal SAR images captured in the same geographical area to obtain a difference map;

[0009] Step 2, pre-classification is performed on the difference map to construct a training data set and a test data set;

[0010] Step 3, a global dynamic convolutional neural network model is constructed;

[0011] Step 4, the global dynamic convolutional neural network model is trained by using the training set enhanced by the two-stage mixed sample data enhancement method;

[0012] Step 5, the trained network model is used to test the test data set, so as to obtain the change detection result of the whole image.

[0013] The specific steps of the application include the following:

[0014] 1. Difference analysis is performed on two multi-temporal SAR images captured in the same geographical area to obtain a difference map:

[0015] The two multi-temporal SAR images are subjected to difference analysis by using a logarithmic ratio operator to obtain a difference map of the two multi-temporal SAR images;

[0016] The calculation formula of the difference map is:

[0017] I DI = |logI1-logI2|

[0018] Wherein, I1 and I2 respectively represent two multi-temporal SAR images in the same geographical area, I DI represents the difference map of the two multi-temporal SAR images I1 and I2, |·| represents absolute value operation, and log represents logarithmic operation with base 10;

[0019] 2. Pre-classification is performed on the difference map I DI to construct a training data set and a test data set:

[0020] (2.1) The difference map I DI is pre-classified by using a hierarchical FCM (fuzzy C-means) clustering algorithm to obtain a pseudo-label matrix, and the pseudo-label values are 0, 1 and 0.5, that is, high-probability samples that can be determined as unchanged or changed and uncertain samples are generated after pre-classification;

[0021] (2.2) p% of pixels are randomly selected from the pixels with pseudo-label values of 0 and 1, the value of p is an integer not greater than 10, and the spatial positions of these pixels are extracted; in the corresponding spatial positions of the original two multi-temporal SAR images, the neighborhood pixel blocks of r x r around the pixel points are taken as the training set, and the domain pixel blocks are extracted by filling 0 in the domain pixels of the edge pixel points, and the value of r is an odd number not less than 3;

[0022] (2.3) Extract all the r x r neighborhood pixel blocks around each pixel in the original two multi-temporal SAR images as the test set, and the edge pixel points are extracted by filling the domain pixels with 0, and the value of r is an odd number not less than 3;

[0023] 3. Construct a global dynamic convolutional neural network:

[0024] The core architecture of the constructed network is composed of three global dynamic convolutional layers, and the structure is: input layer→low-level global dynamic convolutional layer→middle-level global dynamic convolutional layer→high-level global dynamic convolutional layer→fully connected layer. The sample size input by the input layer is c x r x r, and c represents the number of input data channels;

[0025] (3.1) Construct a global dynamic convolutional layer:

[0026] The structure of the three global dynamic convolutional layers is consistent, and each global dynamic convolutional layer is composed of a global feature encoding module, a context feature projection module and a convolution kernel weight generation module. First, the global feature encoding module encodes the global information through the average pooling layer and the linear layer, and then the context feature projection module projects the encoded features to the output dimension space through the context feature projection. Finally, the convolution kernel weight generation module generates a convolution kernel containing global context information, and uses the generated new convolution kernel to perform regular convolution;

[0027] (3.1.1) Construct a global feature encoding module:

[0028] The global feature encoding module is composed of an average pooling layer, a linear layer, a normalization layer and an activation layer. The average pooling layer extracts the global information of the input data, and reduces the spatial size of the input data to k x k, where k is the size of the convolution kernel; then the linear layer projects the features in all c channels to a vector of size m; and then the normalized and ReLU activation function obtains the output features of the global feature encoding module:

[0029]

[0030] (3.1.2) Construct a context feature projection module:

[0031] The context feature projection module is composed of a linear layer, a normalization layer and an activation layer. The linear layer projects the features G obtained in step 3.1.1 to a space with output dimension n, and then the normalized and ReLU activation function obtains the output features of the context feature projection module:

[0032]

[0033] (3.1.3) Construct a convolution kernel weight generation module:

[0034] The weight generation module first uses two linear layers to convert the feature G obtained in step 3.1.1 and the feature C obtained in step 3.1.2 into the spatial size of the convolution kernel. The first linear layer converts the feature Convert to The second linear layer transforms the features Convert to Then M G Dimension expansion is performed to obtain M C Dimension expansion is performed to obtain Then, M G and M C Add together to generate the convolution kernel weight M, which is calculated as follows:

[0035] M = δ(M′ G +M′ C )

[0036] in, The size is the same as the original convolution kernel W, and δ represents the Sigmoid activation function. Finally, the generated convolution kernel weight M is multiplied element-by-element by the current convolution kernel weight W. Combining global and local information, a new convolution kernel weight W′ that is adaptive to the input data is obtained. The calculation formula is as follows:

[0037] W′=M⊙W

[0038] Among them, ⊙ represents element-by-element multiplication;

[0039] (3.1.4) Perform regular convolution on the current data in the input data using the new convolution kernel generated in step 3.1.3;

[0040] (3.2) Use the low-level global dynamic convolution layer constructed in step 3.1 to extract the low-level features F of the input data l :

[0041] The convolution kernel size in the low-level global dynamic convolution layer is k l ×k l , the number of convolution kernels is n l , where k l The size of n is 3, l The number of low-level features is 48. l The calculation formula is as follows:

[0042] F l =σ(BN(XW l ′+b l ))

[0043] Among them, X represents the input sample in the input layer, which is also the input of the lower global dynamic convolution layer; W lW' represents the sample adaptive convolution kernel weight of the low-layer global dynamic convolution layer obtained through step 3.1.3; b l represents the bias term of the low-layer global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; BN represents the batch normalization operation; and σ represents the ReLU activation function;

[0044] (3.3) extracting the middle-layer feature F of the input data by using the middle-layer global dynamic convolution layer constructed in step 3.1 m :

[0045] The convolution kernel size in the middle-layer global dynamic convolution layer is k m ×k m , and the number of convolution kernels is n m , wherein the size of k m is 3, the number of n m is 96, and the calculation formula of the middle-layer feature F m is as follows:

[0046] F m =σ(BN(F l W m ′+b m ))

[0047] , wherein F l represents the low-layer feature of the input data, which is also the input of the middle-layer global dynamic convolution layer; W m ′ represents the low-layer feature adaptive convolution kernel weight of the middle-layer global dynamic convolution layer obtained through step 3.1.3; and b m represents the bias term of the middle-layer global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; BN represents the batch normalization operation; and σ represents the ReLU activation function;

[0048] (3.4) extracting the high-layer feature F of the input data by using the high-layer global dynamic convolution layer constructed in step 3.1 h :

[0049] The convolution kernel size in the high-layer global dynamic convolution layer is k h ×k h , and the number of convolution kernels is n h , wherein the size of k h is 3, the number of n h is 48, and the calculation formula of the high-layer feature F h is as follows:

[0050] F h =σ(BN(F m W h ′+b h ))

[0051] wherein F m represents the middle-layer feature of the input data, and is also the input of the high-layer global dynamic convolutional layer; W h ' represents the middle-layer feature adaptive convolution kernel weight of the high-layer global dynamic convolutional layer obtained through step 3.1.3; b h represents the bias term of the high-layer global dynamic convolutional layer, which is obtained by random initialization and is optimized through network iteration training; BN represents the batch normalization operation; and σ represents the ReLU activation function;

[0052] (3.5) obtaining the high-layer feature F h through the full connection layer to obtain Y:

[0053] Y = W fc2 (W fc1 F h )

[0054] wherein W fc1 represents the first layer full connection operation, W fc2 represents the second layer full connection operation, and Y obtained after the two full connection operations is a vector with a dimension of 2x1 wherein a represents the probability that the input sample belongs to the invariant class, and b represents the probability that the input sample belongs to the change class, and the prediction label of each sample is output according to the vector When a > b, is equal to the category to which a belongs, that is, When a < b, is equal to the category to which b belongs, that is,

[0055] 4. Using the training data set obtained in step 2 to train the global dynamic convolutional neural network constructed in step 3:

[0056] During training, a two-stage mixed sample data enhancement method is used for training to increase the number of training samples, enrich the diversity of samples, and alleviate the overfitting problem.

[0057] (4.1) Let the training round be t, and from the first round to the t / 2 round, use the mixed sample data enhancement method to construct virtual samples for all samples in the batch when training in batches:

[0058] The calculation formula of the mixed sample data enhancement is:

[0059]

[0060]

[0061] wherein x i and x j ​Represents two input sample data, y i and y j correspond to their sample labels respectively, and (x i ,y i ) and (x j ,y j ) are two samples randomly selected from the training data set constructed in step 2.2, so and It represents the data and label of the new virtual sample after the two samples are mixed. λ is the mixing coefficient obtained by random sampling from the Beta distribution, and its value range is [0,1]:

[0062] λ~Beta(α,β)

[0063] Among them, α and β are the parameters of the Beta distribution. In this method, α = β = 0.5;

[0064] (4.2) From round (t / 2+1) to round t, the mixed sample data augmentation method gradually transitions to the basic data augmentation method, and the linearly decreasing probability ε is used to control whether to use mixed sample data augmentation. The calculation formula of ε is as follows:

[0065] ε=(ti) / 2t

[0066] Here, i represents the current round number. A random number θ is set as the threshold, and the random range of θ is [0, 1]. When θ < ε, the mixed sample data augmentation in step 4.1 is used; otherwise, basic data augmentation is used to expand the sample, such as cropping, flipping, rotating the sample, etc.

[0067] (4.3) During the training process, the loss function of the global dynamic convolutional neural network is calculated as follows:

[0068]

[0069] in,

[0070]

[0071]

[0072] If mixed sample data enhancement is performed: y i and y j For the two random input samples x described in step 4.1 i and x j The corresponding sample label, is x i and x j Mixed samples The predicted label obtained after the network constructed in step 3; if no mixed sample data enhancement is performed, y i =y j =y, y is the true label of the current input sample; is the predicted label obtained after the network constructed in step 3 for the current input sample; L CE represents a cross-entropy loss function, λ represents a mixing coefficient described in step 4.1, and log represents a logarithm operation with base 10;

[0073] (4.4) using a stochastic gradient descent (SGD) algorithm to optimize the global dynamic convolutional neural network parameters;

[0074] 5, input the test data set in step 2.3 into the optimized global dynamic convolutional neural network, and obtain the predicted label of the test data set according to the processes described in steps 3.2 to 3.5; obtain the change result map of the place described in step 1.

[0075] Compared with the prior art, the advantages and positive effects of the present application are that:

[0076] The SAR image change detection method based on the global dynamic convolutional neural network provided by the present application processes the SAR image through the difference map generation, unsupervised pre-classification and neural network model training and classification, and utilizes the characteristics of high classification accuracy and high robustness to noise of the global dynamic convolutional neural network classifier:

[0077] 1. The log ratio operator is used for difference analysis to obtain the difference map of the two multi-temporal SAR images. The log ratio operator can effectively suppress speckle noise, enhance the contrast of change classes and non-change classes, and thus enhance the difference of samples;

[0078] 2. The hierarchical FCM (fuzzy C-means) clustering algorithm is used for pre-classification to obtain a pseudo label matrix. The hierarchical FCM clustering algorithm has good clustering effect and high clustering efficiency, so that the accuracy and speed of pre-classification can be improved;

[0079] 3. The global dynamic convolutional neural network effectively combines global features and local features through adaptive convolution kernels, so that more robust feature representation can be obtained, and the classification accuracy of the neural network classifier can be improved;

[0080] 4. The two-stage mixed sample data enhancement method can increase the number and diversity of training samples, prevent overfitting problems, and improve the generalization ability of the network, so that more stable classification results can be effectively generated. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 is a flowchart schematic diagram of the SAR image change detection method described in the embodiments of the present application;

[0082] Figure 2 The method principle block diagram of the embodiment of the present application is shown in the figure;

[0083] Figure 3 The schematic diagram of the neural network structure of the embodiment of the present application is shown in the figure;

[0084] Figure 4 The schematic diagram of the global dynamic convolution module of the embodiment of the present application is shown in the figure;

[0085] Figure 5 The schematic diagram of the input data of the embodiment of the present application is shown in the figure;

[0086] Figure 6 The effect comparison diagram of the method of the embodiment of the present application and the prior art is shown in the figure. DETAILED DESCRIPTION

[0087] In the following description, a plurality of different aspects of the present application will be described, however, the present application can be implemented by only using some or all structures or processes of the present application for the ordinary skilled in the art. In order to clarify, specific numbers, configurations and sequences are set forth, however, it is obvious that the present application can be implemented without these specific details. In other cases, in order not to confuse the present application, some well-known features will not be described in detail.

[0088] Reference Figure 1 The specific steps implemented by the present application are as follows:

[0089] Step 1: difference analysis is performed on two multi-temporal SAR images captured in the same geographical area to obtain a difference image:

[0090] Difference analysis is performed on the two multi-temporal SAR images by using a logarithmic ratio operator to obtain a difference image of the two multi-temporal SAR images;

[0091] The calculation formula of the difference image is:

[0092] I DI = |logI1-logI2|

[0093] Wherein, I1 and I2 respectively represent two multi-temporal SAR images of the same geographical area, I DI represents the difference image of the two multi-temporal SAR images I1 and I2, |·| represents absolute value operation, and log represents logarithmic operation with base 10;

[0094] Step 2: pre-classification is performed on the difference image I DI to construct a training data set and a test data set:

[0095] Step 21: the difference image I DIThe hierarchical FCM (fuzzy C-means) clustering algorithm is used for pre-classification to obtain a pseudo-label matrix, and the pseudo-label values are 0, 1 and 0.5, that is, the pre-classification produces high-probability samples that can be determined as unchanged or changed and uncertain samples;

[0096] Step 22: Randomly select p% of pixels from the pixels with pseudo-label values of 0 and 1, p is an integer not greater than 10 (the optimal value in the data set used in the method is 6), extract the spatial positions of these pixels, and take the r x r neighborhood pixel block around the pixel point as the training set on the corresponding spatial position pixels of the original two multi-temporal SAR images, and the edge pixel block is extracted by filling the edge pixel with 0, and r is an odd number not less than 3 (the optimal value in the data set used in the method is 9 or 11);

[0097] Step 2.3: Extract all r x r neighborhood pixel blocks around the pixels in the original two multi-temporal SAR images as the test set, and extract the edge pixel block by filling the edge pixel with 0, and r is an odd number not less than 3 (the optimal value in the data set used in the method is 9 or 11);

[0098] Step 3: Construct a global dynamic convolutional neural network:

[0099] The core architecture of the constructed network is composed of three global dynamic convolutional layers, and the structure is: input layer→low-level global dynamic convolutional layer→middle-level global dynamic convolutional layer→high-level global dynamic convolutional layer→fully connected layer. The sample size input by the input layer is c x r x r, and c represents the number of input data channels;

[0100] Step 3.1: Construct a global dynamic convolutional layer:

[0101] The structure of the three global dynamic convolutional layers is consistent, and each global dynamic convolutional layer is composed of a global feature encoding module, a context feature projection module and a convolution kernel weight generation module. First, the global feature encoding module encodes the global information through the average pooling layer and the linear layer, and then the context feature projection module projects the encoded features to the output dimension space through the context feature projection. Finally, the convolution kernel weight generation module generates a convolution kernel containing global context information, and uses the generated new convolution kernel to perform regular convolution;

[0102] Step 3.1.1: Construct a global feature encoding module:

[0103] The global feature encoding module is composed of an average pooling layer, a linear layer, a normalization layer and an activation layer. The average pooling layer extracts the global information of the input data, and reduces the spatial size of the input data to k x k, where k is the size of the convolution kernel; then the features in all c channels are projected onto a vector of size m through the linear layer; and then the output features of the global feature encoding module are obtained through normalization and ReLU activation function:

[0104]

[0105] Step 3.1.2: Constructing the context feature projection module:

[0106] The context feature projection module is composed of a linear layer, a normalization layer and an activation layer. The linear layer projects the features G obtained in step 3.1.1 into a space with output dimension n, and then the output features of the context feature projection module are obtained through normalization and ReLU activation function:

[0107]

[0108] Step 3.1.3: Constructing the convolution kernel weight generation module:

[0109] The weight generation module first converts the features G obtained in step 3.1.1 and the features C obtained in step 3.1.2 into the spatial size of the convolution kernel using two linear layers. The first linear layer converts the feature G into The second linear layer converts the feature C into Then, M G is dimensionally expanded to obtain M C is dimensionally expanded to obtain Then, M G and M C are added to generate the convolution kernel weight M, and the calculation formula is as follows:

[0110] M = δ (M' G + M' C )

[0111] wherein, is the same size as the original convolution kernel W, and δ represents the Sigmoid activation function. Finally, the generated convolution kernel weight M and the current convolution kernel weight W are multiplied element by element to combine the global information and the local information, and the new adaptive convolution kernel weight W' of the input data is obtained, and the calculation formula is as follows:

[0112] W' = M

[0113] wherein, represents element-wise multiplication;

[0114] ​​Step 3.1.4: regular convolution is performed on the current data in the input data using the new convolution kernel generated in step 3.1.3;

[0115] Step 3.2: low-level features F of the input data are extracted using the low-level global dynamic convolution layer constructed in step 3.1 l :

[0116] The size of the convolution kernel in the low-level global dynamic convolution layer is k l ×k l , and the number of convolution kernels is n l , wherein the size of k l is 3, the number of n l is 48, and the calculation formula of the low-level features F l is as follows:

[0117] F l =σ(BN(XW l ′+b l ))

[0118] Wherein, X represents the input sample in the input layer, which is also the input of the low-level global dynamic convolution layer; W l ′ represents the sample adaptive convolution kernel weight of the low-level global dynamic convolution layer obtained by step 3.1.3; b l represents the bias term of the low-level global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; BN represents batch normalization operation; σ represents ReLU activation function;

[0119] Step 3.3: middle-level features F of the input data are extracted using the middle-level global dynamic convolution layer constructed in step 3.1 m :

[0120] The size of the convolution kernel in the middle-level global dynamic convolution layer is k m ×k m , and the number of convolution kernels is n m , wherein the size of k m is 3, the number of n m is 96, and the calculation formula of the middle-level features F m is as follows:

[0121] F m =σ(BN(F l W m ′+b m ))

[0122] Wherein, F l represents the low-level features of the input data, which is also the input of the middle-level global dynamic convolution layer; W′ mrepresents the low-level feature adaptive convolution kernel weight of the middle global dynamic convolution layer obtained by step 3.1.3; b m represents the bias term of the middle global dynamic convolution layer, which is obtained by random initialization and optimized through network iterative training; BN represents the batch normalization operation; σ represents the ReLU activation function;

[0123] Step 3.4: Use the high-level global dynamic convolutional layer constructed in step 3.1 to extract the high-level features F of the input data h :

[0124] The convolution kernel size in the high-level global dynamic convolution layer is k h ×k h , the number of convolution kernels is n h , where k h The size of n is 3, h The number of high-level features F is 48. h The calculation formula is as follows:

[0125] F h =σ(BN(F m W′ h +b h ))

[0126] Among them, F m Represents the middle-level features of the input data and is also the input of the high-level global dynamic convolution layer; W′ h represents the adaptive convolution kernel weight of the middle-level feature of the high-level global dynamic convolution layer obtained by step 3.1.3; b h represents the bias term of the high-level global dynamic convolution layer, which is obtained by random initialization and optimized through network iterative training; BN represents the batch normalization operation; σ represents the ReLU activation function;

[0127] Step 3.5: The high-level features F obtained in step 3.4 h Through the fully connected layer, we get Y:

[0128] Y=W fc2 (W fc1 F h )

[0129] Among them, W fc1 Represents the first layer of fully connected operations, W fc2 Represents the second layer of full connection operation. After two full connection operations, the Y obtained is a vector of dimension 2×1 Where a represents the probability that the input sample belongs to the unchanged class, and b represents the probability that the input sample belongs to the changed class. To output the predicted label for each sample When a>b, equal to the class to which a belongs when a < b equal to the class to which b belongs

[0130] Step 4: use the training data set obtained in step 2 to train the global dynamic convolutional neural network constructed in step 3:

[0131] During training, a two-stage mixed sample data augmentation method is used for training to increase the number of training samples, enrich the diversity of samples, and alleviate the problem of overfitting.

[0132] Step 4.1: Let the number of training rounds be t. From the first round to the t / 2th round, use the mixed sample data augmentation method to construct virtual samples for all samples in each batch during batch training:

[0133] The calculation formula of the mixed sample data augmentation is:

[0134]

[0135]

[0136] where x i and x j represent two input sample data, y i and y j correspond to their sample labels, respectively, and (x i , y i ) and (x j , y j ) are two samples randomly selected from the training data set constructed in step 2.2, so and represent the data and labels of the new virtual sample after mixing the two samples. λ is a mixing coefficient randomly sampled from the Beta distribution, with a value range of [0, 1]:

[0137] λ ~ Beta (α, β)

[0138] where α and β are parameters in the Beta distribution, and in this method, α = β = 0.5.

[0139] Step 4.2: From the (t / 2+1)th round to the tth round, gradually transition the mixed sample data augmentation method to the basic data augmentation method,

[0140] use a linearly decreasing probability ε to control whether to use mixed sample data augmentation, and the calculation formula of ε is as follows:

[0141] ε = (t-i) / 2t

[0142] Wherein, i represents the current round. And set a random number theta as a threshold, the random range of theta is [0, 1], when theta < epsilon, the mixed sample data enhancement in step 4.1 is used; otherwise, the basic data enhancement is used to expand the sample, such as cutting, flipping, rotating sample, etc.

[0143] Step 4.3: the loss function of the global dynamic convolutional neural network is calculated during the training process, and the calculation formula is as follows:

[0144]

[0145] Wherein,

[0146]

[0147]

[0148] If mixed sample data enhancement is performed: y i And y j The sample labels corresponding to the two random input samples x i And x j In step 4.1, The sample x i And x j After mixing The prediction label obtained after the network constructed in step 3; if mixed sample data enhancement is not performed, y i =y j =y, y is the true label of the current input sample; The prediction label obtained after the current input sample passes through the network constructed in step 3; L CE Indicates the cross-entropy loss function, lambda indicates the mixing coefficient described in step 4.1, and log indicates the logarithm operation with base 10;

[0149] Step 4.4: using the stochastic gradient descent (SGD) algorithm to optimize the global dynamic convolutional neural network parameters;

[0150] Step 5: input the test data set in step 2.3 into the optimized global dynamic convolutional neural network, and obtain the prediction label of the test data set according to the processes described in steps 3.2 to 3.5;

[0151] Step 6: according to the prediction label obtained in step 5, the change result map of the place described in step 1 is obtained.

[0152] The effect of the application will be further described in combination with the simulation experiment:

[0153] The simulation experiment of the present application is carried out under the hardware environment of Intel Xeon E5-2609, GeForce RTX 2080, memory 32 GB and the software environment of Ubuntu 16.04.6, PyTorch and Matlab2016a. The simulation experiment data of the present application are shown in Table 1, wherein, Figure 5 Figure 5 (a) is a real SAR image photographed at time 1; Figure 5 (b) is a real SAR image photographed at time 2; Figure 5 (c) is a simulation change detection result reference map of the real SAR image, which is carefully labeled by an expert considering prior knowledge. The experimental objects of the present application are three groups of multi-temporal SAR image data sets, which are Sulzberger data set, Chaohu I data set and Chaohu II data set. The Sulzberger data set is photographed by ENVISAT satellite on March 11, 2011 and March 16, 2011 respectively, and the size is 256*256 pixels, as shown in Table 1, Figure 5 The first row. The Chaohu I data set and the Chaohu II data set are photographed by Sentinel-1 satellite in May 2020 and July 2020 respectively, and the size is 384*384 pixels, as shown in Table 1, Figure 5 The second row and the third row.

[0154] The comparison results of the method of the present application and the existing advanced change detection method are shown in Table 2. Figure 6 ​The PCAKM method in the comparative experiment is proposed in the article "Unsupervised change detection in satellite images using principal component analysis and k-means clustering"; the GaborPCANet method is proposed in the article "Automatic change detection in synthetic aperture radar images based on PCANet"; the NR-ELM method is proposed in the article "Change detection from synthetic aperture radar images based on neighborhood-based ratio and extreme learning machine"; the CWNN method is proposed in the article "Sea ice change detection in sar images based on convolutional-wavelet neural networks"; and the DDNet method is proposed in the article "Change detection in synthetic aperture radar images using a dual-domain network".

[0155] As shown in the Sulzberger data set, the method of the application can retain the details of the change and suppress the speckle noise on the data set; for the Chaohu I and Chaohu II data sets, the influence of the speckle noise is much stronger, and the comparative methods show different sensitivities to the noise in different regions, so their performances are no longer consistent like the Sulzberger data set, however, the method of the application can still achieve good performance, which shows that the method of the application has better robustness to the noise. Figure 6 The application uses the classification accuracy (PCC) and the Kappa coefficient (KC) to compare with the above-mentioned methods, and the calculation is as follows:

[0156]

[0157]

[0158]

[0159] wherein,

[0160] OE = FP + FN

[0161] ​PRE = [(TP+FP-FN) x TP + (TN+FN-FP) x TN] / (N x N)

[0162] N is the total number of pixels, OE is the total number of errors, FP is the number of false detections, indicating the number of pixels in the reference image that belong to the unchanged class but are detected as the changed class in the final change image; FN is the number of missed detections, indicating the number of pixels in the reference image that belong to the changed class but are detected as the unchanged class in the final change image. PRE represents the number and proportion relationship of false detections and missed detections, wherein TP is the number of truly changed pixels, and TN is the number of truly unchanged pixels. Larger PCC and KC values indicate that the change detection result is more accurate and the noise suppression capability is stronger.

[0163] Tables 1, 2 and 3 show the comparison experimental results of the present application and the above method. As can be seen from the tables, the PCC and KC values of the method of the present application are the highest, and good performance is achieved on the three data sets, which shows that the method of the present application can accurately detect the change information in the input image, and has good robustness to noise.

[0164] Table 1 Change detection experimental results of Sulzberger data set

[0165] Method FP FN OE PCC (%) KC (%) PKAKM 3308 701 4009 93.88 84.49 GaborPCANet 2485 494 2979 95.45 88.34 NR-ELM 2386 646 3032 95.37 88.07 CWNN 1598 1132 2730 95.83 88.98 DDNet 1932 494 2426 96.30 90.40 The method of the present invention 1398 731 2129 96.75 91.44

[0166] Table 2 Change detection experimental results of Chaohu I data set

[0167] Method FP FN OE PCC (%) KC (%) PKAKM 13199 2996 16195 89.02 43.99 GaborPCANet 16777 1500 18277 87.61 45.01 NR-ELM 2252 3552 5804 96.06 69.63 CWNN 2908 2808 5716 96.12 71.85 DDNet 4983 1217 6200 95.80 73.54 The method of the present invention 3029 1034 4063 97.24 81.46

[0168] Table 3 Change detection experimental results of Chaohu II data set

[0169] Method FP FN OE PCC (%) KC (%) PKAKM 8521 2248 10769 92.70 65.58 GaborPCANet 2946 1771 4717 96.80 82.66 NR-ELM 595 3836 4431 97.00 81.27 CWNN 959 2397 3356 97.72 86.63 DDNet 3107 779 3886 97.36 86.18 The method of the present invention 792 1218 2010 98.64 92.24

[0170] The method based on the global dynamic convolutional neural network provided by the present application is mainly proposed to improve the performance of multi-temporal SAR image change detection. However, the method is also applicable to the analysis of images taken by ordinary imaging devices such as digital cameras and mobile phones, and the beneficial effects obtained are similar.

[0171] The above is only a preferred embodiment of the present application, and is not intended to limit the present application in other forms. Any person skilled in the art can modify or reform the above disclosed technical content to equivalent embodiments applied to other fields, but any simple modification, equivalent change and reform made on the basis of the technical essence of the present application to the above embodiments still belongs to the protection scope of the technical solution of the present application.

Claims

1. A SAR image change detection method based on a global dynamic convolutional neural network, characterized in that, Comprising the following steps: Step 1, difference analysis of two multi-temporal SAR images captured for the same geographical area, obtaining difference image I DI ; Step 2, Difference map I DI Pre-classification, training data set and test data set construction; Step 3, constructing a global dynamic convolutional neural network; The constructed global dynamic convolutional neural network core architecture comprises three globally consistent global dynamic convolutional layers, each of which is composed of a global feature encoding module, a context feature projection module and a convolution kernel weight generation module, the global feature encoding module encodes the global information through an average pooling layer and a linear layer, then the context feature projection module projects the encoded features to the output dimension space through context feature projection, finally, the convolution kernel weight generation module generates a convolution kernel containing global context information, and a new convolution kernel is generated to perform regular convolution; the global dynamic convolutional neural network structure is: input layer→low-level global dynamic convolutional layer→middle-level global dynamic convolutional layer→high-level global dynamic convolutional layer→fully connected layer, wherein the sample size input by the input layer is c×r×r, c represents the number of input data channels; The construction process of each module of the global dynamic convolutional layer is as follows: 1) Constructing a global feature encoding module: the global feature encoding module comprises an average pooling layer, a linear layer, a normalization layer and an activation layer, the average pooling layer extracts the global information of the input data, and reduces the spatial size of the input data to k×k, wherein k is the size of the convolution kernel; then the features in all c channels are projected to a vector of size m through the linear layer; and the output features of the global feature encoding module are obtained through normalization and ReLU activation function: 2) Constructing a context feature projection module: The context feature projection module comprises a linear layer, a normalization layer and an activation layer, the linear layer projects the features G obtained in 1) to a space with an output dimension of n, and then the output features of the context feature projection module are obtained through normalization and ReLU activation function: 3) Constructing a convolution kernel weight generation module: (1) The weight generation module first converts the features G and C to the spatial size of the convolution kernel by using two linear layers. The first linear layer converts the feature to The second linear layer converts the feature to Then, M G is dimensionally expanded to obtain M C is dimensionally expanded to obtain (2) Then, M G and M C are added to generate the convolution kernel weight M, and the calculation formula is as follows: M=δ(M′ G +M′ C ) wherein, δ denotes a Sigmoid activation function, of the same size as the original convolution kernel W; (3) Finally, the generated convolution kernel weight M and the current convolution kernel weight W are multiplied element by element, the global information and the local information are combined, and a new convolution kernel weight W' of the input data is obtained, and the calculation formula is as follows: W'=M⊙W Where, ⊙ represents element-wise multiplication; then the generated new convolution kernel is used to perform regular convolution on the current data in the input data; Step 4: using the training data set obtained in step 2 to train the global dynamic convolutional neural network constructed in step 3, and using a two-stage mixed sample data enhancement method to train during training; Step 5: using the trained network model to test the test data set, obtaining the predicted label of the test data set, and thus obtaining the change detection result of the whole image.

2. The SAR image change detection method based on global dynamic convolutional neural network according to claim 1, characterized in that: The step 3 of constructing a global dynamic convolutional neural network is specifically implemented by the following method: Step 31, using the constructed low-level global dynamic convolution layer to extract the low-level features F of the input data l l l l wherein X represents the input sample in the input layer, which is also the input of the low-level global dynamic convolution layer; W l ' represents the sample adaptive convolution kernel weight of the low-level global dynamic convolution layer; b l represents the bias term of the low-level global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; BN represents the batch normalization operation; σ represents the ReLU activation function; Step 32: using the constructed middle-level global dynamic convolution layer to extract the middle-level features F m m l m m wherein F l represents the low-level features of the input data, which is also the input of the middle-level global dynamic convolution layer; W m ' represents the low-level feature adaptive convolution kernel weight of the middle-level global dynamic convolution layer; b m represents the bias term of the middle-level global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; Step 33: using the constructed high-level global dynamic convolution layer to extract the high-level features F h h m h h wherein F m represents the middle-level features of the input data, which is also the input of the high-level global dynamic convolution layer; W h ' represents the middle-level feature adaptive convolution kernel weight of the high-level global dynamic convolution layer; b h represents the bias term of the high-level global dynamic convolution layer, which is obtained by random initialization and optimized through network iteration training; Step 34: passing the high-level features F h obtained in step 33 through a fully connected layer to obtain Y: Y = W fc2 (W fc1 F h ) wherein W fc1 represents the first layer fully connected operation, W fc2 represents the second layer fully connected operation, and the Y obtained after two fully connected operations is a vector with a dimension of 2x1 wherein a represents the probability that the input sample belongs to the invariant class, and b represents the probability that the input sample belongs to the change class, and the prediction label of each sample is output according to the vector when a > b,​​​​​​​​​​​​ equal to the class to which a belongs, i.e. when a < b, equal to the class to which b belongs, i.e.

3. The SAR image change detection method based on global dynamic convolutional neural network according to claim 1, characterized in that: The step 4 of training using a two-stage mixed sample data enhancement method comprises the following steps: Step 41: record the training round as t, from the first round to the t / 2 round, use the mixed sample data enhancement method to construct virtual samples for all samples in batches during batch training, and expand the training samples; The calculation formula of the mixed sample data enhancement is: where x i and x j represent two input sample data, y i and y j correspond to their sample labels respectively, and (x i , y i ) and (x j , y j ) are two samples randomly drawn from the training dataset constructed in step 2, so and represent the data and label of the new virtual sample after mixing the two samples, and λ is a mixing coefficient randomly sampled from the Beta distribution with the value range of [0, 1]: λ~Beta(α,β) Wherein, α and β are parameters in the beta distribution; Step 42: From (t / 2+1) round to the t round, the mixed sample data augmentation mode is gradually transitioned to the basic data augmentation mode, and whether to use the mixed sample data augmentation is controlled by the linearly decreasing probability epsilon, and the calculation formula of epsilon is as follows: Epsilon = (t-i) / 2t Wherein, i represents the current round number, and a random number theta is set as a threshold value, and the random range of theta is [0, 1], when theta < epsilon, the mixed sample data augmentation in step 41 is used; otherwise, the basic data augmentation is used to expand the sample; Step 43: Calculate the loss function of the global dynamic convolutional neural network in the training process, and the calculation formula is as follows: Wherein, If mixed sample data augmentation is performed: y i and y j are the sample labels corresponding to the two random input samples x i and x j described in step 41, are the sample labels corresponding to the two random input samples x i and x j after mixing the predicted labels obtained after the network constructed by step 3; If no mixed sample data augmentation is performed, y i = y j = y, y is the true label of the current input sample; is the predicted label of the current input sample obtained after the network constructed in step 3; L CE represents the cross-entropy loss function, and log represents the logarithm operation with base 10; Step 44: The stochastic gradient descent (SGD) algorithm is used to optimize the global dynamic convolutional neural network parameters.

4. The SAR image change detection method based on global dynamic convolutional neural network according to claim 1, characterized in that: In the step 1, the difference between the two multi-temporal SAR images is analyzed by using the logarithmic ratio operator to obtain the difference map of the two multi-temporal SAR images, and the calculation formula of the difference map is as follows: I DI = |log I1 - log I2| where I1and I2denote two multi-temporal SAR images of the same geographical area, respectively, and I DI denotes the difference image of the two multi-temporal SAR images I1and I2, | · | denotes the absolute value operation, and log denotes the logarithm operation with base 10.

5. The SAR image change detection method based on global dynamic convolutional neural network according to claim 1, characterized in that: The step 2 specifically includes the following steps: Step 21, difference map I DI The pre-classification is performed using the hierarchical FCM clustering algorithm to obtain a pseudo-label matrix, and the pseudo-label values are 0, 1 and 0.5, that is, high-probability samples determined as unchanged or changed and uncertain samples are generated after pre-classification. Step 22, randomly select p% of the pixels from the pixels with pseudo-label values of 0 and 1, the value of p is an integer not greater than 10, extract the spatial positions of these pixels, and take the r x r neighborhood pixel block around the pixel points as the training set on the corresponding spatial positions of the original two multi-temporal SAR images, and the edge pixel points are extracted by filling 0 in the domain pixel block, and the value of r is an odd number not less than 3; Step 23, extract all the r x r neighborhood pixel blocks around the pixels in the original two multi-temporal SAR images as the test set, and the edge pixel points are extracted by filling 0 in the domain pixel block, and the value of r is an odd number not less than 3.

Images