Remote sensing image change detection method based on neural network structure search

Abstract

The present invention proposes a remote sensing image change detection method based on neural network structure search, and the implementation steps are: obtaining training, verification, and test sample sets; constructing a super neural network model; iteratively training the super neural network model; using a genetic algorithm to search the trained super neural network model to obtain structure search parameters; constructing a remote sensing image change detection model based on the structure search parameters; iteratively training the remote sensing image change detection model; and obtaining remote sensing image change detection results. The present invention uses a genetic algorithm to search the trained super network neural model, and determines the structure of the remote sensing image change detection model through the structure search parameters obtained by the search. The structure of the remote sensing image change detection model can be highly matched with the characteristics of the remote sensing image, and the remote sensing image features can be more fully extracted, effectively improving the change detection accuracy of the remote sensing image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a remote sensing image change detection method, and specifically to a remote sensing image change detection method based on neural network structure search, which can be used in the fields of geological disaster monitoring, land cover investigation, urban planning, etc. Background Art

[0002] Remote sensing images are images obtained by collecting and processing the electromagnetic radiation energy of ground objects by remote sensing imaging instruments on remote sensing platforms. Two remote sensing images taken at the same location at two different times can reflect the changes of ground objects. Remote sensing image change detection refers to detecting the changed areas between two remote sensing images taken at the same location at different times. How to improve the accuracy of change detection is the focus and difficulty of remote sensing image change detection.

[0003] Existing remote sensing image change detection methods are mainly divided into traditional change detection methods and deep learning-based change detection methods. Traditional change detection methods include methods based on direct pixel comparison, methods based on feature level, methods based on object-oriented image analysis, etc. These methods have low detection accuracy on remote sensing images. With the widespread application of deep learning in various fields, a variety of change detection methods based on deep learning have emerged, such as change detection methods based on twin neural networks, change detection methods based on autoencoders, etc. Change detection methods based on deep learning have achieved high detection accuracy on remote sensing images and have been widely used.

[0004] The neural networks used in existing remote sensing image change detection methods based on deep learning are often artificially designed. The structure of the artificially designed neural network cannot match the characteristics of the remote sensing image well, and the remote sensing image features are not extracted sufficiently, which reduces the detection accuracy. For example, a patent application with application publication number CN112613352A and name "A remote sensing image change detection method based on a twin network" discloses a remote sensing image change detection method based on a twin convolutional neural network. This method first uses the encoding end of the twin convolutional neural network to extract the features of two images at different times, and then uses the decoding end of the twin convolutional neural network to decode the features extracted by the encoding end to obtain a feature map, and then uses the LogSoftmax classifier to predict the category probability of each pixel in the feature map to obtain a predicted probability map, and then predicts the predicted probability. Figure 2 The method is disadvantageous in that the structure of the twin convolutional neural network is artificially designed and cannot match the characteristics of remote sensing images well, resulting in weak feature extraction capability of remote sensing images, which in turn affects the further improvement of detection accuracy.

[0005] Neural Architecture Search is a technology that uses data characteristics to automatically search for neural network structures. Its principle is to use a certain optimization algorithm to use data characteristics to search for structure search parameters on a given hyper neural network model, and then determine the structure of the neural network model based on the structure search parameters. This structure can be highly matched with the data characteristics. The neural network model using this structure can fully extract the features of the data and has a higher detection accuracy. Therefore, if the neural network structure search is applied to the remote sensing image change detection method, the detection accuracy of the remote sensing image change detection method can be improved. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to propose a remote sensing image change detection method based on neural network structure search to solve the technical problem of low detection accuracy in the prior art.

[0007] To achieve the above object, the technical solution adopted by the present invention includes the following steps:

[0008] (1) Obtain training sample set, verification sample set and test sample set:

[0009] (1a) Obtain T remote sensing images of size R×R at time A and B from the remote sensing image change detection dataset HA={HA1,···,HA t ,···,HA T}、HB={HB1,···,HB t ,···,HB T}, and the label image HL of the region where HB changes relative to HA = {HL1, ···, HL t ,···,HL T}, where R ≥ 256, T ≥ 200, HA t HB t represents the t-th remote sensing image at time A and time B, HL t Indicates HB t Compared with HA t Label image of the area where changes occurred;

[0010] (1b) Each remote sensing image HA at time A and time B t HB t , and the label image HL t They are cropped into Z image blocks of size p×p respectively, and the remote sensing image block set HA′={HA1′, ···, HA t ′,···,HA′ T}、HB corresponding to the remote sensing image block set HB′={HB1′,···,HBt ′,···,HB′ T}, and the set of label image blocks corresponding to HL ′={HL1′, ···, HL t ′,···,HL′ T},in, HA t ′ indicates HA t The corresponding remote sensing image block set, HB t ' indicates HB t The corresponding remote sensing image block set, HL t ' indicates HL t The corresponding set of label image patches, Respectively represent HA t ′, HB t ′, HL t The zth image block in ′;

[0011] (1c) randomly extracting more than half of the image blocks from the image block sets HA′, HB′, and HL′ to form a training sample set, and selecting half of the image blocks from the remaining image blocks to form a verification sample set, and the other half of the image blocks to form a test sample set;

[0012] (2) Constructing a hyper-neural network model M for remote sensing image change detection:

[0013] A remote sensing image change detection super neural network model M is constructed, which includes a feature extraction super network, a feature fusion network, and a detection network connected in sequence, wherein the feature extraction super network includes a convolution layer and U blocks connected in sequence thereto, each block includes v units arranged in parallel and composed of x convolution layers connected in sequence; the feature fusion network includes a plurality of fusion blocks connected in sequence, each fusion block includes an upsampling layer, a concatenate layer, and a convolution layer connected in sequence; the detection network includes a cascaded encoding layer and an upsampling layer;

[0014] (3) Iteratively train the remote sensing image change detection hyper-neural network model M:

[0015] (3a) The number of initialization iterations is n, the maximum number of iterations is N, N ≥ 10000, and the current remote sensing image change detection hyper neural network model is M n , and let n = 1, M n =M;

[0016] (3b) The training sample set is used as the remote sensing image change detection super neural network model M n Forward propagation of the input:

[0017] (3b1) The feature extraction super network performs feature extraction on any b remote sensing image blocks at time A and b remote sensing image blocks at time B in the training sample set, and obtains feature maps of b remote sensing image blocks at time A and feature maps of b remote sensing image blocks at time B, wherein b≥16, each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B include an equal number of d sub-feature maps, and U>d≥2;

[0018] (3b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain b fused feature maps at time A, and at the same time fuses the feature maps of each remote sensing image block at time B to obtain b fused feature maps at time B;

[0019] (3c) The detection network encodes the fused feature map at each time point A and its corresponding fused feature map at each time point B, and upsamples the encoded detection feature map to obtain b detection maps;

[0020] (3d) Using the contrast loss function, M is calculated using b detection images and their corresponding b label image patches. n The detection loss value l n , and then use the gradient descent method, and detect the loss value l n To M n The weight parameter w n Make updates;

[0021] (3e) Determine whether n=N is true. If so, obtain the trained remote sensing image change detection super neural network model M′. Otherwise, set n=n+1 and execute step (3b);

[0022] (4) Genetic algorithm is used to search the trained remote sensing image change detection hypernetwork neural model M′ to obtain the structural search parameter y:

[0023] The genetic algorithm is used to search the trained remote sensing image change detection hyper-neural network model M′ to obtain the structural search parameters in, The value is The gene position corresponding to the u-th block, is an integer;

[0024] (5) Construct a remote sensing image change detection model Y based on the structure search parameter y:

[0025] Construct a remote sensing image change detection model Y that includes a sequentially connected feature extraction network, a feature fusion network, and a detection network; the feature extraction network includes a convolutional layer and U ceil blocks sequentially connected to it, each ceil block consists of three sequentially connected convolutional layers, and the convolution kernel size of the three convolutional layers contained in the uth ceil block is is determined by the structure search parameter y:

[0026]

[0027]

[0028]

[0029] The structure of the feature fusion network and the detection network is the same as that of the feature fusion network and the detection network in step (2);

[0030] (6) Iteratively train the remote sensing image change detection model Y:

[0031] (6a) The number of initial iterations is o, the maximum number of iterations is O, O ≥ 10000, and the current remote sensing image change detection model is Y o , and let o = 1, Y o =Y;

[0032] (6b) The training sample set is used as the remote sensing image change detection model Y o Forward propagation of input:

[0033] (6b1) The feature extraction network extracts features from any r remote sensing image blocks at time A and r remote sensing image blocks at time B in the training sample set, and obtains feature maps of the r remote sensing image blocks at time A and feature maps of the remote sensing image blocks at time B, wherein r ≥ 16, and each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B include an equal number of d sub-feature maps, and U>d≥2;

[0034] (6b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain r fused feature maps at time A, and simultaneously fuses the feature maps of each remote sensing image block at time B to obtain r fused feature maps at time B;

[0035] (6c) The detection network encodes each fusion feature map at time A and its corresponding fusion feature map at time B, and upsamples the encoded detection feature map to obtain r detection maps;

[0036] (6d) Using the contrast loss function, we calculate Y through r detection images and their corresponding r label image patches. o The detection loss value q o , and then use the gradient descent method and detect the loss value q o Y o The weight parameter θ o Make updates;

[0037] (6e) Determine whether o=O is true. If so, obtain the trained remote sensing image change detection model Y′. Otherwise, set o=o+1 and execute step (6b);

[0038] (7) Obtaining change detection results of remote sensing images:

[0039] The test sample set is used as the input of the remote sensing image change detection model Y′ for forward propagation to obtain the detection map of all test samples, and the pixel points with pixel values ​​less than 2 in the detection map are assigned to 0, and the remaining pixel points are assigned to 255 to obtain the detection results of all test samples.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] The present invention uses a genetic algorithm to search for a trained remote sensing image change detection hypernetwork neural model, and determines the structure of the remote sensing image change detection model through the structural search parameters obtained by the search. The structure of the remote sensing image change detection model can be highly matched with the characteristics of the remote sensing image, avoiding the defect of the prior art that the artificially designed detection model is used to extract insufficient remote sensing image features, and effectively improving the change detection accuracy of the remote sensing image. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart for implementing the present invention.

[0043] Figure 2 Schematic diagram of feature extraction of the hyper-neural network model for remote sensing image change detection in which the hyper-network extracts features from remote sensing image blocks and obtains feature maps of the remote sensing image blocks.

[0044] Figure 3 A schematic diagram showing that the feature fusion network of the super neural network model for remote sensing image change detection fuses the feature maps of remote sensing image blocks and obtains the fused feature maps.

[0045] Figure 4 The detection network of the super neural network model for remote sensing image change detection encodes the fused feature maps at time A and B, and then upsamples the encoded detection feature map to obtain a schematic diagram of the detection map.

[0046] Figure 5 A schematic diagram of a feature extraction network of a remote sensing image change detection model extracting features from a remote sensing image block and obtaining a feature map of the remote sensing image block. DETAILED DESCRIPTION

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0048] Reference Figure 1 , the present invention comprises the following steps:

[0049] Step 1) Obtain training sample set, validation sample set and test sample set:

[0050] (1a) Obtain T remote sensing images of size R×R at time A and B from the remote sensing image change detection dataset HA = {HA1, ···, HA t ,···,HA T}、HB={HB1,···,HB t ,···,HB T}, and the label image HL of the region where HB changes relative to HA = {HL1, ···, HL t ,···,HL T}, where R ≥ 256, T ≥ 200, HA t HB t represents the t-th remote sensing image at time A and time B, HL t Indicates HB t Compared with HA t Label image of the area where changes occurred;

[0051] In this embodiment, the remote sensing images and label images at time A and B are obtained from the LEVIR-CD dataset, R = 1024, T = 637;

[0052] (1b) Each remote sensing image HA at time A and time B t HB t , and the label image HL t They are cropped into Z image blocks of size p×p respectively, and the remote sensing image block set HA′={HA1′, ···, HA t ′,···,HA′ T}、HB corresponding to the remote sensing image block set HB′={HB1′,···,HB t ′,···,HB′ T}, and the set of label image blocks corresponding to HL ′={HL1′, ···, HL t ′,···,HL′ T},in, HA t ′ indicates HA t The corresponding remote sensing image block set, HB t ' indicates HB t The corresponding remote sensing image block set, HL t ' indicates HL t The corresponding set of label image patches, Respectively represent HA t′, HB t ′, HL t The zth image block in ′;

[0053] In this embodiment, p=256, Z=16;

[0054] (1c) randomly extracting more than half of the image blocks from the image block set HA′, HB′, and HL′ to form a training sample set, and half of the image blocks in the remaining image blocks to form a verification sample set, and the other half of the image blocks to form a test sample set. In this embodiment, two-thirds of the image blocks are randomly extracted from HA′, HB′, and HL′ to form a training sample set;

[0055] Step 2) Construct a remote sensing image change detection hyper-neural network model M:

[0056] A remote sensing image change detection super neural network model M is constructed, which includes a feature extraction super network, a feature fusion network, and a detection network connected in sequence, wherein:

[0057] The feature extraction super network includes a convolutional layer and U blocks connected to it in sequence, each block includes v units arranged in parallel and composed of x convolutional layers connected in sequence; the feature fusion network includes multiple fusion blocks connected in sequence, each fusion block includes an upsampling layer, a concatenate layer and a convolutional layer connected in sequence; the detection network includes a cascaded encoding layer and an upsampling layer;

[0058] In this embodiment, U=20, v=4, x=3;

[0059] The convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer included in the feature extraction hypernetwork are 3×3, 32, and 2 respectively;

[0060] The feature extraction hypernetwork contains 20 blocks, each block contains 4 unit blocks, each unit block contains 3 convolutional layers, the convolution kernel sizes of the 3 convolutional layers contained in the first unit block are 1×1, 3×3, and 1×1 respectively, the convolution kernel sizes of the 3 convolutional layers contained in the second unit block are 1×1, 5×5, and 1×1 respectively, the convolution kernel sizes of the 3 convolutional layers contained in the third unit block are 1×1, 7×7, and 1×1 respectively, and the convolution kernel sizes of the 3 convolutional layers contained in the fourth unit block are all 3×3;

[0061] In the feature extraction hypernetwork, except for the second convolutional layer of the three convolutional layers contained in the four unit blocks contained in the first, fifth, ninth, and fifteenth blocks, the convolutional step size of the three convolutional layers contained in the four unit blocks contained in each block is 1;

[0062] The number of convolution kernels in the three convolution layers of the four unit blocks contained in the first to fourth blocks in the feature extraction hypernetwork is 64, the number of convolution kernels in the three convolution layers of the four unit blocks contained in the fifth to eighth blocks is 128, the number of convolution kernels in the three convolution layers of the four unit blocks contained in the ninth to sixteenth blocks is 256, and the number of convolution kernels in the three convolution layers of the four unit blocks contained in the sixteenth to twentieth blocks is 512;

[0063] The feature fusion network includes 4 sequentially connected fusion blocks. The convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer contained in the first fusion block are 3×3, 768, and 1 respectively; the convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer contained in the second fusion block are 3×3, 384, and 1 respectively; the convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer contained in the third fusion block are 3×3, 192, and 1 respectively; the convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer contained in the fourth fusion block are 3×3, 96, and 1 respectively; the upsampling layer contained in each fusion block adopts the bilinear interpolation method;

[0064] The detection network, the upsampling layer it contains uses the bilinear interpolation method, and the encoding calculation formula of the encoding layer it contains is:

[0065]

[0066] Among them, FA and FB represent the two feature maps input to the encoding layer, and their height, width, and number of channels are H, W, and C respectively. FO represents the feature map output by the encoding layer, and its height, width, and number of channels are H, W, and 1 respectively. h,w,c represents the value of the cth channel in the hth row, wth column, and FB h,w,c Indicates the value of the cth channel at the hth row, wth column in FB, FO h,w,1 It represents the value of the first channel at the hth row, wth column in FB;

[0067] The role of the remote sensing image change detection super neural network model is to provide multiple candidate remote sensing image change detection sub-network models for the subsequent search process. When each block of the feature extraction super network of the super neural network model only retains one unit block, a candidate remote sensing image change detection sub-network model can be formed; the super neural network model contains 4 20 candidate sub-network models, the subsequent search process will start from 4 20 Find the sub-network model with high detection accuracy among the candidate sub-network models;

[0068] Step 3) Iteratively train the remote sensing image change detection hyper-neural network model M:

[0069] (3a) The number of initialization iterations is n, the maximum number of iterations is N, N ≥ 10000, and the current remote sensing image change detection hyper neural network model is M n , and let n = 1, M n =M;

[0070] In this embodiment, N=30000. The purpose of designing N=30000 is to make the model training more sufficient.

[0071] (3b) The training sample set is used as the remote sensing image change detection super neural network model M n Forward propagation of the input:

[0072] (3b1) The feature extraction super network performs feature extraction on any b remote sensing image blocks at time A and b remote sensing image blocks at time B in the training sample set, and obtains feature maps of b remote sensing image blocks at time A and feature maps of b remote sensing image blocks at time B, wherein b≥16, and the number of sub-feature maps contained in each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B is d=5;

[0073] In this embodiment, b=32, and the five sub-feature maps included in the feature map of each remote sensing image block at time A and B come from the convolutional layer, the fourth block, the eighth block, the sixteenth block, and the twentieth block of the feature extraction super network respectively.

[0074] (3b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain b fused feature maps at time A, and at the same time fuses the feature maps of each remote sensing image block at time B to obtain b fused feature maps at time B;

[0075] (3c) The detection network encodes the fused feature map at each time point A and its corresponding fused feature map at each time point B, and upsamples the encoded detection feature map to obtain b detection maps;

[0076] (3d) Using the contrast loss function, M is calculated using b detection images and their corresponding b label image patches. n The detection loss value l n , and then use the gradient descent method, and detect the loss value l n To M n The weight parameter w n Make updates;

[0077] The detection loss value l in step (3d) n , and the weight parameter w nThe update formulas are:

[0078]

[0079]

[0080] Where b represents the number of detection images and label image blocks, E represents the number of pixels contained in each detection image and each label image block, Represents the pixel value of the e-th pixel in the λ-th detection image, represents the pixel value of the e-th pixel in the λ-th label image block, b u represents the number of pixels with a pixel value of 0 in b label image blocks, b c represents the number of pixels with a pixel value of 1 in the b label image blocks, Max represents the maximum value function; η represents the learning rate, 1e-6≤η≤0.1, l n represents the detection loss value, w′ represents w n The updated results, Represents partial derivative calculation.

[0081] In this embodiment, the initial learning rate η=0.001, when the iteration reaches the 10,000th time, the learning rate η=0.0001, and when the iteration reaches the 20,000th time, the learning rate η=0.00001. The optimizer function uses stochastic gradient descent SGD. The learning rate is decayed when the network iterates to a certain number of times in order to prevent the loss function from falling into a local minimum.

[0082] (3e) Determine whether n=N. If so, obtain the trained remote sensing image change detection hyper-neural network model M′. Otherwise, set n=n+1 and execute step (3b);

[0083] Iterative training of the hyper-neural network model for remote sensing image change detection is equivalent to the 4 20 The trained remote sensing image change detection super neural network model contains 4 candidate remote sensing image change detection sub-network models. 20 A trained candidate remote sensing image change detection sub-network model;

[0084] Step 4) Genetic algorithm is used to search the trained remote sensing image change detection hypernetwork neural model M′ to obtain the structure search parameter y. The implementation steps are:

[0085] (4a) Initialize the genetic algorithm parameters: the number of iterations is g, the maximum number of iterations is G, the number of elites to be retained is K, and the parent population is P = {p1, ···, p i ,···,p NP}, the subpopulation is S = {}, where NP represents the number of individuals in the parent population, pi The i-th parent population individual represented by Indicates that the value corresponding to the u-th block is The gene position, is an integer, and let g = 1;

[0086] In this embodiment, G=20, NP=50, K=2. G=20 is designed to reduce the search time, and NP=50, K=2 are designed to prevent premature phenomenon.

[0087] (4b) For each parent population individual p, PC is the crossover probability and PM is the mutation probability. i Perform crossover and mutation operations, and add the new individuals generated by crossover and mutation to the sub-population S, and obtain the sub-population S = {s1,···,s α ,···,s NS}, where NS represents the number of individuals in the subpopulation, s α represents the αth subpopulation individual;

[0088] In this embodiment, PC=0.4, PM=0.1, the crossover operation adopts a single-point crossover method, and the mutation operation adopts a basic bit mutation method;

[0089] (4c) Combine the NP parent population individuals in the parent population P and the NS sub-population individuals in the sub-population S into a temporary population T = {t1, ···, t β ,···,t NT}, where NT represents the number of individuals in the temporary population, NT = NP + NS, t β represents the βth temporary population individual, Indicates that the value corresponding to the u-th block is The gene position;

[0090] (4d) Get each temporary population individual t β Corresponding remote sensing image change detection subnetwork model M β :

[0091] Remove the uth block of the feature extraction hypernetwork in the trained remote sensing image change detection hyperneural network model M′ except the uth block. Unit blocks outside the unit block, get t β Corresponding remote sensing image change detection subnetwork model M β ;

[0092] (4e) Use the validation sample set as the sub-network model M βThe input is forward propagated to obtain the detection map of each verification sample, and the pixel points with pixel values ​​less than 2 in each detection map are assigned to 0, and the remaining pixels are assigned to 1, and the detection result map of all verification samples is obtained. M is calculated through all the detection result maps and their corresponding label image blocks. β The F1 score is then used as t β The fitness value f β , where the F1 score is an evaluation indicator of change detection results;

[0093] The larger the F1 score, the better the change detection result, and the higher the detection accuracy of the remote sensing image change detection sub-network model;

[0094] (4f) Select K temporary population individuals with the largest fitness value in the temporary population T, and use The probability of selecting NP-K individuals from the remaining NT-K temporary population in T to form a new parent population P′={p1′,···,p i ′,···,p′ NP};

[0095] (4g) Determine whether g = G. If so, replace the U genes of the individual with the highest fitness value in P′ As the structure search parameter y, where Indicates that the value corresponding to the u-th block is The gene position, is an integer, otherwise, let g=g+1 and execute step (4b).

[0096] The genetic algorithm is used to search the trained remote sensing image change detection super network neural model in order to find a remote sensing image change detection sub-network model with high detection accuracy: the genetic algorithm can adaptively eliminate individuals with low fitness and generate individuals with higher fitness during the iteration process. When the iteration terminates, the individual with the highest fitness in the parent population is the individual with the highest fitness among all individuals generated during the entire iteration process; the fitness of an individual is equal to the F1 score of its corresponding sub-network model. The higher the F1 score, the higher the detection accuracy of the sub-network model. Therefore, the sub-network model corresponding to the individual with the highest fitness in the parent population has high detection accuracy.

[0097] The reason why the U gene positions of the individual with the highest fitness value in the parent population are used as the structural search parameter y at the end of the iteration is that the structure of the remote sensing image change detection sub-network model with high detection accuracy can be highly matched with the characteristics of the remote sensing image, and the remote sensing image features can be extracted more fully; the U gene positions of the individual with the highest fitness value in the parent population are used as the structural search parameters, and the remote sensing image change detection model with the same structure as the sub-network model with high detection accuracy can be constructed according to the structural search parameters later;

[0098] Step 5) Construct a remote sensing image change detection model Y based on the structure search parameter y:

[0099] Construct a remote sensing image change detection model Y that includes a sequentially connected feature extraction network, a feature fusion network, and a detection network; the feature extraction network includes a convolutional layer and U ceil blocks sequentially connected to it, each ceil block consists of three sequentially connected convolutional layers, and the convolution kernel size of the three convolutional layers contained in the uth ceil block is is determined by the structure search parameter y:

[0100]

[0101]

[0102]

[0103] The convolution kernel size, number of convolution kernels, and convolution step size of the convolution layer included in the feature extraction network are 3×3, 32, and 2 respectively;

[0104] In the feature extraction network, except for the second convolutional layer of the three convolutional layers contained in the first, fifth, ninth, and fifteenth ceil blocks, the convolutional step size of each ceil block is 1.

[0105] The number of convolution kernels in the three convolution layers contained in the first to fourth ceil blocks in the feature extraction network is 64, the number of convolution kernels in the three convolution layers contained in the fifth to eighth ceil blocks is 128, the number of convolution kernels in the three convolution layers contained in the ninth to sixteenth ceil blocks is 256, and the number of convolution kernels in the three convolution layers contained in the sixteenth to twentieth ceil blocks is 512;

[0106] The structure of the feature fusion network and the detection network is the same as that of the feature fusion network and the detection network in step (2);

[0107] The structure of the remote sensing image change detection model constructed according to the structural search parameters is the same as the structure of the sub-network model corresponding to the individual with the highest fitness in step (4). Therefore, the remote sensing image change detection model can highly match the characteristics of the remote sensing image and fully extract the features of the remote sensing image, thereby improving the change detection accuracy.

[0108] Step 6) Iteratively train the remote sensing image change detection model Y:

[0109] (6a) The number of initial iterations is o, the maximum number of iterations is O, O ≥ 10000, and the current remote sensing image change detection model is Y o , and let o = 1, Y o =Y;

[0110] In this embodiment, O = 30000. The purpose of designing O = 30000 is to make the model training more sufficient;

[0111] (6b) The training sample set is used as the remote sensing image change detection model Y o Forward propagation of the input:

[0112] (6b1) The feature extraction network performs feature extraction on any r remote sensing image blocks at time A and r remote sensing image blocks at time B in the training sample set, and obtains feature maps of the r remote sensing image blocks at time A and feature maps of the remote sensing image blocks at time B, wherein r ≥ 16, and the number of sub-feature maps contained in each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B is d = 5;

[0113] In this embodiment, r=32, and the five sub-feature maps included in the feature map of each remote sensing image block at time A and B come from the convolution layer, the fourth ceil block, the eighth ceil block, the sixteenth ceil block, and the twentieth ceil block of the feature extraction network respectively.

[0114] (6b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain r fused feature maps at time A, and simultaneously fuses the feature maps of each remote sensing image block at time B to obtain r fused feature maps at time B;

[0115] (6c) The detection network encodes each fusion feature map at time A and its corresponding fusion feature map at time B, and upsamples the encoded detection feature map to obtain r detection maps;

[0116] (6d) Using the contrast loss function, we calculate Y through r detection images and their corresponding r label image patches. o The detection loss value q o , and then use the gradient descent method and detect the loss value q o Y o The weight parameter θ o Make updates;

[0117] The detection loss value q in step (6d) o , and the weight parameter θ o The update formulas are:

[0118]

[0119]

[0120] Where r represents the number of detection images and label image blocks, E represents the number of pixels contained in each detection image and each label image block, Represents the pixel value of the e-th pixel in the λ-th detection image, represents the pixel value of the e-th pixel in the λ-th label image block, r u represents the number of pixels with a pixel value of 0 in the r label image blocks, r c represents the number of pixels with a pixel value of 1 in the r label image blocks, Max represents the maximum value function; γ represents the learning rate, 1e-6≤γ≤0.1, q o represents the detection loss value, θ′ represents the result after θo is updated, Represents partial derivative calculation.

[0121] In this embodiment, the initial learning rate γ=0.001, when the iteration reaches the 10,000th time, the learning rate γ=0.0001, and when the iteration reaches the 20,000th time, the learning rate γ=0.00001. The optimizer function uses stochastic gradient descent SGD. The learning rate is decayed when the network iterates to a certain number of times in order to prevent the loss function from falling into a local minimum.

[0122] (6e) Determine whether o=O is true. If so, obtain the trained remote sensing image change detection model Y′. Otherwise, set o=o+1 and execute step (6b);

[0123] Step 7) Obtain the change detection results of the remote sensing image:

[0124] The test sample set is used as the input of the remote sensing image change detection model Y′ for forward propagation to obtain the detection map of all test samples, and the pixel points with pixel values ​​less than 2 in the detection map are assigned to 0, and the remaining pixel points are assigned to 255 to obtain the detection results of all test samples.

[0125] The following simulation experiments are used to further illustrate the technical effects of the present invention:

[0126] 1. Simulation conditions and contents:

[0127] 637 remote sensing images at time A and B and the corresponding labeled images were obtained from the LEVIR-CD dataset for simulation experiments;

[0128] The simulation experiment was conducted on a server with an Intel Xeon E5-2678 CPU and a GeForce GTX1080 GPU. The operating system was UBUNTU 18.04, the deep learning framework was PyTorch, and the programming language was Python 3.6.

[0129] The present invention and an existing remote sensing image change detection method based on twin neural networks are compared and simulated. In order to quantitatively compare the change detection results, the experiment uses two remote sensing image change detection result evaluation indicators, namely F1 score and mIoU. The higher the two evaluation indicators, the better the change detection results. The simulation results are shown in Table 1.

[0130] 2. Analysis of simulation results:

[0131] Table 1

[0132]

[0133] It can be seen from Table 1 that compared with the existing remote sensing image change detection method based on twin neural networks, the F1 and mIoU of the present invention are significantly improved, indicating that the remote sensing image change detection model constructed by the present invention can highly match the remote sensing graphic characteristics and fully extract the remote sensing image features, thereby improving the change detection accuracy, and thus has important practical significance.

Claims

1. A remote sensing image change detection method based on neural network structure search, characterized in that: The steps include: (1) Obtain training sample set, verification sample set and test sample set: (1a) Obtain T remote sensing images of size R×R at time A and B from the remote sensing image change detection dataset HA={HA1,…,HA t ,…,HA T }、HB={HB1,…,HB t ,…,HB T }, and the label image HL of the region where HB changes relative to HA = {HL1,…,HL t ,…,HL T }, where R ≥ 256, T ≥ 200, HA t HB t represents the t-th remote sensing image at time A and time B, HL t Indicates HB t Compared with HA t Label image of the area where changes occurred; (1b) Each remote sensing image HA at time A and time B t HB t , and the label image HL t They are cropped into Z image blocks of size p×p respectively, and the remote sensing image block set HA′ corresponding to HA is obtained as follows: HA′={HA′1,…,HA′ t ,…,HA′ T }、HB corresponding to the remote sensing image block set HB′={HB′1,…,HB′ t ,…,HB′ T }, and the set of label image blocks corresponding to HL ′={HL′1,…,HL′ t ,…,HL′ T },in, HA′ t Indicates HA t The corresponding remote sensing image block set, HB′ t Indicates HB t The corresponding remote sensing image block set, HL′ t Indicates HL t The corresponding set of label image patches, Respectively represent HA′ t HB′ t HL′ t The zth image block in ; (1c) randomly extracting more than half of the image blocks from the image block sets HA′, HB′, and HL′ to form a training sample set, and selecting half of the image blocks from the remaining image blocks to form a verification sample set, and the other half of the image blocks to form a test sample set; (2) Constructing a hyper-neural network model M for remote sensing image change detection: A remote sensing image change detection super neural network model M is constructed, which includes a feature extraction super network, a feature fusion network, and a detection network connected in sequence, wherein the feature extraction super network includes a convolution layer and U blocks connected in sequence thereto, each block includes v units arranged in parallel and composed of x convolution layers connected in sequence; the feature fusion network includes a plurality of fusion blocks connected in sequence, each fusion block includes an upsampling layer, a concatenate layer, and a convolution layer connected in sequence; the detection network includes a cascaded encoding layer and an upsampling layer; (3) Iteratively train the remote sensing image change detection hyper-neural network model M: (3a) The number of initialization iterations is n, the maximum number of iterations is N, N ≥ 10000, and the current remote sensing image change detection hyper neural network model is M n , and let n = 1, M n =M; (3b) The training sample set is used as the remote sensing image change detection super neural network model M n Forward propagation of the input: (3b1) The feature extraction super network performs feature extraction on any b remote sensing image blocks at time A and b remote sensing image blocks at time B in the training sample set, and obtains feature maps of b remote sensing image blocks at time A and feature maps of b remote sensing image blocks at time B, wherein b≥16, each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B include an equal number of d sub-feature maps, and U>d≥2; (3b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain b fused feature maps at time A, and at the same time fuses the feature maps of each remote sensing image block at time B to obtain b fused feature maps at time B; (3c) The detection network encodes the fused feature map at each time point A and its corresponding fused feature map at each time point B, and upsamples the encoded detection feature map to obtain b detection maps; (3d) Using the contrast loss function, M is calculated using b detection images and their corresponding b label image patches. n The detection loss value l n , and then use the gradient descent method, and detect the loss value l n To M n The weight parameter w n Make updates; (3e) Determine whether n=N. If so, obtain the trained remote sensing image change detection hyper-neural network model M′. Otherwise, set n=n+1 and execute step (3b); (4) Genetic algorithm is used to search the trained remote sensing image change detection hypernetwork neural model M′ to obtain the structural search parameter y: The genetic algorithm is used to search the trained remote sensing image change detection hyper-neural network model M′ to obtain the structural search parameters in, The value is The gene position corresponding to the u-th block, is an integer; (5) Construct a remote sensing image change detection model Y based on the structure search parameter y: Construct a remote sensing image change detection model Y that includes a sequentially connected feature extraction network, a feature fusion network, and a detection network; the feature extraction network includes a convolutional layer and U ceil blocks sequentially connected to it, each ceil block consists of three sequentially connected convolutional layers, and the convolution kernel size of the three convolutional layers contained in the uth ceil block is is determined by the structure search parameter y: The structure of the feature fusion network and the detection network is the same as that of the feature fusion network and the detection network in step (2); (6) Iteratively train the remote sensing image change detection model Y: (6a) The number of initial iterations is o, the maximum number of iterations is O, O ≥ 10000, and the current remote sensing image change detection model is Y o , and let o = 1, Y o =Y; (6b) The training sample set is used as the remote sensing image change detection model Y o Forward propagation of the input: (6b1) The feature extraction network extracts features from any r remote sensing image blocks at time A and r remote sensing image blocks at time B in the training sample set, and obtains feature maps of the r remote sensing image blocks at time A and feature maps of the remote sensing image blocks at time B, wherein r ≥ 16, and each feature map of the remote sensing image block at time A and each feature map of the remote sensing image block at time B include an equal number of d sub-feature maps, and U>d≥2; (6b2) The feature fusion network fuses the feature maps of each remote sensing image block at time A to obtain r fused feature maps at time A, and simultaneously fuses the feature maps of each remote sensing image block at time B to obtain r fused feature maps at time B; (6c) The detection network encodes each fusion feature map at time A and its corresponding fusion feature map at time B, and upsamples the encoded detection feature map to obtain r detection maps; (6d) Using the contrast loss function, we calculate Y through r detection images and their corresponding r label image patches. o The detection loss value q o , and then use the gradient descent method and detect the loss value q o Y o The weight parameter θ o Make updates; (6e) Determine whether o=O is true. If so, obtain the trained remote sensing image change detection hyper-neural network model Y′. Otherwise, set o=o+1 and execute step (6b); (7) Obtaining change detection results of remote sensing images: The test sample set is used as the input of the remote sensing image change detection model Y′ for forward propagation to obtain the detection map of all test samples, and the pixel points with pixel values ​​less than 2 in the detection map are assigned to 0, and the remaining pixel points are assigned to 255 to obtain the detection results of all test samples.

2. The remote sensing image change detection method based on neural network structure search according to claim 1 is characterized in that: The remote sensing image change detection hyper-neural network model M described in step (2), wherein: The feature extraction hypernetwork contains a convolution kernel size of 3×3 in its convolution layer. The number of blocks contained in the feature extraction hypernetwork is 20, the number of unit blocks contained in each block is 4, the number of convolution layers contained in each unit block is 3, the convolution kernel sizes of the three convolution layers contained in the first unit block are 1×1, 3×3, and 1×1 respectively, the convolution kernel sizes of the three convolution layers contained in the second unit block are 1×1, 5×5, and 1×1 respectively, the convolution kernel sizes of the three convolution layers contained in the third unit block are 1×1, 7×7, and 1×1 respectively, and the convolution kernel sizes of the three convolution layers contained in the fourth unit block are all 3×3; The feature fusion network contains 4 fusion blocks, and the convolution kernel size of the convolution layer contained in each fusion block is 3×3.

3. The remote sensing image change detection method based on neural network structure search according to claim 1 is characterized in that: The detection loss value l in step (3d) n , and the weight parameter w n The update formulas are: Where b represents the number of detection images and label image blocks, E represents the number of pixels contained in each detection image and each label image block, Represents the pixel value of the e-th pixel in the λ-th detection image, represents the pixel value of the e-th pixel in the λ-th label image block, b u represents the number of pixels with a pixel value of 0 in b label image blocks, b c represents the number of pixels with a pixel value of 1 in the b label image blocks, Max represents the maximum value function; η represents the learning rate, 1e-6≤η≤0.1, l n represents the detection loss value, w′ represents w n The updated results, Represents partial derivative calculation.

4. The remote sensing image change detection method based on neural network structure search according to claim 1 is characterized in that: The genetic algorithm described in step (4) is used to search the trained remote sensing image change detection hyper-neural network model M′, and the implementation steps are: (4a) Initialize the genetic algorithm parameters: the number of iterations is g, the maximum number of iterations is G, the number of elites to be retained is K, and the parent population is P = {p1, ..., p i ,…,p NP }, the subpopulation is S = {}, where NP represents the number of individuals in the parent population, p i The i-th parent population individual represented by Indicates that the value corresponding to the u-th block is The gene position, is an integer, and let g = 1; (4b) For each parent population individual p, PC is the crossover probability and PM is the mutation probability. i Perform crossover and mutation operations, and add the new individuals generated by crossover and mutation to the sub-population S, and obtain the sub-population S = {s1,…,s α ,…,s NS }, where NS represents the number of individuals in the subpopulation, s α represents the αth subpopulation individual; (4c) Combine the NP parent population individuals in the parent population P and the NS sub-population individuals in the sub-population S into a temporary population T = {t1,…,t β ,…,t NT }, where NT represents the number of individuals in the temporary population, NT = NP + NS, t β represents the βth temporary population individual, Indicates that the value corresponding to the u-th block is The gene position, is an integer; (4d) Get each temporary population individual t β Corresponding remote sensing image change detection subnetwork model M β : Remove the uth block of the feature extraction hypernetwork in the trained remote sensing image change detection hyperneural network model M′ except the uth block. Unit blocks outside the unit block, get t β Corresponding remote sensing image change detection subnetwork model M β ; (4e) Use the validation sample set as the sub-network model M β The input is forward propagated to obtain the detection map of each verification sample, and the pixel points with pixel values ​​less than 2 in each detection map are assigned to 0, and the remaining pixels are assigned to 1, and the detection result map of all verification samples is obtained. M is calculated through all the detection result maps and their corresponding label image blocks. β The F1 score is then used as t β The fitness value f β , where the F1 score is an evaluation indicator of change detection results; (4f) Select K temporary population individuals with the largest fitness value in the temporary population T, and use The probability is to select NP-K individuals from the remaining NT-K temporary population in T to form a new parent population P′={p′1,…,p′ i ,…,p′ NP }; (4g) Determine whether g = G. If so, replace the U gene positions of the individual with the highest fitness value in P′ As the structure search parameter y, where Indicates that the value corresponding to the u-th block is The gene position, is an integer, otherwise, let g=g+1 and execute step (4b).

5. The remote sensing image change detection method based on neural network structure search according to claim 1 is characterized in that: The detection loss value q in step (6d) o , and the weight parameter θ o The update formulas are: Where r represents the number of detection images and label image blocks, E represents the number of pixels contained in each detection image and each label image block, Represents the pixel value of the e-th pixel in the λ-th detection image, represents the pixel value of the e-th pixel in the λ-th label image block, r u represents the number of pixels with a pixel value of 0 in the r label image blocks, r c represents the number of pixels with a pixel value of 1 in the r label image blocks, Max represents the maximum value function; γ represents the learning rate, 1e-6≤γ≤0.1, q o represents the detection loss value, θ′ represents θ o The updated results, Represents partial derivative calculation.

Images