A method and device for detecting heterogeneous dual-time image changes

Abstract

The present invention discloses a heterogeneous dual-time image change detection method and device, which relates to the field of image change detection technology, including: using a dual-stream encoder to obtain the first latent space feature map and the prior change map of two heterogeneous dual-time images to be detected that are mapped to the same latent feature space, determining the change difference map according to the L2 norm of the pixel-by-pixel position difference of the two first latent space feature maps, determining the first pseudo-label map according to the change difference map, using a feedforward neural network to process the first latent space feature map after channel weighting and channel dimension splicing, obtaining the initial change probability map, determining the second pseudo-label map according to the initial change probability map, replacing the prior change map with the second pseudo-label map, and iteratively updating the second pseudo-label map. The method uses the encoding of the dual-time image in the same feature space as input, uses reliable pseudo-labels as supervision to guide the model to autonomously infer incorrect areas in the pseudo-labels, and can effectively suppress noise spots therein.

Description

Technical Field

[0001] The present invention relates to the technical field of image change detection, and more specifically to a heterogeneous dual-time image change detection method and device. Background Art

[0002] Change detection is to compare images collected from an area at two different times to determine the type, magnitude and location of changes, including changes in land cover types caused by human activities, natural disasters or climate change. Remote sensing image change detection has been a research hotspot in the field of remote sensing for many years. It is a key link in tasks such as land use, geological disaster monitoring, and ecological and environmental protection.

[0003] However, in practical applications, it is difficult to obtain ideal homogeneous dual-phase images due to the limitations of various internal (sensor performance, satellite orbit, etc.) and external (weather, climate, etc.) factors. Change detection between heterogeneous images highlights its importance.

[0004] SAR (Synthetic Aperture Radar) actively emits microwaves and receives reflected waves from objects to form images, which is an active imaging method; while optical sensors form images by capturing light reflected from objects, which is a passive imaging method; the two different imaging methods determine that the two are complementary in expressing the characteristics of objects. For example, optical images have richer semantic information and object details, but are easily affected by weather and light, while SAR images are formed all day and all weather, allowing SAR images to be formed in bad weather or at night, but because they are accompanied by a large amount of noise, the readability of the image content is not strong.

[0005] Unsupervised change detection between optical images and SAR images has great research significance. In the case of cloud cover, optical sensors cannot obtain complete and clear ground images, while microwaves emitted by SAR can penetrate the clouds and fog, thus capturing ground images without being affected. At this time, a pair of heterogeneous images can be formed by SAR images and optical images of another phase for change detection; in addition, remote sensing images are large in size, SAR images are difficult to interpret, and the cost of manual change analysis and annotation is very high, so unsupervised change detection methods will be more valuable for research.

[0006] Since heterogeneous images have very different representations of the same object, direct feature comparison is impossible. Therefore, current unsupervised heterogeneous change detection methods are mostly based on image conversion (i.e., conversion between dual-temporal image domains) and latent space mapping (i.e., converting dual-temporal image domains to latent space). The converted homogeneous images or features are then simply differentiated, and finally a binary change map is obtained using automatic threshold segmentation technology or unsupervised binary classification technology such as clustering.

[0007] Since the differences in the representation of objects between optical images and SAR images are huge and complex, and the mapping relationships between different objects in optical and SAR images are completely inconsistent, it is difficult for unsupervised methods to correctly distinguish the changed areas from the complex heterogeneous differences and change differences. Neither image conversion nor latent space mapping can obtain a good difference map. Moreover, the final change map is completely dependent on the distribution properties of the difference map, and the obtained change map will have a large number of noise spots. Even if some methods use simple denoising methods to improve the accuracy of the final change map, this method is not smart enough and can only be used in specific scenarios. Summary of the invention

[0008] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the first aspect of the present invention proposes a heterogeneous dual-time image change detection method, the method comprising:

[0009] A dual-stream encoder is used to obtain a first latent space feature map and a priori change map of two heterogeneous dual-time images to be detected that are mapped to the same latent feature space; the dual-stream encoder is trained based on a first loss function composed of a first objective function determined by the two first latent space feature maps and the priori change map and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder, wherein the heterogeneous dual-time images include optical images and SAR images acquired at different time periods;

[0010] Determine a change difference map according to the L2 norm of the pixel-by-pixel position difference of the two first latent space feature maps, and determine a first pseudo label map according to the change difference map;

[0011] A feedforward neural network is used to process the first latent space feature map after channel weighting and channel dimension splicing to obtain an initial probability map, and the feedforward neural network is supervised and trained based on a cross entropy loss determined by the first pseudo label map and the initial change probability map and a second loss function determined by the sum of the L2 norm regularization of the parameters of the feedforward neural network;

[0012] Determine a second pseudo label map according to the initial probability map;

[0013] The prior change map is replaced by the second pseudo label map, the second pseudo label map is iteratively updated, and a probability map of the change detection results of the two heterogeneous dual-time images is determined.

[0014] Optionally, the dual-stream encoder includes two autoencoders, each of which includes:

[0015] The first n-1 encoding layers of the sub-encoder all contain two-dimensional convolution with multiple output channels, batch normalization, and LeakyReLU activation function, and the nth encoding layer contains two-dimensional convolution with multiple output channels, batch normalization, and Sigmoid activation function. The size of the convolution kernel of each layer is 3x3, and the padding method is used to make the spatial dimensions of the input and output the same;

[0016] The structure of the sub-decoder is the same as the structure of the first n-1 coding layers of the sub-encoder.

[0017] Optionally, using a dual-stream encoder to obtain a first latent space feature map of two heterogeneous dual-time images to be detected that is mapped to the same latent feature space includes:

[0018] Get an initialized prior change image;

[0019] Input two heterogeneous dual-time images into a dual-stream encoder and output two initial latent space feature images;

[0020] Determine the first objective function according to the quotient of the number of pixels of the two prior change images and the cumulative sum of the L1 norms of the pixels of the initial latent space feature images and the number of pixels of the two prior change images;

[0021] Determine a second objective function according to the sum of the difference between the pixel points of the two heterogeneous dual-time images and the pixel points of the two reconstructed images reconstructed and outputted by the dual-stream encoder;

[0022] Determine a first loss function according to the first objective function and the second objective function;

[0023] Train the two-stream encoder according to the first loss function;

[0024] Two heterogeneous dual-time images are input into a dual-stream encoder to obtain two first latent space feature maps, which are mapped to the same latent feature space.

[0025] Optionally, obtain an initialized prior change image, including:

[0026] Add the L1 norms of each pixel point of the two initial latent space feature images to obtain a cumulative sum, and use the quotient of the cumulative sum and the number of pixel points of the initial latent space feature image to determine the third objective function;

[0027] Using the third objective function, train the sub-encoder of the two-stream autoencoder;

[0028] Input the initial latent space feature image into the trained sub-encoder to obtain two second latent space feature maps;

[0029] The L2 norm value of the pixel-by-pixel position difference of the two second latent space feature maps is obtained, the differential image composed of the pixel points corresponding to the L2 norm value is determined, the differential image is threshold segmented, and the initialized prior change image is determined.

[0030] Optionally, determining the first pseudo label map according to the change difference map includes:

[0031] Get the L2 norm value of each pixel of the two first latent space feature maps;

[0032] Processing the L2 norm value and the first threshold of each pixel according to the percentile function, determining a plurality of pixel points in the first latent space feature map whose L2 norm value is greater than the first threshold;

[0033] A first pseudo label map is determined according to the plurality of pixels.

[0034] Optionally, a feedforward neural network is used to process the first latent space feature map after channel weighting and channel dimension concatenation to obtain an initial probability map, including:

[0035] Using the channel attention in CBAM, determine the channel attention weight of the first latent space feature map;

[0036] Using a multi-layer feed-forward neural network FFN, determine the changed probability map and the unchanged probability map of the output after channel-dimensional splicing;

[0037] An initial probability map is determined based on the changed probability map.

[0038] Optionally, determining a second pseudo label map according to the initial probability map includes:

[0039] Normalizing each pixel of the initial probability map to determine a normalized probability map of each pixel;

[0040] A second pseudo label map is determined according to a change region in the normalized probability map whose probability is greater than a second threshold.

[0041] The present invention also provides a heterogeneous dual-time image change detection device, comprising:

[0042] A feature map acquisition module is used to acquire a first latent space feature map and a priori change map of two heterogeneous dual-time images to be detected, which are mapped to the same latent feature space, using a dual-stream encoder; the dual-stream encoder is trained based on a first loss function composed of a first objective function determined by the two first latent space feature maps and the priori change map and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder, wherein the heterogeneous dual-time images include optical images and SAR images acquired at different time periods;

[0043] A first pseudo label map acquisition module is used to determine a change difference map according to the L2 norm of the pixel-by-pixel position difference between the two first latent space feature maps, and determine the first pseudo label map according to the change difference map;

[0044] An initial probability map acquisition module is used to use a feedforward neural network to process the first latent space feature map after channel weighting and channel dimension splicing to obtain an initial probability map, and the feedforward neural network is supervised and trained based on a cross entropy loss determined by the first pseudo label map and the change probability map and a second loss function determined by the sum of the L2 norm regularization of the parameters of the feedforward neural network;

[0045] A second pseudo label map acquisition module, used to determine a second pseudo label map according to the initial probability map;

[0046] The iterative output module is used to replace the prior change map with the second pseudo label map, iteratively update the second pseudo label map, and determine the probability map of the change detection results of the two heterogeneous dual-time images.

[0047] The present invention also provides an electronic device, comprising:

[0048] processor;

[0049] a memory for storing instructions executable by the processor;

[0050] The processor is configured to execute the instructions to implement the heterogeneous dual-time image change detection method as described in the first aspect.

[0051] The present invention also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of a server, the server can execute the heterogeneous dual-time image change detection method as described in the first aspect.

[0052] The embodiment of the present invention provides a heterogeneous dual-time image change detection method and device, which have the following beneficial effects compared with the prior art:

[0053] The present invention constructs an unsupervised change detection method by iteratively updating pseudo labels. Compared with the existing methods, the results effectively reduce the noise in the change map through model autonomous reasoning, effectively improve the detection accuracy, and do not require other complex post-processing. Compared with image conversion, which requires difficult conversion learning, the method of converting to latent space in this method is easier to train and takes up less resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0055] Figure 1 A flow chart of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention;

[0056] Figure 2 It is a structural diagram of the encoding part of the autoencoder of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention;

[0057] Figure 3 It is a channel attention module proposed from the CBAM model in a heterogeneous dual-time image change detection method provided by an embodiment of the present invention;

[0058] Figure 4 It is a schematic diagram of the entire optical and SAR image change detection method of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention;

[0059] Figure 5 It is a diagram of the training stages and pseudo-label iteration structure of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention;

[0060] Figure 6 It is a sample data 1 and experimental results of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention, wherein (a) is an optical image, (b) is a SAR image, (c) is a true change label, and (d) is a detection result;

[0061] Figure 7 This is sample data 2 of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention and its experimental results, (a) is an optical image, (b) is a SAR image, (c) is a true change label, and (d) is a detection result;

[0062] Figure 8 It is an intermediate result on sample data 2 of a heterogeneous dual-time image change detection method provided by an embodiment of the present invention, (a) is a difference image, (b) is the result of using Otsu threshold segmentation on the difference image, and (c) is the final result of the method;

[0063] Fig. 9 It is a structural diagram of a heterogeneous dual-time image change detection device provided by the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] This specification provides method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on conventional or non-creative work. When the actual system or server product is executed, the method shown in the embodiments or figures may be executed sequentially or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0066] Figure 1 A flow chart showing a heterogeneous dual-time image detection method provided by the present invention.

[0067] like Figure 1 As shown, the steps of the detection method include:

[0068] Step 101, using a dual-stream encoder to obtain a first latent space feature map and a priori change map of two heterogeneous dual-time images to be detected that are mapped to the same latent feature space; the dual-stream encoder is trained based on a first loss function consisting of a first objective function determined by the two first latent space feature maps and the priori change map and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder, wherein the heterogeneous dual-time images include optical images and SAR (Synthetic Aperture Radar) images acquired at different time periods;

[0069] Specifically, in the case of cloud cover, the optical sensor cannot obtain a complete and clear ground image, while the microwaves emitted by SAR can penetrate the cloud, thereby capturing the ground image without being affected. At this time, a pair of heterogeneous images can be composed of SAR image and optical image of another phase for change detection. Due to the huge distribution difference between optical and SAR images, the shared feature extractor cannot be uniformly encoded and represented. Two encoders must be used to encode them separately, and then uniformly encoded. The two encoders used in the present invention are called dual-stream encoders. A spatial scale-invariant CNN (Convolutional Neural Network) dual-stream encoder is constructed to encode and represent dual-phase images; and the loss function L1 described in the invention content is established to constrain the heterogeneous dual-phase images to be projected into the same feature space.

[0070] In a possible implementation, the dual-stream encoder includes two autoencoders, each of which includes:

[0071] The first n-1 encoding layers of the sub-encoder all contain two-dimensional convolution with multiple output channels, batch normalization, and LeakyReLU activation function, and the nth encoding layer contains two-dimensional convolution with multiple output channels, batch normalization, and Sigmoid activation function. The size of the convolution kernel of each layer is 3x3, and the padding method is used to make the spatial dimensions of the input and output the same;

[0072] The sub-decoder has a structure that is the same as the structure of the first n-1 coding layers of the sub-encoder.

[0073] In the embodiment provided by the present invention, the sub-encoder has six layers, and the convolution output channels of the first five layers are 12, 48, 96, 48, and 12 respectively. Each layer also includes a normalization process BN (Batch Normalization) and a LeakyReLU activation function. The last layer of the encoder is called the alignment layer, which includes a convolution with an output channel of 5 and a normalization process and a Sigmoid activation function. The Sigmoid activation function is used to limit its numerical range to the range of (0, 1). The structure of the sub-decoder is similar to that of the encoder, but there is no alignment layer.

[0074] Figure 2 This is the structural diagram of the encoding part of the autoencoder.

[0075] like Figure 2 As shown, the autoencoder of the autoencoder includes 6 layers, and the last layer is the alignment layer.

[0076] In a possible implementation, the step of using a dual-stream encoder to obtain a first latent space feature map of two heterogeneous dual-time images to be detected that is mapped to the same latent feature space includes:

[0077] Step 1011, obtaining an initialized prior change image;

[0078] Step 1012: input the two heterogeneous dual-time images into the dual-stream encoder, and output two initial latent space feature images;

[0079] Specifically, the two heterogeneous dual-time images are input into the dual-stream encoder, and two initial latent space feature images are output. This process is expressed as:

[0080] Z X =F enc (X), X∈R m×n

[0081] Z Y =G enc (Y), Y∈R m×n

[0082] Z X Represents the first initial latent space feature image of the original image X in the latent space Z, Z Y Represents the second initial latent space feature image of the original image Y in the latent space Z, and m and n represent the width and height of the spatial dimensions of X and Y.

[0083] Step 1013, determining a first objective function according to the quotient of the number of pixels of the two prior change images and the cumulative sum of the pixel values ​​of the initialized prior change image and the corresponding L1 norm of each pixel of the two initial latent space feature images;

[0084] Specifically, in order to encode two heterogeneous dual-time images into the same feature space, it is necessary to constrain the unchanged regions to have the same encoding representation (that is, to project them into the same latent feature space). Since there are also changed regions in the image, in order to make the feature representation of the changed regions show differences, the prior change map P is introduced u As a mask, to perform more accurate feature zooming, the following formula needs to be optimized:

[0085]

[0086] P u (i, j)∈0,1

[0087] L pull is the first objective function to be determined, L pull The minimum value of is the optimization goal.

[0088] Step 1014: determining a second objective function according to the sum of the difference between the pixel points of the two heterogeneous dual-time images and the pixel points of the two reconstructed images reconstructed and outputted by the dual-stream encoder;

[0089] Specifically, in order to retain sufficient implicit information in the encoded result, an autoencoder is used here. The autoencoder structure requires that the data reconstructed during the decoding process is the same as the input, which needs to be optimized:

[0090] lm rec =(XF dec (Z x ))+(YG dec (Z Y ))

[0091] L rec Denotes the determined second objective function, F enc (·), F dec (·), G enc (·), G dec(·) represent the dual-stream encoder and decoder respectively. Both the dual-stream encoder and decoder are implemented using CNN.

[0092] Step 1015: determining a first loss function according to the first objective function and the second objective function;

[0093] Specifically, the loss function of this stage is:

[0094] L1=L pull +λL rec

[0095] Where λ represents the linear combination parameter.

[0096] Steps 1011 to 1015 establish a loss function for measuring the training results of the dual-stream encoder.

[0097] Step 1016: training the dual-stream encoder according to the first loss function;

[0098] Specifically, the data used in training comes from optical remote sensing images and single-polarization SAR amplitude images that have been processed by radiation correction and geometric correction. The images of the two phases are spatially registered with each other.

[0099] During training, two heterogeneous dual-time images are divided into patches. The size of the patch is related to the properties of the image itself, the receptive field of the model, and the performance of the training device. It can be set to 96, 64, 32, etc., and there must be overlap between patches to ensure that each pixel is fully trained. The setting of the patch cropping step is related to the size of the patch and can be set to 32, 16, etc. Use symmetric padding to fill the image to a size that can be cut;

[0100] Each patch needs to be enhanced and normalized before being sent to training. The image enhancement strategy is to randomly rotate, scale, flip and introduce random noise to the bi-phase patch.

[0101] Step 1016 starts training the dual-stream encoder to obtain a suitable dual-stream encoder.

[0102] Step 1017: input the two heterogeneous dual-time images into the dual-stream encoder to obtain two first latent space feature maps, and the two first latent space feature maps are mapped to the same latent feature space.

[0103] Specifically, step 1017 obtains two aligned encoded first latent space feature maps mapped to the same latent feature space.

[0104] In a possible implementation, the step of obtaining an initialized priori change image includes:

[0105] Step 10111: Add the L1 norms of the pixels of the two initial latent space feature images to obtain a cumulative sum, and determine a third objective function using the quotient of the cumulative sum and the number of pixels of the initial latent space feature image;

[0106] Specifically, the third objective function is defined as:

[0107] Step 10112: Using the third objective function, train the sub-encoder of the dual-stream autoencoder;

[0108] Step 10113: input the initial latent space feature image into the trained sub-encoder to obtain two second latent space feature maps;

[0109] Specifically, step 10112-step 10113, use the unmasked Only the encoder of the two-stream autoencoder is trained (i.e., warm-up training) to obtain a difference map that extracts significant features without using reconstruction constraints, i.e., two second latent space feature maps.

[0110] Step 10114: obtain the L2 norm value of the pixel-by-pixel position difference between the two second latent space feature maps, determine the differential image composed of the pixel points corresponding to the L2 norm value, perform threshold segmentation on the differential image, and determine the initialized prior change image.

[0111] Specifically, no reconstruction constraints are used in the above steps, and the encoded result only extracts significant features, so the obtained difference map is not accurate. We use the threshold segmentation method, specifically the maximum inter-class variance algorithm Otsu (Otsu method) to obtain a rough change map, which we use as the prior change image This process can be expressed as:

[0112]

[0113] Among them, Otsu() represents the threshold segmentation algorithm, D (0) Represents the difference image without using reconstruction constraints.

[0114] Step 102: determining a change difference map according to the L2 norm of the pixel-by-pixel position difference between the two first latent space feature maps, and determining a first pseudo label map according to the change difference map;

[0115] Step 103: Use the feedforward neural network supervised by the first pseudo-label map to process the first latent space feature map after channel weighting and channel dimension splicing to obtain an initial change probability map;

[0116] In a possible implementation, the step of using the first pseudo-label map to supervise the trained feedforward neural network to process the first latent space feature map after channel weighting and channel dimension splicing to obtain the initial change probability map includes:

[0117] Step 1031: using the channel attention in the Convolutional Block Attention Module (CBAM) to determine the channel attention weights of the two first latent space feature maps;

[0118] Step 1032: using a multi-layer feed-forward neural network (FFN), determine a probability map of changes and a probability map of invariance of the output after channel-dimensional splicing;

[0119] Step 1033, determining an initial probability map according to the changed probability map;

[0120] Step 1034: Determine a second loss function by using the first pseudo label map, the determined cross entropy loss, and the sum of regularizing parameters of the multilayer feedforward neural network FFN using the L2 norm;

[0121] Step 1035: training the multi-layer feed-forward neural network FFN according to the second loss function;

[0122] Step 1036: Output the initial change probability map in the multi-layer feed-forward neural network FFN.

[0123] In the embodiment provided by the present invention, after obtaining the first latent space feature map of two heterogeneous dual-time images, the features at this time are easy to distinguish and more suitable for supervised learning. Since the result of the encoding link has the constraint of reconstruction, the encoding result must retain enough original image information. Compared with the final change information, its feature representation has some redundancy. Therefore, before pseudo-label supervised learning, channel attention is first used to Z X and Z Y This method uses the channel attention implementation method in CBAM to suppress channel information that is irrelevant to the change information. Then, the image features of the two phases are fused and supervised learning is performed using FFN.

[0124] Specifically, the fusion method uses channel-dimensional feature concatenation. FFN is a three-layer convolutional layer. The first two layers are each Conv-BN-ReLU, and the third layer is Conv-BN-Sigmoid, called the direct supervision layer. The Sigmoid activation function is used to map the output to a probability value in the [0, 1] interval. This process can be expressed as:

[0125] p=FFN(cat(Z X ·att(Z X ), Z Y ·att(Z Y )))

[0126] Among them, att(·) represents the input channel attention weight, cat(·, ·) represents the channel-dimensional concatenation operation, and p is the initial probability map output by FFN.

[0127] Specifically, supervised learning is performed through cross entropy loss. Since the labels are inaccurate, in order to maximize the generalization ability of FFN and infer more accurate results, the L2 norm is used to regularize the parameters of FFN to prevent overfitting of pseudo labels.

[0128] The optimization goals of this stage are:

[0129]

[0130] W FFN represents the parameters of FFN, β is the coefficient of the regularization term, and the cross entropy loss is:

[0131]

[0132] The output of the FNN after training is 2 channels, representing probability maps of two categories (change and unchanged), where the probability map of change is the initial change probability map obtained in this application.

[0133] Figure 3 It is a channel attention module proposed from the CBAM model.

[0134] like Figure 3 As shown in the figure, after the channel attention module proposed in the CBAM model inputs the feature F, it passes through the parallel maximum pool MaxPool and average pool AvgPool, passes through the shared multi-layer perception mechanism Shared MLP, and then passes through the maximum pool MaxPool and average pool AvgPool respectively, accumulates and integrates, and outputs the channel attention.

[0135] Step 104: determining a second pseudo label map according to the initial change probability map;

[0136] Step 105: Use the second pseudo label map to replace the prior change map, iteratively update the second pseudo label map, and determine a probability map of the change detection results of the two heterogeneous dual-time images.

[0137] Specifically, in the iterative update process, the method provided by the present invention is first divided into two stages:

[0138] Phase I: construct two spatial scale-invariant CNN autoencoders to encode and represent the dual-phase images respectively; and establish the loss function L1 described in the invention content to constrain the dual-phase images to be projected into the same feature space.

[0139] Phase II: Construct an FFN reasoning structure with channel attention, take the feature representation generated in Phase I as input, and use the reliable change information generated in Phase I as pseudo-label; and establish the loss function L2 described in the invention content, which is used to help FFN infer the change area more accurately than the pseudo-label.

[0140] Then, the method is trained, and the training method includes:

[0141] Two SGD (stochastic gradient descent) optimizers are used for optimization, one for updating the parameters of the two stages respectively. The initial learning rate is set to 1e-4. A learning rate decay strategy is set. Since stage II may have overfitting problems, the learning rate decay degree of stage II is larger. The batch size can be 32, 64, etc.

[0142] The hyperparameter λ in L1 and the hyperparameter β in L2 in the training method are λ=2 and β=0.5, respectively.

[0143] Training begins with using only Loss training for several epochs to perform warm-up training and obtain the initialization prior change graph After the formal start of training, stage I and stage II are trained alternately. During the training process, the number of epochs of each iteration of stage I and stage II is fixed to ensure that each iteration of stage I training converges. The number of epochs of stage II training should not be too many, otherwise it will over-fit with the pseudo-labels. Only in the case of under-convergence can the reasoning ability of FFN be fully utilized. With the increase of the number of iterations, stage II with a fixed number of epochs will slowly converge, and all losses will converge after multiple rounds of iterations.

[0144] After the training is completed, two heterogeneous dual-time images are input into the model to obtain the probability map of the FFN output change detection results, namely the initial change probability map.

[0145] In summary, the present invention uses pseudo labels throughout the entire method, and one iteration is divided into two clear stages. The first stage is mainly to obtain a unified feature representation of heterogeneous images based on the prior change map, and obtain pseudo labels in a TopN manner. The second stage uses the encoding results of the first stage and the obtained pseudo labels for supervised learning. The key points of this method are as follows:

[0146] (1) Use the prior change map to obtain better encoding results. This is an important part of the iterative results of this method, and the encoding results are continuously corrected through iteration. The existing methods also use a similar approach, but the prior change map of the existing methods is continuously updated through the encoding results or differential results, while the pseudo-labels of this method will go through the second stage of reasoning, which is obviously more reliable and reasonable.

[0147] (2) The method of initializing the prior change image is also very clever. The substructure of this model is used to obtain a better prior change map than other traditional methods without introducing additional complex operations. At the same time, the encoder is initialized, making the encoder training faster and more stable.

[0148] (3) Obtain reliable pseudo labels in the first stage. Although this pseudo label is obtained by a simple TopN method, this method is consistent with the characteristics of the model: the farther the feature measurement distance is, the more likely it is a change area. Since the second stage can autonomously infer more accurate results based on the pseudo labels, this method is less sensitive to N and only needs to maintain a relatively small value. There are few existing methods that use binary pseudo labels. A similar approach is to use differential Figure 2 However, this approach will introduce a lot of errors.

[0149] (4) Use pseudo labels for supervised learning. This step can fully reason about the change area, which is the most critical step to improve the detection results and the root of the iterative mode. There is no method in the existing methods to use pseudo labels for supervised learning to obtain the change map.

[0150] Figure 4 It is a schematic diagram of the entire optical and SAR image change detection method.

[0151] like Figure 4 As shown in the figure, the diagram includes the flow of data in training and inference, as well as the status of the network's intermediate feature maps. The dotted line is mainly used to generate and update the changes on the entire image. Since the model is a full CNN, it has no requirements on the spatial size of the image input, and the changes in the entire image need to be generated at one time, so it cannot be used as an input in the same way as in training (the distribution of each patch is different). In this method, the process of obtaining the probability map of the entire image change by inputting the entire image is called the verification process. All D and P u All are obtained in the verification process, and P is used in training. u When participating in the recalculation, it is necessary to slice and index the patch according to its location (see Appendix Figure 4 'Slice' operation in .

[0152] Figure 5 It is a diagram of the training stages and pseudo-label iteration structure.

[0153] like Figure 5 As shown in FIG. 1 , firstly, prepare the optical image X and the SAR image Y, then preheat the encoder of the autoencoder to generate an initialized prior change image, construct the L1 loss function according to the initialized prior change image, and use the L1 loss function to train the aligned autoencoder to obtain the encoded image Z of the latent space mapped from the output of the autoencoder X and Y. X and Z Y , according to the encoded image Z X and Z Y Get the first pseudo label, use the L2 loss function to supervise the pseudo label and the result of FFN inference, get the second pseudo label map of pseudo label supervised learning, and then determine whether the L2 loss function converges. If not, use the second pseudo label map to update the initialized prior change image and continue with the subsequent steps. If the L2 loss function converges, the training ends.

[0154] Figure 6 This is sample data 1 and its experimental results.

[0155] like Figure 6 As shown, (a) is the optical image, (b) is the SAR image, (c) is the true change label, and (d) is the detection result.

[0156] Figure 7 This is sample data 2 and its experimental results.

[0157] like Figure 7 As shown, (a) is the optical image, (b) is the SAR image, (c) is the true change label, and (d) is the detection result.

[0158] Figure 8 It is the intermediate result on sample data 2.

[0159] like Figure 8 As shown, (a) is the difference image, (b) is the result of using Otsu threshold segmentation on the difference image, and (c) is the final result of this method.

[0160] In a possible implementation, the step of determining the first pseudo label map according to the change difference map includes:

[0161] Step 1021: Obtain the L2 norm value of each pixel of the two first latent space feature maps;

[0162] Step 1022: Process the L2 norm value and the first threshold of each pixel according to the percentile function to determine a plurality of pixel points in the first latent space feature map whose L2 norm value is greater than the first threshold;

[0163] Step 1023: Determine the first pseudo label image according to the multiple pixel points.

[0164] In the embodiment provided by the present invention, after obtaining the encoded representation of the two images in the same feature space, that is, two first latent space feature maps, the feature distance of the unchanged area will be much smaller than that of the changed area. Therefore, the L2 norm can be used to measure the degree of change of each position. At this time, the measurement value is between [0, 1], thereby obtaining a change probability map:

[0165] D(i, j)=||Z X (i, j)-Z Y (i,j)||2,D(i,j)∈[0,1]

[0166] Among them, Z X (i, j) and Z Y (i, j) represent the first latent space feature maps of two heterogeneous dual-time images, and D(i, j) represents the value of each pixel in the change probability map;

[0167] In the obtained D(i, j), the position with a farther feature distance is more likely to be a change area, so the second pseudo label map P u (n) By selecting pixel positions in D whose distance is greater than TopN, considering that the changed area usually occupies a small part, N can select a smaller first threshold value according to the situation, such as 5%. That is:

[0168] α=percentile(D,N)

[0169] Among them, percentile(·,·) means calculating the quantile;

[0170] Specifically, the pixel values ​​D(i, j) of the change probability map are obtained, and the pixel values ​​whose percentiles are greater than α are set to 1, and the values ​​of other pixels are set to 0, so as to obtain the pixel values ​​of the first pseudo-label map.

[0171] In a possible implementation, determining the second pseudo label map according to the initial change probability map includes:

[0172] Step 1041, normalizing each pixel of the initial change probability map to determine a normalized probability map of each pixel;

[0173] Step 1042: Determine a second pseudo label map according to a change region in the normalized probability map whose probability is greater than a second threshold.

[0174] In the embodiment provided by the present invention, since the input feature change region is easy to distinguish, the change region also shows a very high change probability, while the probability range of the unchanged region is larger. Therefore, the channel p1 representing the change probability is taken, and after normalization, the probability is selected to be greater than the second threshold pt (The second threshold p in this method t =0.95) is the changed area, and the others are unchanged areas, and P is updated. u Used for the next iteration:

[0175]

[0176]

[0177] The present invention also provides a heterogeneous dual-time image change detection device 2, comprising:

[0178] The feature map acquisition module 201 is used to acquire the first latent space feature map and the prior change map of the two heterogeneous dual-time images to be detected mapped to the same latent feature space by using a dual-stream encoder; the dual-stream encoder is trained based on a first loss function composed of a first objective function determined by the two first latent space feature maps and the prior change map, and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder;

[0179] A first pseudo label map acquisition module 202 is used to determine a change difference map according to the L2 norm of the pixel-by-pixel position difference between the two first latent space feature maps, and determine a first pseudo label map according to the change difference map;

[0180] An initial probability map acquisition module 203 is used to process the first latent space feature map after channel weighting and channel dimension splicing processing using the feedforward neural network supervised by the first pseudo label map to obtain an initial change probability map;

[0181] A second pseudo label map acquisition module 204, configured to determine a second pseudo label map according to the initial change probability map;

[0182] The iterative output module 205 is used to replace the prior change map with the second pseudo label map, iteratively update the second pseudo label map, and determine a probability map of the change detection results of the two heterogeneous dual-time images.

[0183] Fig. 9 It is a structural diagram of a heterogeneous dual-time image change detection device provided by the present invention.

[0184] like Fig. 9 The heterogeneous dual-time image change detection device 2 shown includes: a feature map acquisition module 201, a first pseudo label map acquisition module 202, an initial probability map acquisition module 203, a second pseudo label map acquisition module 204, and an iterative output module 205.

[0185] In a possible implementation, the dual-stream encoder includes two autoencoders, each of which includes:

[0186] The first n-1 encoding layers of the sub-encoder all contain two-dimensional convolution with multiple output channels, batch normalization, and LeakyReLU activation function, and the nth encoding layer contains two-dimensional convolution with multiple output channels, batch normalization, and Sigmoid activation function. The size of the convolution kernel of each layer is 3x3, and the padding method is used to make the spatial dimensions of the input and output the same;

[0187] The sub-decoder has a structure that is the same as the structure of the first n-1 coding layers of the sub-encoder.

[0188] In a possible implementation, the feature map acquisition module 201 specifically includes:

[0189] The first image acquisition unit 2011 is used to acquire an initialized priori change image;

[0190] An initial image acquisition unit 212 is used to input the two heterogeneous dual-time images into the dual-stream encoder and output two initial latent space feature images;

[0191] A first function determination unit 2013 is used to determine a first objective function according to the quotient of the number of pixels of the two prior change images and the cumulative sum of the pixel values ​​of the initialized prior change image and the L1 norm of the corresponding pixel of the two initial latent space feature images;

[0192] A second function determination unit 2014 is used to determine a second objective function according to the sum of differences between the pixel points of the two heterogeneous dual-time images and the pixel points of the two reconstructed images reconstructed and outputted by the dual-stream encoder respectively;

[0193] A first loss function determining unit 2015, configured to determine a first loss function according to the first objective function and the second objective function;

[0194] A first training unit 2016, configured to train the dual-stream encoder according to the first loss function;

[0195] The first feature map acquisition unit 2017 is used to input the two heterogeneous dual-time images into the dual-stream encoder to obtain two first latent space feature maps, and the two first latent space feature maps are mapped to the same latent feature space.

[0196] In a possible implementation manner, the first image acquisition unit 2011 specifically includes:

[0197] A third function determination unit 20111 is configured to add the L1 norms of the pixels of the two initial latent space feature images to obtain a cumulative sum, and determine a third objective function using a quotient of the cumulative sum and the number of pixels of the initial latent space feature image;

[0198] The second training unit 20112 is used to train the sub-encoder of the dual-stream autoencoder using the third objective function;

[0199] A second feature map acquisition unit 20113 is used to input the initial latent space feature image into the trained sub-encoder to obtain two second latent space feature maps;

[0200] The prior change image acquisition unit 20114 obtains the L2 norm value of the pixel-by-pixel position difference between the two second latent space feature maps, determines the differential image composed of the pixel points corresponding to the L2 norm value, performs threshold segmentation on the differential image, and determines the initialized prior change image.

[0201] In a possible implementation manner, the first pseudo label map acquisition module 202 specifically includes:

[0202] An L2 norm acquisition unit 2021 is used to acquire the L2 norm value of each pixel of the two first latent space feature maps;

[0203] A first processing unit 2022 is used to process the L2 norm value and the first threshold of each pixel according to the percentile function, and determine a plurality of pixel points whose L2 norm value is greater than the first threshold in the first latent space feature map;

[0204] The generating unit 2023 is configured to determine the first pseudo label map according to the multiple pixel points.

[0205] In a possible implementation manner, the initial change probability map acquisition module 203 specifically includes:

[0206] A weight determination unit 2031 is used to determine the channel attention weight of the first latent space feature map by using the channel attention in CBAM;

[0207] The probability map output unit 2032 is used to determine the changed probability map and the unchanged probability map of the output after channel dimension splicing by using the multi-layer feed-forward neural network FFN;

[0208] The probability map determining unit 2033 is used to determine an initial probability map according to the changed probability map;

[0209] A second loss function determining unit 2034 is used to determine a second loss function by using the first pseudo label map and the determined cross entropy loss and the L2 norm to regularize the parameters of the multilayer feedforward neural network FFN;

[0210] The third training unit 2035 is used to train the multi-layer feed-forward neural network FFN according to the second loss function;

[0211] The initial change probability map output unit 2036 is used to output the initial change probability map in the multi-layer feed-forward neural network FFN.

[0212] In a possible implementation manner, the second pseudo label map acquisition module 204 specifically includes:

[0213] The second image acquisition unit 2041 is used to normalize each pixel of the initial change probability map to determine a normalized probability map of each pixel;

[0214] The second processing unit 2042 is used to determine a second pseudo label map according to a change area in the normalized probability map whose probability is greater than a second threshold.

[0215] In another embodiment provided by the present invention, a device is also provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the heterogeneous dual-time image change detection method described in the embodiment of the present invention.

[0216] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the heterogeneous dual-time image change detection method described in the embodiment of the present invention.

[0217] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0218] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0219] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0220] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A heterogeneous dual-time image change detection method, characterized in that: include: A dual-stream encoder is used to obtain a first latent space feature map and a priori change map of two heterogeneous dual-time images to be detected that are mapped to the same latent feature space; the dual-stream encoder is trained based on a first loss function composed of a first objective function determined by the two first latent space feature maps and the priori change map and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder, wherein the heterogeneous dual-time images include optical images and SAR images acquired at different time periods; Determine a change difference map according to an L2 norm of a pixel-by-pixel position difference between the two first latent space feature maps, and determine a first pseudo label map according to the change difference map; Using the first pseudo-label map to supervise the trained feedforward neural network to process the first latent space feature map after channel weighting and channel dimension splicing, to obtain an initial change probability map; Determine a second pseudo label map according to the initial change probability map; The prior change map is replaced by the second pseudo label map, the second pseudo label map is iteratively updated, and a probability map of change detection results of two heterogeneous dual-time images is determined.

2. The method according to claim 1, characterized in that The dual-stream encoder includes two autoencoders, each of which includes: The first n-1 encoding layers of the sub-encoder all contain two-dimensional convolution with multiple output channels, batch normalization, and LeakyReLU activation function, and the nth encoding layer contains two-dimensional convolution with multiple output channels, batch normalization, and Sigmoid activation function. The size of the convolution kernel of each layer is 3x3, and the padding method is used to make the spatial dimensions of the input and output the same; The sub-decoder has a structure that is the same as the structure of the first n-1 coding layers of the sub-encoder.

3. The method according to claim 2, characterized in that The method of using a dual-stream encoder to obtain a first latent space feature map of two heterogeneous dual-time images to be detected that are mapped to the same latent feature space includes: Get an initialized prior change image; Inputting the two heterogeneous dual-time images into the dual-stream encoder, and outputting two initial latent space feature images; Determine a first objective function according to the quotient of the cumulative sum of the pixel values ​​of the initialized priori change image and the L1 norm of each pixel of the corresponding two initial latent space feature images and the number of pixels of the two initial latent space feature images; Determine a second objective function according to the sum of differences between pixel points of the two heterogeneous dual-time images and pixel points of two reconstructed images reconstructed and outputted by the dual-stream encoder respectively; Determine a first loss function according to the first objective function and the second objective function; Training the dual-stream encoder according to the first loss function; The two heterogeneous dual-time images are input into the dual-stream encoder to obtain two first latent space feature maps, and the two first latent space feature maps are mapped to the same latent feature space.

4. The method according to claim 3, characterized in that The step of obtaining an initialized priori change image comprises: Adding the L1 norms of the pixels of the two initial latent space feature images to obtain a cumulative sum, and determining a third objective function using the quotient of the cumulative sum and the number of pixels of the initial latent space feature image; Using the third objective function, training a sub-encoder of the two-stream autoencoder; Inputting the initial latent space feature image into the trained sub-encoder to obtain two second latent space feature maps; Obtain the L2 norm value of the pixel-by-pixel position difference between the two second latent space feature maps, determine the differential image composed of the pixel points corresponding to the L2 norm value, perform threshold segmentation on the differential image, and determine the initialized prior change image.

5. The method according to claim 1, characterized in that Determining a first pseudo label map according to the change difference map includes: Obtaining the L2 norm value of each pixel of the two first latent space feature maps; Processing the L2 norm value and the first threshold of each pixel according to the percentile function, determining a plurality of pixel points in the first latent space feature map whose L2 norm value is greater than the first threshold; The first pseudo label map is determined according to the multiple pixel points.

6. The method according to claim 1, characterized in that The method of using the first pseudo-label map to supervise the trained feedforward neural network to process the first latent space feature map after channel weighting and channel dimension splicing to obtain an initial change probability map includes: Using the channel attention in CBAM, determine the channel attention weights of the two first latent space feature maps; Using a multi-layer feed-forward neural network FFN, determine the changed probability map and the unchanged probability map of the output after channel-dimensional splicing; determining an initial probability map according to the changed probability map; Determine a second loss function based on the first pseudo label graph, the determined cross entropy loss, and the sum of the results of regularizing the parameters of the multilayer feedforward neural network FFN using the L2 norm; Training the multi-layer feedforward neural network FFN according to a second loss function; Output the initial change probability map in the multi-layer feed-forward neural network FFN.

7. The method according to claim 1, characterized in that The determining of the second pseudo label map according to the initial change probability map comprises: Normalizing each pixel of the initial change probability map to determine a normalized probability map of each pixel; A second pseudo label map is determined according to a change region in the normalized probability map whose probability is greater than a second threshold.

8. A heterogeneous dual-time image change detection device, characterized in that: include: A feature map acquisition module is used to acquire a first latent space feature map and a priori change map of two heterogeneous dual-time images to be detected mapped to the same latent feature space using a dual-stream encoder; the dual-stream encoder is trained based on a first loss function composed of a first objective function determined by the two first latent space feature maps and the priori change map and a second objective function determined by input and output during the reconstruction process of the dual-stream encoder; A first pseudo label map acquisition module, used to determine a change difference map according to the L2 norm of the pixel-by-pixel position difference between the two first latent space feature maps, and determine a first pseudo label map according to the change difference map; An initial probability map acquisition module is used to process the first latent space feature map after channel weighting and channel dimension splicing processing using the feedforward neural network supervised by the first pseudo label map to obtain an initial change probability map; A second pseudo label map acquisition module, used to determine a second pseudo label map according to the initial change probability map; An iterative output module is used to replace the prior change map with the second pseudo label map, iteratively update the second pseudo label map, and determine a probability map of the change detection results of the two heterogeneous dual-time images.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the heterogeneous dual-time image change detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to execute the heterogeneous dual-time image change detection method as described in any one of claims 1 to 7.

Images