Deep neural network image change detection method fused with confidence corresponding estimation

Abstract

The invention discloses a deep neural network image change detection method fused with confidence correspondence estimation, which belongs to the technical field of image processing and is realized by adopting two cascaded deep sub-neural networks of a deep confidence correspondence estimation network and a change detection network. The method comprises the following steps: inputting two images to be detected into a depth confidence correspondence estimation network, outputting a correspondence between the two images and the confidence of the correspondence, and extracting image features and uncertainty features of the correspondence; and inputting the confidence, the image features and the corresponding relation uncertainty features into a change detection network, performing further processing through a multi-stage convolutional neural network, and outputting a change detection result. According to the method, image corresponding relation estimation, corresponding relation confidence coefficient estimation and change detection are organically fused, so that image registration and change detection are unified into a network framework, and meanwhile, the image change detection effect is improved.

Description

Technical Field

[0001] The present invention relates to image change detection technology, and in particular to a deep neural network image change detection method integrating confidence correspondence estimation. Background Art

[0002] Image change detection refers to finding the changed parts of two images by analyzing and comparing them. Image change detection technology has a wide range of application scenarios, such as video surveillance, remote sensing image analysis, medical image analysis and other fields.

[0003] Traditional image change detection methods mainly include threshold-based methods, difference map-based methods, and feature extraction-based methods:

[0004] The threshold-based method is one of the simplest and most direct image change detection methods. This method first grayscales the two images and then compares them at the pixel level. If the difference between the two pixel values ​​exceeds the set threshold, the pixel is considered to have changed. This method is sensitive to illumination changes and noise.

[0005] The difference map-based method represents the difference information of two images as a difference map. This method first grayscales the image sequence, and then performs pixel-level difference calculation to obtain a difference map. In the difference map, areas with larger pixel values ​​indicate that a larger change has occurred. By setting a threshold, the changed and unchanged areas can be segmented. This method is somewhat robust to illumination changes and noise, but it is not effective for detecting image changes in complex backgrounds.

[0006] The feature extraction-based method uses features in the image sequence to detect changes. Features can be structural, texture, color, or motion features. This method first extracts features from the image sequence and then uses a machine learning algorithm to classify it. In the training phase, a classifier is established by learning known changed and unchanged samples. In the testing phase, the image sequence to be detected is input into the classifier to obtain the changed and unchanged results. The advantage of this method is that it can adapt to complex backgrounds and lighting changes, but it requires a large number of samples for training and has a high computational complexity.

[0007] Existing image change detection methods usually assume that the corresponding pixels of the two input images also correspond to each other in the imaging target space, that is, the same position in the two images reflects the information of the same position in the imaging target space. However, in actual use, due to the inconsistency of the imaging positions and viewing angles of the two images, the pixel coordinates corresponding to the same position in the imaging target space in the two images are not consistent. Before applying the traditional image change detection method, it is necessary to perform an additional image registration algorithm to correct the spatial inconsistency of the two images. Usually, both image registration and change detection require related processing such as extracting image features and feature matching. Therefore, if image registration and change detection are performed independently, there will be repeated processing problems, resulting in low processing efficiency of the entire method. Summary of the invention

[0008] In view of the above-mentioned deficiencies in the prior art, the deep neural network image change detection method integrating confidence correspondence estimation provided by the present invention solves the problem that image registration and change detection in the existing change detection are performed independently, resulting in repeated processing, thereby causing low processing efficiency of the entire method.

[0009] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0010] A deep neural network image change detection method integrating confidence correspondence estimation is provided. The method is implemented by two cascaded deep neural networks, namely a deep confidence correspondence estimation network and a change detection network. The method comprises the following steps:

[0011] S1. Input the image A and the image B to be detected into the feature extraction part of the depth confidence correspondence estimation network for feature extraction to obtain the full-size feature map of the image A to be detected. And the pyramid feature F of the image B to be detected ΔB , full-size feature map

[0012] S2, full-size feature map and full-size feature maps After the correlation operation, the feature correlation body V is obtained, and the feature correlation body V is input into the correspondence estimation part and the corresponding uncertainty feature extraction part of the deep confidence correspondence estimation network respectively to obtain the correspondence M and the corresponding uncertainty feature F between the image to be detected A and the image to be detected B. P ;

[0013] S3. The corresponding relationship M and the corresponding uncertainty feature F PInput the probability regression part of the deep confidence correspondence estimation network to obtain the parameters of the corresponding uncertainty probability model; then, calculate the confidence map T of the correspondence M between the image to be detected A and the image to be detected B according to the parameters of the corresponding uncertainty probability model;

[0014] S4, the corresponding uncertainty feature F P The features after stacking with the confidence map T and the pyramid features F of the image B to be detected ΔB The image is input into the change detection network, and after processing by the change detection network, the change detection result between the image to be detected A and the image to be detected B is obtained.

[0015] Furthermore, the method of performing correlation operation on the image features of two images to be detected includes:

[0016] A1. For a pixel at any position (i, j) in the image to be detected B, find all the pixels in the neighborhood around the pixel at the corresponding position (i, j) in the image to be detected A. The size of the neighborhood is d×d, where d is a constant selected according to the application background.

[0017] A2. Full-size feature map of the pixel at position (i, j) The feature vector in step A1 is the full-size feature map of all pixels found in step A1 The matrix composed of all the correlation operation results corresponding to the pixel points at position (i, j) is used as the correlation sub-body v ij ∈R d×d ;

[0018] A3, performing steps A1 and A2 on the remaining pixel points in the image B to be detected, to obtain relevant sub-volumes corresponding to the pixel points at all positions in the image B to be detected;

[0019] A4. The feature correlation body V of the input uncertainty feature extraction part and the corresponding relationship estimation part is formed by using the correlation sub-body corresponding to the pixel points at all positions in the image B to be detected.

[0020] Further, the method of obtaining the corresponding uncertainty feature by using the corresponding uncertainty feature extraction part includes:

[0021] The feature correlation body V is expanded to 5 dimensions h×w×d×d×1 and input into the corresponding uncertainty feature extraction part composed of a multi-stage convolutional network;

[0022] The feature correlation body V of dimension h×w×d×d×1 passes through several convolutional layers corresponding to the uncertainty feature extraction part, and the output dimension is h×w×1×1×n p Characteristics, n p is the dimension corresponding to the uncertainty characteristic;

[0023] The removed dimensions are h×w×1×1×n p The dimension of the feature is 1 in the middle two dimensions, so that it can be reconstructed into h×w×n p The corresponding uncertainty characteristic F P ; h and w are the height and width of the images A and B to be detected respectively.

[0024] Furthermore, the method of obtaining the corresponding relationship between the two images A and B to be detected by using the corresponding relationship estimation part includes:

[0025] The last two dimensions of the feature correlation body V are flattened to obtain a 3D tensor with dimensions of h×w×(d×d); h and w are the height and width of the images A and B to be detected, respectively;

[0026] The 3D tensor is passed through the correspondence estimation part consisting of a multi-level convolutional neural network with residual connections to obtain a correspondence M with a dimension of h×w×2:

[0027] The corresponding relationship between the pixel with the same name as any pixel B(u,j) in the image to be detected B and the pixel with the same name in the image to be detected A is M(i,j)=(l―i,k―j), where (l,k) is the position of the pixel with the same name as B(i,j) in the image to be detected A;

[0028] The multi-level convolutional neural network with residual connection in the correspondence estimation part includes a convolutional layer C0, a Relu layer R0, a convolution+Relu layer C1 and a convolutional layer C2 connected in sequence. The output of the convolutional layer C0 is superimposed with the output of the convolutional layer C2 after passing through the residual connection layer S0 to obtain a superimposed feature F S1 , superimposed feature F S1 Input the Relu layer R1, convolution + Relu layer C3 and convolution layer C4 connected in sequence; superimpose feature F S1 After passing through the residual connection layer S1, the output of the convolution layer C4 is superimposed to obtain the superimposed feature F S2 , superimposed feature F S2 After the output of the Relu layer R2 and the convolution layer C5, the corresponding relationship is obtained.

[0029] Furthermore, step S3 further includes:

[0030] S31, the corresponding relationship M and the corresponding uncertainty feature F P After stacking, the probability regression part with a multi-level convolutional neural network as the architecture is input;

[0031] S32, the stacked features are passed through multiple convolutional layers to obtain the mixed probability model parameters where α′1∈R h×w ,α′2∈Rh×w are the unnormalized weights of the two independent probability models in the mixed probability model, are the variances of the two independent probability models in the mixed probability model respectively; for any pixel (i, j) in the image B to be detected, the corresponding mixed probability model parameters are The corresponding element in the correspondence relationship M is m ij , is taken as the mean μ of the probability density function ij , that is, μ ij =m ij ;

[0032] S33, unnormalized weights α′ of two independent probability models 1,ij , α′ 2,ij After the softmax layer, the corresponding mixed probability model is obtained Normalized model weights α 1,ij , α 2,ij , where y ij is a random variable, which represents the possible value of the correspondence between the two images to be detected at the position of the pixel point (i, j) in the image to be detected B, and it obeys the mixed probability density distribution: in is the independent probability density function, α 1,ij +α 2,ij =1, is the probability distribution parameter;

[0033] S34. According to the probability distribution parameters of the model Calculate the correspondence m between the two images to be detected at the pixel point (i, j) in the image to be detected B ij The confidence level P R,ij :

[0034]

[0035] Where R is a real number, which is determined according to the application context; ‖y ij ―m ij ‖ is y ij 、m ij The Euclidean distance between .

[0036] Furthermore, when the probability density function adopts the Laplace distribution, the correspondence relationship m between the two images to be detected at the position of the pixel point (i, j) in the image to be detected B is ij The confidence level P R,ij The expression is:

[0037]

[0038] Here, k is a variable with values ​​of 1 and 2, and exp(·) is an exponential function.

[0039] Furthermore, the feature extraction part uses VGG-16 as the skeleton network; during feature extraction, the convolution layer output before each pooling layer in the VGG-16 skeleton network is taken to form the pyramid feature F of the image to be detected A and the image to be detected B ΔA and pyramid feature F ΔB , which contains level IV image features; then upsampled to the same height and width as the input image by nearest neighbor interpolation and stacked to form a feature map of dimension h×w×1472, respectively obtaining the full-size feature maps of the image to be detected A and the image to be detected B and full-size feature maps After feature extraction of the image A to be detected, the pyramid feature F is obtained. ΔA and full-size feature maps After feature extraction of the image B to be detected, the pyramid feature F is obtained ΔB and full-size feature maps

[0040] Furthermore, the loss function of the deep confidence correspondence estimation network is The expression is:

[0041]

[0042] Among them, θ is the network parameter; is the training set for training the deep confidence correspondence estimation network, X is an input image pair consisting of image A and image B, is the true correspondence of X; N is the total number of input image pairs; is the probability density function; and are the normalized independent probability density model weights corresponding to the image B(i,j) estimated by the nth input image pair; and are the probability model variances corresponding to the image B(i,j) estimated by the nth input image pair; is the correspondence at image B(i,j) estimated from the nth input image pair; for The true correspondence between pixels (i, j) in .

[0043] Furthermore, the change detection network in step S4 adopts a U-Net-like architecture and an encoder-decoder structure; in the process of gradually reducing the encoder, the pyramid feature F of the corresponding image B to be detected is introduced according to the scale size. ΔBThe IV level image features in the decoder are used; at the end of the decoder, a 1×1 convolution layer is used to obtain the probability value of each pixel in the image B to be detected belonging to the changed area.

[0044] Furthermore, the loss function of the change detection network is The expression is:

[0045]

[0046] in, is the training set for training the change detection network, F C is the pyramid image feature, uncertainty feature, and confidence of the corresponding relationship of the input image pair A and image B, G is the change region label map; N is the total number of input image pairs; The pixel at (i, j) of image B in the nth image pair is the label value of the changed area, which is 1 if changed and 0 if not changed; The probability that the pixel at (i, j) of image B in the image pair predicted by the network is a changed area; w n is the weight corresponding to the nth input image pair, which is equal to the number of pixels in the sample that do not belong to the changed area divided by the number of pixels that belong to the changed area.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This scheme integrates the correspondence between two images when performing change monitoring. This eliminates the need to align the two input images before detection, thereby improving detection efficiency. The change detection process integrates the correspondence information between images extracted by the correspondence estimation network, thereby improving the accuracy of change area detection.

[0049] 2. The image feature C fused in the change detection process overcomes the influence of unclear correspondence between weak texture areas. The change detection process adopts an end-to-end deep neural network structure to overcome the negative impact of manually selected image features in traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the deep neural network image change detection method with fused confidence correspondence estimation.

[0051] Figure 2 Detailed overall technical solution flow chart of the deep neural network image change detection method integrating confidence correspondence estimation.

[0052] Figure 3 This is a schematic diagram of using VGG-16 as the skeleton network for feature extraction.

[0053] Figure 4 This is a schematic diagram of the architecture of the uncertainty feature extraction part.

[0054] Figure 5 Schematic diagram of the architecture of the correspondence estimation part.

[0055] Figure 6 This is a schematic diagram of the architecture of the probability parameter regression part.

[0056] Figure 7 Schematic diagram of the architecture of the changing region segmentation network. DETAILED DESCRIPTION

[0057] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0058] refer to Figure 1 , Figure 1 A deep neural network image change detection method integrating confidence correspondence estimation is shown; Figure 1 As shown, the method S is implemented by two cascaded deep sub-neural networks, namely a deep confidence correspondence estimation network and a change detection network. The method S includes steps S1 to S4. The detailed overall technical solution flow chart of the method can be referred to Figure 2 .

[0059] In step S1, the image to be detected A and the image to be detected B are input into the feature extraction part of the depth confidence correspondence estimation network for feature extraction to obtain the pyramid feature F of the image to be detected A. ΔA , full-size feature map And the pyramid feature F of the image B to be detected ΔB , full-size feature map Among them, the image A to be detected is an early image, and the image B to be detected is a late image, that is, the acquisition time of the image A to be detected is earlier than that of the image B to be detected.

[0060] like Figure 3 As shown in the figure, the feature extraction part uses VGG-16 as the skeleton network; when extracting features, the convolution layer output before each pooling layer in the VGG-16 skeleton network is taken to form the pyramid feature F of the image to be detected A and the image to be detected B. ΔA and pyramid feature F ΔB, which contains level IV image features; then upsampled to the same height and width as the input image by nearest neighbor interpolation and stacked to form a feature map of dimension h×w×1472, respectively obtaining the full-size feature maps of the image to be detected A and the image to be detected B and full-size feature maps After feature extraction of the image A to be detected, the pyramid feature F is obtained. ΔA and full-size feature maps After feature extraction of the image B to be detected, the pyramid feature F is obtained ΔB and full-size feature maps

[0061] In step S2, the full-size feature map and full-size feature maps After the correlation operation, the feature correlation body V is obtained, and the feature correlation body V is input into the correspondence estimation part and the corresponding uncertainty feature extraction part of the deep confidence correspondence estimation network respectively to obtain the correspondence M and the corresponding uncertainty feature F between the image to be detected A and the image to be detected B. P .

[0062] In one embodiment of the present invention, a method for performing correlation operation on image features of two images to be detected includes:

[0063] A1. For a pixel point at any position (i, j) in the image to be detected B, search for all pixel points in the neighborhood around the pixel point at the corresponding position (i, j) in the image to be detected A. The neighborhood size is d×d, where d is a constant selected according to the application background. In this embodiment, d is 17, and the corresponding relationship offset is constrained within ±8 pixels.

[0064] A2. Full-size feature map of the pixel at position (i, j) The feature vector in step A1 is the full-size feature map of all pixels found in step A1 The matrix composed of all the correlation operation results corresponding to the pixel points at position (i, j) is used as the correlation sub-body v ij ∈R 17×17 ;

[0065] A3, performing steps A1 and A2 on the remaining pixel points in the image B to be detected, to obtain relevant sub-volumes corresponding to the pixel points at all positions in the image B to be detected;

[0066] A4. Use the correlation sub-volumes corresponding to the pixel points at all positions in the image B to be detected to form a feature correlation volume V that is input into the corresponding uncertainty feature extraction part and the corresponding relationship estimation part.

[0067] During implementation, the scheme preferably adopts the corresponding uncertainty feature extraction part to obtain the corresponding uncertainty feature, including:

[0068] The feature correlation body V is expanded to 5 dimensions h×w×17×17×1 and input into the corresponding uncertainty feature extraction part composed of a multi-stage convolutional network;

[0069] The feature correlation body V with dimension h×w×17×17×1 passes through several convolutional layers corresponding to the uncertainty feature extraction part, and the output dimension is h×w×1×1×n p Characteristics, n p is the dimension corresponding to the uncertainty feature. In this embodiment, n p Take 64;

[0070] The removed dimensions are h×w×1×1×n p The dimension of the feature is 1 in the middle two dimensions, so that it can be reconstructed into h×w×n p The corresponding uncertainty characteristic F P ; h and w are the height and width of the images A and B to be detected respectively. Figure 4 As shown in the figure, in the four sequentially connected convolutional layers of the uncertainty feature extraction part, the convolution kernel of convolution layer 1 is 5×5, Stride is 2, Pad is 0, the number of channels is 8, and the output dimension is h×w×7×7×8; the convolution kernel of convolution layer 2 is 3×3, Stride is 1, Pad is 0, the number of channels is 16, and the output dimension is h×w×5×5×16;

[0071] The convolution kernel of convolution layer 3 is 3×3, Stride is 1, Pad is 0, the number of channels is 32, and the output dimension is h×w×3×3×32; the convolution kernel of convolution layer 4 is 3×3, Stride is 1, Pad is 0, the number of channels is 64, and the output dimension is h×w×1×1×64.

[0072] During implementation, the method for obtaining the corresponding relationship between the two images A and B to be detected by the corresponding relationship estimation part preferably includes:

[0073] The last two dimensions of the feature correlation body V are flattened to obtain a 3D tensor with dimensions of h×w×(d×d); h and w are the height and width of the images A and B to be detected, respectively;

[0074] The tensor is passed through a multi-level convolutional neural network with residual connections as the correspondence estimation part to obtain a correspondence M with a dimension of h×w×2:

[0075] The corresponding relationship between the pixel with the same name as any pixel B(i,j) in the image to be detected B and the pixel with the same name in the image to be detected A is M(i,j)=(l―i,k―j), where (l,k) is the position of the pixel with the same name as B(i,j) in the image to be detected A;

[0076] like Figure 5 As shown in the figure, the correspondence estimation part includes the convolution layer C0, Relu layer R0, convolution + Relu layer C1 and convolution layer C2 connected in sequence. The output of convolution layer C0 is superimposed with the output of convolution layer C2 after passing through the residual connection layer S0 to obtain the superimposed feature F S1 , superimposed feature F S1 Input the Relu layer R1, convolution + Relu layer C3 and convolution layer C4 connected in sequence; superimpose feature F S1 After passing through the residual connection layer S1, the output of the convolution layer C4 is superimposed to obtain the superimposed feature F S2 , superimposed feature F S2 After the output of the Relu layer R2 and the convolution layer C5, the corresponding relationship is obtained.

[0077] The convolution kernels of convolutional layers C0, C2, C4, and C5 are 3×3, Stride is 1, and Padding is 1; the convolution kernels of convolutional + Relu layers C1 and C are 3×3, Stride is 1, and Padding is 1; the convolution kernels of residual connection layers S0 and S1 are 1×1, Stride is 1, and Padding is 0.

[0078] In step S3, the corresponding relationship M and the corresponding uncertainty feature F P The probability regression part of the deep confidence correspondence estimation network is input to obtain the parameters of the corresponding uncertainty probability model; then, according to the parameters of the corresponding uncertainty probability model, the confidence map T of the correspondence M between the image to be detected A and the image to be detected B is calculated.

[0079] In one embodiment of the present invention, step S3 further comprises:

[0080] S31, the corresponding relationship M and the corresponding uncertainty feature F P After stacking, the probability regression part with a multi-level convolutional neural network as the architecture is input;

[0081] S32, the stacked features are passed through multiple convolutional layers to obtain the mixed probability model parameters where α′1∈R h×w ,α′2∈R h×w are the unnormalized weights of the two independent probability models in the mixed probability model, are the variances of the two independent probability models in the mixed probability model; for any pixel (i, j) in the image B to be detected, the corresponding mixed probability model parameter is α′ 1,ij , α′ 2,ij , The corresponding element of the correspondence relationship M is m ij , taken as the mean μ of the probability density function ij , that is, μ ij =m ij ;

[0082] S33, unnormalized model weight α′ 1,ij , α′ 2,ij After the softmax layer, the corresponding mixed probability model is obtained Normalized model weights α 1,ij , α 2,ij , where y ij is a random variable, which represents the possible value of the correspondence between the two images to be detected at the position of the pixel point (i, j) in the image to be detected B, and it obeys the mixed probability density distribution: in (·) is the independent probability density function, α 1,ij +α 2,ij =1, is the probability distribution parameter;

[0083] S34. According to the probability distribution parameters of the model Calculate the correspondence m between the two images to be detected at the pixel point (i, j) in the image to be detected B ij The confidence level P R,ij :

[0084]

[0085] Where R is a real number, which is determined according to the application context; ‖y ij ―m ij ‖ is y ij 、m ij The Euclidean distance between .

[0086] like Figure 6 As shown in the figure, C0 and C1 in the probability regression network are convolution + Relu layers, whose convolution kernel is 3×3, Stride is 1, and Padding is 1; C2 is a convolution layer, whose convolution kernel is 3×3, Stride is 1, and Padding is 1.

[0087] When the probability density function adopts Laplace distribution, the corresponding relationship m between the two images to be detected at the position of the pixel point (i, j) in the image to be detected B is ij The confidence level PR,ij The expression is:

[0088]

[0089] Where k is a variable with values ​​of 1 and 2, and exp(·) is an exponential function.

[0090] In step S4, the corresponding uncertainty feature F P The features after stacking with the confidence map T and the pyramid features F of the image B to be detected ΔB The image is input into the change detection network, and after processing by the change detection network, the change detection result between the image to be detected A and the image to be detected B is obtained.

[0091] The detailed implementation process of step S4 is as follows: The change detection network consists of two parts: hybrid feature synthesis SN 21 and Variation Region Segmentation Network SN 22 SN 21 The corresponding uncertainty feature F P And the confidence map T is stacked, and then the stacked features and the pyramid features F of the image B to be detected are combined. ΔB Segmentation of the network SN by changing regions 22 , perform inference calculation to obtain the change area of ​​the image B to be detected relative to the image A to be detected, thereby obtaining the change detection result.

[0092] The above-mentioned change region segmentation network SN 22 It is a semantic segmentation network composed of a multi-level convolutional neural network. The input of the network is the stacked features and the pyramid features F of the image B to be detected. ΔB , output the probability that each pixel in the image to be detected B belongs to the change area. In this embodiment, the change detection network adopts a U-Net-like architecture, and the network structure is as follows Figure 7 As shown in the figure, an encoder-decoder structure is adopted. In the process of gradually reducing the encoder, the corresponding image pyramid feature F is introduced according to the scale size. ΔB IV image features in ( Figure 3 IV level image features shown in the figure); at the end of the decoder, a 1×1 convolution layer is used to obtain the probability value of each pixel in the image to be detected belonging to the change area. The number of channels in each convolution layer in the network is as follows Figure 7 The upper or lower number of the convolution result.

[0093] When implemented, this scheme prefers the loss function of the deep confidence correspondence estimation network The expression is:

[0094]

[0095] Among them, θ is the network parameter; is the training set for training the deep confidence correspondence estimation network, X is an input image pair consisting of image A and image B, is the true correspondence of X; N is the total number of input image pairs; is the probability density function; and are the normalized independent probability density model weights corresponding to the image B(i,j) estimated by the nth input image pair; and are the probability model variances corresponding to the image B(i,j) estimated by the nth input image pair; is the correspondence at image B(i,j) estimated from the nth input image pair; for The true correspondence between pixels (i, j) in .

[0096] When the loss function When the probability density function is Laplace probability density distribution, the loss function The expression is:

[0097]

[0098] in, and are the unnormalized independent probability model weights corresponding to the image B(i,j) estimated by the nth input image pair; k is a variable, equal to 1 or 2; e is the natural logarithm; |·|1 is the 1 norm.

[0099] The loss function of the change detection network is The expression is:

[0100]

[0101] in, is the training set for training the change detection network, F C is the pyramid image feature, uncertainty feature, and confidence of the corresponding relationship of the input image pair A and image B, G is the change region label map; N is the total number of input image pairs; The pixel at (i, j) of image B in the nth image pair is the label value of the changed area, which is 1 if changed and 0 if not changed; The probability that the pixel at (i, j) of image B in the image pair predicted by the network is a changed area; w n is the weight corresponding to the nth input image pair, which is equal to the number of pixels in the sample that do not belong to the changed area divided by the number of pixels that belong to the changed area.

[0102] In summary, this scheme organically integrates image correspondence estimation, correspondence confidence estimation and change detection, unifies image registration and change detection into a single network framework, and improves the image change detection effect.

Claims

1. A deep neural network image change detection method integrating confidence correspondence estimation, characterized in that: The method is implemented by using two cascaded deep neural networks, a deep confidence correspondence estimation network and a change detection network, and the method comprises the steps of: S1. Input the image A and the image B to be detected into the feature extraction part of the depth confidence correspondence estimation network for feature extraction to obtain the full-size feature map of the image A to be detected. And the pyramid feature F of the image B to be detected ΔB , full-size feature map S2, for the full-size feature map and full-size feature maps After the correlation operation, the feature correlation body V is obtained, and the feature correlation body V is input into the correspondence estimation part and the corresponding uncertainty feature extraction part of the deep confidence correspondence estimation network respectively to obtain the correspondence M and the corresponding uncertainty feature F between the image to be detected A and the image to be detected B. P ; S3. The corresponding relationship M and the corresponding uncertainty feature F P Input the probability regression part of the deep confidence correspondence estimation network to obtain the parameters of the corresponding uncertainty probability model; then, calculate the confidence map T of the correspondence M between the image to be detected A and the image to be detected B according to the parameters of the corresponding uncertainty probability model; S4, the corresponding uncertainty feature F P The features after stacking with the confidence map T and the pyramid features F of the image B to be detected ΔB The image is input into the change detection network, and after processing by the change detection network, the change detection result between the image to be detected A and the image to be detected B is obtained.

2. The deep neural network image change detection method according to claim 1, characterized in that: The method of performing correlation operation on the image features of two images to be detected includes: A1. For a pixel at any position (i, j) in the image to be detected B, find all the pixels in the neighborhood around the pixel at the corresponding position (i, j) in the image to be detected A. The neighborhood size is d×d, where d is a constant. A2. Full-size feature map of the pixel at position (i, j) The feature vector in step A1 is the full-size feature map of all pixels found in step A1 The matrix composed of all the correlation operation results corresponding to the pixel points at position (i, j) is used as the correlation sub-body v ij ∈R d×d ; A3, performing steps A1 and A2 on the remaining pixel points in the image B to be detected, to obtain relevant sub-volumes corresponding to the pixel points at all positions in the image B to be detected; A4. Use the correlation sub-volumes corresponding to the pixel points at all positions in the image B to be detected to form a feature correlation volume V input to the uncertainty feature extraction part and the corresponding relationship estimation part.

3. The deep neural network image change detection method according to claim 2, characterized in that: Methods for obtaining corresponding uncertainty features by using the corresponding uncertainty feature extraction part include: The feature correlation body V is expanded to 5 dimensions h×w×d×d×1 and input into the corresponding uncertainty feature extraction part composed of a multi-stage convolutional network; The feature correlation body V of dimension h×w×d×d×1 passes through several convolutional layers corresponding to the uncertainty feature extraction part, and the output dimension is h×w×1×1×n p Characteristics, n p is the dimension corresponding to the uncertainty characteristic; The removed dimensions are h×w×1×1×n p The dimension of the feature is 1 in the middle two dimensions, so that it can be reconstructed into h×w×n p The corresponding uncertainty characteristic F P ; h and w are the height and width of the images A and B to be detected respectively.

4. The deep neural network image change detection method according to claim 2, characterized in that: The method of obtaining the corresponding relationship between two images A and B to be detected by using the corresponding relationship estimation part includes: The last two dimensions of the feature correlation body V are flattened to obtain a 3D tensor with dimensions of h×w×(d×d); h and w are the height and width of the images A and B to be detected, respectively; The 3D tensor is passed through the correspondence estimation part composed of a multi-level convolutional neural network with residual connections to obtain a correspondence M with a dimension of h×w×2: The corresponding relationship between the pixel with the same name as any pixel B(u,j) in the image to be detected B and the pixel with the same name in the image to be detected A is M(i,j)=(l―i,k―j), where (l,k) is the position of the pixel with the same name as B(i,j) in the image to be detected A; The multi-level convolutional neural network with residual connection in the correspondence estimation part includes a convolutional layer C0, a Relu layer R0, a convolution+Relu layer C1 and a convolutional layer C2 connected in sequence. The output of the convolutional layer C0 is superimposed with the output of the convolutional layer C2 after passing through the residual connection layer S0 to obtain a superimposed feature F S1 , superimposed feature F S1 Input the Relu layer R1, convolution + Relu layer C3 and convolution layer C4 connected in sequence; superimpose feature F S1 After passing through the residual connection layer S1, the output of the convolution layer C4 is superimposed to obtain the superimposed feature F S2 , superimposed feature F S2 After the output of the Relu layer R2 and the convolution layer C5, the corresponding relationship M is obtained.

5. The deep neural network image change detection method according to claim 4, characterized in that: Step S3 further comprises: S31, the corresponding relationship M and the corresponding uncertainty feature F P After stacking, the probability regression part with a multi-level convolutional neural network as the architecture is input; S32, the stacked features are passed through multiple convolutional layers to obtain the mixed probability model parameters where α′1∈R h×w ,α′2∈R h×w are the unnormalized weights of the two independent probability models in the mixed probability model, are the variances of the two independent probability models in the mixed probability model; for any pixel (i, j) in the image B to be detected, the corresponding mixed probability model parameter is α′ 1,ij , α′ 2,ij , The corresponding element m in the corresponding relationship M ij is taken as the mean μ of the probability density function ij , that is, μ ij =m ij ; S33, unnormalized weights α′ of two independent probability models 1,ij , α′ 2,ij After the softmax layer, the corresponding mixed probability model is obtained Normalized model weights α 1,ij , α 2,ij , where y ij is a random variable, which represents the possible value of the correspondence between the two images to be detected at the position of the pixel point (i, j) in the image to be detected B, and it obeys the mixed probability density distribution: in is the independent probability density function, α 1,ij +α 2,ij =1, is the probability distribution parameter; S34. According to the probability distribution parameters of the model Calculate the correspondence between the two images to be detected at the pixel point (i, j) in the image to be detected B as m ij The confidence level P R,ij : Where R is a real number, which is determined according to the application context; ‖y ij ―m ij ‖ is y ij 、m ij The Euclidean distance between .

6. The deep neural network image change detection method according to claim 5, characterized in that: When the probability density function adopts Laplace distribution, the corresponding relationship m between the two images to be detected at the pixel point (i, j) in the image to be detected B is ij The confidence level P R,ij The expression is: Here, k is a variable with values ​​of 1 and 2, and exp(·) is an exponential function.

7. The deep neural network image change detection method according to any one of claims 1 to 6, characterized in that: The feature extraction part uses VGG-16 as the skeleton network; when extracting features, the convolution layer output before each pooling layer in the VGG-16 skeleton network is taken to form the pyramid feature F of the image to be detected A and the image to be detected B ΔA and pyramid feature F ΔB , which contains level IV image features; then upsampled to the same height and width as the input image by nearest neighbor interpolation and stacked to form a feature map of dimension h×w×1472, respectively obtaining the full-size feature maps of the image to be detected A and the image to be detected B and After feature extraction of the image A to be detected, the pyramid feature F is obtained. ΔA and full-size feature maps After feature extraction of the image B to be detected, the pyramid feature F is obtained. ΔB and full-size feature maps 8. The deep neural network image change detection method according to any one of claims 1 to 6, characterized in that: Loss function of the deep confidence correspondence estimation network The expression is: Among them, θ is the network parameter; is the training set for training the deep confidence correspondence estimation network, X is an input image pair consisting of image A and image B, is the true correspondence of X; N is the total number of input image pairs; is an independent probability density function; and are the normalized independent probability density model weights corresponding to the image B(i,j) estimated by the nth input image pair; and are the variances of the independent probability density models corresponding to the image B(i,j) estimated by the nth input image pair; is the correspondence at image B(i,j) estimated from the nth input image pair; for The true correspondence between pixels (i, j) in .

9. The deep neural network image change detection method according to claim 7, characterized in that: The change detection network in step S4 adopts a U-Net-like architecture and an encoder-decoder structure; the encoder gradually reduces the size of the pyramid feature F of the image B to be detected according to the scale. Δb The IV level image features in the decoder are used; at the end of the decoder, a 1×1 convolution layer is used to obtain the probability value of each pixel in the image B to be detected belonging to the changed area.

10. The deep neural network image change detection method according to any one of claims 1 to 6, characterized in that: The loss function of the change detection network is The expression is: in, is the training set for training the change detection network, F C is a mixed feature consisting of the pyramid image features, uncertainty features and confidence of the corresponding relationship corresponding to the input image pair A and image B, G is the change region label map; N is the total number of input image pairs; The pixel at (i, j) of image B in the nth image pair is the label value of the changed area, which is 1 if changed and 0 if not changed; is the probability value of the pixel at (i, j) of image B in the image pair predicted by the network as a changed area; w n is the weight corresponding to the nth input image pair, which is equal to the number of pixels in the sample that do not belong to the changed area divided by the number of pixels that belong to the changed area.

Images