Heterogeneous image change detection method and device based on structural regression fusion

Abstract

The present application relates to a heterogeneous image change detection method and device based on structural regression fusion. The method comprises: constructing a forward regression model using the regularized hypergraph Laplace matrix of the pre-event image; constructing a backward regression model using the regularized hypergraph Laplace matrix of the post-event image; fusing the hypergraph of the pre-event image with the hypergraph of the post-event image, and calculating the regularized hypergraph Laplace matrix of the fused hypergraph to obtain a smooth constraint of the change image; constructing a fused regression model based on the forward regression model, the backward regression model, the smooth constraint, and the change alignment constraint, solving the fused regression model using the alternating direction multiplier method to obtain a difference matrix between the pre-event image and the post-event image; mapping the difference matrix to obtain a difference image; the difference image is the result of transformation detection. The present method can improve the accuracy of image change detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer equipment and storage medium for detecting heterogeneous image changes based on structural regression fusion. Background Art

[0002] The goal of change detection in remote sensing images is to identify changed regions by jointly analyzing two (or more) remote sensing images of the same scene acquired at different times. Change detection plays a crucial role in Earth observation applications such as disaster assessment, urban development, and environmental monitoring. Heterogeneous change detection is a common problem in remote sensing, as it is difficult to detect changes by directly comparing heterogeneous images from different domains.

[0003] However, a number of graph-based unsupervised heterogeneous change detection methods have emerged. These methods exploit the structural consistency of multi-temporal images in invariant regions. These methods can be roughly divided into two categories: the first category is based on structural transformation and comparison, constructing a graph for each image to capture its structural information and then comparing the structures through graph fusion and graph mapping; the second category is based on structural regression, which requires that the structures of the original and transformed images are similar. However, these structure-based methods have a major drawback: they ignore structural asymmetry. That is, the structural complexity of the images before and after the event is different, especially when a new type of feature appears or when a type of feature completely disappears from one of the two images. In such cases, they can only complete the mapping comparison or image regression in one direction and fail in the other. That is, they cannot map or transform images with complex structures into images with simpler structures, resulting in low image change detection accuracy. Summary of the Invention

[0004] Based on this, it is necessary to provide a heterogeneous image change detection method, device, computer equipment and storage medium based on structural regression fusion that can improve the accuracy of image change detection in order to address the above technical problems.

[0005] A heterogeneous image change detection method based on structural regression fusion, the method comprising:

[0006] obtaining a pre-event image and a post-event image to be compared;

[0007] Preprocessing the pre-event image and the post-event image according to the superpixel segmentation method of the Gaussian mixture model to obtain the superpixel segmentation results of the pre-event image and the post-event image;

[0008] The superpixel segmentation results of the pre-event image and the post-event image are used to construct the K-nearest neighbor graph of the pre-event image and the post-event image, and the weight matrix of the pre-event image and the weight matrix of the post-event image are obtained;

[0009] Constructing a hypergraph of the pre-event image and a hypergraph of the post-event image according to the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculating a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image;

[0010] A forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image, and the post-event image is decomposed into a forward regression image and a forward change image; a backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image, and the pre-event image is decomposed into a backward regression image and a backward change image;

[0011] The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smoothness constraint of the changed image;

[0012] A fusion regression model is constructed based on the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint. The fusion regression model is solved using the alternating direction multiplier method to obtain the difference matrix between the pre-event image and the post-event image.

[0013] The difference matrix is mapped to obtain a difference image; the difference image is the result of transformation detection.

[0014] In one embodiment, superpixel segmentation results of the pre-event image and the post-event image are used to construct a K-nearest neighbor graph of the pre-event image and the post-event image, and a weight matrix of the pre-event image and a weight matrix of the post-event image are obtained, including:

[0015] The superpixel segmentation results of the pre-event image are used to construct the K nearest neighbor graph of the pre-event image, and the weight matrix of the pre-event image is obtained as follows:

[0016]

[0017] in, express and The distance between two superpixels, α i >0 is the equilibrium parameter, k i represents the number of neighbor nodes of the i-th superpixel, represents the i-th superpixel in the image before the event, represents the jth superpixel in the image before the event.

[0018] In one embodiment, a hypergraph includes a vertex set, a hyperedge set, and hyperedge weights; a hypergraph of a pre-event image and a hypergraph of a post-event image are fused, and a regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain a smooth constraint of the changed image, including:

[0019] The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smooth constraint of the change image:

[0020]

[0021] in, represents the hyperedge of the fused hypergraph, E f represents the hypergraph hyperedge set after fusion, represents the hyperedge weight, represents a hyperedge, The super-edge degree, represents the j,lth element in the hypergraph incidence matrix H^f, represents the feature vector of the i-th superpixel in the backward-varied image, Represents the feature vector of the i-th superpixel of the forward change image, Δ x Represents the feature matrix of the backward-changing image, Δ y Represents the feature matrix of the forward change image, L f represents the regularized hypergraph Laplacian matrix of the fused hypergraph, T represents the transpose operation, and Tr represents the trace of the matrix.

[0022] In one embodiment, constructing a hypergraph of the pre-event image and a hypergraph of the post-event image based on a weight matrix of the pre-event image and a weight matrix of the post-event image, and calculating a regularized hypergraph Laplacian matrix to obtain a regularized hypergraph Laplacian matrix of the pre-event image and a regularized hypergraph Laplacian matrix of the post-event image includes:

[0023] According to the weight matrix of the pre-event image, the hypergraph of the pre-event image is constructed, and the regularized hypergraph Laplace matrix is calculated. The regularized hypergraph Laplace matrix of the pre-event image is obtained as

[0024]

[0025] in, The i-th hyperedge of the hypergraph representing the image before the event, E t1 represents the hyperedge set, w(e t1 ) represents the hyperedge weight, Represents a hyperedge The super-edge degree, Represents the incidence matrix of the pre-event image hypergraph, Y′ i Represents the feature vector of the i-th superpixel of the forward regression image, Y′ j Represents the feature vector of the jth superpixel of the forward regression image, L t1represents the regularized hypergraph Laplacian matrix of the pre-event image, and T represents the transpose operation.

[0026] In one embodiment, a forward regression model is constructed using a regularized hypergraph Laplacian matrix of a pre-event image, comprising:

[0027] The forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image:

[0028]

[0029] Among them, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, Y represents the feature matrix of the post-event image, Y′ represents the feature matrix of the forward regression image, and λ represents the sparsity penalty parameter.

[0030] In one embodiment, a backward regression model is constructed using a regularized hypergraph Laplacian matrix of a post-event image, comprising:

[0031] The backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image:

[0032]

[0033] Among them, L t2 represents the regularized hypergraph Laplacian matrix of the post-event image, X represents the feature matrix of the pre-event image, and X′ represents the feature matrix of the backward regression image.

[0034] In one embodiment, the change alignment constraint is

[0035]

[0036] Among them, || ||2 means Norm, N s represents the total number of superpixels, and φ represents the variation alignment constraint function.

[0037] In one embodiment, a fusion regression model is constructed based on a forward regression model, a backward regression model, a smooth constraint, and a change alignment constraint, including:

[0038] According to the forward regression model, backward regression model, smooth constraint and change alignment constraint conditions, the fusion regression model is constructed as follows:

[0039]

[0040] Among them, β represents the change smooth penalty parameter, η represents the change alignment penalty parameter, st represents the constraint condition, |||| 2,1 express norm.

[0041] A heterogeneous image change detection device based on structural regression fusion, the device comprising:

[0042] A superpixel segmentation module is used to obtain a pre-event image and a post-event image to be compared; the pre-event image and the post-event image are pre-processed according to a superpixel segmentation method of a Gaussian mixture model to obtain superpixel segmentation results of the pre-event image and the post-event image;

[0043] A weight matrix construction module is used to construct a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtain a weight matrix of the pre-event image and a weight matrix of the post-event image;

[0044] a hypergraph Laplace matrix calculation module, configured to construct a hypergraph of the pre-event image and a hypergraph of the post-event image based on the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculate a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image;

[0045] A regression model construction module is used to construct a forward regression model using the regularized hypergraph Laplace matrix of the pre-event image, and decompose the post-event image into a forward regression image and a forward change image; and to construct a backward regression model using the regularized hypergraph Laplace matrix of the post-event image, and decompose the pre-event image into a backward regression image and a backward change image;

[0046] The hypergraph fusion module is used to fuse the hypergraph of the pre-event image and the hypergraph of the post-event image, and calculate the regularized hypergraph Laplacian matrix of the fused hypergraph to obtain the smoothness constraint of the changed image;

[0047] The fusion regression model construction and solution module is used to construct a fusion regression model based on the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint conditions, and solve the fusion regression model using the alternating direction multiplier method to obtain the difference matrix between the pre-event image and the post-event image;

[0048] The mapping module is used to map the difference matrix to obtain a difference image; the difference image is the result of transformation detection.

[0049] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0050] obtaining a pre-event image and a post-event image to be compared;

[0051] Preprocessing the pre-event image and the post-event image according to the superpixel segmentation method of the Gaussian mixture model to obtain the superpixel segmentation results of the pre-event image and the post-event image;

[0052] The superpixel segmentation results of the pre-event image and the post-event image are used to construct the K-nearest neighbor graph of the pre-event image and the post-event image, and the weight matrix of the pre-event image and the weight matrix of the post-event image are obtained;

[0053] Constructing a hypergraph of the pre-event image and a hypergraph of the post-event image according to the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculating a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image;

[0054] A forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image, and the post-event image is decomposed into a forward regression image and a forward change image; a backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image, and the pre-event image is decomposed into a backward regression image and a backward change image;

[0055] The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smoothness constraint of the changed image;

[0056] A fusion regression model is constructed based on the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint. The fusion regression model is solved using the alternating direction multiplier method to obtain the difference matrix between the pre-event image and the post-event image.

[0057] The difference matrix is mapped to obtain a difference image; the difference image is the result of transformation detection.

[0058] The above-mentioned heterogeneous image change detection method, device, computer equipment and storage medium based on structural regression fusion, this application uses a hypergraph to capture the high-order information of the image to obtain a more accurate image structure representation, and calculates the smoothness constraint conditions after fusing the hypergraph of the image before the event and the hypergraph of the image after the event. A fusion regression model is constructed according to the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint conditions, and the changes in the regression model are fused with some constraints to make the changes in the output image more accurate, thereby mutually improving the image regression performance and detection performance, overcoming the defects of the structural regression method caused by structural asymmetry, so that when performing heterogeneous image change detection, a more accurate difference matrix can be obtained, thereby improving the accuracy of heterogeneous image change detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 11 is a flow chart of a heterogeneous image change detection method based on structural regression fusion in one embodiment;

[0060] Figure 2 1 is a structural block diagram of a heterogeneous image change detection device based on structural regression fusion in one embodiment;

[0061] Figure 3 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0063] In one embodiment, Figure 1 As shown, a heterogeneous image change detection method based on structure regression fusion is provided, which includes the following steps:

[0064] Step 102: Obtain a pre-event image and a post-event image to be compared; pre-process the pre-event image and the post-event image according to a superpixel segmentation method of a Gaussian mixture model to obtain superpixel segmentation results of the pre-event image and the post-event image.

[0065] The pre-event image and the post-event image are preprocessed using the superpixel segmentation method of the Gaussian mixture model. First, a pseudo RGB image is constructed, in which the channels include the grayscale image of the pre-event image, the grayscale image of the post-event image, and a zero channel. Then, the pseudo RGB image is segmented using the GMMSP algorithm to obtain Ns segmentation regions Λ. The superpixel segmentation results of the pre-event image and the post-event image are then obtained, denoted as

[0066]

[0067] Then, and Represent the same area, they are and The multi-temporal images are all internally homogeneous. Once the superpixels are obtained, the different information represented by the superpixels can be represented by the extracted different features, such as spectral (intensity), spatial and textual information. This application simply extracts the mean and median of each band as the features of the superpixel, and then the feature matrix of the multi-temporal image can be obtained, which is recorded as and Each column (X i and Y i ) represents the feature vector of the superpixel ( and ). The superpixel segmentation result has two advantages: first, superpixels can capture background information and preserve the edges of objects; second, the computational complexity can be greatly reduced by reducing the size of the graph.

[0068] Step 104 : constructing a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtaining a weight matrix of the pre-event image and a weight matrix of the post-event image.

[0069] In order to capture the structural information of the image, a K-nearest neighbor graph is constructed. For example, in SCASC, the weight matrix S is used t1 is the K nearest neighbor graph of the image before the event The graph divides each superpixel With its weight The k nearest neighbors are connected and S is calculated using the following minimization model t1

[0070]

[0071] express and The distance between two superpixels, α i >0 is the equilibrium parameter, k i represents the number of neighbor nodes of the i-th superpixel, represents the i-th superpixel in the image before the event, represents the jth superpixel of the image before the event. According to SCASC, S t1 The closed-form solution of can be calculated by the following formula:

[0072]

[0073] Among them, Sort in ascending order j represents The position of the jth minimum value in k i Adaptively determined by the in-degree-based strategy proposed in SCASC.

[0074] Step 106: construct a hypergraph of the pre-event image and a hypergraph of the post-event image based on the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculate the regularized hypergraph Laplace matrix to obtain the regularized hypergraph Laplace matrix of the pre-event image and the regularized hypergraph Laplace matrix of the post-event image.

[0075] The weight matrix of the image before the event and the weight matrix of the image after the event (the construction method is similar to ) based on which the pre-event image is constructed. and post-event images of and In contrast to pairwise graphs, hypergraphs can link more than two vertices, which can preserve high-order neighborhood relationships between superpixels and capture more comprehensive structural information.

[0076] Use V t1 、E t1 Representing a hypergraph The vertex set and superedge set of each superpixel are set as a vertex v∈V t1 , and treat each superpixel and its neighbors as a hyperedge to relate to w(e t1 ), as a subset of the vertex set To build Specifically, using S t1 , Hypergraph Built as

[0077]

[0078] Using a weighted incidence matrix H t1 , that is, H t1 =S t1 , is connected to the center superpixel Superpixels Assign different weights and change the hyperedge weight w(e t1 ) is set to the average value of the thermal kernel similarity of superpixels within the class, as follows:

[0079]

[0080] Based on H t1 ,w(e t1 ), each vertex The vertex degree and each hyperedge The hyperedge degree is Let d t1 , ψ t1 and W t1 The diagonal matrices representing vertex degree, hyperedge degree, and hyperedge weight, d t1 =diag{d(1),…,d(N S )}、 and The regularized hypergraph Laplacian matrix is defined as:

[0081] L t1 =d t1 -H t1 W t1 (ψ t1 ) -1 (Ht1 ) T

[0082] Hypergraph similar to pre-event image Similarly, the post-event image is constructed of and obtain d t2 , ψ t2 , W t2 and L t2 .

[0083] The main goal of regression-based heterogeneous image change detection is to transform an image into another image domain and obtain a difference image between the transformed image and the target image. This application uses superpixels as vertices (basic units) and extracts features to represent superpixels, so it is necessary to find a regression function between two feature matrices. The transformation function between the feature matrix domains is defined as: and The feature extraction operator is defined as Define the pixel value extraction operator as For example, extract the average feature from the feature matrix as the pixel value of each pixel within the superpixel, represent X and Y as the transformed feature matrix, and and Defined as the transformed image. The forward transformation is

[0084]

[0085] The backward transformation is

[0086] Two superpixels in the pre-event image ( and ) represents the same ground feature (shown to be very similar), then the transformed superpixel and It should also represent the same type of ground objects (the difference / distance is very small). To represent, in within the same hyperedge and The corresponding transformed superpixel and Right now and should be similar. Therefore, the regularized hypergraph Laplacian matrix of the pre-event image can be obtained as

[0087]

[0088] Similarly, the regularized hypergraph Laplacian matrix of the post-event image can be obtained.

[0089] Step 108: Use the regularized hypergraph Laplace matrix of the pre-event image to construct a forward regression model, and decompose the post-event image into a forward regression image and a forward change image; use the regularized hypergraph Laplace matrix of the post-event image to construct a backward regression model, and decompose the pre-event image into a backward regression image and a backward change image.

[0090] For post-event images Decompose it into forward regression image and the forward change image We get Y = Y′ - Δ y ,in Represents the feature matrix of the forward change image. There is a prior sparsity-based regularization (PSR) for the changed image, which is based on the fact that only a small part of the ground objects in practice have changed. Sparse regularization of ||Δ y || 2，1 Defined as This is the original The convex relaxation of y The number of non-zero columns (the number of superpixels that change). By combining the regularized hypergraph Laplacian matrix of the pre-event image and sparse regularization, the forward transformation model is obtained as:

[0091]

[0092] Similar to the forward transform, the hypergraph Post-event image represented Similarity relationships within the image should be regressed The retained superpixels after transformation and Corresponding to The same hypergraph of in and That is to say, and should be similar to each other. By combining the regularized hypergraph Laplacian matrix of the post-event image and sparse regularization, the backward transformation model is obtained as:

[0093]

[0094] Step 110 , fusing the hypergraph of the image before the event and the hypergraph of the image after the event, and calculating the regularized hypergraph Laplacian matrix of the fused hypergraph to obtain the smoothness constraint of the changed image.

[0095] Forward and backward changing images and Since these images come from different domains, they cannot be directly fused to obtain a difference image. For example, merging these changed images by simply averaging them may not improve detection accuracy. This application fuses the forward and backward transformations during the regression process, rather than performing post-fusion on the changed images as in previous methods, to improve regression performance and obtain a more accurate changed image.

[0096] In the regression process, the forward and backward transformations are fused, first in the fusion hypergraph The changing image is smoothed at the same time. In this application, the fusion hypergraph The definition is as follows: Assumptions and Otherwise, set the fused hyperedge weight to:

[0097] Here, card(·) represents the cardinality of the set.

[0098] because but and Indicates the same type of landform (from 's construction), and and also represent the same features (from the structural consistency constraint of the backward transformation). Similarly, since but and Also refers to the same type of land feature (from 's construction), and It also represents the same type of ground feature (from the structural consistency constraint of the forward transformation). Then we get and and Also represents the same type of change. A more intuitive mathematical description: Since ||X i -X j ||2 and ||Y i -Y j ||2 is very small (because and ), and ||X′ i -X′ j ||2 and ||Y′ i -Y′ j ||2 is also very small (because of HGLR based on structural consistency), so we can conclude that and is also very small. Therefore, we can obtain the fused hypergraph-based smoothness regularization (FHSR), namely

[0099]

[0100] Among them, L f Represents the regularized hypergraph Laplacian matrix of the fused hypergraph.

[0101] Step 112 : construct a fusion regression model based on the forward regression model, the backward regression model, the smoothness constraint, and the change alignment constraint. The fusion regression model is solved using the alternating direction multiplier method to obtain a difference matrix between the pre-event image and the post-event image.

[0102] The change images are aligned on the support set. Because the two change images describe the same change event, that is, and The change area in is the same, then there are the following constraints

[0103]

[0104] However, it is very difficult to use this constraint directly in the regression model. Therefore, this application uses the following variation alignment based regularization (CAR) Its function φ is defined as

[0105]

[0106] In CAR, use and Respectively and The probability of change of the i-th superpixel in . If the i-th superpixel ( or ) has changed during the event, then and should be large, then the obtained CAR is small. On the contrary, if this area does not change during the event, then and should be close to zero, then the obtained CAR is large. Minimizing CAR requires that When any one level of change is larger, the other level of change should also be larger, which means and Align.

[0107] Therefore, the fusion regression model is constructed based on the forward regression model, backward regression model, smooth constraint and change alignment constraint conditions:

[0108]

[0109] From the fusion regression model, it can be found that the forward and backward transformation constraints tend to X′=Y′=0, and the sparse constraint smooth constraint tends to Δ x =Δ y = 0, the solution of the variation alignment constraint (CAR) tends to be Δ x and Δ x ≠0, which means that these regularization terms have an adversarial balancing effect. From the fusion regression model, we can also find that X and X′ are in the same field for two reasons: First, X′ is decomposed from X, and there are only a few columns between them that are different (Δ x is sparse in columns), which means that ideally X′ i =X i Suitable for most Second, for the changing X j , the regression model constrains it to be similar to the neighborhood through backward transformation, thus preventing abnormal X′ j . Similarly, Y and Y′ are also in the same domain. From the fused regression model, it can be found that by using CAR based on change alignment, the forward and backward transformations are fused in one model. By introducing CAR, the two changed images are aligned in the SRF: structural regression fusion model, which can also be regarded as change fusion. In addition, through this change fusion in SRF, one regression (such as forward) can use the supervision information (change information) from another regression (such as backward). Therefore, this allows the forward regression to overcome the influence of structural asymmetry, thereby improving the effect of fused regression and obtaining a more accurate change image.

[0110] This application uses the alternating direction multiplier method (ADMM) to solve the minimization problem of the fusion regression model. x and P2 = Δ y The auxiliary constraints of the fusion regression model are as follows:

[0111]

[0112] in and is the Lagrangian multiplier, μ1, μ2, μ3, μ4>0 are penalty parameters. The minimization of the augmented Lagrangian function of the fusion regression model can be further divided into the following problems.

[0113] 1) X′ and Y′ problem. The minimization of the augmented Lagrangian function of the fusion regression model with respect to X′ can be written as

[0114]

[0115] This can be solved by taking the first derivative of the objective function to zero. Then X′ can be updated as

[0116]

[0117] Among them I NS Indicates N S ×N S Scalar matrix. Similarly, for the regression Y′, it can be updated as

[0118]

[0119] 2)Δ x and Δ y Problem. The augmented Lagrangian function of the fusion regression model is about Δ x The minimization of can be written as

[0120]

[0121] This minimization problem uses the gradient descent method. Set the number of iterations of the inner loop Δ x N i , the step size is τ, then we can get

[0122] △ x =△ x -τg(△ x )

[0123]

[0124] where ⊙ represents the Hadamard product, The calculation formula is

[0125]

[0126] 3) P1 and P2 problems. The minimization of the augmented Lagrangian function of the fusion regression model with respect to P1 can be written as

[0127]

[0128] It can be interpreted as:

[0129] Similarly, for P2, it can be updated as

[0130] Finally, the Lagrange multipliers R1, R2, R3, and R4 can be updated as

[0131] R1←R1+μ1(X′-X-△ x )

[0132] R2←R2+μ2(△ x -P1)

[0133] R3←R3+μ3(Y′-Y-△y )

[0134] R4←R4+μ4(△ y -P2)

[0135] This application obtains the regression feature matrix of X and Y and Δ by integrating the regression model. x , Δ y The change feature matrix of and Calculate, p x and P y The change of the horizontal vector can be obtained by Calculation, the corresponding DI can be calculated by the following formula:

[0136]

[0137] Step 114 , mapping the difference matrix to obtain a difference image; the difference image is the result of transformation detection.

[0138] Finally calculate DI x and DI y The difference image of DI x and DI y Segmentation is performed to obtain a binary change map, which is the result of change detection.

[0139] In the above-mentioned heterogeneous image change detection method based on structural regression fusion, the present application uses a hypergraph to capture the high-order information of the image to obtain a more accurate image structure representation, and calculates the smoothness constraint conditions after fusing the hypergraph of the image before the event and the hypergraph of the image after the event. A fusion regression model is constructed according to the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint conditions, and the changes in the regression model are fused with some constraints to make the changes in the output image more accurate, thereby mutually improving the image regression performance and detection performance, overcoming the defects of the structural regression method caused by structural asymmetry, so that when performing heterogeneous image change detection, a more accurate difference matrix can be obtained, thereby improving the accuracy of heterogeneous image change detection.

[0140] In one embodiment, superpixel segmentation results of the pre-event image and the post-event image are used to construct a K-nearest neighbor graph of the pre-event image and the post-event image, and a weight matrix of the pre-event image and a weight matrix of the post-event image are obtained, including:

[0141] The superpixel segmentation results of the pre-event image are used to construct the K nearest neighbor graph of the pre-event image, and the weight matrix of the pre-event image is obtained as follows:

[0142]

[0143] in, express and The distance between two superpixels, α i >0 is the equilibrium parameter, k i represents the number of neighbor nodes of the i-th superpixel, represents the i-th superpixel in the image before the event, represents the jth superpixel in the image before the event.

[0144] In one embodiment, a hypergraph includes a vertex set, a hyperedge set, and hyperedge weights; a hypergraph of a pre-event image and a hypergraph of a post-event image are fused, and a regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain a smooth constraint of the changed image, including:

[0145] The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smooth constraint of the change image:

[0146]

[0147] in, represents the hyperedge of the fused hypergraph, E f represents the hypergraph hyperedge set after fusion, represents the hyperedge weight, represents a hyperedge, The super-edge degree, represents the j,lth element in the hypergraph incidence matrix H^f, represents the feature vector of the i-th superpixel in the backward-varied image, Represents the feature vector of the i-th superpixel of the forward change image, Δ x Represents the feature matrix of the backward-changing image, Δ y Represents the feature matrix of the forward change image, L f represents the regularized hypergraph Laplacian matrix of the fused hypergraph, T represents the transpose operation, and Tr represents the trace of the matrix.

[0148] In one embodiment, constructing a hypergraph of the pre-event image and a hypergraph of the post-event image based on a weight matrix of the pre-event image and a weight matrix of the post-event image, and calculating a regularized hypergraph Laplacian matrix to obtain a regularized hypergraph Laplacian matrix of the pre-event image and a regularized hypergraph Laplacian matrix of the post-event image includes:

[0149] According to the weight matrix of the pre-event image, the hypergraph of the pre-event image is constructed, and the regularized hypergraph Laplace matrix is calculated. The regularized hypergraph Laplace matrix of the pre-event image is obtained as

[0150]

[0151] in, The i-th hyperedge of the hypergraph representing the image before the event, E t1 represents the hyperedge set, w(e t1 ) represents the hyperedge weight, Represents a hyperedge The super-edge degree, Represents the incidence matrix of the pre-event image hypergraph, Y′ i Represents the feature vector of the i-th superpixel of the forward regression image, Y′ j Represents the feature vector of the jth superpixel of the forward regression image, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, and T represents the transpose operation.

[0152] In one embodiment, a forward regression model is constructed using a regularized hypergraph Laplacian matrix of a pre-event image, comprising:

[0153] The forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image:

[0154]

[0155] Among them, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, Y represents the feature matrix of the post-event image, Y′ represents the feature matrix of the forward regression image, and λ represents the sparsity penalty parameter.

[0156] In one embodiment, a backward regression model is constructed using a regularized hypergraph Laplacian matrix of a post-event image, comprising:

[0157] The backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image:

[0158]

[0159] Among them, L t2 represents the regularized hypergraph Laplacian matrix of the post-event image, X represents the feature matrix of the pre-event image, and X′ represents the feature matrix of the backward regression image.

[0160] In one embodiment, the change alignment constraint is

[0161]

[0162] Among them, || ||2 means Norm, N S represents the total number of superpixels, and φ represents the variation alignment constraint function.

[0163] In one embodiment, a fusion regression model is constructed based on a forward regression model, a backward regression model, a smooth constraint, and a change alignment constraint, including:

[0164] According to the forward regression model, backward regression model, smooth constraint and change alignment constraint conditions, the fusion regression model is constructed as follows:

[0165]

[0166] Among them, β represents the change smooth penalty parameter, η represents the change alignment penalty parameter, st represents the constraint condition, |||| 2,1 express norm.

[0167] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0168] In one embodiment, Figure 2 As shown, a heterogeneous image change detection device based on structural regression fusion is provided, comprising: a superpixel segmentation module 202, a weight matrix construction module 204, a hypergraph Laplacian matrix calculation module 206, a regression model construction module 208, a hypergraph fusion module 210, a fusion regression model construction and solution module 212, and a mapping module 214, wherein:

[0169] The superpixel segmentation module 202 is used to obtain a pre-event image and a post-event image to be compared; pre-process the pre-event image and the post-event image according to a superpixel segmentation method of a Gaussian mixture model to obtain superpixel segmentation results of the pre-event image and the post-event image;

[0170] A weight matrix construction module 204 is used to construct a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtain a weight matrix of the pre-event image and a weight matrix of the post-event image;

[0171] A hypergraph Laplacian matrix calculation module 206 is configured to construct a hypergraph of the pre-event image and a hypergraph of the post-event image based on the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculate a regularized hypergraph Laplacian matrix to obtain a regularized hypergraph Laplacian matrix of the pre-event image and a regularized hypergraph Laplacian matrix of the post-event image;

[0172] The regression model construction module 208 is used to construct a forward regression model using the regularized hypergraph Laplace matrix of the pre-event image to decompose the post-event image into a forward regression image and a forward change image; and to construct a backward regression model using the regularized hypergraph Laplace matrix of the post-event image to decompose the pre-event image into a backward regression image and a backward change image;

[0173] The hypergraph fusion module 210 is used to fuse the hypergraph of the pre-event image and the hypergraph of the post-event image, and calculate the regularized hypergraph Laplacian matrix of the fused hypergraph to obtain the smoothness constraint of the changed image;

[0174] A fusion regression model construction and solution module 212 is used to construct a fusion regression model based on the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint, and solve the fusion regression model using the alternating direction multiplication method to obtain a difference matrix between the pre-event image and the post-event image;

[0175] The mapping module 214 is used to map the difference matrix to obtain a difference image; the difference image is the result of transformation detection.

[0176] In one embodiment, the weight matrix construction module 204 is further configured to construct a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtain a weight matrix of the pre-event image and a weight matrix of the post-event image, including:

[0177] The superpixel segmentation results of the pre-event image are used to construct the K nearest neighbor graph of the pre-event image, and the weight matrix of the pre-event image is obtained as follows:

[0178]

[0179] in, express and The distance between two superpixels, α i >0 is the equilibrium parameter, k i represents the number of neighbor nodes of the i-th superpixel, represents the i-th superpixel in the image before the event, represents the jth superpixel in the image before the event.

[0180] In one embodiment, the hypergraph includes a vertex set, a hyperedge set, and hyperedge weights; the hypergraph fusion module 210 is further configured to fuse the hypergraph of the pre-event image and the hypergraph of the post-event image, and calculate a regularized hypergraph Laplacian matrix of the fused hypergraph to obtain smooth constraints of the changed image, including:

[0181] The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smooth constraint of the change image:

[0182]

[0183] in, represents the hyperedge of the fused hypergraph, E f represents the hypergraph hyperedge set after fusion, represents the hyperedge weight, Represents a hyperedge The super-edge degree, represents the j,lth element in the hypergraph incidence matrix H^f, represents the feature vector of the i-th superpixel in the backward-varied image, Represents the feature vector of the i-th superpixel of the forward change image, Δ x Represents the feature matrix of the backward-changing image, Δ y Represents the feature matrix of the forward change image, L f represents the regularized hypergraph Laplacian matrix of the fused hypergraph, T represents the transpose operation, and Tr represents the trace of the matrix.

[0184] In one embodiment, the hypergraph Laplacian matrix calculation module 206 is further configured to construct a hypergraph of the pre-event image and a hypergraph of the post-event image based on the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculate a regularized hypergraph Laplacian matrix to obtain a regularized hypergraph Laplacian matrix of the pre-event image and a regularized hypergraph Laplacian matrix of the post-event image, including:

[0185] According to the weight matrix of the pre-event image, the hypergraph of the pre-event image is constructed, and the regularized hypergraph Laplace matrix is calculated. The regularized hypergraph Laplace matrix of the pre-event image is obtained as

[0186]

[0187] in, The i-th hyperedge of the hypergraph representing the image before the event, E t1 represents the hyperedge set, w(e t1 ) represents the hyperedge weight, Represents a hyperedge The super-edge degree, Represents the incidence matrix of the pre-event image hypergraph, Y′ i Represents the feature vector of the i-th superpixel of the forward regression image, Y′ j Represents the feature vector of the jth superpixel of the forward regression image, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, and T represents the transpose operation.

[0188] In one embodiment, the regression model building module 208 is further configured to build a forward regression model using a regularized hypergraph Laplacian matrix of the pre-event image, including:

[0189] The forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image:

[0190]

[0191] Among them, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, Y represents the feature matrix of the post-event image, Y′ represents the feature matrix of the forward regression image, and λ represents the sparsity penalty parameter.

[0192] In one embodiment, the regression model building module 208 is further configured to build a backward regression model using a regularized hypergraph Laplacian matrix of the post-event image, including:

[0193] The backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image:

[0194]

[0195] Among them, L t2 represents the regularized hypergraph Laplacian matrix of the post-event image, X represents the feature matrix of the pre-event image, and X′ represents the feature matrix of the backward regression image.

[0196] In one embodiment, the change alignment constraint is

[0197]

[0198] Among them, ||||2 means Norm, N S represents the total number of superpixels, and φ represents the variation alignment constraint function.

[0199] In one embodiment, the fusion regression model construction and solution module 212 is further configured to construct a fusion regression model based on the forward regression model, the backward regression model, the smoothness constraint, and the variation alignment constraint, including:

[0200] According to the forward regression model, backward regression model, smooth constraint and change alignment constraint conditions, the fusion regression model is constructed as follows:

[0201]

[0202] Among them, β represents the change smooth penalty parameter, η represents the change alignment penalty parameter, st represents the constraint condition, |||| 2,1 express norm.

[0203] Regarding the specific definition of a heterogeneous image change detection device based on structural regression fusion, please refer to the definition of a heterogeneous image change detection method based on structural regression fusion above, and will not be repeated here. Each module in the above-mentioned heterogeneous image change detection device based on structural regression fusion can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

[0204] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a heterogeneous image change detection method based on structural regression fusion is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0205] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0206] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0207] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0208] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A heterogeneous image change detection method based on structural regression fusion, characterized in that: The method comprises: obtaining a pre-event image and a post-event image to be compared; Preprocessing the pre-event image and the post-event image according to a superpixel segmentation method of a Gaussian mixture model to obtain superpixel segmentation results of the pre-event image and the post-event image; Constructing a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtaining a weight matrix of the pre-event image and a weight matrix of the post-event image; Constructing a hypergraph of the pre-event image and a hypergraph of the post-event image according to the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculating a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image; A forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image, and the post-event image is decomposed into a forward regression image and a forward change image; a backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image, and the pre-event image is decomposed into a backward regression image and a backward change image; Fusing the hypergraph of the pre-event image and the hypergraph of the post-event image, and calculating the regularized hypergraph Laplacian matrix of the fused hypergraph to obtain a smooth constraint of the changed image; A fusion regression model is constructed according to the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint, and the fusion regression model is solved using an alternating direction multiplier method to obtain a difference matrix between a pre-event image and a post-event image; Mapping is performed on the difference matrix to obtain a difference image; the difference image is the result of transformation detection.

2. The method according to claim 1, characterized in that The K-nearest neighbor graphs of the pre-event image and the post-event image are constructed using the superpixel segmentation results of the pre-event image and the post-event image, and a weight matrix of the pre-event image and a weight matrix of the post-event image are obtained, including: The K nearest neighbor graph of the image before the event is constructed using the superpixel segmentation result of the image before the event, and the weight matrix of the image before the event is obtained as in, express and The distance between two superpixels, α i >0 is the equilibrium parameter, k i represents the number of neighbor nodes of the i-th superpixel, represents the i-th superpixel in the image before the event, represents the jth superpixel in the image before the event.

3. The method according to claim 2, characterized in that The hypergraph includes a vertex set, a hyperedge set, and a hyperedge weight; the hypergraph of the pre-event image and the hypergraph of the post-event image are fused, and a regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain a smooth constraint of the changed image, including: The hypergraph of the image before the event and the hypergraph of the image after the event are fused, and the regularized hypergraph Laplacian matrix of the fused hypergraph is calculated to obtain the smooth constraint of the change image: in, represents the hyperedge of the fused hypergraph, E f represents the hypergraph hyperedge set after fusion, represents the hyperedge weight, Represents a hyperedge The super-edge degree, represents the j,lth element in the hypergraph incidence matrix H^f, represents the feature vector of the i-th superpixel in the backward-varied image, Represents the feature vector of the i-th superpixel of the forward change image, Δ x Represents the feature matrix of the backward-changing image, Δ y Represents the feature matrix of the forward change image, L f represents the regularized hypergraph Laplacian matrix of the fused hypergraph, T represents the transpose operation, and Tr represents the trace of the matrix.

4. The method according to claim 3, characterized in that Constructing a hypergraph of the pre-event image and a hypergraph of the post-event image according to the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculating a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image, including: The hypergraph of the pre-event image is constructed according to the weight matrix of the pre-event image, and the regularized hypergraph Laplace matrix is calculated to obtain the regularized hypergraph Laplace matrix of the pre-event image: in, The i-th hyperedge of the hypergraph representing the image before the event, E t1 represents the hyperedge set, w(e t1 ) represents the hyperedge weight, represents a hyperedge, The super-edge degree, Represents the incidence matrix of the pre-event image hypergraph, Y′ i Represents the feature vector of the i-th superpixel of the forward regression image, Y′ j Represents the feature vector of the jth superpixel of the forward regression image, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, and T represents the transpose operation.

5. The method according to claim 4, characterized in that A forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image, comprising: The forward regression model is constructed using the regularized hypergraph Laplacian matrix of the pre-event image: Among them, L t1 represents the regularized hypergraph Laplacian matrix of the pre-event image, Y represents the feature matrix of the post-event image, Y′ represents the feature matrix of the forward regression image, and λ represents the sparsity penalty parameter.

6. The method according to claim 4, characterized in that A backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image, comprising: The backward regression model is constructed using the regularized hypergraph Laplacian matrix of the post-event image: Among them, L t2 represents the regularized hypergraph Laplacian matrix of the post-event image, X represents the feature matrix of the pre-event image, and X′ represents the feature matrix of the backward regression image.

7. The method according to claim 6, characterized in that The change alignment constraint is Among them, || ||2 means Norm, N s represents the total number of superpixels, and φ represents the variation alignment constraint function.

8. The method according to claim 7, characterized in that A fusion regression model is constructed according to the forward regression model, the backward regression model, the smoothness constraint and the change alignment constraint, including: According to the forward regression model, backward regression model, smooth constraint and change alignment constraint conditions, a fusion regression model is constructed as follows: Among them, β represents the change smooth penalty parameter, η represents the change alignment penalty parameter, st represents the constraint condition, || || 2,1 express norm.

9. A heterogeneous image change detection device based on structural regression fusion, characterized in that: The device comprises: a superpixel segmentation module for obtaining a pre-event image and a post-event image to be compared; preprocessing the pre-event image and the post-event image according to a superpixel segmentation method of a Gaussian mixture model to obtain superpixel segmentation results of the pre-event image and the post-event image; A weight matrix construction module is used to construct a K-nearest neighbor graph of the pre-event image and the post-event image using the superpixel segmentation results of the pre-event image and the post-event image, and obtain a weight matrix of the pre-event image and a weight matrix of the post-event image; a hypergraph Laplace matrix calculation module, configured to construct a hypergraph of the pre-event image and a hypergraph of the post-event image based on the weight matrix of the pre-event image and the weight matrix of the post-event image, and calculate a regularized hypergraph Laplace matrix to obtain a regularized hypergraph Laplace matrix of the pre-event image and a regularized hypergraph Laplace matrix of the post-event image; A regression model construction module is used to construct a forward regression model using the regularized hypergraph Laplace matrix of the pre-event image, and decompose the post-event image into a forward regression image and a forward change image; and to construct a backward regression model using the regularized hypergraph Laplace matrix of the post-event image, and decompose the pre-event image into a backward regression image and a backward change image; A hypergraph fusion module, configured to fuse the hypergraph of the pre-event image and the hypergraph of the post-event image, and calculate a regularized hypergraph Laplacian matrix of the fused hypergraph to obtain a smooth constraint of the changed image; a fusion regression model construction and solution module, configured to construct a fusion regression model based on the forward regression model, the backward regression model, the smoothness constraint, and the change alignment constraint, and solve the fusion regression model using an alternating direction multiplier method to obtain a difference matrix between the pre-event image and the post-event image; The mapping module is used to map the difference matrix to obtain a difference image; the difference image is the result of transformation detection.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images