A Multidimensional Visual Data Repair Method Based on Weighted Hybrid Graph Laplacian

Abstract

The present invention discloses a multi-dimensional visual data repair method based on weighted hybrid graph Laplacian, which includes calculating the similarity between each pixel and its four surrounding adjacent pixels, constructing a first-order local Laplacian operator, calculating the similarity between a single pixel and non-local adjacent pixels, constructing a first-order non-local Laplacian operator, jointly adding the first-order local Laplacian operator and the first-order non-local Laplacian operator to obtain a first-order Laplacian operator, generalizing the first-order Laplacian operator to a second-order form to obtain a second-order Laplacian operator, respectively performing weighted joint addition on the first-order Laplacian operator and the second-order Laplacian operator to obtain a hybrid graph Laplacian operator, and using the hybrid graph Laplacian operator to construct a repair model for multi-dimensional visual data. The present invention repairs missing data in multi-dimensional visual data by improving the traditional graph Laplacian regular term.

Description

Technical Field

[0001] The present invention belongs to the field of multi-dimensional visual data, and particularly relates to a multi-dimensional visual data repair method based on weighted hybrid graph Laplacian. Background Art

[0002] The integrity of multi-dimensional visual data is the basis for its wide use. However, due to limitations of the actual environment, such as poor imaging conditions, malfunction of acquisition devices, and bandwidth limitations, natural visual data may be incomplete or severely damaged. The task of recovering potential complete data from such incomplete observed data can be classified as a multi-dimensional visual data repair problem, and its key work depends on correctly describing the internal connections hidden in multi-dimensional data, that is, mining the prior information in the data, such as smoothness, low rankness, non-local similarity, and so on.

[0003] Existing research has shown that using only a single type of prior knowledge is not sufficient to cover all data information. For example, the low-rank property mainly focuses on the global structure and rarely pays attention to local or non-local information. Multiple existing graph theory-based regular terms only focus on non-local features. Both local structures and long-term correlations are lacking in consideration.

[0004] In order to inherit the advantages of previous models and make up for the corresponding deficiencies, this study combines multiple attributes hidden in visual data and proposes a multi-dimensional visual data repair method based on weighted hybrid graph Laplacian. Compared with previous techniques, the visual data elements reconstructed by the technical method proposed in this paper have richer correlations inside and are closer to the underlying data structure. Summary of the Invention

[0005] The purpose of the present invention is to propose a multi-dimensional visual data repair method based on weighted hybrid graph Laplacian for the problem that multi-dimensional visual data repair in the prior art only focuses on non-local features and lacks consideration of local structures and long-term correlations.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A multi-dimensional visual data repair method based on weighted hybrid graph Laplacian proposed by the present invention, the multi-dimensional visual data repair method based on weighted hybrid graph Laplacian includes:

[0008] Obtain the observed multi-dimensional visual data, calculate the similarity between a single pixel point and its four surrounding adjacent pixel points, construct a first-order local Laplacian operator, calculate the similarity between a single pixel point and non-local adjacent pixel points, construct a first-order non-local Laplacian operator, and obtain a first-order Laplacian operator by jointly adding the first-order local Laplacian operator and the first-order non-local Laplacian operator;

[0009] The first-order Laplacian operator is decomposed into the sum of a labeled set and an unlabeled set, the labeled set is weighted to obtain the first-order weighted graph Laplacian operator, the first-order Laplacian operator is generalized to the second-order form to obtain the second-order Laplacian operator, the second-order Laplacian operator is decomposed into the sum of a labeled set and an unlabeled set, the labeled set is weighted to obtain the second-order weighted graph Laplacian operator, and the first-order weighted graph Laplacian operator and the second-order weighted graph Laplacian operator are jointly added to obtain the mixed graph Laplacian operator;

[0010] A repair model for multi-dimensional visual data is constructed using the mixed graph Laplacian operator, optimized using the variational method, and iteratively solved using the generalized minimal residual method until convergence to obtain the repaired multi-dimensional visual data.

[0011] Preferably, calculating the similarity between a single pixel point and its four surrounding adjacent pixel points to construct a first-order local Laplacian operator includes:

[0012] For each point v N1×N2×B in the observed multi-dimensional visual data b ∈ R x , the similarity with its four surrounding adjacent points is calculated using the following formula:

[0013]

[0014] where N1, N2, and B are the length, width, and number of spectral bands of the spectral image respectively, w l (x, y) is the similarity between point v x and the local adjacent point v y , τ l is a parameter, all w l (x, y) values are incorporated into a matrix denoted as W l , and the first-order local Laplacian operator is obtained through the formula Δ l = D l - W l , where D l = diag[d1, d2,..., d N , diag represents the diagonalization operation, x ∈ [1, N], N = N1 × N2;

[0015] Calculating the similarity between a single pixel point and non-local adjacent pixel points to construct a first-order non-local Laplacian operator includes:

[0016] For each point v x , the approximate nearest neighbor search algorithm ANN and the kd-tree method are used to search for the similarity between non-local adjacent points and v x , and the similarity is calculated using the following formula:

[0017]

[0018] where w nl (x, y) is the similarity between point v x and non-local neighboring point v y , τ p and τ v are two parameters, p x and p y represent the positions of point v x and v y in the image space respectively. Similarly, all w nl (x, y) values will be incorporated into a matrix, denoted as W nl . Through the formula Δ nl = D nl - W nl the first-order non-local Laplacian operator can be obtained. D nl = diag[d′1, d′2,..., d′ N , x ∈ [1, N];

[0019] The first-order Laplacian operator is obtained by jointly adding the first-order local Laplacian operator and the first-order non-local Laplacian operator: where, is denoted as For the value of a certain point (x, y) on , it can be expressed as

[0020] Preferably, the first-order Laplacian operator is decomposed into the sum of a labeled set and an unlabeled set, and the labeled set is weighted to obtain the first-order weighted graph Laplacian operator, including:

[0021] The expression of the first-order weighted graph Laplacian operator with respect to the unknown hyperspectral image u to be reconstructed:

[0022]

[0023] where, u ∣S represents the known pixel points in u, u ∣P\S represents the missing pixel points in P except for the known pixel points S in u, and γ is a parameter,

[0024]

[0025] and:

[0026]

[0027] Generalizing the first-order Laplacian operator to the second-order form to obtain the second-order Laplacian operator includes: when γ equals 1, The formula for a certain point x in u can be expressed as:

[0028]

[0029] where g and k respectively represent certain adjacent points of x and y, and respectively represent the sets of adjacent points of x and y. According to Equation (6), satisfies:

[0030]

[0031] The second-order Laplacian operator is disassembled into the sum of a labeled set and an unlabeled set, and the labeled set is weighted to obtain the second-order weighted graph Laplacian operator:

[0032]

[0033] The first-order weighted graph Laplacian operator and the second-order weighted graph Laplacian operator are jointly added to obtain the hybrid graph Laplacian operator:

[0034]

[0035] where β is a parameter used to balance the first-order weighted graph Laplacian term and the second-order weighted graph Laplacian term.

[0036] Preferably, constructing a repair model for multi-dimensional visual data using the hybrid graph Laplacian operator includes:

[0037] Incorporating the hybrid graph Laplacian operator into the repair model of multi-dimensional visual data to obtain the following formula to be optimized:

[0038]

[0039] u(s)=b(s) means assigning the known pixel values in b to u. Using Equations (4), (7), and (9) to expand the above formula, we can obtain:

[0040]

[0041] Combining the third and fourth terms of the above formula, (11) can be re-expressed as:

[0042]

[0043] where, E = diag{e1,...,e n}, where when u y ∈P\S, e y =1, uy When it belongs to S, e y = γ; and:

[0044]

[0045] Thus, Equation (7) can be re-expressed as:

[0046]

[0047] where η is a penalty parameter, is a mapping operator, which can be specifically expressed as:

[0048]

[0049] By optimizing through the variational method, solving (14) can be transformed to obtain:

[0050]

[0051] Using to represent the last two terms of the above formula, and using the symmetry of the graph Laplacian operator, the first term of the above formula can be simply transformed to obtain:

[0052]

[0053] Adding the first term and the third term, we can get:

[0054]

[0055] According to the solution formula of the above formula can be finally updated to:

[0056]

[0057] where μ = γ - 1, and Equation (19) is solved by the generalized minimal residual method iteratively to obtain the finally reconstructed image, and finally repair the multi-dimensional visual data.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows: By using any pixel point in the multi-dimensional visual data to construct a local similarity matrix and a non-local similarity matrix, and constructing a local and non-local joint graph Laplacian operator, through the improvement of the traditional graph Laplacian regularization term, the problem that the multi-dimensional visual data repair in the prior art only focuses on non-local features and lacks consideration of local structures and long-term correlations is solved. Furthermore, the internal elements of the reconstructed visual data have richer correlations and are closer to the underlying data structure. The weighted method is used to increase the continuity of its reconstruction function. The experimental results on multiple data sets show the superiority of the proposed technical solution. Description of the Drawings

[0059] Figure 1 This is the flowchart of the multi - dimensional visual data repair method of the present invention;

[0060] Figure 2 This is the cloud - removing and repairing diagram of the present invention based on weighted hybrid graph Laplacian regularization on a real cloud - covered multi - spectral image with a sampling rate of 10%;

[0061] Figure 3 This is the reconstruction effect diagram of the present invention based on weighted hybrid graph Laplacian regularization on a hyperspectral data set with a loss rate of 90%;

[0062] Figure 4 This is the reconstruction effect diagram of the present invention based on weighted hybrid graph Laplacian regularization on video data with a loss rate of 80%. Detailed implementation manners

[0063] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application herein are only for the purpose of describing specific embodiments, and are not intended to limit this application.

[0065] As Figures 1-4 shown, a multi - dimensional visual data repair method based on weighted hybrid graph Laplacian includes:

[0066] Step 1: Obtain the observed multi - dimensional visual data, calculate the similarity between a single pixel point and its four adjacent pixel points around it, construct a first - order local Laplacian operator, calculate the similarity between a single pixel point and non - local adjacent pixel points, construct a first - order non - local Laplacian operator, and obtain a first - order Laplacian operator by jointly adding the first - order local Laplacian operator and the first - order non - local Laplacian operator.

[0067] Specifically, it includes: Given the known incomplete observation map of the unknown hyperspectral image where N1, N2, and B are the length, width, and number of spectra of the spectral image respectively. Let P = {p1, p2,..., p n} be all the pixel points in b, p1 to p n represent pixel values, and n = N1×N2×B. S = {s1, s2,..., s m}\ is a subset of P, where s1 to s m are known true pixel values, m < n. For any point x ∈ P, a point v of size 1×1×B in a multi-dimensional space can be defined x . For each point v x , the following formula is used to calculate the similarity with its four surrounding adjacent points:

[0068]

[0069] where w l (x,y) is the similarity between the point v x and the local adjacent point v y , τ l is a parameter. Incorporate all the w l (x,y) values into a matrix, denoted as W l . Use four local similar image patches to construct a local similarity matrix. Through the formula Δ l = D l - W l the first-order local Laplacian operator can be obtained, where D l = diag[d1, d2,..., d N , diag is the diagonalization operation, x ∈ [1, N], N = N1×N2. Similarly, for each point v x , use the approximate nearest neighbor search algorithm ANN and the kd-tree method to search for the similarity between the non-local adjacent point and v x , and calculate the similarity using the following formula:

[0070]

[0071] where w nl (x,y) is the similarity between the point v x and the non-local adjacent point v y , τ p and τ v are two parameters, p x and p y represent the positions of the points v x and v y in the image space respectively. Similarly, incorporate all the w nl (x,y) values into a matrix, denoted as W nl . Use ten non-local similar image patches to construct a non-local similarity matrix. Through the formula Δ nl = D nl - W nl the first-order non-local Laplacian operator can be obtained, D nl= diag[d′1, d′2, ..., d′ N , x ∈ [1, N]. From this, the local non-local joint first-order Laplacian operator can be obtained for modeling the unknown hyperspectral image u to be reconstructed; where, is expressed as For the value of a certain point (x, y) on

[0072] The non-local region is the position that is not adjacent to v in terms of spatial position; the local region is the point v x adjacent, and the up, down, left, and right of the spatial position of the point v x .

[0073] Step 2: Decompose the first-order Laplacian operator into the sum of the labeled set and the unlabeled set, weight the labeled set to obtain the first-order weighted graph Laplacian operator, generalize the first-order Laplacian operator to the second-order form to obtain the second-order Laplacian operator, decompose the second-order Laplacian operator into the sum of the labeled set and the unlabeled set, weight the labeled set to obtain the second-order weighted graph Laplacian operator, and jointly add the first-order weighted graph Laplacian operator and the second-order weighted graph Laplacian operator to obtain the mixed graph Laplacian operator.

[0074] Specifically, it includes: decomposing the obtained first-order Laplacian operator into the sum of the labeled set and the unlabeled set, weighting the labeled set to obtain the first-order weighted graph Laplacian operator with respect to b, and applying it to the unknown hyperspectral image u to be reconstructed. Then, the expression of the first-order weighted graph Laplacian operator with respect to the unknown hyperspectral image u to be reconstructed can be expressed as:

[0075]

[0076] where, u ∣S represents the known pixel points in u, and u ∣P\S represents the missing pixel points in P except for the known pixel points S in u. γ is a parameter,

[0077]

[0078] And:

[0079]

[0080] Next, generalize the WGL to the second-order form to enhance the constraint on the smoothness of visual data. To better illustrate the second-order graph Laplacian operator, first introduce the second-order form of When γ is equal to 1, The formula for a point x in u can be expressed as:

[0081]

[0082] where g and k represent certain adjacent points of x and y respectively, and represent the sets of adjacent points of x and y respectively. According to Equation (6), satisfies:

[0083]

[0084] Similar to Equation (3), decompose into a labeled set and an unlabeled set, and its second-order weighted graph Laplacian is expressed as:

[0085]

[0086] Combine the first-order weighted graph Laplacian and the second-order weighted graph Laplacian to obtain a regularization term closer to the underlying properties of visual data, which is called the hybrid graph Laplacian regularization term, that is, the hybrid graph Laplacian operator:

[0087]

[0088] where β is a parameter used to balance the first-order weighted graph Laplacian term and the second-order weighted graph Laplacian term.

[0089] Step 3: Construct a restoration model for multi-dimensional visual data using the hybrid graph Laplacian operator, optimize it using the variational method, and iteratively solve it using the Generalized Minimal Residual method (GMRES) until convergence to obtain the restored multi-dimensional visual data.

[0090] Specifically, it includes: incorporating the obtained hybrid graph Laplacian operator into the restoration model of multi-dimensional visual data to obtain the following formula to be optimized:

[0091]

[0092] u(s)=b(s) means assigning the known pixel values in b to u. u represents the restored multi-dimensional visual data. Using Equations (4), (7), and (9) to expand the above formula, we can get:

[0093]

[0094] Combining the 3rd and 4th terms of the above formula, we can re-express (11) as:

[0095]

[0096] where \(E = \text{diag}\{e_1,\cdots,e\}\), when \(u\in P\setminus S\), \(e = 1\); when \(u\in S\), \(e=\gamma\). And n}, y when \(u y \in P\setminus S\), \(e y = 1\); when \(u y \in S\), \(e

[0097]

[0098] Thus, equation (7) can be rewritten as:

[0099]

[0100] where \(\eta\) is a penalty parameter, is a mapping operator, which can be specifically expressed as:

[0101]

[0102] By the standard variational method, the solution of equation (14) can be transformed into:

[0103]

[0104] For simplicity, use to represent the last two terms of the above equation. Using the symmetry of the graph Laplacian operator, the first term of the above equation can be simply transformed, and we can get:

[0105]

[0106] Adding the first term and the third term, we can get:

[0107]

[0108] According to the solution formula of the above equation can be finally updated to:

[0109]

[0110] where \(\mu=\gamma - 1\). Rearranging equation (19) gives:

[0111]

[0112] where, for each row of the matrix set all elements that do not belong to \(S\) to 0, we can get Use GMRES iteration to solve equation (20) to obtain the finally reconstructed image.

[0113] The finally reconstructed image \(u\) can be expressed as:

[0114]

[0115] The generalized minimal residual method (GMRES) belongs to the prior art in this technical field, and the specific process will not be elaborated here. Then, multi-dimensional visual data is obtained.

[0116] The above-described embodiments only represent relatively specific and detailed embodiments of the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A multi-dimensional visual data repair method based on weighted hybrid graph Laplacian, characterized in that, The multi-dimensional visual data repair method based on weighted mixed graph Laplacian includes: Obtain the observed multi-dimensional visual data, calculate the similarity between a single pixel point and its surrounding four adjacent pixel points, construct a first-order local Laplacian operator, calculate the similarity between a single pixel point and non-local adjacent pixel points, construct a first-order non-local Laplacian operator, and obtain a first-order Laplacian operator by jointly adding the first-order local Laplacian operator and the first-order non-local Laplacian operator; Decompose the first-order Laplacian operator into the sum of a marked set and an unmarked set, weight the marked set to obtain a first-order weighted graph Laplacian operator, generalize the first-order Laplacian operator to a second-order form to obtain a second-order Laplacian operator, decompose the second-order Laplacian operator into the sum of a marked set and an unmarked set, weight the marked set to obtain a second-order weighted graph Laplacian operator, and jointly add the first-order weighted graph Laplacian operator and the second-order weighted graph Laplacian operator to obtain a mixed graph Laplacian operator; Construct a repair model for multi-dimensional visual data using the mixed graph Laplacian operator, optimize it using the variational method, and iteratively solve it using the generalized minimal residual method until it is stable to obtain the repaired multi-dimensional visual data; Among them, the calculation of the similarity between a single pixel point and its surrounding four adjacent pixel points and the construction of a first-order local Laplacian operator include: For each point v in the observed multi-dimensional visual data b ∈ R N1×N2×B , the similarity is calculated for its four surrounding adjacent points using the following formula: x ​ where N1, N2, and B are the length, width, and number of spectra of the spectral image, respectively, and w l (x, y) is the similarity between point v x and its local neighboring point v y , and τ l is a parameter. Incorporating all the w l (x, y) values into a matrix, denoted as W l , and through the formula Δ l = D l - W l , the first-order local Laplacian operator can be obtained, where D l = diag[d1, d2,..., d N , diag represents the diagonalization operation, N = N1 × N2; The calculation of the similarity between a single pixel point and non-local adjacent pixel points and the construction of a first-order non-local Laplacian operator include: For each point v x , use the approximate nearest neighbor search algorithm ANN and the kd-tree method to search for non-local neighboring points and the similarity of v x , and calculate the similarity using the following formula: where w nl (x, y) is the point v x and the non-local neighboring point v y the similarity between, τ p and τ v are two parameters, p x and p y represent the point v x and v y in the image space position, similarly, all the w nl (x, y) values will be incorporated into a matrix, denoted as W nl , through the formula Δ nl = D nl - W nl the first-order non-local Laplacian operator can be obtained, D nl = diag[d′1, d′2,..., d′ N , The first-order Laplacian operator is obtained by adding the first-order local Laplacian operator and the first-order non-local Laplacian operator: Among them, It is expressed as For The value of a certain point (x, y) on can be expressed as Among them, the decomposition of the first-order Laplacian operator into the sum of a marked set and an unmarked set and the weighting of the marked set to obtain a first-order weighted graph Laplacian operator include: The formula of the first-order weighted graph Laplacian operator with respect to the unknown hyperspectral image u to be reconstructed: Among them, it is defined that P = {p1, p2,..., p n} is all the pixel points in b, S = {s1, s2,..., s m} is a subset of P, and s1 to s m are known true pixel values, u ∣S represents the known pixel points in u, and u ∣P\S represents the missing pixel points in P except for the known pixel points S in u. γ is a parameter. And: Generalizing the first-order Laplace operator to the second-order form to obtain the second-order Laplace operator includes: when γ equals 1, The formula for a certain point x in u can be expressed as: where g and k respectively represent certain adjacent nodes of x and y, and respectively represent the sets of adjacent nodes of x and y; according to Equation (6), satisfies: The decomposition of the second-order Laplacian operator into the sum of a marked set and an unmarked set and the weighting of the marked set to obtain a second-order weighted graph Laplacian operator: The joint addition of the first-order weighted graph Laplacian operator and the second-order weighted graph Laplacian operator to obtain a mixed graph Laplacian operator: Where β is a parameter used to balance the first-order weighted graph Laplacian term and the second-order weighted graph Laplacian term; Among them, the construction of a repair model for multi-dimensional visual data using the mixed graph Laplacian operator includes: Incorporate the mixed graph Laplacian operator into the repair model of multi-dimensional visual data to obtain the following formula to be optimized: u(s)=b(s) means assigning the known pixel values in b to u. Using Equation (4), Equation (7), and Equation (9) to expand the above formula, we can get: Combining the third and fourth terms of the above formula, (11) can be re-expressed as: Among them, E = diag{e1,..., e n}, where when u y ∈P\S, e y = 1, when u y ∈S, e y = γ; and: Therefore, Equation (7) can be re-expressed as: where η is a penalty parameter, is a mapping operator, which can be specifically expressed as: Through variational method optimization, the solution of (14) can be transformed to obtain: Use to represent the last two terms of the above formula. Utilizing the symmetry of the graph Laplacian, the first term of the above formula can be simply transformed, and we can obtain: Adding the first term and the third term, we can get: According to The solution formula of the above equation can be finally updated to: Where μ = γ - 1, and the final reconstructed image is obtained by iteratively solving Equation (19) using the generalized minimal residual method, and finally the multi-dimensional visual data is repaired.

Images