Image restoration method based on adaptive weighted tensor completion

Abstract

The invention provides an image restoration method based on adaptive weighted tensor completion, and relates to the technical field of image processing and application, and the method comprises the steps: obtaining to-be-restored image data, and carrying out the tensor of the to-be-restored image data, and obtaining input tensor data; constructing a tensor completion model based on an adaptive weighted tensor nuclear norm; wherein a weight matrix in the tensor completion model can be adaptively updated along with input tensor data; and based on an alternating direction multiplier method or an approximate singular value decomposition method based on tensor QR decomposition, solving the tensor completion model, and outputting restored tensor data to realize image restoration. According to the scheme, the image restoration quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and application, and particularly relates to an image restoration method based on adaptive weighted tensor completion. BACKGROUND

[0002] Due to damage of acquisition equipment, interference of noise and difficulty of data acquisition, the collected data is often incomplete or severely damaged. A popular problem of data completion is low-rank tensor completion (LRTC):

[0003]

[0004] where rank(X) represents the rank of tensor X, is an observed variable with missing values, P Ω is an orthogonal projection on the tensor dimension, and its value is 0 outside Ω. It has been widely applied, including but not limited to visual data recovery, super-resolution, big data analysis, anomaly detection, traffic data analysis, neuroimaging, recommendation system and knowledge graph completion.

[0005] As a typical representative of LRTC, the rank minimization model realizes image data completion by minimizing the rank of the tensor, which can reduce the noise and errors that may occur in the completion process while preserving the original data structure. Of course, different tensor ranks lead to different methods of LRTC models, including CANDECOMP / PARAFAC (CP) rank and Tucker rank, which correspond to CP decomposition and Tucker decomposition, respectively. Recently, the tensor tube rank machine convex relaxation alternative tensor nuclear norm (TNN) method based on tensor-tensor product and tensor singular value decomposition more effectively preserves the tensor structure and achieves remarkable results. However, although the tensor nuclear norm method is effective for high-dimensional data completion, it relies on singular value decomposition to obtain the singular values of the incomplete tensor, and it ignores the differences between these singular values based on equal punishment for each singular value of the low-rank tensor, which further leads to the fact that the nuclear norm method often performs poorly when applied to high-dimensional real data sets, and the image data restoration effect is poor. SUMMARY

[0006] Therefore, in view of the above problems, it is necessary to provide an image restoration method based on adaptive weighted tensor completion to improve the quality of image restoration.

[0007] In a first aspect, the present application provides an image restoration method based on adaptive weighted tensor completion, comprising:

[0008] acquiring image data to be restored, and tensorizing the image data to obtain input tensor data;

[0009] constructing a tensor completion model based on an adaptive weighted tensor kernel norm, wherein a weight matrix in the tensor completion model can be updated adaptively with input tensor data;

[0010] solving the tensor completion model based on an alternating direction multiplier method or an approximate singular value decomposition method based on tensor QR decomposition, outputting recovered tensor data, and realizing image inpainting.

[0011] Preferably, the construction of the tensor completion model based on the adaptive weighted tensor kernel norm comprises:

[0012] The construction of the adaptive weighted tensor kernel norm completion model is:

[0013]

[0014] wherein W ki represents a weighting coefficient, represents a tensor a result of discrete Fourier transform along a third dimension n3, is a kth frontal slice of represents a ith singular value of d = min(n1, n2); P Ω is an orthogonal projection on the tensor dimension, and the value is 0 outside Ω, is an observed variable with missing values, P Ω (X) = P Ω (M) is a constraint condition; ψ is a non-convex conjugate function of ρ γ (x) = γ(1-exp(-|x| / γ)), which satisfies and the minimum point of the above completion model is obtained when w = exp(-|x| / γ) for any given independent variable x, and γ is a positive parameter;

[0015] The above model can be reconstructed into the following weighted form of tensor completion model:

[0016]

[0017] wherein, is a tensor weighted kernel norm, and the function The minimum point of the above tensor completion model is obtained when W ki is a weighting coefficient.

[0018] Preferably, the solving of the tensor completion model based on the alternating direction multiplier method comprises:

[0019] ​An auxiliary variable Y is introduced to re-express the weighted form of the tensor completion model as follows:

[0020]

[0021] The Lagrangian function is expressed as follows:

[0022]

[0023] where Z is the Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product.

[0024] The Lagrangian function is solved by using the alternating direction multiplier method, and the variables Y and X are iteratively updated, and the weight matrix W, the Lagrange multiplier Z, and the parameter μ are iteratively updated.

[0025] Preferably, the variables Y and X are updated by using the following calculation formula:

[0026]

[0027] where argmin is used to obtain the variable value when the objective function reaches the minimum value.

[0028] Preferably, the weight matrix W, the Lagrange multiplier Z, and the parameter μ are iteratively updated by using the following calculation formula:

[0029]

[0030] Z k+1 = Z k + μ k (X k+1 -Y k+1 ),

[0031] μ k+1 = ρ * μ k .

[0032] where ρ ≥ 1 is a positive parameter.

[0033] Preferably, the approximate singular value decomposition method based on tensor QR decomposition is used to solve the tensor completion model, comprising:

[0034] The approximate singular value decomposition method based on tensor QR decomposition is used to construct the weighted form of the tensor completion model into the following approximate singular value decomposition form:

[0035]

[0036] where X = L * D * R is the singular value decomposition result of the tensor approximation, is an orthogonal tensor, is an upper triangular tensor, and the first r singular values of each forward leaf slice matrix of the tensor X and the reduced dimension tensor D are the same;

[0037] The approximate singular value decomposition form of the tensor completion model is solved based on an alternating direction multiplier method.

[0038] Preferably, the approximate singular value decomposition form of the tensor completion model is solved based on an alternating direction multiplier method, comprising:

[0039] The Lagrange function of the approximate singular value decomposition form of the tensor completion model is expressed as follows:

[0040]

[0041] wherein Z is a Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product;

[0042] By presetting the value of the tensor tube rank r, L k+1 and R k+1 are updated as follows:

[0043]

[0044] Further, by fixing the variables X k , Z k , W k , μ k , L k+1 and R k+1 , the variable D can be optimized by the following calculation formula:

[0045]

[0046] The weight matrix W is updated by ;

[0047] The variable X is updated by the following calculation formula:

[0048]

[0049] In the formula, Ω C is the complement of Ω, representing the index set of missing data;

[0050] The Lagrange multiplier Z and the parameter μ are iteratively updated by the following calculation formula:

[0051] Z k+1 = Z k + μ k (X k+1 -L k+1 *D k+1 *R k+1 ),

[0052] μ k+1 = p * mu k .

[0053] where p >= 1 is a positive parameter.

[0054] In a second aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method according to any one of the first aspect.

[0055] In a third aspect, the present application provides a computing device comprising a memory and a processor, the memory having stored thereon executable code which, when executed by the processor, implements the method according to any one of the first aspect.

[0056] From the above technical solution, it can be seen that in the image restoration method based on adaptive weighted tensor completion provided by the embodiments of the present application, firstly, the image data to be restored is obtained, and the tensorization is performed to obtain input tensor data, then a tensor completion model based on adaptive weighted tensor kernel norm is constructed, and the weight matrix in the tensor completion model can be updated adaptively with the input tensor data; further, the tensor completion model is solved based on the alternating direction multiplier method or the approximate singular value decomposition method based on tensor QR decomposition, and the recovered tensor data is output, that is, the image restoration is realized. As can be seen, the method does not need to predefine and manually set the weight of the low-rank tensor singular value, and the weight matrix can be updated adaptively, that is, the smaller weight is adaptively allocated, and the main singular value of the low-rank tensor is punished slightly, so that the performance of the kernel norm on the high-dimensional real data set is improved, and the recovery quality of the image data is improved.

[0057] In addition, the present application also considers that the approximate singular value decomposition method based on tensor QR decomposition is used to solve the tensor completion model, which can effectively improve the limitation of the method in the calculation speed by reducing the dimension of the high-dimensional tensor. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of the image restoration method based on adaptive weighted tensor completion provided by the embodiments of the present application.

[0059] Figure 2 A visual comparison of the restoration effect of the present application and other tensor completion methods on color images.

[0060] Figure 3 A histogram display of the restoration effect (relative error) of the present application and other tensor completion methods on color images.

[0061] Figure 4 A histogram display of the restoration efficiency (time) of the present application and other tensor completion methods on color images.

[0062] Figure 5 Visual comparison of the restoration effect of the application and other tensor completion methods on multispectral images.

[0063] Figure 6 Visual comparison of the restoration effect of the application and other tensor completion methods on YUV video sequences. DETAILED DESCRIPTION

[0064] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0065] As shown in Figure 1 The application provides an image restoration method based on adaptive weighted tensor completion, which can include the following steps:

[0066] Step 101: Obtain the image data to be repaired, and tensorize it to obtain input tensor data;

[0067] Step 102: Construct a tensor completion model based on adaptive weighted tensor kernel norm; wherein the weight matrix in the tensor completion model can be updated adaptively with the input tensor data;

[0068] Step 103: Based on the alternating direction multiplier method or the approximate singular value decomposition method based on tensor QR decomposition, the tensor completion model is solved, and the restored tensor data is output, realizing image restoration.

[0069] In this embodiment, based on the method provided above, the weight of the low-rank tensor singular value does not need to be predefined and manually set, the weight matrix can be updated adaptively, that is, to adaptively assign a smaller weight and to lightly punish the main singular value of the low-rank tensor, thereby improving the performance of the kernel norm on the high-dimensional real data set and improving the restoration quality of the image data. In addition, the approximate singular value decomposition method based on tensor QR decomposition is also considered to solve the tensor completion model, which can effectively improve the limitation of the method in calculation speed by reducing the dimension of the high-dimensional tensor.

[0070] For step 101, obtain the image data to be repaired, and tensorize it to obtain input tensor data.

[0071] In this step, consider obtaining the image data X' to be repaired, and representing it as a third-order tensor X ∈ R n1×n2×n3wherein X denotes an input tensor data, R denotes a real number set, n1, n2, n3 respectively denote sizes of first, second and third dimensions of the tensor.

[0072] Step 102: constructing a tensor completion model based on an adaptive weighted tensor core norm; wherein a weight matrix in the tensor completion model can be updated adaptively with input tensor data.

[0073] In this step, first, a tensor completion model based on an adaptive weighted tensor core norm is constructed, and the constructed tensor completion model is expressed as follows:

[0074]

[0075] wherein W ki denotes a weighted coefficient, denotes a tensor , and is a k-th frontal slice of , and denotes an i-th singular value of X (k) , d = min(n1, n2); P Ω is an orthogonal projection on the tensor dimension, and its value is 0 outside Ω, is an observed variable with missing values, and P Ω (X) = P Ω (M) is a constraint condition; ψ is a non-convex conjugate function of ρ γ (x) = γ(1-exp(-|x| / γ)), which satisfies and for any given independent variable x, the minimum point of the above completion model is obtained when w = exp(-|x| / γ), and γ is a positive parameter;

[0076] The above model can be reconstructed in the following weighted form:

[0077]

[0078] wherein is a tensor weighted core norm, and the function is fixed for the tensor X, and the minimum point of the above minimization form of the tensor completion model is obtained when , W ki is a weighted coefficient, and constitutes a weight matrix W. Since the weight matrix W can be updated adaptively with the target tensor X, the above model can be called an adaptive weighted tensor completion (Auto-weighted TC, AWTC).

[0079] Step 103: based on the alternating direction multiplier method or the approximate singular value decomposition method based on tensor QR decomposition, the tensor completion model is solved, and the recovered tensor data is output, and image inpainting is realized.

[0080] After the tensor completion model is constructed, the model needs to be solved. When solving the model, the traditional alternating direction multiplier method can be considered for solving, or the approximate singular value decomposition SVD method based on tensor QR (Qatar Riyal) decomposition can be considered for solving to improve the solving speed.

[0081] Method one: using the traditional alternating direction multiplier method to solve

[0082] In this way, first, the auxiliary variable Y is introduced, and the weighted form of the tensor completion model is represented as:

[0083]

[0084] The Lagrangian function can be represented as follows:

[0085]

[0086] Where Z is the Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product.

[0087] The alternating direction multiplier method is used to solve the Lagrangian function, and the variables Y and X are iteratively updated, and the weight matrix W, the Lagrange multiplier Z and the parameter μ are iteratively updated.

[0088] Specifically, when updating the variable Y, other variables can be fixed, and Y can be updated by first minimizing L(Y,X k ,W k ,Z k ,μ k ), that is, Further, through the soft threshold operator of the tensor weighted kernel norm, the above minimization problem has a global optimal solution: Where argmin is used to find the variable value when the objective function reaches the minimum value.

[0089] Further, by fixing other variables, the update of X is easily obtained, that is, the recovered tensor data is obtained:

[0090] At the same time, the weight matrix W, the Lagrange multiplier Z and the parameter μ can be iteratively updated by the following calculation formula:

[0091]

[0092] Z k+1= Z k + μ k (X k+1 - Y k+1 ),

[0093] μ k+1 = p * μ k .

[0094] where p > 1 is a positive parameter.

[0095] Method two: solving (TQR-AWTC) based on the approximate singular value decomposition method of tensor QR decomposition

[0096] In this method, first, based on the approximate singular value decomposition SVD method of tensor QR decomposition, the weighted form of the tensor completion model is constructed into the following approximate singular value decomposition form:

[0097]

[0098] where X = L * D * R is the result of the singular value decomposition of the tensor approximation, is an orthogonal tensor, is an upper triangular tensor, and the first r singular values of each forward leaf slice matrix of the tensor X and the reduced dimension tensor D are the same;

[0099] Then, based on the alternating direction multiplier method, the tensor completion model in the form of the approximate singular value decomposition is solved.

[0100] Specifically, when solving, the Lagrange function of the tensor completion model in the form of the approximate singular value decomposition can be expressed as follows:

[0101]

[0102] where Z is the Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product.

[0103] First, L k+1 and R k+1 can be updated by the following minimization formula:

[0104]

[0105] Unlike the alternating direction multiplier ADMM algorithm, by presetting the value of the tensor tube rank r, the tensors L k+1 and R k+1 are updated as follows:

[0106]

[0107] Further, by fixing the variable X k , Zk W k ,μ k ,L k+1 and R k+1 , the variable D can be optimized by the following calculation formula:

[0108]

[0109] In this step, the above minimization problem has a global optimal solution by the soft threshold operator of the tensor weighted kernel norm:

[0110]

[0111] The adaptive update of the weight matrix W can be updated by ;

[0112] Further, the variable X is updated by the following calculation formula, that is, the output recovered tensor data:

[0113]

[0114] In the formula, Ω C is the complement of Ω, representing the index set of the missing data.

[0115] Finally, the Lagrange multiplier Z and the parameter μ are iteratively updated by the following calculation formula:

[0116] Z k+1 = Z k + μ k (X k+1 - L k+1 * D k+1 * R k+1 ),

[0117] μ k+1 = ρ * μ k .

[0118] Wherein, ρ≥1, is a positive parameter.

[0119] The effect of the method of the present application will be further described below in combination with specific examples.

[0120] Example 1: Color image data recovery. As Figure 2 , Figure 3 , Figure 4As shown, the AWTC model proposed in the present application has much better recovery effect than the pre-defined weighted wTNN and RWTN, and has more advantages in the methods of removing artifacts and recovering edge texture information than the recently proposed non-convex surrogate function (NTC) and iterative re-weighting technique (IRTNN) methods based on logarithmic function; compared with the AWTC model, the TQR-AWTC fast algorithm proposed in the present application has significantly improved computing efficiency and shortened recovery time by nearly 90% (speedup of about 10 times) on the premise of maintaining the same recovery accuracy.

[0121] Example 2: Recovery of multispectral image data. As shown, Figure 5 As shown, compared with other methods, the AWTC can better recover the edge and texture detail information of the image. For example, the TNN, wTNN and RWTN have poor effect on recovering the edge area (bright part of the chili) of the enlarged image, and the NTC and IRTNN methods can recover the edge, but the recovery effect on the texture detail information (concave and convex part of the bright part of the chili) of the enlarged image is not as good as the model proposed in the present application.

[0122] Example 3: Recovery of YUV video sequence data. As shown, Figure 6 As shown, compared with the wTNN and RWTN methods of the traditional manual setting weighting mode, the AWTC realizes more accurate recovery through adaptive weight learning. Compared with the non-convex optimization methods such as NTC and IRTNN, the dynamic threshold strategy of the AWTC effectively reduces the artifact residues, and better preserves the high-frequency texture details.

[0123] The specification also provides a computer-readable storage medium having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method in any one of the embodiments of the specification.

[0124] The specification also provides a computing device comprising a memory and a processor, the memory having stored thereon executable code, which, when executed by the processor, implements the method in any one of the embodiments of the specification.

[0125] The device embodiments provided by the present application are based on the same inventive concept as the method embodiments of the specification, and specific contents can be referred to the description in the method embodiments of the specification, which will not be described here.

[0126] The modules or units in the device embodiments of the present application can be combined, divided, and deleted according to actual needs. The above disclosure is only the preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application, and those skilled in the art can understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made according to the claims of the present application still belong to the scope covered by the present application.

Claims

1. An image restoration method based on adaptive weighted tensor completion, characterized in that: include: Obtain the image data to be repaired and tensorize it to obtain input tensor data; Construct a tensor completion model based on the adaptive weighted tensor nuclear norm; the weight matrix in the tensor completion model can be adaptively updated with the input tensor data; The tensor completion model is solved based on the alternating direction multiplier method or the approximate singular value decomposition method based on tensor QR decomposition, and the restored tensor data is output to achieve image restoration.

2. The image restoration method based on adaptive weighted tensor completion according to claim 1, characterized in that: The construction of a tensor completion model based on an adaptive weighted tensor nuclear norm includes: The completion model of the adaptive weighted tensor nuclear norm is constructed as follows: Among them, W ki represents the weighting coefficient, Represents a tensor The result of discrete Fourier transform along the third dimension n3 is, yes The kth frontal slice of It means The i-th singular value of , d = min (n1, n2); P Ω is the orthogonal projection on the tensor dimension, and its value is 0 outside Ω. is an observed variable with missing values, P Ω (X) = P Ω (M) is the constraint condition; ψ is ρ γ (x) = γ(1-exp(-|x| / γ)) non-convex conjugate function, satisfying And for any given independent variable x, the minimum point of the above-mentioned completion model is obtained when w = exp(-|x| / γ), where γ is a positive parameter; The above model can be reconstructed into the following weighted tensor completion model: in, is the tensor weighted nuclear norm, function The minimum point of the above tensor completion model is When obtained, W ki is the weighting coefficient.

3. The image restoration method based on adaptive weighted tensor completion according to claim 2, characterized in that: The solving the tensor completion model based on the alternating direction multiplier method includes: Introducing the auxiliary variable Y, the weighted tensor completion model is re-expressed as: Its Lagrangian function is expressed as follows: Where Z is the Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product; The alternating direction multiplier method is used to solve the Lagrangian function, iteratively update the variables Y and X, and iteratively update the weight matrix W, Lagrangian multiplier Z and parameter μ.

4. The image restoration method based on adaptive weighted tensor completion according to claim 3, characterized in that: Update variables Y and X using the following calculation formula: Among them, argmin is used to find the variable value when the objective function reaches the minimum value.

5. The image restoration method based on adaptive weighted tensor completion according to claim 4, characterized in that: The weight matrix W, Lagrange multiplier Z, and parameter μ are iteratively updated using the following formula: Z k+1 =Z k +μ k (X k+1 -Y k+1 ), m k+1 =r*m k . Among them, ρ≥1 is a positive parameter.

6. The image restoration method based on adaptive weighted tensor completion according to claim 2, characterized in that: The tensor completion model is solved by the approximate singular value decomposition method based on tensor QR decomposition, including: Based on the approximate singular value decomposition method of tensor QR decomposition, the weighted tensor completion model is constructed as the following approximate singular value decomposition form: Among them, X = L * D * R is the result of the singular value decomposition of the tensor approximation, is an orthogonal tensor, is an upper triangular tensor, and the first r singular values ​​of each forward leaf slice matrix of the tensor X and the reduced-dimensional tensor D are the same; The tensor completion model in the form of approximate singular value decomposition is solved based on the alternating direction multiplier method.

7. The image restoration method based on adaptive weighted tensor completion according to claim 6, characterized in that: The method of solving the tensor completion model in the form of approximate singular value decomposition based on the alternating direction multiplier method includes: The Lagrangian function of the tensor completion model in the form of approximate singular value decomposition is expressed as follows: Where Z is the Lagrange multiplier, μ is a positive parameter, Ψ represents a non-convex conjugate function, and * is a tensor-tensor product; By presetting the value of tensor tube rank r, L k+1 and R k+1 Updates as follows: Furthermore, by fixing the variable X k ,Z k ,W k ,μ k ,L k+1 and R k+1 , the variable D can be optimized by the following calculation formula: The weight matrix W is obtained by Make updates; Then the variable X is updated by the following calculation formula: Where, Ω C is the complement of Ω, representing the index set of missing data; The Lagrange multiplier Z and the parameter μ are iteratively updated using the following formula: Z k+1 =Z k +μ k (X k+1 -L k+1 *D k+1 *R k+1 ), m k+1 =r*m k . Among them, ρ≥1, which is a positive parameter.

8. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 7.

9. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 7 is implemented.

Images