Unsupervised Network Prior Based Non-Convex Low-Rank Image Decomposition Method

Abstract

The present invention provides a non-convex low-rank image decomposition method based on unsupervised network prior, which relates to the technical field of image processing. The method includes: constructing an image decomposition model based on a deep image prior (DIP) and a non-convex low-rank regular term; inputting the image to be decomposed into the image decomposition model for iterative calculation until the relative error of the evaluation index between the primary output image obtained from the iterative calculation of the current round and the primary output image obtained from the previous round is less than a preset error threshold, and ending the iterative calculation; outputting the final image decomposition result of the image to be decomposed; wherein, after each round of iterative calculation, if the condition for ending the iterative calculation is not met, the optimization variables of the image decomposition model are updated, and the next round of iterative calculation is performed using the image decomposition model with the updated optimization variables. This solution can improve the fineness and integrity of image decomposition, thereby improving the effect of image decomposition.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a non-convex low-rank image decomposition method based on unsupervised network prior. Background Art

[0002] With the rapid progress of computer technology and the full arrival of the digital age, image data has increasingly become one of the important media in our daily life and communication. However, in actual application scenarios, images are often affected by multiple factors such as complex and changeable scenes, noise interference, and illumination changes, which may have a significant impact on the effect of image processing. To address this challenge, methods using image decomposition technology to refine images have received extensive attention.

[0003] The main purpose of image decomposition is to divide different components or features in an image into two parts: structure and texture, so as to deeply understand and utilize image information. Among them, the structure part mainly reflects the main information of the image, constituting the rough outline and main content of the image; while the texture part is composed of image details and noise, reflecting the redundant information of the image. In the process of constructing an image decomposition model, it is mainly through the analysis of prior information such as the sparsity and smoothness of the image, and then an appropriate strategy is selected to describe the structural and textural features of the image to achieve effective decomposition. Through the image decomposition method, useful information in the image can be more accurately extracted, laying a solid foundation for subsequent image processing and applications.

[0004] In recent years, the deep image prior (DIP) technology and many improved DIP networks have been widely applied in the fields of image processing and computer vision. By integrating network prior into the traditional model framework, the performance of the model can be significantly improved. However, these priors often only constrain the structural part of image decomposition, while ignoring the texture components with different characteristics, resulting in a lower degree of refinement and poorer accuracy in image decomposition. Moreover, the distribution of image texture exhibits significant characteristics such as periodicity and oscillation, and these characteristics are usually constrained by low-rank methods to obtain better decomposition effects. However, in the current low-rank constraint methods for processing image matrices, the same penalty is imposed on all singular values, ignoring the problem that the singular values are different in the high-frequency and low-frequency information of the image, which also reduces the fine degree of image decomposition and the image decomposition effect is poor. Summary of the Invention

[0005] In view of this, in order to make up for the above deficiencies, it is necessary to propose a non-convex low-rank image decomposition method based on unsupervised network prior to improve the fineness and integrity of image decomposition, and thus improve the effect of image decomposition.

[0006] The present invention provides a non-convex low-rank image decomposition method based on unsupervised network prior, including:

[0007] Construct an image decomposition model based on the deep image prior (DIP) and the non-convex low-rank regularization term;

[0008] Input the image to be decomposed into the image decomposition model for iterative calculation in a loop until the relative error of the evaluation index between the primary output image obtained from the iterative calculation in the current round and the primary output image obtained in the previous round is less than the preset error threshold, and end the iterative calculation in the loop;

[0009] Output the final image decomposition result of the image to be decomposed;

[0010] Wherein, after each round of iterative calculation, if the condition for ending the iterative calculation in the loop is not met, update the optimization variables of the image decomposition model, and use the image decomposition model with updated optimization variables for the next round of iterative calculation.

[0011] Preferably, the image decomposition model includes:

[0012]

[0013] Wherein, τ i and μ are both positive parameters, ||·|| r represents the truncated nuclear norm, f θ (z) is a network generator with parameter θ, i represents the i-th pixel point, N is the image size, D is the gradient operator, y is the input image to be decomposed, z is the image structure part, and v is the image texture part; is the data fidelity term related to DIP, is the regularization term for gradient constraint on the whole DIP, μ||v|| r is the non-convex low-rank regularization term.

[0014] Preferably, updating the optimization variables of the image decomposition model includes:

[0015] Change the image decomposition model into the following augmented Lagrangian function form:

[0016]

[0017] Wherein, β is the penalty parameter, λ is the Lagrange multiplier, t i =(Df θ (z)) i is to update the optimization variables θ, t i , v and λ of the image decomposition model in the form of the augmented Lagrangian function according to the constraint conditions.

[0018] Preferably, updating the optimization variable θ includes:

[0019] Updating the optimization variable θ using the following calculation formula:

[0020]

[0021] where k represents the k-th iteration.

[0022] Preferably, updating the optimization variable t i includes:

[0023] Expressing the sub-problem of the optimization variable t i as:

[0024]

[0025] where k represents the k-th iteration;

[0026] Obtaining the following explicit representation through the soft threshold function, and updating the optimization variable t i using the following explicit representation:

[0027]

[0028] where, for any c > 0,

[0029] Preferably, updating the optimization variable v includes:

[0030] Expressing the non-convex sub-problem of the optimization variable v as:

[0031]

[0032] Solving it using the singular value decomposition method represented by tensors, obtaining the following explicit representation, and updating the optimization variable v using the following explicit representation:

[0033]

[0034] where SVT ε (Q) = Udiag[max(σ - ε), 0]V T , the singular value decomposition of Q is Q = Udiag(σ)V T , σ = (σ1, σ2, … σ r ) T ∈R r , k represents the k-th iteration, T represents the transpose of the matrix, A = (u1, u2, … u r ) T ∈R r×m , B = (w1, w2, … wr ) T ∈R r×n ,V=(v1,v2,…v n )∈R n×n 。

[0035] Preferably, updating the Lagrange multiplier λ includes:

[0036] Updating the Lagrange multiplier using the following calculation formula:

[0037]

[0038] where k represents the k-th iteration.

[0039] Preferably, the evaluation index includes at least one of signal-to-noise ratio SNR and structural similarity SSIM.

[0040] In a second aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute any one of the methods in the first aspect.

[0041] In a third aspect, the present invention provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, any one of the methods in the first aspect is implemented.

[0042] As can be seen from the above technical solutions, in the non-convex low-rank image decomposition method based on unsupervised network prior provided by this solution, an image decomposition model is first constructed based on the deep image prior DIP and the non-convex low-rank regularization term, and then the image to be decomposed is input into the image decomposition model for iterative calculation. Among them, in each iterative calculation, the optimization variables of the image decomposition model are continuously updated until the error between the evaluation index of the output image and the evaluation index of the image output in the previous iteration is less than a preset error threshold. Thus, it can be seen that this solution uses an unsupervised DIP network as an implicit prior and incorporates it into the model in a plug-and-play manner, which can extract image structure information more accurately, thereby ensuring the detail integrity of the decomposed image. At the same time, the non-convex low-rank regularization term constructed in the model depicts the low-rank property of the image, so that the detail information of the image can be described more precisely. Therefore, through this solution, the fineness and integrity of image decomposition can be improved, and thus the effect of image decomposition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flowchart of a non-convex low-rank image decomposition method based on unsupervised network prior provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the embodiments. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0045] As Figure 1 shown, the present invention provides a non-convex low-rank image decomposition method based on unsupervised network prior, which may include the following steps:

[0046] Step 101: Construct an image decomposition model based on the depth image prior DIP and the non-convex low-rank regularization term;

[0047] Step 102: Input the image to be decomposed into the image decomposition model for iterative calculation in a loop until the relative error of the evaluation index between the primary output image obtained from the iterative calculation in the current round and the primary output image obtained in the previous round is less than the preset error threshold, and end the iterative calculation in a loop; wherein, after each round of iterative calculation, if the condition for ending the iterative calculation in a loop is not met, update the optimization variables of the image decomposition model, and use the image decomposition model with updated optimization variables for the next round of iterative calculation.

[0048] Step 103: Output the final image decomposition result of the image to be decomposed.

[0049] In this embodiment, an unsupervised DIP network is used as an implicit prior and incorporated into the model in a plug-and-play manner, which can more accurately extract the image structure information, thus ensuring the detail integrity of the decomposed image. At the same time, the non-convex low-rank regularization term constructed in the model depicts the low-rank property of the image, so that the detail information of the image can be described more precisely. Therefore, through this solution, the fineness and integrity of image decomposition can be improved, and thus the effect of image decomposition can be improved.

[0050] For step 101, construct an image decomposition model based on the depth image prior DIP and the non-convex low-rank regularization term;

[0051] Regarding the problem of insufficient extraction of image structure information in the image decomposition task, in this step, it is considered to construct a DIP data fidelity term and a non-convex low-rank regularization term to construct a decomposition model. The former can use the network to learn the deep semantic information of the image and then extract the image structure, and the latter can use the non-convex truncated nuclear norm to more precisely describe the image details, thereby improving the fineness and integrity of image decomposition. Specifically, in one embodiment, the constructed image decomposition model can be:

[0052] wherein, τ i and μ are both positive parameters, ||·||r denotes the truncated nuclear norm, and f θ (z) is a network generator with parameter θ, i represents the i-th pixel, N is the image size, D is the gradient operator, y is the input image to be decomposed, z is the image structure part, and v is the image texture part; is the data fidelity term related to DIP, is the regularization term for gradient constraint on the whole DIP, μ||v|| r is the non-convex low-rank regularization term.

[0053] In this embodiment, aiming at the problem of insufficient extraction of image structure information in the image decomposition task, a decomposition model based on DIP and non-convex low-rank regularization term constraints is constructed. The first term of the model is the data fidelity term related to DIP, the second term is the regularization term for gradient constraint on the whole DIP, and the third term is the non-convex low-rank regularization term. The first two terms of the model use the network to learn the deep semantic information of the image and then extract the image structure. The last term uses the non-convex truncated nuclear norm to more accurately describe the details of the image, so as to obtain a more accurate and complete decomposed image.

[0054] For step 102, input the image to be decomposed into the image decomposition model for iterative calculation until the relative error of the evaluation index between the primary output image obtained by the iterative calculation of the current round and the primary output image obtained in the previous round is less than the preset error threshold, and end the iterative calculation;

[0055] In this step, each time of iterative calculation, the relative error of the evaluation index of the decomposed image obtained by the iterative calculation of the previous round is used to determine whether the convergence condition is reached, that is, to determine whether the requirements of image decomposition are met. The optimization variables in the image decomposition model are updated during each iterative calculation, so as to obtain a decomposed image with better quality through continuous update and iteration. Specifically, when updating the optimization variables of the image decomposition model, it can be realized in the following way:

[0056] Change the image decomposition model into the following augmented Lagrangian function form:

[0057]

[0058] where β is the penalty parameter, λ is the Lagrange multiplier, and t i =(Df θ (z)) i is the constraint condition

[0059] Update the optimization variables θ, t i , v and λ of the image decomposition model in the form of augmented Lagrangian function.

[0060] In this embodiment, the alternating direction method of multipliers is considered to optimize and update each optimization variable in the image decomposition model, and the Lagrange multiplier is updated. Specifically, an auxiliary variable t can be introduced. i And the constraint condition t i =(Df θ (z)) i is used to solve by decomposing each optimization variable. The originally constructed image decomposition model can be represented in the form of an augmented Lagrangian function. Updating the optimization variables in the image decomposition model means updating θ, t i , v, and λ.

[0061] For the optimization variable θ;

[0062] The sub-problem for the optimization variable θ can be expressed as: where k represents the k-th iteration.

[0063] In this embodiment, iterative calculation can be performed through the numerical gradient method, and its numerical gradient can be obtained by automatic differentiation of the variable θ. Therefore, the above calculation formula can be used, combined with the numerical gradient method, etc., to iteratively update the optimization variable θ.

[0064] For the optimization variable t i ;

[0065] For the optimization variable t i the sub-problem can be expressed as: where k represents the k-th iteration; this problem can obtain an explicit solution through the soft threshold function where, for any c > 0, In this way, the iterative update of the optimization variable t i can be achieved through the above calculation formula.

[0066] For the optimization variable v;

[0067] The non-convex sub-problem of the optimization variable v can be expressed as: Since the truncated nuclear norm term in the non-convex sub-problem cannot be directly solved, the singular value decomposition method represented by tensors is considered for calculation. The truncated nuclear norm can be rewritten as:

[0068]

[0069] where A=(u1,u2,…u r ) T ∈R r×m , B=(w1,w2,…w r ) T ∈R r×n , V=(v1,v2,…v n )∈Rn×n , and AA T = I r×r , BB T = I r×r , I r×r represents the identity matrix of size r×r, and Tr(·) and σ(·) represent the trace and singular values of a matrix, respectively.

[0070] Since a tensor can be regarded as a high-dimensional extension of a matrix and can represent high-dimensional images more naturally. Therefore, it is possible to consider expanding along the third dimension of the tensor v k+1 ∈R m×n×3 . Then the expression of the non-convex sub-problem of the above optimization variable v can be written in the following form: Further, this equation has an explicit solution where, SVT ε (Q) = Udiag[max(σ - ε), 0]V T , and the singular value decomposition of Q is Q = Udiag(σ)V T . Thus, based on this explicit solution, the update of the optimization variable v can be realized.

[0071] When the sub-problems of the optimization variables θ, t i and v are all solved, the Lagrange multipliers are updated according to the alternating direction method of multipliers. Specifically, the update can be performed using the following calculation formula:

[0072] λ k+1 = λ k + β((Df θk+1 (z)) i - t i k+1 )

[0073] For the evaluation index of the decomposed image, it is possible to consider using at least one of the signal-to-noise ratio (SNR) and the structural similarity (SSIM) to evaluate the decomposition result of the synthesized image. The higher the index value, the closer the decomposition result is to the original image.

[0074] For step 103, output the final image decomposition result of the image to be decomposed;

[0075] When the iterative calculation reaches the convergence condition, output the structure and texture results of the decomposed image.

[0076] This specification also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method in any one of the embodiments in the specification.

[0077] The present specification also provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the method in any one of the embodiments in the specification is implemented.

[0078] The modules or units in the device according to the embodiments of the present invention may be combined, divided, and deleted according to actual needs. The foregoing disclosure is only a preferred embodiment of the present invention, and of course, it cannot be used to limit the scope of rights of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the present invention.

Claims

1. A non-convex low-rank image decomposition method based on unsupervised network prior, characterized in that: include: Construct an image decomposition model based on deep image prior DIP and non-convex low-rank regularization term; The image to be decomposed is input into the image decomposition model for cyclic iterative calculation until the relative error of the evaluation index between the primary output image obtained by the current round of iterative calculation and the primary output image obtained by the previous round is less than a preset error threshold, and the cyclic iterative calculation is terminated; Outputting the final image decomposition result of the image to be decomposed; Wherein, after each round of iterative calculation, if the condition for ending the loop iterative calculation is not met, the optimization variables of the image decomposition model are updated, and the image decomposition model with the updated optimization variables is used to perform the next round of iterative calculation; The image decomposition model includes: in, τ i and μ are both positive parameters, ||·|| r represents the truncated nuclear norm, f θ (z) is a network generator with a parameter of θ, i represents the i-th pixel, N is the image size, D is the gradient operator, y is the input image to be decomposed, z is the image structure part, and v is the image texture part; is the data fidelity item related to DIP, is the regularization term for the gradient constraint of the DIP as a whole, μ||v|| r is a non-convex low-rank regularization term; The updating of the optimization variables of the image decomposition model includes: The image decomposition model is transformed into the following augmented Lagrangian function form: Among them, β is the penalty parameter, λ is the Lagrange multiplier, t i =Df θ (z) i is a constraint condition; The optimization variables θ, t of the image decomposition model in the form of augmented Lagrangian function i , v and λ are updated.

2. The non-convex low-rank image decomposition method based on unsupervised network prior according to claim 1, characterized in that: Update the optimization variable θ, including: Use the following calculation formula to update the optimization variable θ: Here, k represents the kth iteration.

3. The non-convex low-rank image decomposition method based on unsupervised network prior according to claim 1, characterized in that: For the optimization variable t i Updates include: The optimization variable t i The sub-problem is expressed as: Where k represents the kth iteration; The following display representation is obtained through the soft threshold function, and the following display representation is used to optimize the variable t i To update: where, for any c>0, 4. The non-convex low-rank image decomposition method based on unsupervised network prior according to claim 1, characterized in that: Update the optimization variable v, including: The non-convex subproblem of optimizing variable v is expressed as: The singular value decomposition method of tensor representation is used to solve the problem, and the following display representation is obtained, and the optimization variable v is updated using the following display representation: Among them, SVT ε (Q)=Udiag[max(σ-ε),0]V T , the singular value decomposition of Q is Q = Udiag (σ) V T ,σ=(σ1,σ2,…σ r ) T ∈R r , k represents the kth iteration, T represents the transpose of the matrix, A=(u1,u2,…u r ) T ∈R r×m ,B=(w1,w2,…w r ) T ∈R r×n ,V=(v1,v2,…v n )∈R n×n .

5. The non-convex low-rank image decomposition method based on unsupervised network prior according to claim 1, characterized in that: Update the Lagrange multiplier λ, including: Use the following formula to update the Lagrange multiplier: Here, k represents the kth iteration.

6. The non-convex low-rank image decomposition method based on unsupervised network prior according to any one of claims 1 to 5, characterized in that: The evaluation index includes at least one of a signal-to-noise ratio (SNR) and a structural similarity (SSIM).

7. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

8. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 6 is implemented.

Images