An image restoration method based on low-rank matrix filling model

Abstract

This invention relates to an image inpainting method based on a low-rank matrix filling model. First, a novel low-rank regularization constraint term is designed, enabling the proposed matrix filling model to more tightly relax the rank function, thereby learning better low-rank structural relationships to effectively repair elements in damaged matrix regions. Furthermore, a sparsity constraint term is introduced to improve the model's robustness to noise in the image. This invention designs an efficient and simple optimization algorithm to solve the proposed matrix filling model and applies this model to image inpainting methods, effectively solving the problem of repairing damaged or blurred RGB and grayscale images.

Description

Technical Field

[0001] This invention belongs to the field of machine learning technology, specifically relating to an image restoration method based on a low-rank matrix filling model. It utilizes the proposed matrix filling technique based on novel low-rank regularization constraints to effectively restore damaged or blurred images. Background Technology

[0002] Image restoration is a hot research topic in computer vision and image processing. Its principle is to use known pixel information from the effective regions of a damaged image to repair or remove pixels in the damaged areas, restoring the image to a level close to the real scene or the standards required by human vision. Therefore, image restoration methods have extremely wide applications in areas such as blurred image restoration, removal of specific objects from images, high-definition upscaling of film and television materials, and restoration of works of art.

[0003] Image inpainting methods primarily aim to effectively restore damaged images using known information. Based on different inpainting mechanisms, they can be broadly categorized into four types: inpainting algorithms based on Partial Differential Equations (PDEs), patch-based inpainting algorithms, matrix completion (MC)-based inpainting algorithms, and deep learning-based inpainting algorithms. PDE-based methods mainly address the restoration of small-scale damaged images. For large-scale damaged images, patch-based inpainting algorithms show better results. However, when the texture features in the image are irregular, neither of these methods achieves satisfactory restoration results. Matrix completion-based inpainting methods assume that the pixels in the image have a certain correlation, including sparse and low-rank structures. Therefore, this type of method obtains representation coefficients for the effective regions of the image through constraint learning and further reconstructs the damaged areas to achieve image restoration. In real-world scenarios, matrix completion-based image inpainting methods can effectively improve the quality of image restoration under unsupervised conditions. Furthermore, due to its strong feature extraction capabilities, deep learning can effectively extract image information to repair damaged areas, especially large-scale damaged areas. However, it requires a large amount of training data and a long training time, and has high hardware requirements, thus limiting its efficient application in real-world scenarios.

[0004] For matrix filling models, they can be implemented using bilinear factorization (BF) and rank minimization (RM) methods, respectively. Bilinear factorization avoids the singular value decomposition process, resulting in lower computational complexity. However, this method requires pre-estimation of the dimensionality of the factorization factors, thus limiting its practicality in real-world scenarios. Unlike bilinear factorization, rank minimization-based methods combine known element information and low-rank constraints to learn unknown elements in the matrix, ensuring the learned matrix is ​​in an optimal low-rank structure. It can be seen that this method can directly apply rank minimization constraints to the matrix without requiring prior information about its structure. Furthermore, when the given matrix satisfies strong incoherence and the known elements satisfy random consistency sampling, the rank minimization-based matrix filling algorithm can restore the matrix with a high probability.

[0005] However, due to the discrete nature of the rank function, directly solving the matrix rank minimization problem is an NP-hard problem. Since the nuclear norm is the optimal convex enveloping of the matrix rank function, it is often used to perform convex relaxation of the rank minimization problem. However, the nuclear norm-based minimization problem is not a tight relaxation of the rank minimization problem because there is still a significant difference between the nuclear norm and the rank function. Specifically, the nuclear norm is the sum of all singular values ​​of a matrix, while the rank of a matrix is ​​the number of non-zero singular values. When a non-zero singular value in a matrix changes significantly, its nuclear norm also changes significantly, but the rank of the matrix remains unchanged. This causes the nuclear norm-based minimization problem to lose effective control over the matrix rank in some cases. Therefore, a novel low-rank constraint regularization term should be proposed to more tightly relax the rank minimization problem. Summary of the Invention

[0006] Technical problems to be solved

[0007] To effectively solve the rank minimization problem in existing matrix filling models and learn better low-rank structural relationships to effectively repair elements in broken matrix regions, this invention provides an image restoration method based on a low-rank matrix filling model.

[0008] Technical solution

[0009] An image inpainting method based on a low-rank matrix filling model, characterized by the following steps:

[0010] Step 1: Preprocess the given damaged image. For grayscale images, read them directly into matrix form and obtain the set of known effective region element positions. For RGB images, read the data of the three channels into matrix form and obtain the set of known effective region element positions for the corresponding channels.

[0011] Step 2: For each channel input data matrix, under the known set of valid region element positions, use a matrix filling model based on new regular constraints to learn low-rank filling of the data matrix. By iteratively optimizing the objective function, the optimal low-rank matrix is ​​obtained, thereby effectively restoring the elements in the damaged region of the matrix.

[0012] Step 3: After obtaining the restoration matrix for each channel, the grayscale matrix can be directly converted into an image; for the RGB image, the restoration matrices obtained from the three channels are fused to obtain the restored RGB image.

[0013] A further technical solution of the present invention: The matrix filling model based on the novel regularization constraint in step 2 is as follows:

[0014]

[0015] Where ⊙ denotes the Hadamard product, and λ denotes the penalty parameter; for matrix W, when its (i,j)th element belongs to the known set of element positions Ω, then W i,j =1, otherwise W i,j =0; To improve the model's robustness to noise, an error matrix E was introduced during the low-rank matrix learning process, and [the following was used]: Sparsity constraints are applied using norms;

[0016] A further technical solution of the present invention: The method for iteratively optimizing the objective function in step 2 is as follows:

[0017] Problem (3) can be equivalently transformed into the following problem

[0018]

[0019] Furthermore, according to the definition of the augmented Lagrange function, the augmented Lagrange multiplier (ALM) problem for problem (5) is as follows:

[0020]

[0021] Here, Y represents the Lagrange multiplier, and μ > 0 is the Lagrange penalty parameter;

[0022] With variables F, E, and multiplier Y fixed, and the low-rank learning matrix L updated, problem (6) can be transformed into the following convex optimization problem.

[0023]

[0024] Problem (6) is a convex problem with respect to matrix L. Therefore, by differentiating problem (6) with respect to variable L and setting it to 0, we obtain the following equation.

[0025] 2FF T L+μW⊙L=W⊙(Y+μX-μE). (7)

[0026] For equation (7), we define l i w i y i x i and e i Let L, W, Y, X, and E be the i-th row vectors of matrices L, W, Y, X, and E, respectively. Then equation (7) can be written in the following vector form.

[0027] 2FF T ·l i +μdiag(w i )·l i =diag(w i )·(y i +μx i -μe i (8)

[0028] Where diag(w) i ) is a block diagonal matrix whose diagonal elements are formed by row vectors w i The elements of matrix L are composed of ; therefore, according to formula (8), the update formula for each row of matrix L is as follows:

[0029] l i =(2FF) T +μdiag(w i )) -1 [diag(w i )·(yi+μxi-μei)]. (9)

[0030] With fixed matrices L, E, and multipliers Y, and variable F updated, problem (6) is equivalent to the following problem:

[0031]

[0032] For problem (10), its optimal solution F is given by matrix LL. T The first k eigenvectors are composed of the first k smallest eigenvalues;

[0033] With the matrices L, F, and multiplier Y fixed, and the error matrix E updated, the proposed ALM problem (6) can be transformed into the following problem:

[0034]

[0035] The above problem has an analytical solution. Let Q = XW⊙L + Y / μ, then each column of the optimal solution E can be updated using the following formula.

[0036]

[0037] Here j q j These are the j-th column vectors in matrices E and Q, respectively;

[0038] With variables L, E, and F fixed, the update multiplier Y and the penalty parameter μ are updated using the following formula:

[0039] Y=Y+μ(xW⊙LE),μ=max(ρμ,max μ (11)

[0040] Here, ρ represents the learning rate, and max... μ is a given constant.

[0041] A computer system is characterized by comprising: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method described above.

[0042] A computer-readable storage medium is characterized by storing computer-executable instructions, which, when executed, are used to implement the above-described method.

[0043] Beneficial effects

[0044] This invention provides an image inpainting method based on a low-rank matrix filling model. A novel low-rank regularization constraint term is designed, enabling the proposed matrix filling model to more tightly relax the rank function, thereby learning better low-rank structural relationships to effectively repair elements in damaged matrix regions. Furthermore, a sparse constraint term is introduced to improve the model's robustness to noise in the image. This invention designs an efficient and simple optimization algorithm to solve the proposed matrix filling model and applies it to image inpainting methods, effectively solving the problem of repairing damaged or blurred RGB and grayscale images. Attached Figure Description

[0045] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0046] Figure 1 The flowchart of the method of this invention: The left figure is the overall flowchart of the image restoration method, and the right figure is the flowchart of the proposed matrix filling model based on the novel low-rank regularization constraint;

[0047] Figure 2 The proposed image restoration method based on a novel low-rank matrix filling model is applied to damaged images: (a) is the original damaged image, and (b) is the restored image. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0049] This invention provides a novel regularization constraint term applied to a matrix filling model, which better approximates the rank minimization problem by directly minimizing the first k smallest singular values ​​of the matrix. Compared to the nuclear norm, the novel regularization constraint term proposed in this invention can better relax the rank function, solving the problem that traditional matrix filling models cannot effectively control the matrix rank. An efficient and simple optimization algorithm is proposed to solve the optimization problem of this model. Furthermore, based on the matrix filling model based on the novel regularization constraint term, this invention further designs an image restoration method, which can effectively restore damaged color images (RGB images) or grayscale images.

[0050] This invention first proposes a matrix filling model based on novel low-rank regularization constraints, referred to as the novel low-rank matrix filling model, and further applies it to image restoration methods, thereby effectively restoring damaged RGB and grayscale images. In the image restoration process, the image is first preprocessed to obtain the set of known pixel locations. For single-channel data matrices (grayscale images), the proposed model is directly used for low-rank matrix restoration. For RGB images, which have three channels, low-rank restoration is performed on the data matrix of each channel separately, and the restored data from the three channels are then fused to obtain the restored RGB image. Furthermore, the proposed novel low-rank matrix filling model and its optimization method are described below.

[0051] This invention provides a matrix filling model based on a novel low-rank regularization constraint, the implementation of which is as follows:

[0052] The data matrix input to each channel of the image And the set of known element positions Ω of its effective region. The matrix filling model based on the rank minimization problem introduces a low-rank learning matrix. The model formula is as follows: To effectively approximate the original data matrix X using the known set of element positions Ω and perform low-rank recovery of unknown elements in the damaged region, the model formula is as follows:

[0053]

[0054] Here, rank(L) represents the rank function of matrix L, and also represents the number of its non-zero singular values. When matrix X satisfies strong incoherence and the known elements satisfy random consistency sampling, the above rank minimization matrix filling model can restore matrix X with a high probability. Since the rank function rank(·) is discrete with respect to L, directly solving problem (1) is an NP-hard problem.

[0055] Due to the nuclear norm ||L|| * It is the optimal convex hull of the matrix rank function rank(L), so problem (1) can usually be relaxed to the problem of minimizing the nuclear norm for optimization. However, as mentioned in the background art, the nuclear norm is not a close approximation of the rank function, and it cannot well control the non-zero singular values ​​in the matrix, thus failing to effectively guarantee the capture of the global low-rank structure in the data. Therefore, in order to better learn the low-rank structure in matrix X, this invention proposes a new regularization constraint term, which only minimizes the first k singular values ​​of the matrix to ensure that the rank of matrix L is strictly rank(L) = min(m,n)-k, thereby better learning the low-rank structure of the matrix and ensuring the effectiveness of matrix filling. The new low-rank regularization term proposed in this invention can be written as the following formula:

[0056]

[0057] Here, σ i (L) represents the i-th smallest singular value of matrix L. When the proposed novel low-rank regularization term R(L) is minimized, the rank of matrix L can be strictly approximated to min(m,n)-k. Therefore, the proposed regularization term can precisely control the size of the matrix rank. Combining with problem (1), the matrix filling model based on the novel low-rank regularization constraint proposed in this invention can be written as the following formula:

[0058]

[0059] Here, ⊙ denotes the Hadamard product. For a matrix W, when its (i,j)th element belongs to the known set of element positions Ω, then W has i,j =1, otherwise W i,j =0. To improve the model's robustness to noise, an error matrix E was introduced during the low-rank matrix learning process, and [the following was used]. The norm is used to impose sparsity constraints. For the proposed model (3), it can be seen that the regularization term... Since the singular values ​​of matrix L are also discrete, directly solving problem (3) is also an NP-hard problem. Therefore, in this invention, an efficient and simple algorithm is designed to optimize the proposed model.

[0060] Optimization algorithm design

[0061] According to Ky Fan's theorem, the following equation holds.

[0062]

[0063] Based on equation (4), the problem (3) can be equivalently transformed into the following problem.

[0064]

[0065] Furthermore, according to the definition of the augmented Lagrange function, the augmented Lagrange multiplier (ALM) problem for problem (5) is as follows:

[0066]

[0067] Here, Y represents the Lagrange multiplier, and μ > 0 is the penalty parameter. Compared with the original problem (3), it can be seen that problem (6) is a convex problem with respect to matrix L, and therefore easier to solve. Here, an optimization algorithm based on alternating iteration is designed to solve problem (6), and its specific process is as follows.

[0068] With variables F, E and multiplier Y fixed, and the low-rank learning matrix L updated, problem (6) can be transformed into the following convex optimization problem.

[0069]

[0070] Problem (6) is a convex problem with respect to matrix L. Therefore, by differentiating problem (6) with respect to variable L and setting it to 0, we can obtain the following equation.

[0071] 2FF T L+μW⊙L=W⊙(Y+μX-μE). (7)

[0072] For equation (7), we define l i w i y i x i and e i Let L, W, Y, X, and E be the i-th row vectors of matrices L, W, Y, X, and E, respectively. Then equation (7) can be written in the following vector form.

[0073] 2FF T ·l i +μdiag(wi )·l i =diag(w i )·(y i +μx i -μe i (8)

[0074] Where diag(w) i ) is a block diagonal matrix whose diagonal elements are formed by row vectors w i The elements constitute the matrix. Therefore, according to formula (8), the update formula for each row of matrix L is as follows:

[0075] l i =(2FF) T +μdiag(w i )) -1 [diag(w i )·(y i +μx i -μe i (9)

[0076] With fixed matrices L, E, and multipliers Y, and variable F updated, problem (6) is equivalent to the following problem:

[0077]

[0078] For problem (10), its optimal solution F is given by matrix LL. T It consists of the first k eigenvectors, which correspond to the first k smallest eigenvalues.

[0079] With the matrices L, F, and multiplier Y fixed, and the error matrix E updated, the proposed ALM problem (6) can be transformed into the following problem:

[0080]

[0081] The above problem has an analytical solution. Let Q = XW⊙L + Y / μ, then each column of the optimal solution E can be updated using the following formula.

[0082]

[0083] Here j q j These are the j-th column vectors in matrices E and Q, respectively.

[0084] With variables L, E, and F fixed, the update multiplier Y and the penalty parameter μ are updated using the following formula:

[0085] Y=Y+μ(XW⊙LE),μ=max(ρμ,max μ (II)

[0086] Here, ρ represents the learning rate, and max... μ is a given constant.

[0087] Based on the above solution process, this invention proposes an optimization algorithm based on alternating iteration to solve the target problem (3) until the algorithm converges to obtain its optimal solution, namely the recovery matrix L with a global low-rank structure, thereby achieving the purpose of effectively repairing the damaged data matrix X input by the channel.

[0088] from Figure 2 As can be seen, the image restoration method proposed in this invention, based on the proposed novel low-rank matrix filling model, can effectively utilize the known pixel information of the effective region in the image to repair the pixels in the damaged area of ​​the image, thereby effectively restoring the damaged image.

[0089] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. An image inpainting method based on a low-rank matrix filling model, characterized in that... Here are the steps: Step 1: Preprocess the given damaged image. For grayscale images, read them directly into matrix form and obtain the set of known effective region element positions. For RGB images, read the data of the three channels into matrix form and obtain the set of known effective region element positions for the corresponding channels. Step 2: For each channel input data matrix, under the known set of valid region element positions, use a matrix filling model based on new regular constraints to learn low-rank filling of the data matrix. By iteratively optimizing the objective function, the optimal low-rank matrix is ​​obtained, thereby effectively restoring the elements in the damaged region of the matrix. The matrix filling model based on the new regularization constraint is as follows: stX=W⊙L+E (3) where ⊙ represents the Hadamard product and λ represents the penalty parameter; for matrix W, when its (i,j)th element belongs to the known element position set Ω, then W i,j =1, otherwise W i,j =0; To improve the model's robustness to noise, an error matrix E was introduced during the low-rank matrix learning process, and l was used. 2，1 Sparsity constraints are applied using norms; The method for iteratively optimizing the objective function is as follows: Problem (3) can be equivalently transformed into the following problem Furthermore, according to the definition of the augmented Lagrange function, the augmented Lagrange multiplier ALM problem for problem (5) is as follows: Here, Y represents the Lagrange multiplier, and μ > 0 is the Lagrange penalty parameter; With variables F, E and multiplier Y fixed, and the low-rank learning matrix L updated, problem (6) can be transformed into the following convex optimization problem. Problem (6) is a convex problem with respect to matrix L. Therefore, by differentiating problem (6) with respect to variable L and setting it to 0, we obtain the following equation. 2FF T L+μW⊙L=W⊙(Y+μX-μE) (7) For equation (7), we define l i ,w i ,y i ,x i and e i Let L, W, Y, X and E be the vectors of the i-th row of matrices L, W, Y, X and E respectively. Then equation (7) can be written in the following vector form. 2FF T ·l i +μdiag(w i )·l i =diag(w i )·(y i +μx i -μe i ) (8) Where diag(w) i ) is a block diagonal matrix whose diagonal elements are formed by row vectors w i The elements of matrix L are composed of ; therefore, according to formula (8), the update formula for each row of matrix L is as follows: l i =(2FF T +μdiag(w i )) -1 [diag(w i )·(y i +μx i -μe i )] (9) With fixed matrices L, E and multiplier Y, and variable F updated, problem (6) is equivalent to the following problem: For problem (10), its optimal solution F is given by matrix LL. T The first k eigenvectors are composed of the first k smallest eigenvalues; With the matrices L, F and multiplier Y fixed, and the error matrix E updated, the proposed ALM problem (6) can be transformed into the following problem: The above problem has an analytical solution. Let Q = XW⊙L + Y / μ, then each column of the optimal solution E can be updated using the following formula. Here e j ,q j These are the j-th column vectors in matrices E and Q, respectively; With variables L, E, and F fixed, the multiplier Y and the penalty parameter μ are updated using the following formula: Y=Y+μ(XW⊙LE),μ=max(ρμ,max μ (11) Here ρ represents the learning rate, max μ Given a constant; Step 3: After obtaining the restoration matrix for each channel, the grayscale matrix can be directly converted into an image; for the RGB image, the restoration matrices obtained from the three channels are fused to obtain the restored RGB image.

2. A computer system, characterized in that... include: One or more processors, a computer-readable storage medium for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of claim 1.

3. A computer-readable storage medium, characterized in that... The device stores computer-executable instructions, which, when executed, are used to implement the method of claim 1.

Images