Image inpainting method, system, device and storage medium based on tensor singular value decomposition

Abstract

The application provides an image inpainting method based on tensor singular value decomposition. Firstly, image data cannot be directly used for image processing by a tensor singular value decomposition framework due to dimension limitation. In order to mine potential tensor tubal rank prior information of the image data, a data rearrangement scheme is designed, so that the image data elements are rearranged into a data structure suitable for tensor singular value decomposition, and the scheme can ensure that the rearranged data has tensor tubal rank. Then, by constraining the tensor tubal rank and combining the image local smoothness, an image inpainting optimization model based on tensor singular value decomposition is constructed. Finally, a model optimization solving algorithm based on the ADMM method is developed. Experimental results show that the method can effectively repair various typical image damage forms, and is superior to the low-rank matrix filling method and the image inpainting method based on Tucker tensor decomposition in visual and numerical indicators.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image inpainting method, system, device, and storage medium based on tensor singular value decomposition. Background Technology

[0002] Image inpainting technology involves optimizing and restoring image data in situations such as localized damage, impulse interference, random noise, malicious cutout, text overlay, and advertising overlay. Traditional image inpainting techniques typically optimize and repair images by exploiting matrix structure features such as low-rank properties in the transform domain, sparsity in the transform domain, and using optimized filtering or interpolation. However, methods that directly utilize low-rank properties are generally ineffective in restoring images with blocky or parallel-line stripe-like contamination.

[0003] In recent years, several novel tensor decomposition frameworks have emerged in mathematics for processing high-dimensional data. The novel tensor singular value decomposition (t-SVD) framework breaks away from the traditional CANDECOMP / PARAFAC (CP) and Tucker tensor decomposition approaches, further improving computational complexity and the effectiveness of approximating the low-rank properties of tensors. In video restoration applications, the low-tubular rank method defined by the tensor singular value decomposition framework shows a significant improvement in restoration accuracy compared to methods directly utilizing matrix low-rank properties or methods utilizing Tucker low-rank properties. However, RGB image data, due to its dimensionality limitations, cannot be directly processed using the tensor singular value decomposition framework. When traditional CP or Tucker decomposition is applied to RGB image restoration, it generally treats the RGB image directly as a third-order tensor, without deeply exploring the potential prior information of the image's low-rank properties, often resulting in poor restoration performance. Summary of the Invention

[0004] To address the aforementioned problems and further improve image inpainting accuracy, this invention proposes an image inpainting method based on tensor singular value decomposition (SVD). First, to mine the latent tensor low-rank prior information of the image, an image element rearrangement scheme is designed to rearrange the image into a data structure suitable for SVD without changing the image elements. Then, SVD is used for image inpainting modeling and to solve the optimization algorithm.

[0005] The technical solution to achieve the purpose of this invention is as follows:

[0006] An image inpainting method based on tensor singular value decomposition, characterized by the following steps:

[0007] Step 1: Input a polluted RGB image and obtain the image size. and the set of pixel locations in the contaminated area ;

[0008] Step 2: Rearrange the image data elements obtained in Step 1 using an image rearrangement scheme to obtain new third-order tensor data;

[0009] Step 3: Based on the new third-order tensor data, construct an image inpainting model based on the low tube rank constraint of tensor singular value decomposition;

[0010] Step 4: Set the maximum number of iterations, iteration error tolerance, empirical parameters and initialize variables, and solve the image restoration model based on the iterative optimization algorithm using the alternating direction multiplier method;

[0011] Step 5: Output the repaired image.

[0012] Furthermore, the specific operational steps of step 2 include:

[0013] Step 21: Copy the edge pixels of the contaminated RGB image to make the image length... and width It satisfies the geometric multiplication relationship of 2;

[0014] Step 22: Extract the four types of pixels (top left, bottom left, top right, and bottom right) from every other row and every other column, thus dividing the image into four smaller images;

[0015] Step 23: Repeat step 22 for each small image until the last number of sub-images is obtained. The length and width of the subgraph are on the same order of magnitude, base 10.

[0016] Step 24: The subgraphs are arranged into a size of The balanced third-order tensor.

[0017] Furthermore, the image inpainting model constructed in step 3 is as follows:

[0018] (1)

[0019] in, For image rearrangement operations, and For balancing parameters, This represents the tensor nuclear norm defined within the t-SVD decomposition framework. express:

[0020] .

[0021] Furthermore, the specific operational steps of step 4 include:

[0022] Step 41: Introduce auxiliary variables , The image restoration model is transformed into an augmented Lagrangian function minimization problem:

[0023] (2)

[0024] in, and It is the introduced balance coefficient. Describing the Frobenius norm, and This indicates the amount of residual introduced;

[0025] Step 42: When the number of iterations is satisfied and In the distributed iterative solution of problem (2), the following subproblems are solved sequentially in the s-th iteration:

[0026]

[0027]

[0028]

[0029] calculate:

[0030]

[0031]

[0032]

[0033] Step 43: Number of iterations Then return to step 42 until the iteration ends and the restored image is output.

[0034] An image inpainting system based on tensor singular value decomposition, characterized in that it includes an image input module, an image rearrangement module, an image inpainting module, and an image output module;

[0035] The image input module is used to input the polluted RGB image to be repaired;

[0036] The image rearrangement module is used to rearrange the input image data elements using an image rearrangement scheme to obtain new third-order tensor data.

[0037] The image restoration module is used to establish an image restoration model based on the third-order tensor data obtained from the image rearrangement module, and to restore the image by solving the image restoration model using an iterative optimization algorithm with alternating direction multipliers.

[0038] The image output module is used to output the repaired image.

[0039] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an image inpainting method based on tensor singular value decomposition.

[0040] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform an image inpainting method based on tensor singular value decomposition.

[0041] A computer program product includes a computer program that, when executed by a processor, implements an image inpainting method based on tensor singular value decomposition.

[0042] Beneficial effects:

[0043] First, this invention proposes an alternating row and column data rearrangement scheme, which transforms image data into a data structure suitable for tensor singular value decomposition without changing the total number of image data element values, and makes the new data structure have low tube rank, providing a new approach for image data restoration technology that can use the tensor singular value decomposition framework.

[0044] Secondly, this invention proposes a novel image inpainting model based on tensor singular value decomposition and provides an algorithm for solving the model. This method, to some extent, solves the problem that directly utilizing matrix low-rank properties for image inpainting often yields poor results in cases of image contamination such as blocky or striped patterns. Finally, comparative experimental analysis shows that, under various forms of image damage, the proposed inpainting method further improves the visual accuracy and numerical performance of image inpainting compared to traditional tensor decomposition methods and direct matrix low-rank inpainting methods. Attached Figure Description

[0045] Figures 1a-1f This is a schematic diagram of image contamination.

[0046] Figure 2 A schematic diagram illustrating the process of extracting four types of data elements from alternating rows and columns;

[0047] Figure 3 The Lena image is divided into four smaller Lena images;

[0048] Figure 4 The curves are arranged from largest to smallest pixel values ​​for four small Lena images;

[0049] Figure 5 This is a schematic diagram of an image rearrangement scheme;

[0050] Figure 6 The low tube rank property of the balanced tensor obtained after the image is rearranged is verified by the following curve.

[0051] Figure 7 The image shows a comparison of the repair effects of three methods. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0053] This invention addresses the image inpainting problem of low-dimensional image data contaminated by stripes, blocks, random pulses, and random character coverage, proposing an image inpainting method based on tensor singular value decomposition (SVD). First, a data rearrangement scheme is proposed to uncover potential low-rank prior information in the image. Then, image inpainting modeling and algorithm optimization based on SVD are performed. Finally, the program is executed to recover a high-precision image. (See attached reference.) Figures 1a-1f , Figure 1a , Figure 1c , Figure 1e This is the original image. Figure 1b , Figure 1d , Figure 1f The images show salt and pepper pulse pollution, streak pollution, and block pollution, respectively.

[0054] The image inpainting method based on tensor singular value decomposition proposed in this invention includes the following steps:

[0055] Step 1: Input a contaminated image and obtain an image size of [size missing]. Obtain the set of locations of contaminated pixels. ;

[0056] Step 2: Input the maximum number of iterations Iteration error tolerance Empirical parameters Set the initial number of iterations. ,initialization Auxiliary variables and residual variables and It is an all-zero tensor;

[0057] Step 3: Considering that tube-rank approximation of balanced tensors is more effective, a rearrangement scheme is used to transform the imbalanced image into a more balanced third-order tensor data, that is, to rearrange the image data elements to obtain a new third-order data structure.

[0058] Specifically, the rearrangement scheme is as follows:

[0059] Step 31: Copy the edge pixels to make the image length... and width It satisfies the geometric multiplication relationship of 2.

[0060] Step 32: Image Quartering Operation: Extract four categories of pixels—top left, bottom left, top right, and bottom right—from every row and every column. For example... Figure 2 As shown, the four types of pixels—top left, bottom left, top right, and bottom right—correspond to the elements in the blue, green, yellow, and red squares, respectively.

[0061] After extracting four types of data elements by alternating rows and columns, one image can yield four sub-images, such as... Figure 3 The Lena image shown can be divided into four smaller Lena images. Due to the local similarity of the images, the four smaller images obtained are similar, as shown below. Figure 4 The four curves of the four small Lena images shown, arranged from largest to smallest pixel values, have almost overlapped into one curve. This is also the reason why the rearranged "balanced" tensor obtained in the subsequent steps has low tube rank.

[0062] Step 33: Next, for each obtained sub-image, continue to extract the four types of pixels every other row and every other column, i.e., perform the image quartering operation. In this way, each sub-image can obtain four more sub-images, and so on. Through multi-level image quartering operations, when the number of sub-images obtained in the last level reaches a certain level, the image will be divided into four sub-images. Length of the last level subgraph and width When the order of magnitude is the same as that of base 10, the continued quadrature image operation ends.

[0063] Arrange all the last-level subgraphs in chronological order, stacking them into a format of size [size missing]. A third-order tensor. The size of this tensor. , , Longer than the original image tensor ,Width Depth 3 is more balanced (because) , and On the same order of magnitude, while , (The difference is significant, on the order of 3). This "balanced" third-order tensor is significantly larger than the "unbalanced" one. Third-order tensors are more suitable for t-SVD decomposition. For example... Figure 5 As shown, the "unbalanced" 256×256×3 Lena image can be transformed into a more "balanced" 64×64×48 third-order tensor through a three-level quartic image operation. At this point, the operation operator is defined. This indicates the process of steps 31-33.

[0064] To illustrate that the obtained balanced third-order tensor possesses the low-rank property, singular value curves are plotted, such as... Figure 6 As shown in the figure, the vertical axis It is represented as follows,

[0065] ,

[0066] Among them, let Indicates size is The image, then We obtain this from the following two formulas:

[0067]

[0068] ,

[0069] That is, first make the size of The "balanced" third-order tensor The tensor is obtained by performing a 1D-FFT along the third dimension. Then to Each slice Perform singular value decomposition to obtain a diagonal matrix. ,in Finally, it forms a tensor. . Figure 6 The horizontal axis represents .draw Figure 6 When selecting an image A Lena image of size 256×256×3.

[0070] According to the definition of tube rank in the tensor t-SVD decomposition framework Figure 6 The curves verify that the "balanced" third-order tensor obtained after rearrangement has low tube rank property.

[0071] Step 4: Construct an image inpainting model based on low tube rank constraints of tensor singular value decomposition;

[0072] By combining the low-rank property and the sparsity of the total variation domain, and introducing a TV (total variation) constraint term, the repaired model can be obtained:

[0073] (1)

[0074] in, and For balancing parameters, express

[0075] ,

[0076] This represents the tensor nuclear norm (TNN) defined within the t-SVD decomposition framework. The definition and calculation are as follows:

[0077]

[0078] in Indicates along a certain tensor The new tensor obtained by unfolding the third dimension into a 1D Fourier transform (1D-FFT) is, i.e. . Represents the nuclear norm. Indicates tensor Block diagonalization, i.e.

[0079] ,

[0080] in Indicates fixed The third dimension The slice matrix obtained at each position, i.e. , .

[0081] Step 5: Execute the iterative optimization algorithm based on the alternating direction multiplier method to solve model (1);

[0082] Introducing auxiliary variables and ,in , It is a difference operator, that is... ,in

[0083] ,

[0084] , .

[0085] This represents the sum of the absolute values ​​of all elements, i.e. .

[0086] Transform model (1) into an augmented Lagrangian function minimization problem:

[0087] (2)

[0088] in, and It is the introduced balance coefficient. Describing the Frobenius norm, and This represents the amount of residual introduced.

[0089] when as well as In the distributed iterative solution of problem (2), during the s-th iteration, the following subproblems are solved sequentially:

[0090] (3)

[0091] (4)

[0092] (5)

[0093] calculate: ,

[0094] , .

[0095] Specifically, solving problem (3) yields :

[0096] ,

[0097] Where F represents the two-dimensional Fourier transform (2D-FFT), express inverse transform, express The reverse operation, , Indicates the inverse difference operator. ,in

[0098] , .

[0099] Solving problem (4) yields The shrinkage threshold algorithm can be used to solve this problem in one step.

[0100] ,

[0101] in This indicates taking the absolute value. This represents the matrix dot product.

[0102] Solving problem (5) yields The following problem can be solved using the shrinking singular value thresholding algorithm to obtain... .

[0103] ,

[0104] The above solution algorithm is shown in Table 1.

[0105] Table 1 Solution Algorithm Based on ADMM Method

[0106]

[0107] Step 6: Output the repaired image: display and save.

[0108] Figure 7 Each column represents the contaminated graph, the low-rank matrix repair graph, the tensor Tucker decomposition repair graph, and the repair graph using the method of this invention. Figure 7 As can be seen, under salt-and-pepper pulse contamination, blocky contamination, and parallel line stripe contamination, the proposed restoration method can further improve the image restoration accuracy compared to traditional tensor decomposition methods and direct matrix low-rank property restoration methods. This method, to some extent, solves the problem that directly utilizing matrix low-rank properties for image restoration often yields poor results in cases of blocky or stripe-like contamination.

[0109] Contents not described in detail in this specification are existing technologies known to those skilled in the art. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of image inpainting based on tensor singular value decomposition, characterized in that, The method comprises the following steps: Step 1: input the contaminated RGB image, get the image size and the set of pixel positions of the contaminated area ; Step 2: rearranging the image data elements obtained in step 1 by using an image rearrangement scheme to obtain new third-order tensor data; Step 3: constructing an image inpainting model based on tensor singular value decomposition low tube rank constraint based on the new third-order tensor data; Step 4: setting the maximum number of iterations, iteration error tolerance, empirical parameters and initializing variables, and solving the image inpainting model based on an iterative optimization algorithm using an alternating direction multiplier method; Step 5: outputting the inpainted image; The image inpainting model constructed in step 3 is: (1) where is an image rearrangement operator, and is a balancing parameter, denotes the tensor nuclear norm defined under the t-SVD decomposition framework, denotes: ； The specific operation steps of step 4 include: Step 41 : Introduce auxiliary variables , Transform the image inpainting model into a minimization problem of an augmented Lagrangian function: （2） wherein and are introduced balance coefficients, denotes the Frobenius norm, and denote introduced residual quantities; Step 42: When the number of iterations and is satisfied, solve the problem (2) in a distributed manner, at the s-th iteration, solve the following sub-problems in turn: ； ； ； Calculate: ； ； ； Step 43: iteration number and return to step 42 until the iteration is finished and the recovered image is output.

2. The image inpainting method based on tensor singular value decomposition according to claim 1, characterized in that, The specific operation steps of step 2 include: Step 21: copy the edge pixels of the contaminated RGB image to make the image length and width satisfy the geometric multiplication relationship of 2; Step 22: every other row and every other column extracts left top, left bottom, right top and right bottom four types of pixel points, and divides a picture into four small pictures; Step 23: repeat step 22 for each small image until the last obtained number of sub-images with the length and width of the sub-images being on the same order of magnitude of 10. Step 24: Arrange the subgraphs into a balanced third-order tensor of size .

3. A system for image inpainting based on tensor singular value decomposition, characterized in that, The method implementation based on claim 1 comprises an image input module, an image rearrangement module, an image inpainting module and an image output module; The image input module is used for inputting a polluted RGB image to be inpainted; The image rearrangement module is used for rearranging the input image data elements by using an image rearrangement scheme to obtain new third-order tensor data; The image inpainting module is used for establishing an image inpainting model according to the third-order tensor data obtained by the image rearrangement module, solving the image inpainting model by using an iterative optimization algorithm of an alternating direction multiplier method, and inpainting the image; The image output module is used for outputting the inpainted image.

4. An electronic device, comprising: It comprises: At least one processor; And a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-2.

5. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein the computer instructions, when executed by a processor, cause the processor to perform the method of any one of claims 1-4. Wherein, The computer instructions are used to enable the computer to execute the method of any one of claims 1-2.

6. A computer program product, characterised in that, It comprises a computer program, which, when executed by a processor, implements the method of any one of claims 1-2. It comprises a computer program, which, when executed by a processor, implements the method of any one of claims 1-2.

Images