An image inpainting method based on a quartering rearrangement scheme

Abstract

The application discloses an image repairing method based on a four-part rearrangement scheme, and solves the repairing problem of an image under multiple typical damages, especially under a stripe scratch. Traditional image repairing technology based on a matrix low-rank constraint has low repairing precision under a stripe scratch damage form. In order to fully exploit the potential low-rank prior information of the image, the application first proposes an image data four-part rearrangement scheme. The image data is rearranged into a strong low-rank matrix without changing the total number of image data element values; then the matrix low-rank constraint of the rearranged data is restrained, and a novel image repairing mathematical model is proposed in combination with the local smoothness prior constraint of the image; finally, an optimization solving algorithm is given. The application can achieve good repairing effect of the image, and can provide a technical idea based on a data structure rearrangement scheme for image repairing under multiple typical image damage forms, especially under a stripe scratch.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image restoration method based on a quartering rearrangement scheme. Background Art

[0002] Image data, such as satellite imagery, infrared imagery, remote sensing imagery, and visible light imagery, holds significant intelligence value in a wide range of applications. During the image formation, acquisition, communication, and storage processes, image data is susceptible to various factors, resulting in localized information loss or corruption. Typical forms of image corruption include streaks, blocks, random pulses, random symbols, and random streaks. Image restoration for these various types of corruption—filling missing pixels or repairing damaged image regions—is a classic ill-posed inverse problem.

[0003] Traditional image restoration methods generally use image filtering, interpolation, and constrained optimization modeling methods to solve the problem. Among them, the constrained optimization modeling method solves the problem by constraining the image prior knowledge (local smoothness of the image, sparsity of part of the image, low rank of the image, etc.), such as constructing sparse coding subspace constraints, sparsity constraints, total variation constraints, matrix low rank constraints, etc. Restoration models based on matrix low rank constraints, such as in video restoration and dynamic magnetic resonance imaging methods, are widely used to first expand the video or dynamic image into a matrix by frame arrangement (since each frame that changes slightly along the time dimension has similarity, the expanded matrix has low rank), and then constrain the low rank of the matrix for image restoration modeling.

[0004] These traditional image restoration methods suffer from issues such as inability to address diverse image damage forms and insufficient utilization of prior image features. Restoration methods based on matrix low-rank properties often fail to fully exploit the underlying low-rank prior structural features within the image. Furthermore, existing restoration methods produce poor results for data damage forms such as streaks and random lines. Summary of the Invention

[0005] To address these issues, this paper proposes an image restoration method based on a quartering scheme, leveraging and exploiting the low-rank prior of image data. This method first breaks down the original arrangement of the image data, proposes an effective quartering scheme, and then provides an image restoration model and solution algorithm based on the low-rank nature of matrices.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] An image restoration method based on a quartering rearrangement scheme, characterized by comprising the following steps:

[0008] Step 1: Input the damaged image, obtain the damaged area of ​​the image, and establish a set of damaged pixel locations

[0009] Step 2: Get the size of the damaged image and pre-adjust the image size;

[0010] Step 3: According to the data quartering rearrangement scheme, the pre-adjusted image is transformed into a low-rank structure matrix, and this process is represented by the operation operator MΦ;

[0011] Step 4: Establish an image restoration model;

[0012] Step 5: Construct a repair algorithm based on the alternating direction multiplier method ADMM algorithm;

[0013] Step 6: Use the repair algorithm to solve the image repair model established in step 4, perform data repair, and finally obtain the repaired image.

[0014] Furthermore, when performing pre-adjustment of the image size in step 2, the length and width of the image are both made to satisfy a geometric multiplication relationship of 2, and the insufficient length and width are expanded by copying the edge pixels. The specific steps include:

[0015] Step 21: Get the damaged RGB image, assuming the size is a×b×3;

[0016] Step 22: According to the requirements of the quartering rearrangement scheme, the damaged RGB image is pre-adjusted to an image of size m×n×3, where m≥a,n≥b.

[0017] Furthermore, the steps of step 3 include:

[0018] Step 31: According to the quartering scheme, the m×n×3 image is divided into 4 blocks of size subgraph of ;

[0019] Step 32: Perform the quartering scheme again on each subgraph obtained in step 31 to obtain 16 subgraphs of size subgraph of ;

[0020] Step 33: Perform the quartering scheme again on each subgraph obtained in step 32 to obtain 64 subgraphs of size subgraph of ;

[0021] Step 34: Arrange the pixel values ​​of each frame of the 64 sub-images in the order of rows first and columns to form 192 pixels of size vector of

[0022] Step 35: Expand the vectors obtained side by side according to the order of extraction by the quarter-rearrangement scheme. The side-by-side expansion process is represented by the operation operator M, and the steps 31-34 are represented by the operation operator Φ, and the expanded matrix is ​​the final low-rank structure matrix.

[0023] Furthermore, the steps of the four-part rearrangement scheme in step 31 are:

[0024] Step 311: for an image matrix L of given image data, the size of L is l×l;

[0025] Step 312: Extract element values ​​of the matrix L in every other row and column, and rearrange them in the order of extraction to obtain four smaller matrices P. The size of each matrix P is p×p, and p=l / 2.

[0026] Furthermore, the steps of step 4 include:

[0027] Step 41: Based on the obtained low-rank structure matrix, an image restoration model based on the low-rank structure matrix under the quartering rearrangement scheme is given:

[0028] satisfy

[0029] Where X represents the repaired image, Φ represents the operator that splits the image into multiple sub-image data through multiple image quartering schemes, M represents the operator that expands multiple sub-image data into a low-rank structure matrix in order, Ω is the position set of undamaged pixels, and Y is the damaged image;

[0030] Step 42: In model (1), use formula (2) to constrain the low rank of the matrix:

[0031]

[0032] Among them, U and V are obtained by decomposing the MΦX matrix, H represents the matrix transpose conjugate, ||·|| F represents the Frobenius norm;

[0033] Step 43: Combine model (1) with the TV term to propose an image restoration model with sparsity constraints

[0034] satisfy

[0035] where β is the regularization parameter.

[0036] Furthermore, the steps of step 6 include:

[0037] Step 61: Introduce an auxiliary variable Z = DX, where D is a finite difference operator, and rewrite model (3) as an unconstrained convex optimization problem, that is:

[0038]

[0039] where τ Ω (X) means:

[0040]

[0041] L and Λ are variables Z and UV respectively H The Lagrange multiplier of ; the regularization parameters ρ1>0 and ρ2>0 are used to balance the penalty parameters of low rank and sparsity constraints respectively;

[0042] Step 62: Adopting the ADMM solution algorithm, iteratively solve the unconstrained convex optimization problem in step 61, and perform image restoration based on the obtained optimal solution to obtain a restored image.

[0043] Furthermore, the solution steps of step 62 are:

[0044] Step 621: Set Y,Ω,ρ1,β,ρ2, and the maximum number of iterations t max , convergence condition η tol ;

[0045] Step 622: Initialize U by solving equation (4) (0) and V (0) , let L (0) =0, Z (0) =0, t=0:

[0046] satisfy

[0047] Step 623: When t <t max and η<η tol When , the loop iterates and calculates in the tth iteration:

[0048]

[0049]

[0050]

[0051]

[0052] Λ t =Λ t-1 +MΦ1X t -U t V (t)H (9)

[0053] L t =L t-1+DX t -Z t (10);

[0054] Step 624: Obtain X by solving (5)-(10) * , where * represents the optimal solution;

[0055] Step 625: Update

[0056] Step 626: When t>t max or η>η tol When , the loop iteration ends and the optimal solution X is output * .

[0057] The beneficial effects of the present invention are:

[0058] The present invention proposes an image restoration method based on a quartering rearrangement scheme, which searches for potential low-rank structural prior features of RGB images by rearranging the data structure of the original image. The quartering rearrangement scheme proposed by the present invention can arrange the image into a new data structure form, and the rearranged data has a corresponding low-rank structure. In addition, the present invention also proposes an image restoration model based on the quartering rearrangement scheme. Considering that the constraint on low rank alone cannot effectively utilize the local smoothness and segmentation prior information of the image, the present invention adds local smoothness prior information constraints of visual data by combining two-dimensional total variation.

[0059] Experimental data show that the expanded matrix constructed by the present invention under the quartering rearrangement scheme without changing the total number of elements in the original image can exhibit good low-rank prior characteristics. In addition, according to experimental data, the proposed image data restoration technology based on the quartering rearrangement scheme can achieve the purpose of improving image restoration accuracy and can realize restoration under various typical image damage forms, especially striped and randomly drawn damage forms. This overcomes the problem of low restoration accuracy of traditional image restoration technology based on matrix low-rank constraints under various typical damage forms, especially striped and randomly drawn damage forms, and provides a new idea for data rearrangement schemes for solving image restoration processing problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is an example of a typical form of image damage;

[0061] Figure 2 This is an example of a matrix four-partitioning scheme;

[0062] Figure 3 This is an example of a four-division scheme for a two-dimensional image;

[0063] Figure 4 is the normalized pixel value curve of the four sub-images under the quarter-division scheme;

[0064] Figure 5 This is an example of the process of quartering an RGB image into a low-rank structure matrix;

[0065] Figure 6 Normalized singular value curve of the low-rank structure matrix of the RGB image under the quarter-permutation scheme (used to demonstrate the low-rank property of the matrix);

[0066] Figure 7 This is a comparison chart of the repair effects of the method of the present invention. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0068] The present invention proposes an image restoration method based on a quartering rearrangement scheme, which comprises the following steps:

[0069] Step 1: Input the damaged image, obtain the damaged area of ​​the image, and establish a set of damaged pixel locations

[0070] Step 2: Obtain the size of the damaged image and pre-adjust the image data size so that the adjusted image meets the quartering rearrangement scheme;

[0071] Step 3: According to the data quartering scheme, the pre-adjusted image is transformed into a low-rank structure matrix (the total number of pixels remains unchanged, but the structure changes), and this process is defined as an operation operator MΦ;

[0072] Step 4: Build an image data restoration model:

[0073] Using the constructed potential low-rank structure matrix of the image, an image restoration model based on the low-rank structure matrix under the quartering rearrangement scheme is proposed:

[0074] satisfy

[0075] Where X represents the image to be repaired, Φ represents the image splitting into multiple sub-image data by the cubic image quartering scheme, M represents the operator that expands the multiple sub-image data into a low-rank structure matrix in order, Ω is the position set of the undamaged pixel points, and Y is the damaged image;

[0076] In order to reduce the computational complexity, in the proposed model (1), the minimum Frobenius norm of the two factorized matrices is used instead of the nuclear norm to reduce the time of singular value decomposition (SVD), that is, the following relationship is used to constrain the low rank of the unfolded matrix:

[0077]

[0078] In addition to low rank, another common prior information is the local smoothness of visual image data. Combining the proposed model (1) with the TV term, an image restoration model is proposed:

[0079] satisfy

[0080] Where β is the regularization parameter;

[0081] Step 5: Construct the restoration algorithm: Use the alternating direction method of multipliers (ADMM) algorithm to solve the proposed image restoration model (1) and model (3);

[0082] Step 6: Execute the algorithm, perform iterative operations, perform data repair, and finally obtain the repaired image.

[0083] Example

[0084] The present invention aims to solve the problem that the traditional image restoration technology based on matrix low rank constraint has low restoration accuracy in a variety of typical damage forms (such as stripes, blocks, random pulses, random symbols, random line stripes, etc.), especially stripes and random line damage forms. These typical image damage forms are as shown in the attached figure. Figure 1 As shown, Figure 1 a is a random point, Figure 1 b is a random symbol, Figure 1 c is striped, Figure 1 d is a random line, Figure 1 e is blocky, Figure 1 f is a random pulse.

[0085] For these damaged images, during the processing, the image pre-adjustment size should try to meet the geometric multiplication relationship of 2. The image size can only be increased and not decreased. When the image size is increased, the insufficient pixel entries are copied and expanded with the edge pixels. Establish a data quartering rearrangement scheme, such as Figure 2 Taking a simple matrix as an example, the steps of the data quartering scheme include:

[0086] S1: Matrix M of given data, size of M is 2 q ×2 p ;

[0087] S2: Extract the element values ​​of the matrix M in every other row and column, and rearrange them in the order of extraction to obtain four smaller matrices, each of which has a size of 2 q-1 ×2 p-1 .

[0088] Furthermore, the present invention is mainly used for RGB image restoration, so it is necessary to establish a data quartering scheme for RGB images. Since RGB images can be regarded as three channel matrices of R, G, and B, according to experience, each matrix requires three quartering rearrangements, that is, executing steps S1-S2 three times.

[0089] Applying the data quartering scheme on a single-channel image, the image can be divided into four smaller images, as shown in the attached figure. Figure 3 As shown in the figure, since the pixel value elements are adjacent during the quartering process, the four smaller images are similar to each other, as shown in the figure. Figure 4 As shown. Figure 4 It can be seen from the figure that after the pixel values ​​of the four smaller images are normalized, the arrangement curves from large to small overlap and become one, which shows that the four smaller images are similar to each other in pixel value structure.

[0090] For example, in order to mine the potential low-rank structural features of an image, the potential low-rank structural matrix of the image is constructed according to the quartering rearrangement scheme, such as Figure 5 The specific steps are as follows:

[0091] 1. Get the RGB image Lena image, size 256×256×3;

[0092] 2. The Lena graph is rearranged into four subgraphs of size 128×128×3 using the proposed quartering scheme;

[0093] 3. Then, the four subgraphs are rearranged into 64×64×3 subgraphs using the proposed quartering scheme (the number of subgraphs is 16 at this time);

[0094] 4. Then, the 16 subgraphs are rearranged into 32×32×3 subgraphs using the proposed quartic rearrangement scheme (the number of subgraphs is 64 at this time);

[0095] 5. Arrange the pixel values ​​of each frame of these 64 sub-images (each sub-image is divided into three channels of R, G, and B, and the 64 sub-images have a total of 192 frames) in the order of row first and then column (or column first and then row, etc.) to form a 1024×1 vector;

[0096] 6. Finally, the 192 vectors of the 192 frames are expanded side by side in the order of the above-mentioned quarter-rearrangement scheme into a 1024×192 size matrix. Since the sub-graphs obtained according to the quarter-rearrangement scheme have high similarity, the rank of the expanded matrix is ​​low. Figure 6 The singular values ​​of the 1024×192 expanded matrix are normalized and arranged from large to small. The number of singular values ​​close to zero accounts for the majority, so it can be seen that the matrix has a strong low rank property.

[0097] According to experience, after three quarter-rearrangement schemes, the low-rank effect of the expanded structure matrix is ​​better, which can be explained as the similarity between the adjacent 8 pixel elements makes the expanded structure matrix have obvious low-rank characteristics;

[0098] The process of constructing the potential low-rank structure matrix of the image without changing the total number of pixel elements is defined as the operation operator. Correspondingly, the inverse process is defined as the inverse operation operator.

[0099] Furthermore, a restoration algorithm based on the alternating direction method of multipliers (ADMM) is constructed to solve the low-rank expansion matrix-based image restoration model (1) and the joint TV-based image restoration model (3). Taking the solution of model (3) as an example, the specific operations are as follows:

[0100] Introduce an auxiliary variable Z = DX, where D is a finite difference operator, and rewrite model (3) as an unconstrained convex optimization problem as follows:

[0101]

[0102] where τ Ω (X) means:

[0103]

[0104] Where L and Λ are variables Z and UV respectively. H The regularization parameters ρ1>0 and ρ2>0 are used to balance the penalty parameters of low-rank and sparsity constraints (i.e. TV), respectively, and usually affect the convergence of the algorithm.

[0105] The ADMM iterative solution algorithm is expanded below. Each subproblem is executed as follows in the tth iteration:

[0106]

[0107]

[0108]

[0109]

[0110] Λ t =Λ t-1 +MΦ1X t -U t V (t)H (8)

[0111] L t =L t-1 +DX t -Z t (9)

[0112] The initial values ​​of U and V can be determined by solving the following optimization problem.

[0113] satisfy

[0114] Table 1 ADMM solution algorithm

[0115]

[0116]

[0117] The main problem solved by this invention is the low repair accuracy of typical damaged images. Figure 7 The figure shows a comparison of the effects of the striped damaged Lena image, the random salt and pepper pulse interference House image, and the random block damaged Baboon image after being repaired by the method of the present invention. The image low-rank constraint repair method refers to a method of directly performing low-rank constraint repair on the image matrix.

[0118] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An image restoration method based on a quarter-permutation scheme, characterized in that: The following steps are involved: Step 1: Input the damaged image, obtain the damaged area of ​​the image, and establish a set of damaged pixel locations ; Step 2: Get the size of the damaged image and pre-adjust the image size; Step 3: According to the data quartering scheme, the pre-adjusted image is transformed into a low-rank structure matrix, and this process is performed using the operator express; Step 4: Establish an image restoration model; Step 5: Construct a repair algorithm based on the alternating direction multiplier method ADMM algorithm; Step 6: Using the inpainting algorithm to solve the image inpainting model established in step 4, perform data inpainting, and finally obtain the inpainted image; The specific steps of the four-part rearrangement scheme are as follows: Step 311: for an image matrix L of given image data, the size of L is l×l; Step 312: Extract element values ​​from every other row and column of the matrix L, and rearrange them in the order of extraction to obtain four smaller matrices P, each of which has a size of p×p, and p=1 / 2; And, the steps of step 4 include: Step 41: Based on the obtained low-rank structure matrix, an image restoration model based on the low-rank structure matrix under the quartering rearrangement scheme is given: （1） in, represents the restored image, Represents an operator that splits an image into multiple sub-image data through multiple image quartering schemes. Represents an operator that expands multiple subgraph data into a low-rank structure matrix in order. is the set of locations of undamaged pixels, It is a damaged image; Step 42: In model (1), use formula (2) to constrain the low rank of the matrix: （2） Among them, U and V are Matrix decomposition is obtained, H represents the matrix transpose conjugate, represents the Frobenius norm; Step 43: Combine model (1) with the TV term to propose an image restoration model with sparsity constraints （3） in is the regularization parameter.

2. The image restoration method based on the quartering rearrangement scheme according to claim 1, characterized in that: When performing pre-adjustment of the image size in step 2, the length and width of the image are both made to satisfy the geometric multiplication relationship of 2. If the length and width are insufficient, the edge pixels are copied and expanded. The specific steps include: Step 21: Get the broken RGB image, assuming the size is ; Step 22: According to the requirements of the quartering scheme, the damaged RGB image is pre-adjusted to the size of images, where .

3. The image restoration method based on the quartering rearrangement scheme according to claim 1, characterized in that: The steps in Step 3 include: Step 31: According to the quarter-rearrangement scheme, The image is divided into 4 parts of size subgraph of ; Step 32: Perform the quartering scheme again on each subgraph obtained in step 31 to obtain 16 subgraphs of size subgraph of ; Step 33: Perform the quartering scheme again on each subgraph obtained in step 32 to obtain 64 subgraphs of size subgraph of ; Step 34: Arrange the pixel values ​​of each frame of the 64 sub-images in the order of rows first and columns to form 192 pixels of size vector of Step 35: Expand the vectors obtained side by side according to the order of extraction by the quarter-rearrangement scheme. The size of the matrix, the side-by-side expansion process is performed using the operator Indicates that steps 31-34 are processed with operators Represented by , and the expanded matrix is ​​the final low-rank structure matrix.

4. The image restoration method based on the quartering rearrangement scheme according to claim 3, characterized in that: The steps in Step 6 include: Step 61: Introduce an auxiliary variable ,in is a finite difference operator, and model (3) is rewritten as an unconstrained convex optimization problem, namely: in express: ， and They are variables and Lagrange multiplier of ; regularization parameter and Penalty parameters used to balance low-rank and sparsity constraints respectively; Step 62: Adopting the ADMM solution algorithm, iteratively solve the unconstrained convex optimization problem in step 61, and perform image restoration based on the obtained optimal solution to obtain a restored image.

5. The image restoration method based on the quartering rearrangement scheme according to claim 4, characterized in that: The solution steps for step 62 are: Step 621: Setup , maximum number of iterations , convergence condition ; Step 622: Initialize by solving equation (4) and ,make , , : （4）； Step 623: When as well as When , the loop iterates and calculates in the tth iteration: （5） （6） （7） （8） （9） （10）； Step 624: Obtain by solving (5)-(10) ,in represents the optimal solution; Step 625: Update ; Step 626: When or When , the loop iteration ends and the optimal solution is output .

Images