A method for image local texture stabilization field reconstruction based on structure tensor

Abstract

This invention discloses a stable field reconstruction method for local image texture based on structure tensor. First, the stable field is combined with image texture to construct a stable field reconstruction equation for the local texture of each missing pixel. Then, a priority weight function is defined using the eigenvalues ​​of the structure tensor, allowing missing points at the edges of the local missing region to be reconstructed according to the strength order of the structural components. Finally, the average coherence factor of the local region is constructed using the structure tensor, enabling missing pixels located in different structural regions of the image to adaptively select the corresponding effective field source range. Experimental results demonstrate that this method achieves more accurate reconstruction of images with different types of defects, especially showing excellent performance in reconstructing the edges and textures of image missing regions.

Description

Technical Field

[0001] This invention belongs to the technical field of damaged image reconstruction, specifically relating to a method for reconstructing local texture stable fields of images based on structure tensors. Background Technology

[0002] Image reconstruction refers to the process of reconstructing areas of an image that are missing information using known information in the image. Its goal is to restore the image to its original state and meet the visual requirements of the human eye. Early research mainly focused on the removal or filling of specific targets, but later it expanded to the digital reconstruction of precious cultural relics, as well as various interferences (such as occlusion, light spots, etc.) that occur during the image acquisition process.

[0003] Traditional reconstruction algorithms are mainly divided into two categories: one is reconstruction based on geometric features (Inpainting), and its representative algorithms include BSCB algorithm (Bertalmio M, Sapiro G, Caseles V, et al. Image inpainting[C]. Proceedings of the 27th annual conference on Computer graphics and interactive techniques.2000:417-424), TV algorithm (Shen J, Chan TF. Mathematical models for local nontexture inpaintings[J]. SIAM Journal on Applied Mathematics,2002,62(3):1019-1043) and CDD algorithm (Chan TF, Shen J. Nontexture inpainting by curvature-driven dif-fusions[J]. Journal of visual communication and image representation,2001,12(4):436-449). This type of algorithm uses partial differential equations (PDEs) as its core, spreading information from the known region to the inside of the missing region through multiple iterations and calculations of higher-order partial derivatives to complete the reconstruction. Another type is texture-filling-based reconstruction (Completion), with the Criminisi algorithm being a representative example (Criminisi A, Pérez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on image processing,2004,13(9):1200-1212). This type of algorithm targets large texture regions in the image, first selecting the block with the highest priority to be repaired within the missing region, and then traversing the entire image to find the block most similar to the missing region, completing the reconstruction by block filling. Both of these traditional image reconstruction algorithms require a large number of iterations during the reconstruction process, and do not perform detailed reconstruction for every missing pixel in the missing region, resulting in low reconstruction efficiency and unsatisfactory reconstruction accuracy.

[0004] To meet the higher requirements of image reconstruction in terms of accuracy and efficiency, Ye Xueyi et al. proposed a stable field-based image reconstruction model by using a stable field to describe the local texture of the image (Ye Xueyi, Qi Zhenzhen, He Zhiwei et al. Local region reconstruction based on the directional derivative of the image field [J]. Journal of Image and Graphics, 2014, 19(7): 998-1005), and first proposed the concept of accurate reconstruction. Li Xiaofei et al. considered the energy transfer relationship between known points and missing points, and assumed that the energy transferred from the known points is first transferred to virtual points in the nearest neighbor region around the missing points, and then the reconstruction is completed by nearest neighbor interpolation, bilinear interpolation and cubic convolution interpolation respectively (Li Xiaofei, Ye Xueyi, Chen Huiyun et al. Study on transfer function in stable field image reconstruction [J]. Journal of Image and Graphics, 2018, 23(3): 333-345). However, the image reconstruction method based on the stable field model mentioned above does not consider the reconstruction order of the missing pixels at the edge of the local defect area during the reconstruction process. In addition, it uses all known points in a fixed-size area as the effective field source around the defect point, without considering the structural features of the local image area where different defect points are located. This leads to the phenomenon of partial structural discontinuity and erroneous extension when reconstructing defect areas with relatively complex structures. Summary of the Invention

[0005] To address the accuracy issues of current image local texture stable field reconstruction algorithms, this invention provides an image local texture stable field reconstruction method based on structure tensors. Building upon the image local texture stable field reconstruction model, this method incorporates the characteristics of structure tensors during the selection of effective field sources, enabling more accurate reconstruction of missing pixels located in different structural feature regions of the image.

[0006] A method for reconstructing local texture stability fields in images based on structure tensors, comprising the following steps:

[0007] Step (1). Construct the stable field equation for the local texture of the image based on the stable field model;

[0008] By combining knowledge of image processing, the stable field in the mathematical physics model is combined with the local texture of the image to establish the stable field equation of the local texture of the image; then, the specific expression of the field source function of the equation is obtained by using the differential approximation method.

[0009] Step (2). Construct the priority weight function based on the structure tensor;

[0010] The structure tensor is used to analyze the geometric structure of local texture regions in an image. The structure tensor of each pixel at the edge of the local defect region and its two corresponding non-negative eigenvalues ​​are calculated. Then, the two eigenvalues ​​are used to construct a priority weight function to determine the reconstruction order of each defective pixel. The larger the priority weight function, the higher the reconstruction priority.

[0011] Step (3). Adaptively select the range of effective field sources based on the structure tensor;

[0012] Using the defect point at the edge of the local defect area as the center, a 9×9 template block is constructed. Two eigenvalues ​​of the structure tensor of each known point in the template block are calculated. Then, the average coherence factor of the template block is defined using these two average eigenvalues. The image is divided into edge regions, texture regions and flat regions. Different sizes of effective field source templates are used in different regions.

[0013] Step (4). Substitute the effective field source template selected in step (3) into the stable field equation of the local texture of the image in step (1) to obtain the final stable field reconstruction equation of the image. Use this equation to complete the reconstruction of each missing pixel in the local defect area after it has been sorted according to the priority formula proposed in step (2).

[0014] Step (1). The specific method is as follows:

[0015] Step (1.1). The local texture of the image is described using the steady-field model from mathematical physics. The specific formula is expressed as follows:

[0016] L·I(r)=f,(r∈D) (1)

[0017] In the formula, L represents the linear differential operator, r represents the spatial coordinates of the missing texture in the image, i.e. the missing pixel, I(r) is a function describing the missing texture region D in the image (for grayscale images, the value of I(r) represents the grayscale value of the corresponding spatial point; for color images, such as RGB images, it represents the value of any color channel), and f represents the field source that provides energy to the missing pixel from the known pixel.

[0018] Step (1.2). Select the defect point r located at the boundary of the defect area and the pixel point r located in the known area. i Substituting into equation (1), the specific formula is expressed as follows:

[0019] LG(r,r i )=δ(rr i (2)

[0020] In the formula, δ(rr) i This indicates that when only known pixel points r with unit intensity are considered... i The magnitude of the field source acting alone at the defect point r, G(r,r) i ) represents a known pixel r i The source function between the defect point r and the field source function.

[0021] Equation (2) is solved using the superposition principle in the Green's function method to obtain the stable field equation of the local texture of the image. The specific formula is expressed as follows:

[0022]

[0023] In the formula, n represents the total number of effective field sources around the defect point r, and I(r) i ) is r i The specific pixel value of a point.

[0024] Step (1.3). If a second-order partial derivative exists at a certain effective field source, that is, when the field source and its eight surrounding neighboring pixels are all known pixels, then the second-order Taylor formula can be used to obtain the approximate differential value at the defect point using the pixel value of the effective field source. The specific formula is expressed as follows:

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031] In the formula, (x,y) represents the coordinates of the defect point, (x0,y0) represents the coordinates of the effective field source, and p and q represent the normalized values ​​of the effective field source relative to the x-axis and y-axis coordinates of the defect point, respectively.

[0032] When there are missing pixels in the neighborhood of the effective source, if there is at least one known point in either the horizontal or vertical direction, then a first-order Taylor expansion is used to obtain the differential approximation of the missing pixel. The specific formula is expressed as follows:

[0033]

[0034]

[0035]

[0036] Step (1.4). The similarity factor sim(r,r) is designed using the difference ratio between the Taylor expansion estimate obtained in step (1.3) and the corresponding source pixel value. i The distance factor dst(r,r) is designed based on the fact that the effective field source attenuates relative to the square of the distance during the transmission of information to the defect point. iCombining the two influencing factors, a weighted average is used to obtain the final source function G(r,r). i The specific formula is expressed as follows:

[0037]

[0038]

[0039]

[0040] In the formula, g(r) i This can avoid including defect points in the calculation, when r i When it is a missing pixel, g(r) i When r = 0, i When the number of pixels is known, g(r) i ) = 1.

[0041] Step (2). The specific method is as follows:

[0042] Step (2.1). Calculate the structure tensor J of each defect point r at the edge of the local defect region. ρ The specific formula is expressed as follows:

[0043]

[0044]

[0045]

[0046] In the formula, I represents the gradient of the image. x and I y The partial derivatives in the x and y directions, respectively; G ρ It is a 2-dimensional Gaussian function with a mean of 0 and a variance of ρ. Because matrix J... ρ It is a symmetric and positive semi-definite 2D matrix, therefore it has two non-negative eigenvalues, as expressed in the following formula:

[0047]

[0048] In the formula, λ1 and λ2 represent the maximum and minimum eigenvalues ​​of the structure tensor at the pixel, respectively.

[0049] Step (2.2). Construct a priority weight function using the structural tensor eigenvalues ​​obtained in step (2.1) to determine the reconstruction order of each missing point. The larger the priority weight function, the earlier the point will be reconstructed. The specific formula is expressed as follows:

[0050] P(r)=1-exp[-(λ1-λ2)] 2 (20)

[0051] In the formula, when pixel r is at the edge of the structure, λ1-λ2>>0, then P(r)≈1; when r is in a flat region, λ1-λ2≈0, then P(r)≈0; when r is in a textured region, λ1-λ2>0, then 0<P(r)<1.

[0052] Step (3). The specific method is as follows:

[0053] Using the defect point at the edge of the local defect region as the center, construct a 9×9 template block. Calculate the two eigenvalues ​​of the structure tensor at each known point of the template block to obtain the average coherence factor (MOA) of the template block. ave The specific formula is expressed as follows:

[0054]

[0055] In the formula, MOA is the average eigenvalue of the structure tensor of known pixels in the template block. Based on extensive experiments, it is found that when the pixel block is located in the edge region, MOA... ave ≥0.9, in this case, select a distance r from the defect point that is no greater than The region is defined as the effective field source range; when in a textured region, 0.7 ≤ MOA ave <0.9, select a distance r from the defect point that is not greater than The region is used as the effective field source range; while in the texture region, MOA ave <0.7, select a distance r from the defect point that is not greater than The region is taken as the effective source range.

[0056] The beneficial effects of this invention are as follows:

[0057] This invention, based on an image texture stable field reconstruction model, fully utilizes the characteristic of structure tensors to reflect the local geometric structure of images. First, it designs a priority weight function using the eigenvalues ​​of the structure tensor, ensuring that missing points are reconstructed according to the strength of structural components. Then, it constructs a local coherence factor using the structure tensor, enabling adaptive selection of effective source regions of different sizes for reconstruction of different structural regions in the image, thus overcoming the drawbacks of structural discontinuities and erroneous extensions. Experimental results demonstrate that this method achieves more accurate reconstruction of images with different types of defects, particularly excelling in reconstructing edges and textures within defective regions. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method of the present invention;

[0059] Figure 2 This is a diagram of the local texture stabilization field model of the image in the method of the present invention;

[0060] Figure 3 According to MOA in the method of this invention ave The effective source range corresponding to the change; Detailed Implementation

[0061] The invention will be further described below with reference to the accompanying drawings.

[0062] like Figure 1 As shown, the specific steps of the method of the present invention are as follows:

[0063] Step (1). Construct the stable field equation for the local texture of the image based on the stable field model;

[0064] Using the steady-field model from mathematical physics to describe the local texture in an image, we can obtain, for example... Figure 2 The image shows a stable field model of local texture, where Ω is the boundary (Ω∈D) surrounding the image defect region; Ω1 is a circle with center r and radius arbitrarily small distance Δl, where r... i Let be any point on Ω1.

[0065] Step (1.1). The local texture of the image is described using the steady-field model from mathematical physics. The specific formula is expressed as follows:

[0066] L·I(r)=f,(r∈D) (1)

[0067] In the formula, L represents the linear differential operator, r represents the spatial coordinates of the missing texture in the image, i.e. the missing pixel, I(r) is a function describing the missing texture region D in the image (for grayscale images, the value of I(r) represents the grayscale value of the corresponding spatial point; for color images, such as RGB images, it represents the value of any color channel), and f represents the field source that provides energy to the missing pixel from the known pixel.

[0068] Step (1.2). Select the defect point r located at the boundary of the defect area and the pixel point r located in the known area. i Substituting into equation (1), the specific formula is expressed as follows:

[0069] LG(r,r i )=δ(rr i (2)

[0070] In the formula, δ(rr) i This indicates that when only known pixel points r with unit intensity are considered... i The magnitude of the field source acting alone at the defect point r, G(r,r) i ) represents a known pixel r i The source function between the defect point r and the field source function.

[0071] Equation (2) is solved using the superposition principle in the Green's function method to obtain the stable field equation of the local texture of the image. The specific formula is expressed as follows:

[0072]

[0073] In the formula, n represents the total number of effective field sources around the defect point r, and I(r) i ) is r i The specific pixel value of a point.

[0074] Step (1.3). If a second-order partial derivative exists at a certain effective field source, that is, when the field source and its eight surrounding neighboring pixels are all known pixels, then the second-order Taylor formula can be used to obtain the approximate differential value at the defect point using the pixel value of the effective field source. The specific formula is expressed as follows:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] In the formula, (x,y) represents the coordinates of the defect point, (x0,y0) represents the coordinates of the effective field source, and p and q represent the normalized values ​​of the effective field source relative to the x-axis and y-axis coordinates of the defect point, respectively.

[0082] When there are missing pixels in the neighborhood of the effective source, if there is at least one known point in either the horizontal or vertical direction, a first-order Taylor expansion is used to obtain the differential approximation of the missing pixel. The specific formula is expressed as follows:

[0083]

[0084]

[0085]

[0086] Step (1.4). The similarity factor sim(r,r) is designed using the difference ratio between the Taylor expansion estimate obtained in step (1.3) and the corresponding source pixel value. i The distance factor dst(r,r) is designed based on the fact that the effective field source attenuates relative to the square of the distance during the transmission of information to the defect point. iCombining the two influencing factors, a weighted average is used to obtain the final source function G(r,r). i The specific formula is expressed as follows:

[0087]

[0088]

[0089]

[0090] In the formula, g(r) i This can avoid including defect points in the calculation, when r i When it is a missing pixel, g(r) i When r = 0, i When the number of pixels is known, g(r) i ) = 1.

[0091] Step (2). Construct the priority weight function based on the structure tensor;

[0092] Step (2.1). Calculate the structure tensor J of each defect point r at the edge of the local defect region. ρ The specific formula is expressed as follows:

[0093]

[0094]

[0095]

[0096] In the formula, I represents the gradient of the image. x and I y The partial derivatives in the x and y directions, respectively; G ρ It is a 2-dimensional Gaussian function with a mean of 0 and a variance of ρ. Because matrix J... ρ It is a symmetric and positive semi-definite 2D matrix, therefore it has two non-negative eigenvalues, as expressed in the following formula:

[0097]

[0098] In the formula, λ1 and λ2 represent the maximum and minimum eigenvalues ​​of the structure tensor at the pixel, respectively.

[0099] Step (2.2). Construct a priority weight function using the structural tensor eigenvalues ​​obtained in step (2.1) to determine the reconstruction order of each missing point. The larger the priority weight function, the earlier the point will be reconstructed. The specific formula is expressed as follows:

[0100] P(r)=1-exp[-(λ1-λ2)] 2 (20)

[0101] In the formula, when pixel r is at the edge of the structure, λ1-λ2>>0, then P(r)≈1; when r is in a flat region, λ1-λ2≈0, then P(r)≈0; when r is in a textured region, λ1-λ2>0, then 0<P(r)<1.

[0102] Step (3). Adaptively select the range of effective field sources based on the structure tensor;

[0103] Using the defect point at the edge of the local defect region as the center, construct a 9×9 template block. Calculate the two eigenvalues ​​of the structure tensor at each known point of the template block to obtain the average coherence factor (MOA) of the template block. ave The specific formula is expressed as follows:

[0104]

[0105] In the formula, This represents the average eigenvalue of the structure tensor for known pixels in the template block. Based on extensive experiments, the following is obtained: Figure 3 As shown in MOA ave The effective field source range corresponding to the change. When the pixel block is in the edge region, MOA ave ≥0.9, in this case, select a distance r from the defect point that is no greater than The region is defined as the effective field source range (r1~r8); when in the texture region, 0.7≤MOA ave <0.9, select a distance r from the defect point that is not greater than The region is taken as the effective source range (r1~r 20 When in a textured area, MOA ave <0.7, select a distance r from the defect point that is not greater than The region is taken as the effective source range (r1~r 44 ).

[0106] Step (4). Substitute the effective field source template selected in step (3) into the stable field equation of the local texture of the image in step (1) to obtain the final stable field reconstruction equation of the image. Use this equation to complete the reconstruction of each missing pixel in the local defect area after it has been sorted according to the priority formula proposed in step (2).

[0107] It should be noted that the above embodiments can be freely combined as needed. The above description is only a detailed explanation of the preferred embodiments and principles of the present invention, but is not intended to limit the present invention. For those skilled in the art, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A method for reconstructing local texture stability fields in images based on structure tensors, characterized in that, The steps are as follows: Step (1). Construct the stable field equation for the local texture of the image based on the stable field model; By combining knowledge of image processing, the stable field in the mathematical physics model is combined with the local texture of the image to establish the stable field equation of the local texture of the image. Then, the specific expression of the field source function in the equation is obtained by using the differential approximation method; Step (2). Construct the priority weight function based on the structure tensor; The structure tensor is used to analyze the geometric structure of local texture regions in an image. The structure tensor of each pixel at the edge of the local defect region and its two corresponding non-negative eigenvalues ​​are calculated. Then, the two eigenvalues ​​are used to construct a priority weight function to determine the reconstruction order of each defective pixel. The larger the priority weight function, the higher the reconstruction priority. Step (3). Adaptively select the range of effective field sources based on the structure tensor; Construct a system centered on the defect point at the edge of the local defect area. The template block of a certain size is used to calculate the two average eigenvalues ​​of the structure tensor of each known point within the template block. Then, the average coherence factor of the template block is defined using these two average eigenvalues. The image is divided into edge regions, texture regions, and flat regions, and different sizes of effective field source templates are used in different regions. Step (4). Substitute the effective field source template selected in step (3) into the stable field equation of the local texture of the image in step (1) to obtain the final stable field reconstruction equation of the image. Use this equation to complete the reconstruction of each defective pixel in the local defective region after it has been sorted according to the priority function proposed in step (2).

2. The image local texture stabilization field reconstruction method based on structure tensor according to claim 1, characterized in that, Step (1). The specific method is as follows: Step (1.1). The local texture of the image is described using the steady-field model from mathematical physics. The specific formula is expressed as follows: In the formula, Represents the linear differential operator. The spatial coordinates of the missing texture in a localized area of ​​an image, i.e., the missing pixels. It describes the local texture loss area of ​​an image. The function, for grayscale images, The value represents the grayscale value of the corresponding spatial point; for a color image, it represents the value of any one of its color channels. This indicates that a known pixel provides energy to a missing pixel. Step (1.2). Select the defect point located at the boundary of the defect area. and pixels located in the known area Substituting into equation (1), we obtain the following formula: In the formula, This indicates that only known pixels with unit intensity are considered. Acting alone on the defect point The size of the field source at that time, Represents known pixel points With defects The source functions between; Equation (2) is solved using the superposition principle in the Green's function method to obtain the stable field equation of the local texture of the image; the specific formula is expressed as follows: In the formula, Indicates the defect point The total number of effective field sources in the surrounding area, yes Determine the pixel value of the point; Step (1.3). If a second-order partial derivative exists at a certain effective field source, that is, when the field source and its 8 surrounding neighboring pixels are all known pixels, then the second-order Taylor formula can be used to obtain the differential approximation at the defect point using the pixel value of the effective field source; the specific formula is expressed as follows: In the formula, Indicates the coordinates of the missing point. Represents the coordinates of the effective field source. , These represent the effective field sources relative to the defect points. axis coordinates and Normalized values ​​of axis coordinates; When there are missing pixels in the neighborhood of the effective source, if there is at least one known point in either the horizontal or vertical direction, the differential approximation of the missing point is obtained using a first-order Taylor expansion; the specific formula is expressed as follows: Step (1.4). The similarity factor is designed using the ratio of the difference between the Taylor expansion estimate obtained in step (1.3) and the corresponding source pixel value. The distance factor is designed based on the fact that the effective field source attenuates relative to the square of the distance during the transmission of information to the defect point. The final source function is obtained by combining the two influencing factors and applying a weighted average. The specific formula is expressed as follows: In the formula, This avoids including defect points in the calculation. When it is a missing pixel, ,when When the number of pixels is known, .

3. The image local texture stabilization field reconstruction method based on structure tensor according to claim 2, characterized in that, Step (2). The specific method is as follows: Step (2.1). Calculate each defect point at the edge of the local defect area. Structure tensor The specific formula is expressed as follows: In the formula, Represents the gradient of an image. and These are the partial derivatives in the x and y directions, respectively; It is a 2D Gaussian function with a mean of 0 and a variance of K; due to the matrix It is a symmetric and positive semi-definite 2D matrix, therefore it has two non-negative eigenvalues, as expressed in the following formula: In the formula, , These represent the maximum and minimum eigenvalues ​​of the structure tensor at the pixel point, respectively; Step (2.2). Construct a priority weight function using the structural tensor eigenvalues ​​obtained in step (2.1) to determine the reconstruction order of each missing point. The larger the priority weight function, the earlier the point will be reconstructed. The specific formula is expressed as follows: In the formula, when the pixel point When at the edge of the structure, ,but ;when When in a flat area, ,but ;when When in a textured area, ,but .

4. The image local texture stabilization field reconstruction method based on structure tensor according to claim 3, characterized in that, Step (3). The specific method is as follows: Construct a system centered on the defect point at the edge of the local defect area. Given a template block of a certain size, calculate the two average eigenvalues ​​of the structure tensor at each known point of the template block to obtain the average coherence factor of the template block. The specific formula is expressed as follows: In the formula, , The average eigenvalue of the structure tensor of known pixels in the template block; Based on extensive experiments, it was concluded that when a pixel block is located in an edge region, At this point, select the distance from the defect point. Not greater than The region is considered the effective source area; when in a flat region, Select the distance to the defect point Not greater than The region is considered the effective field source range; while in the texture region, Select the distance to the defect point Not greater than The region is taken as the effective source range.

Images