Correction method of creased document images based on multiple views

Abstract

The present invention discloses a method for correcting creased document images based on multiple views, comprising the following steps: Step 1: Acquire a multi-view image; Step 2: Obtain three-dimensional point cloud data of the document; Step 3: Generate an image space grid; Step 4: Generate an initial three-dimensional grid; Step 5: Generate an initial unfolded grid; Step 6: Optimize the grid's unfolding; and Step 7: Generate an image. The present invention provides a computational framework for solving the document image correction problem, which simultaneously computes a three-dimensional grid M and a two-dimensional unfolded grid Ω, and applies image structural texture constraints (such as straight lines) to the two-dimensional unfolded grid Ω. Therefore, the method of the present invention can directly obtain a textured two-dimensional unfolded result under the guidance of the structural texture information of the document image, avoiding the errors introduced by the independent grid unfolding step in existing methods.

Description

Technical Field

[0001] The present invention relates to a correction method for a deformed document image, and in particular to a correction method for a creased document image based on multiple views. Background Art

[0002] Document objects in images often exhibit geometric deformations, such as bending or folding of paper. Geometric correction of deformed document images processes the input document image to produce an undistorted document image. This problem has important applications in camera-based scanner algorithms, text recognition in document images, and the digitization of ancient documents.

[0003] Document image correction based on images captured by consumer cameras or mobile phone cameras has attracted extensive research due to its practicality. Existing methods have demonstrated that two types of information in the input image can be exploited to improve document image correction. One type is the document image's texture information, such as the arrangement of text within the document, table structure, and other structural information; the other type is the document's three-dimensional geometric information, leveraging the geometric fact that a document's three-dimensional model is a developable model. Existing methods include estimating pixel depth information based on the brightness of pixels in the image, thereby exploiting this three-dimensional geometric information; some methods utilize image boundary information to guide document image correction; and some methods guide document image correction by calculating the flow field of the texture structure within the document image. In recent years, deep learning methods have also been used to address document image correction, achieving superior results compared to traditional methods. However, these existing methods generally target curved documents and are less effective for documents with significant creases.

[0004] Multi-view-based methods use input images of a document object from multiple angles and, based on a multi-view image model, utilize stereo vision to compute a 3D point cloud representation of the model. Existing multi-view-based methods generally involve two steps: the first step is to generate a 3D mesh M of the paper model; the second step is to unfold the 3D mesh onto a plane to obtain a 2D mesh Ω, and then perform texture mapping. Existing methods focus on obtaining a 3D mesh M with good expandability, treating the 2D unfolding of the 3D mesh M as a separate step. A drawback of these methods is that the 2D unfolding of M introduces errors, and these errors cannot be controlled using the document image's texture structure (e.g., text lines and table lines). Summary of the Invention

[0005] In view of the above-mentioned problems existing in the existing multi-view based methods, the present invention provides a multi-view based method for correcting creased document images.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] A method for correcting creased document images based on multiple views comprises the following steps:

[0008] Step 1: Acquire multi-view images

[0009] Use a consumer-grade digital camera or mobile phone camera to take pictures of the document object from multiple perspectives to obtain multiple images I′={I′ i |i=1,2,...,K,K≥3}, where I′ i is an image, K is the number of images;

[0010] Step 2: Get the 3D point cloud data of the document

[0011] Step 2.1: Use the structure-from-motion algorithm to obtain the point cloud data D′ in a unified coordinate system and the image I after removing camera distortion = {I i |i=1,2,...,K,K≥3} and the camera parameters corresponding to each image, where I i is an image with camera distortion removed;

[0012] Step 2.2: De-noise the point cloud data D′ to obtain point cloud data D;

[0013] Step 3: Generation of image space grid

[0014] Select an image I from the image sequence I r As a reference image, generate a quadrilateral image grid M within the range of the image I , whose mesh vertex is C ij , 0≤i≤N1, 0≤j≤N2, where N1 and N2 represent the number of mesh vertices in the width and height directions;

[0015] Step 4: Generation of initial 3D mesh

[0016] Step 4.1: Convert the data point representation in the point cloud data D to the reference image I r In the corresponding camera coordinate system, point D in the point cloud data D p Projection to reference image I r On the image, we get the projection point П p , and delete the data points whose projection points are outside the document area to obtain the set of projection points on the image П;

[0017] Step 4.2: For the quadrilateral image grid M I A vertex P, assuming its pixel coordinates are (u p ,v p ), find M on the image I The set of projected data points Φ in the neighborhood of point P p={A|A∈Π and d(A,P)<ε}, where d(A,B) represents the Euclidean distance between two points A and B in the pixel coordinate system of the image, ε is a threshold, let A∈Φ p The depth of point P is z(A), then the depth value z p Estimate using the following formula:

[0018]

[0019] Step 4.3: Calculate the three-dimensional coordinates of vertex P (x p ,y p ,z p ), get the initial three-dimensional grid M 3D , where: x coordinates of mesh vertices p 、y p Calculate using the following formula:

[0020]

[0021] Among them, (c u ,c v ) is the coordinate of the center point of the image, (k u ,k v ) is the image scaling factor, and d is the focal length of the camera;

[0022] Step 5: Initial unwrapped mesh generation

[0023] The existing length-preserving mesh expansion algorithm is used to expand the three-dimensional mesh M 3D Expand it onto the plane and get the two-dimensional grid M 2D ;

[0024] Step 6: Unfolding and optimizing the mesh

[0025] Step 6.1: Use the Gauss-Newton algorithm to solve the minimization problem of the objective function F. When the optimization converges, the grid reconstruction 3D grid M is obtained. 3D and plane mesh M 2D If the document image generated by expanding the grid is satisfactory, or the document image result of this iteration is not much different from that of the previous iteration, then proceed to step 7; otherwise, proceed to step 6.2, where the objective function F is defined as:

[0026]

[0027] Among them, w iso 、w Fair3D 、w Fair2D 、w PointClose 、w Line3D 、w Line2D They are the isometric transformation objective function Fiso , three-dimensional grid M 3D The smoothing function F Fair3D , two-dimensional grid M 2D The smoothing function F Fair2D , point cloud distance objective function F PointClose , three-dimensional grid M 3D The straight line constraint F Line3D and the two-dimensional grid M 2D The straight line constraint F Line2D The coefficients of the corresponding functions;

[0028] Step 6.2: Transform the 3D mesh M 3D and the two-dimensional grid M 2D Subdivide at the same time to increase the number of mesh vertices and execute step 6.1;

[0029] Step 7: Image Generation

[0030] The reference image I r As a texture, the optimized two-dimensional grid M 2D Perform texture mapping to obtain the corrected document image.

[0031] Compared with the prior art, the present invention has the following advantages:

[0032] The present invention provides a computational framework for document image correction. This framework simultaneously computes a three-dimensional mesh M and a two-dimensional expanded mesh Ω, and applies image structural and texture constraints (such as straight lines) to the two-dimensional expanded mesh Ω. Consequently, the method of the present invention can directly produce a textured two-dimensional expanded result, guided by the structural and texture information of the document image, avoiding the errors introduced by the separate mesh expansion step in existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the point cloud data obtained from multi-view images based on the SFM method;

[0034] Figure 2 Control grid for quadrilaterals on document images;

[0035] Figure 3 is the initial three-dimensional mesh;

[0036] Figure 4 Unfold the mesh for a 3D mesh and its plane;

[0037] Figure 5 A flowchart of a method for correcting creased document images based on multiple views;

[0038] Figure 6 is the reference image of Example 1;

[0039] Figure 7 This is the point cloud data of Example 1;

[0040] Figure 8 This is the image correction result of Example 1;

[0041] Figure 9 is the straight line feature on the corrected image of Example 1;

[0042] Figure 10 This is the 3D reconstructed mesh of Example 1;

[0043] Figure 11 is the reference image of Example 2;

[0044] Figure 12 This is point cloud data for example 2;

[0045] Figure 13 This is the image correction result of Example 2;

[0046] Figure 14 The straight line features on the corrected image of Example 2;

[0047] Figure 15 This is the 3D reconstruction result of Example 2;

[0048] Figure 16 is the reference image of Example 3;

[0049] Figure 17 This is the image correction result of Example 3;

[0050] Figure 18 is the reference image of Example 4;

[0051] Figure 19 This is the image correction result of Example 4. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0053] The present invention provides a correction method for creased document images based on multiple views, such as Figure 5 As shown, the method includes the following steps:

[0054] Step 1: Acquire multi-view images.

[0055] Use a consumer-grade digital camera or a mobile phone camera to take pictures of the document object from multiple perspectives to obtain multiple images I′={I′ i |i=1,2,...,K,K≥3}, where I′ iis an image, K is the number of images;

[0056] Step 2: Get the 3D point cloud data of the document.

[0057] The point cloud data D′ in a unified coordinate system is obtained by using the Structure From Motion (SFM) algorithm, and the image I={I i |i=1,2,...,K,K≥3}, where I i is an image with camera distortion removed, K is the number of images, and the camera parameters corresponding to each image. The initial point cloud generally contains a certain amount of noise. First, a simple denoising process is performed on the point cloud data D′ to obtain the point cloud data D. Figure 1 A point cloud data D obtained by the above method is given. Although the accuracy of these point cloud data is not high, they provide certain three-dimensional shape information of the document object.

[0058] Step 3: Generation of image space grid.

[0059] Select an image I from the image sequence I r As a reference image, generate a quadrilateral mesh M within the range of the image I , whose mesh vertex is C ij , 0≤i≤N1, 0≤j≤N2, where N1 and N2 represent the number of mesh vertices in the width and height directions, such as Figure 2 shown.

[0060] Step 4: Generation of initial 3D mesh.

[0061] This step is done by calculating the image grid M I The depth of the vertices is used to construct the initial three-dimensional mesh of the document model in the space. The steps are as follows:

[0062] (4.1) Convert the data point representation in the point cloud data D to the reference image I r In the corresponding camera coordinate system, each data point D in the point cloud data D p Projection to reference image I r On the image, we get the projection point П p And delete the data points whose projection points are outside the document area to obtain the point set П.

[0063] (4.2) Estimate the depth value of the mesh vertex. Since each projection point Π p The depth of is known (given by the SFM method), and now we use these projection points with known depth to estimate the image grid M I The depth value of the vertex.

[0064] For MI A vertex P of p ,v p ), find M on the image I The set of projected data points in the neighborhood of Φ p ={A|A∈Π and d(A,P)<ε}, where d(A,B) represents the Euclidean distance between two points A and B in the image pixel coordinate system, and ε is a threshold. Let A∈Φ p The depth of point P is z(A), then the depth value z p Estimated by formula (1):

[0065]

[0066] in, is a radial basis function.

[0067] (4.3) Calculate the three-dimensional coordinates (x p ,y p ,z p ), get the initial three-dimensional grid M 3D . Get the depth value z p After that, the x coordinates of the mesh vertex p 、y p Given by formula (2):

[0068]

[0069] Among them, (c u ,c v ) is the coordinate of the center point of the image, (k u ,k v ) is the image scaling factor, d is the focal length of the camera, these data are given by the SFM method. I The corresponding three-dimensional point coordinates of each vertex are obtained to obtain the initial three-dimensional mesh M 3D . Figure 3 A 3D mesh obtained by the above method is given.

[0070] Step 5: Generation of the initial unfolded mesh.

[0071] The three-dimensional grid M 3D Expand to the plane to obtain a two-dimensional grid M 2D , which can be obtained using the existing length-preserving grid expansion method. Assume that an M 3D The coordinates of a vertex in is (x i ,y i ,z i ), the expanded mesh M on the plane 2D The corresponding vertex coordinates are (x′ i,y′ i ). Figure 4 A three-dimensional mesh and the corresponding planar unfolded mesh are given.

[0072] Step 6: Unfolding and optimizing the mesh.

[0073] Get the initial three-dimensional mesh surface M 3D and plane mesh M 2D After that, the two meshes are optimized while maintaining some constraints. The constraints include:

[0074] (6.1) Isometric transformation constraints

[0075] The method of the present invention simultaneously optimizes the mesh surface M 3D and its equally spaced grid M 2D , keep M 3D and M 2D The isometric transformation between them gives the developable mesh surface M 3D At the same time, its two-dimensional expanded grid M is obtained 2D The constraints of the isometric transformation are defined as follows:

[0076] Assume M 3D The four vertices of a quadrilateral face f are arranged clockwise as v0, v1, v2, v3, M 2D The vertices of the corresponding patch f′ are v′0, v′1, v′2, and v′3, and the isometric transformation constraint on the patch f is shown in formula (3):

[0077]

[0078] The objective function F of the isometric transformation of the entire grid iso Defined as:

[0079]

[0080] Where F is the grid M 3D The set of all faces.

[0081] (6.2) Point cloud distance control

[0082] In order to ensure the optimized three-dimensional mesh M 3D The shape of the point cloud data is consistent with the shape of the point cloud data, and the point cloud distance objective function F is defined. PointClose for:

[0083] F PointClose =∑ P∈D ||P-π(P)|| 2 (5);

[0084] Among them, π(P) represents the data point P in the grid M 3DThe foot point on , D represents the data point set obtained in step 2 with noise points removed.

[0085] (6.3) Smoothing control of mesh

[0086] Define the three-dimensional mesh M 3D The smoothing function F Fair3D for:

[0087]

[0088] Among them, v i 、v j 、v k is the grid M 3D Three adjacent mesh vertices in the same row or column;

[0089] Similarly, define the plane mesh M 2D The smoothing function F Fair2D for:

[0090]

[0091] Among them, v′ i 、v′ j 、v′ k is the grid M 2D 3 adjacent mesh vertices in the same row or column.

[0092] (6.4) Linear constraints

[0093] Documents often have some geometric features, such as straight lines, etc. The method of the present invention allows the introduction of constraints, requiring that pixels located on the same straight line in the input image also lie on the same straight line on the expanded grid, thereby improving the effect of document image correction.

[0094] The extraction of feature lines on the reference image can use existing image processing algorithms to obtain short straight line segments by detecting straight line features, and then obtain longer feature lines by merging straight line segments. A feature line is represented by l r , which is a set of pixels These pixels are called feature pixels, and G is the number of feature points. These feature lines include three categories: (1) document boundaries; (2) straight lines in the document, such as the boundaries of a table or rectangular area; (3) text lines, that is, straight lines along the direction of the text. The three types of feature lines are treated in the same way as constraints. The set of all feature lines on the reference image is represented by L r .

[0095] The feature lines on the reference image are calculated as follows: 3D and grid M 2DThe corresponding characteristic line on L can be calculated by projection transformation. r The characteristic line l r In the three-dimensional grid M 3D The corresponding characteristic line l on M 3D The set L of all characteristic lines on r Feature pixels Calculate the image projection surface through the viewpoint The equation of the line corresponding to the point The straight line and the three-dimensional grid M 3D The intersection point is denoted as s j , s j is a characteristic point of the characteristic line l, then l is the characteristic point s1,s2,…,s G Set of . Let s j Located in triangle S, and the coordinates of the center of gravity are W, then the plane mesh M 2D The point in the corresponding triangle S' with the same center of gravity coordinate W is denoted as s' j , s′ j is the characteristic point of the plane characteristic line l'. In this way, the plane mesh M 2D The corresponding characteristic line l' on the feature point s'1, s'2, ..., s' G The collection of M 2D The set of all characteristic lines on the plane is recorded as L'. Calculate the characteristic points s'1, s'2, ..., s' that fit l' on the plane G The equation of the straight line A l' x=0.

[0096] Based on the above expression, the three-dimensional grid M 3D The straight line constraint F Line3D and plane mesh M 2D The straight line constraint F Line2D The definition of is shown in formula (8) and formula (9):

[0097]

[0098] F Line2D =∑ l∈L′ ∑ j＝1...G (A l′ s′ j ) 2 (9);

[0099] in, is the coefficient matrix of the projection line equation corresponding to a point on the characteristic line, A l′ is the coefficient matrix of the straight line fitting equation of the plane characteristic line l', s j and s′ jThey are the corresponding feature points of a feature point of the feature line on the three-dimensional grid and the plane grid, and the feature point is the centroid coordinate representation of the grid vertex of the triangular area where it is located.

[0100] (6.5) Construction and solution of objective function

[0101] The overall optimization objective function F is a linear combination of the above constraint functions (including the functions defined by formula (4), formula (5), formula (6), formula (7), formula (8), and formula (9)), as defined by formula (10):

[0102]

[0103] Among them, w iso 、w Fair3D 、w Fair2D 、w PointClose 、w Line3D 、w Line2D are the coefficients of the corresponding functions, which are used to adjust the components of each function. In the experiment, we choose w iso =1,w Fair3D =0.0001, w Fair2D =0.1, w PointClose =1,w Line3D =1,w Line2D =1.

[0104] The present invention uses the Gauss-Newton algorithm to solve the minimization problem of function (10), and the optimization variable is the three-dimensional grid M 3D and the two-dimensional grid M 2D The vertex position of the optimal solution, in each iteration, the foot point π(P) in formula (5), the characteristic point fitting straight line equation coefficient A′ in formula (9) l′ , all need to be recalculated. When the optimization converges, the grid reconstruction grid M is obtained 3D and the plane expanded mesh M 2D .

[0105] (6.6) Grid subdivision

[0106] When generating the initial mesh, we usually select a smaller number of faces to make optimization easier. After the optimization is completed, we will use the 3D mesh M 3D and the two-dimensional grid M 2D Subdivide the mesh using the Catmull-Clark method to increase the number of mesh vertices. After subdivision, perform a new round of optimization, iterating until satisfactory results are achieved. In our experiments, the initial mesh size was 30×20, and generally, three rounds of subdivision and optimization were sufficient to achieve satisfactory results.

[0107] Step 7: Image generation.

[0108] To generate an image, we need to optimize the plane mesh M 2D For texture mapping, use the reference image I r As a texture. For M 2D A point on M 3D and M 2D The corresponding relationship of M 3D The corresponding points on the image are then projected to obtain the image I r The method of the present invention gives the corresponding points on the two-dimensional grid M 2D The point on the reference image I r The position on the document is obtained by texture mapping to obtain the corrected document image.

[0109] The experimental results are as follows Figures 6 to 19 As shown. Figures 6 to 19 It can be seen that these experimental data are all document images with large folding deformations. The document correction results obtained by existing methods for such data are not good, while the method of the present invention can better restore the original unfolded image of the crease-deformed document, and the document image correction effect is better than the existing method.

Claims

1. A method for correcting creased document images based on multiple views, characterized in that The method comprises the following steps: Step 1: Acquire multi-view images The document object is photographed from multiple perspectives to obtain multiple images I′={I i ′|i=1,2,...,K,K≥3}, where I i ′ is an image, K is the number of images; Step 2: Get the 3D point cloud data of the document Step 2.1: Use the structure-from-motion algorithm to obtain the point cloud data D′ in a unified coordinate system and the image I with camera distortion removed = {I i |i=1,2,...,K,K≥3} and the camera parameters corresponding to each image, where I i is an image with camera distortion removed; Step 2.2: De-noise the point cloud data D′ to obtain point cloud data D; Step 3: Generation of image space grid Select an image I from the image sequence I r As a reference image, generate a quadrilateral image grid M within the range of the image I , whose mesh vertex is C ij , 0≤i≤N1, 0≤j≤N2, where N1 and N2 represent the number of mesh vertices in the width and height directions; Step 4: Generation of initial 3D mesh Step 4.1: Convert the data point representation in the point cloud data D to the reference image I r In the corresponding camera coordinate system, point D in the point cloud data D p Projection to reference image I r On the image, we get the projection point П p , and delete the data points whose projection points are outside the document area to obtain the set of projection points on the image П; Step 4.2: For the quadrilateral image grid M I A vertex P, assuming its pixel coordinates are (u p ,v p ), find image M I The set of projected data points Φ in the neighborhood of the upper point P p ={A|A∈Π and d(A,P)<ε}, where d(A,B) represents the Euclidean distance between two points A and B in the pixel coordinate system of the image, and ε is a threshold; Step 4.3: Calculate the three-dimensional coordinates (x p ,y p ,z p ), get the initial three-dimensional grid M 3D ; The x of the vertex P p 、y p Calculate using the following formula: Among them, (c u ,c v ) is the coordinate of the center point of the image, (k u ,k v ) is the image scaling factor, and d is the focal length of the camera; The depth value z of the vertex P p Estimate using the following formula: Where z(A) is A∈Φ p The depth, is a radial basis function; Step 5: Initial unwrapped mesh generation The three-dimensional mesh M is expanded using the length-preserving mesh expansion algorithm. 3D Expand it onto the plane and get the two-dimensional grid M 2D ; Step 6: Unfolding and optimizing the mesh Step 6.1: Use the Gauss-Newton algorithm to solve the minimization problem of the objective function F. When the optimization converges, the grid reconstruction 3D grid M is obtained. 3D and plane mesh M 2D If the document image generated by expanding the grid is satisfactory, or the document image result of this iteration is not much different from that of the previous iteration, then proceed to step 7. Otherwise, proceed to step 6.

2. The objective function F is defined as: Among them, w iso 、w Fair3D 、w Fair2D 、w PointClose 、w Line3D 、w Line2D They are the isometric transformation objective function F iso , three-dimensional grid M 3D The smoothing function F Fair3D , two-dimensional grid M 2D The smoothing function F Fair2D , point cloud distance objective function F PointClose , three-dimensional grid M 3D The straight line constraint F Line3D and the two-dimensional grid M 2D The straight line constraint F Line2D The coefficients of the corresponding functions; Step 6.2: Transform the 3D mesh M 3D and the two-dimensional grid M 2D Subdivide at the same time to increase the number of mesh vertices and execute step 6.1; Step 7: Image Generation The reference image I r As a texture, the optimized two-dimensional grid M 2D Perform texture mapping to obtain the corrected document image.

2. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The isometric transformation objective function F iso is defined as: Where F is the grid M 3D The set of all faces of ; Assume M 3D The four vertices of a quadrilateral face f are arranged clockwise as v0, v1, v2, v3, M 2D The four vertices of the corresponding quadrilateral patch f′ are v′0, v1′, v′2, and v3′, so the isometric transformation constraint on the quadrilateral patch f is:

3. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The point cloud distance objective function F PointClose Defined as: F PointClose =∑ P∈D ||P-π(P)|| 2 ; Among them, π(P) represents the data point P in the grid M 3D The foot point on .

4. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The three-dimensional grid M 3D The smoothing function F Fair3D Defined as: Among them, v i 、v j 、v k It's M 3D 3 adjacent mesh vertices in the same row or column.

5. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The plane grid M 2D The smoothing function F Fair2D Defined as: Among them, v i ′、v′ j 、v′ k It's M 2D 3 adjacent mesh vertices in the same row or column.

6. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The three-dimensional grid M 3D The straight line constraint F Line3D is defined as: Where l is a feature line on the reference image in the grid M 3D The corresponding characteristic line, s j is a feature point of the feature line l, G is the number of feature points, is a characteristic point s on the characteristic line l j The coefficient matrix of the corresponding projection line equation, L is M 3D The set of all feature lines on .

7. The method for correcting creased document images based on multiple views according to claim 1, characterized in that The plane grid M 2D The straight line constraint F Line2D is defined as: F Line2D =∑ l∈L′ ∑ j＝1...G (A l' s′ j ) 2 ; Among them, A l' is the coefficient matrix of the straight line fitting equation of the characteristic line plane characteristic line l', s' j is the feature point of the plane feature line l', G is the number of feature points, L' is M 2D The set of all feature lines on .

Images