Deep learning based document wrinkle repair method

Abstract

The application discloses a document wrinkle repairing method based on deep learning, and relates to the technical field of document image repairing, which comprises the following steps: obtaining a two-dimensional image and point cloud data of a document to be processed, and aligning each pixel point of the two-dimensional image with the point cloud data; calculating the curvature change rate of the neighborhood where each point in the point cloud data is located, and obtaining a trace line according to the curvature change rate of the neighborhood where each point is located; dividing the point cloud data according to the trace line and generating a plurality of sub-surfaces, performing local affine transformation on each sub-surface to obtain a corresponding flattened sub-surface, and splicing each flattened sub-surface to obtain a three-dimensional surface model; extracting image modal features in the two-dimensional image and extracting geometric modal features of the three-dimensional surface model, fusing the image modal features and the geometric modal features to obtain fused features; and based on the fused features, repairing the two-dimensional image in a cross-modal collaborative manner to obtain a target image. The application realizes the improvement of the repairing effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of document image restoration, in particular to a document wrinkle restoration method based on deep learning. BACKGROUND

[0002] In the image processing scenarios of document restoration and cultural relic digitization, the processing demand for wrinkled and deformed documents is very urgent.

[0003] The current document wrinkle restoration method usually regards the deformed document as a single continuous surface, and forcibly flattens it through a global affine transformation algorithm, which results in loss of key details and fragmented restoration effect, leading to poor restoration effect. SUMMARY

[0004] The main purpose of the present application is to provide a document wrinkle restoration method based on deep learning, which aims to solve the technical problem of poor restoration effect of wrinkled and deformed documents.

[0005] To achieve the above purpose, the present application provides a document wrinkle restoration method based on deep learning, comprising:

[0006] Obtaining the two-dimensional image and point cloud data of the document to be processed, and aligning each pixel point of the two-dimensional image with the point cloud data;

[0007] Calculating the curvature change rate of the neighborhood where each point in the point cloud data is located, and calculating the trace line according to the curvature change rate of the neighborhood where each point is located;

[0008] Dividing the point cloud data according to the trace line and generating a plurality of sub-surfaces, performing local affine transformation on each sub-surface to obtain a corresponding flattened sub-surface, and splicing each flattened sub-surface to obtain a three-dimensional surface model;

[0009] Extracting image modal features from the two-dimensional image, and extracting geometric modal features from the three-dimensional surface model, and fusing the image modal features and the geometric modal features to obtain fused features;

[0010] Based on the fused features, the two-dimensional image is cross-modally and collaboratively restored to obtain a target image.

[0011] In an embodiment, the point cloud data of the document to be processed is obtained, comprising:

[0012] Obtaining the first-view two-dimensional image, the second-view two-dimensional image and the TOF depth information of the document to be processed;

[0013] Calculating the binocular depth information according to the parallax of the first-view two-dimensional image and the second-view two-dimensional image;

[0014] The noise variance of the binocular depth information and the noise variance of the TOF depth information are calculated, and the fusion weight of the binocular depth information and the fusion weight of the TOF depth information are calculated according to the noise variance of the binocular depth information and the noise variance of the TOF depth information;

[0015] The binocular depth information and the TOF depth information are Kalman-filtered and fused according to the fusion weight of the binocular depth information and the fusion weight of the TOF depth information to obtain fused depth information;

[0016] The fused depth information is converted into point cloud data.

[0017] In an embodiment, the binocular depth information is calculated according to the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image, and comprises:

[0018] The first-view two-dimensional image and the second-view two-dimensional image showing the structured light fringe are collected;

[0019] The first-view structured light phase image and the second-view structured light phase image are calculated by the Fourier transform method according to the first-view two-dimensional image and the second-view two-dimensional image showing the structured light fringe;

[0020] The parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image is calculated according to the first-view gray-scale image, the second-view gray-scale image, the first-view structured light phase image and the second-view structured light phase image;

[0021] The binocular depth information is calculated according to the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image.

[0022] In an embodiment, the curvature change rate of the neighborhood of each point in the point cloud data is calculated, and the trace line is calculated according to the curvature change rate of the neighborhood of each point.

[0023] The neighborhood point set of the neighborhood of each point in the point cloud data is established;

[0024] The covariance matrix of the neighborhood point set is constructed, and the eigenvalue decomposition of the covariance matrix is performed to obtain the dispersion values of each point in the neighborhood point set in three orthogonal directions;

[0025] The principal curvatures of the neighborhood of each point in the point cloud data are calculated according to the dispersion values in the three orthogonal directions, and the curvature change rate of the neighborhood of each point in the point cloud data is calculated according to the principal curvatures of the neighborhood of each point in the point cloud data;

[0026] The trace point is calculated according to the curvature change rate of the neighborhood of each point in the point cloud data and the preset curvature change rate threshold, wherein the curvature change rate of the neighborhood of the trace point is greater than the preset curvature change rate threshold;

[0027] A straight line is fitted to each trace point to obtain a trace line.

[0028] In an embodiment, generating the plurality of sub-faces comprises:

[0029] The sub-faces are reconstructed using a greedy triangle mesh reconstruction method.

[0030] In an embodiment, each sub-face is locally affine transformed to obtain a corresponding flattened sub-face, comprising:

[0031] The sub-face is locally affine transformed to obtain a flattened sub-face;

[0032] The flattened sub-face is optimized using a method of solving a minimization energy function to obtain an optimized flattened sub-face, wherein the energy function comprises a first constraint term and a second constraint term, the energy function is used to measure a distortion degree of the flattened sub-face, the first constraint term is used to measure a deviation of a normal vector of the flattened sub-face and a normal vector of each point of the corresponding sub-face, and the second constraint term is used to measure a deviation of vector lengths before and after flattening.

[0033] In an embodiment, the image modality feature and the geometry modality feature are fused to obtain a fused feature, comprising:

[0034] The image modality feature and the geometry modality feature are fused based on a gating mechanism to obtain a fused feature;

[0035] An initial neighbor point set of each point in the three-dimensional surface model is obtained, and principal curvatures of the initial neighbor point set are calculated;

[0036] A dynamic neighborhood radius is calculated according to the principal curvatures of the initial neighbor point set;

[0037] A neighborhood point set is determined according to the dynamic neighborhood radius;

[0038] An attention weight is calculated according to a point distance and a normal vector angle of each point in the neighborhood point set;

[0039] A feature aggregation output is obtained based on the attention weight to obtain an optimized geometry feature;

[0040] A principal folding axis vector of the three-dimensional surface model is extracted;

[0041] A curvature Fourier transform is performed on the principal curvatures of the initial neighbor point set to obtain a curvature high-dimensional feature;

[0042] After the principal folding axis vector of the three-dimensional surface model and the curvature high-dimensional feature are superimposed, the geometry modality feature in the fused feature is enhanced according to an attention mechanism and the optimized geometry feature. In an embodiment, based on the fused feature, a two-dimensional image is cross-modality collaboratively repaired to obtain a target image, comprising:

[0043] Geometric modal features are corrected using image modal features, and image modal features are corrected using geometric modal features, to obtain corrected image modal features and corrected geometric modal features;

[0044] Based on the corrected image modal features and corrected geometric modal features, geometric inpainting, texture inpainting, and structural inpainting are performed on the two-dimensional image to obtain the target image. In one embodiment, obtaining the point cloud data of the document to be processed includes:

[0045] Acquire first-view point cloud data and second-view point cloud data;

[0046] Based on point distance constraints and normal vector gradient consistency constraints, first-view point cloud data and second-view point cloud data are registered and stitched together to obtain fused point cloud data.

[0047] Based on the density changes in the fused point cloud data, locate the seam area of ​​the fused point cloud data;

[0048] RBF interpolation is used to process the seam area to obtain optimized fused point cloud data. In one embodiment, it further includes:

[0049] Curvature analysis was performed on the point cloud data to identify curled surfaces, where the curvature of the curled surfaces exhibited a ring-shaped distribution.

[0050] The curled surface is segmented according to the growth algorithm;

[0051] Establish the parametric equations for the rotation axis of the curled sub-surface;

[0052] Based on the rotation axis parameter equation of the curled surface, the curled surface is unfolded at equal intervals to obtain the flattened surface.

[0053] One or more technical solutions proposed in this application have at least the following technical effects:

[0054] A deep learning-based document wrinkle restoration method is proposed. This method identifies trace lines, then divides the point cloud data into several sub-faces based on these lines. Each sub-face undergoes a local affine transformation to obtain a corresponding flattened sub-face. These flattened sub-faces are then stitched together to obtain a 3D surface model. Image modal features are extracted from the 2D image, and geometric modal features from the 3D surface model are extracted as well. The image modal features and geometric modal features are fused to obtain a fused feature. Based on this fused feature, the 2D image is collaboratively restored across modalities to obtain the target image, thus improving the restoration effect. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the accompanying drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0057] Figure 1 The flowchart of the first embodiment of the document wrinkle repair method based on deep learning provided in the present application is shown in the figure.

[0058] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and not to limit the present application. In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below with reference to the drawings and specific embodiments.

[0060] In the traditional three-dimensional reconstruction method, a parameterized model is usually used to fit the wrinkle surface of the document, but the parameterized model with fixed parameters cannot adapt to complex or multi-directional wrinkles, such as multi-fold and random curling documents.

[0061] At present, the document repair method based on deep learning method has the following technical ideas, one, DocUNet, uses stacked U-Net to directly regress pixel displacement field, only through two-dimensional deformation training, and the repair result deviating from the physical geometric principle is easy to produce in the real scene; two, DewarpNet, introduces three-dimensional coordinate graph regression, but the fixed radius query aggregation neighborhood points are used for middle layer feature extraction, which cannot adapt to the scale difference between the fold edge and the flat area, resulting in inaccurate aggregation of local curvature, normal vector and other features; three, Transformer, a neural network model based on attention mechanism, the position encoding of the encoder of the model only depends on the sine and cosine function, which is difficult to model the overall deformation rule of the document, and the repair effect of the local wrinkle area without complete boundary (such as local tearing and edge curling) is poor.

[0062] In the first embodiment of the document wrinkle repair method based on deep learning of the present application, referring to Figure 1 , Figure 1 The flowchart of the first embodiment of the document wrinkle repair method based on deep learning of the present application is shown in the figure. The document wrinkle repair method based on deep learning can include steps S10-S50:

[0063] Step S10, acquiring the two-dimensional image and point cloud data of the document to be processed, and aligning each pixel point of the two-dimensional image and the point cloud data.

[0064] It should be noted that the two-dimensional image of the document to be processed is an RGB image, and the point cloud data can be complete point cloud data directly collected, that is, complete point cloud data obtained by single scanning, or complete point cloud data spliced from multi-view point cloud data.

[0065] In a feasible implementation, the step S10 of "obtaining point cloud data of the document to be processed" can include steps S101-S105.

[0066] In step S101, a first-view two-dimensional image, a second-view two-dimensional image and TOF depth information of the document to be processed are obtained.

[0067] It should be noted that the first view is different from the second view, that is, the first-view two-dimensional image is different from the second-view two-dimensional image.

[0068] In a specific implementation, the first-view two-dimensional image is a left-eye two-dimensional image collected by a binocular camera, and the second-view two-dimensional image is a right-eye two-dimensional image collected by the binocular camera.

[0069] In step S102, binocular depth information is calculated according to the parallax of the first-view two-dimensional image and the second-view two-dimensional image.

[0070] It should be noted that the parallax is the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image, and the first-view two-dimensional image and the second-view two-dimensional image can include only the original image of the document to be processed, or can be superimposed with a structured light stripe.

[0071] In a specific implementation, the two-dimensional image is superimposed with a structured light stripe, and the step S102 can include steps S1021-S1024.

[0072] In step S1021, the first-view two-dimensional image and the second-view two-dimensional image with the structured light stripe displayed are collected.

[0073] In step S1022, the first-view structured light phase image and the second-view structured light phase image are calculated by the Fourier transform method according to the first-view two-dimensional image and the second-view two-dimensional image with the structured light stripe displayed.

[0074] In step S1023, the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image is calculated according to the first-view gray-scale image, the second-view gray-scale image, the first-view structured light phase image and the second-view structured light phase image.

[0075] Specifically, the preset stereo matching optimization model is used, and the first-view gray-scale image, the second-view gray-scale image, the first-view structured light phase image and the second-view structured light phase image are taken as inputs to calculate the parallax.

[0076] The stereo matching optimization model includes a disparity cost calculation model for measuring the matching reliability of pixels p in the disparity d The mathematical expression of the disparity cost calculation model is:

[0077]

[0078]

[0079] In the formula, is the matching reliability, and the matching reliability of pixels p in the disparity d is represented by pixel coordinates p is the disparity, d is a first-view gray-scale image, is a second-view gray-scale image, is a first-view structured light phase image, is a second-view structured light phase image, is a weight coefficient, is a phase difference decay coefficient.

[0080] is a gray-scale term, and by calculating the sum of L1 norms of left and right image gray-scale differences in a 3*3 window, the characteristics of the L1 norm that is not sensitive to illumination noise are used to ensure the disparity calculation accuracy of a rich-texture area (such as a text or seal area) and avoid the influence of a single pixel error on the overall matching.

[0081] is a structured light term, and is used to extract the first-view structured light phase image and the second-view structured light phase image by Fourier transform, measure the phase difference between the first-view structured light phase image and the second-view structured light phase image, and range [0, 1], and the value closer to 1 indicates a higher similarity.

[0082] When the phase difference between the first-view structured light phase image and the second-view structured light phase image is less than 5°, tends to 1, and provides additional phase constraints for a low-texture area (such as a blank document), and solves the disparity ambiguity problem caused by “no feature to match” in the traditional algorithm in a low-texture area.

[0083] It should be noted that, is used to balance the contributions of the gray-scale term and the structured light term, the rich-texture area is mainly based on the gray-scale term, the low-texture area is mainly based on the structured light term, the disparity corresponding to the pixel p with the highest matching reliability is taken as the optimal disparity, the robustness of the overall disparity calculation is improved, and an accurate foundation is laid for subsequent depth data generation. ​​

[0084] In step S1024, the binocular depth information is calculated according to the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image.

[0085] In step S103, the noise variance of the binocular depth information and the noise variance of the TOF depth information are calculated, and the fusion weight of the binocular depth information and the fusion weight of the TOF depth information are calculated according to the noise variance of the binocular depth information and the noise variance of the TOF depth information.

[0086] Specifically, the mathematical expression of S103 is as follows:

[0087]

[0088]

[0089] In the formula, is the fusion weight of the binocular depth information, is the fusion weight of the TOF depth information, is the noise variance of the binocular depth information, is the noise variance of the TOF depth information, is the confidence of the binocular depth information, is the confidence of the TOF depth information.

[0090] In step S104, the binocular depth information and the TOF depth information are Kalman-filtered and fused according to the fusion weight of the binocular depth information and the fusion weight of the TOF depth information, so as to obtain the fused depth information.

[0091] Specifically, the mathematical expression of step S104 is as follows:

[0092]

[0093] In the formula, is the fused depth information, is the binocular depth information, is the TOF depth information.

[0094] It should be noted that the calculation accuracy of the binocular depth information in the reflective region is low, and the noise of the TOF depth information is large under strong light. Through the Kalman filtering to realize the adaptive fusion of the confidence of the binocular depth information and the TOF depth information, more stable and higher-precision depth data can be output, and the accuracy of the subsequent three-dimensional reconstruction is improved.

[0095] In step S105, the fused depth information is converted into point cloud data.

[0096] Step S20, the rate of change of curvature of the neighborhood where each point in the point cloud data is located is calculated, and the trace line is calculated according to the rate of change of curvature of the neighborhood where each point is located.

[0097] In a feasible implementation, step S20 can include steps S201-S205:

[0098] Step S201, a neighborhood point set of the neighborhood where each point in the point cloud data is located is established.

[0099] It should be noted that the neighborhood radius is set according to actual needs, and in this embodiment, the neighborhood radius is set to 5mm, that is, for each point in the point cloud data, the neighborhood points within a radius of 5mm of the point are selected to construct the neighborhood point set.

[0100] Step S202, a covariance matrix of the neighborhood point set is constructed, and the covariance matrix is subjected to eigenvalue decomposition to obtain the dispersion values of each point in the neighborhood point set in three orthogonal directions.

[0101] It should be noted that the covariance matrix Cov is subjected to eigenvalue decomposition to obtain , wherein , and respectively represent the dispersion degrees of each point in the neighborhood point set in three orthogonal directions.

[0102] Step S203, the principal curvatures of the neighborhood where each point in the point cloud data is located are calculated according to the dispersion values in three orthogonal directions, and the rate of change of curvature of the neighborhood where each point in the point cloud data is located is calculated according to the principal curvatures of the neighborhood where each point in the point cloud data is located.

[0103] Specifically, the mathematical expression of step S203 is:

[0104]

[0105] In the formula, is the first principal curvature, is the second principal curvature, is the rate of change of curvature, wherein the first principal curvature and the second principal curvature reflect the bending degree of the local surface, and the greater the value, the more intense the bending, and the rate of change of curvature is used to distinguish high curvature change areas and low curvature change areas.

[0106] Step S204, the trace point is calculated according to the rate of change of curvature of the neighborhood where each point in the point cloud data is located and the preset rate of change of curvature threshold, wherein the rate of change of curvature of the neighborhood where the trace point is located is greater than the preset rate of change of curvature threshold.

[0107] It should be noted that the preset curvature change rate threshold is a preset value, and in this embodiment, the preset curvature change rate threshold is 0.8. If the curvature change rate of the neighborhood of a point is greater than 0.8, the point is set as a trace point.

[0108] In step S205, a straight line is fitted for each trace point to obtain a trace line.

[0109] In a specific embodiment, two trace pixels can be randomly selected as an initial straight line, the distances of other trace points to the straight line are calculated, the trace points with a distance less than 0.3 mm are selected as inliers, the straight line segment with an inlier ratio greater than 60% is reserved after 50 iterations, the included angle of adjacent straight line segments is calculated, and if the included angle is less than 5° and the distance between the end points is less than 2 mm, the two straight line segments are segments of the same trace line. Each trace line is refitted by the least square method to form a continuous trace line network.

[0110] In step S30, the point cloud data is divided and a plurality of sub-surfaces are generated according to the trace lines, each sub-surface is subjected to a local affine transformation to obtain a corresponding flattened sub-surface, and each flattened sub-surface is spliced to obtain a three-dimensional surface model.

[0111] It should be noted that the sub-surface is a local three-dimensional surface model, and after the point cloud data is divided into a plurality of local point clouds according to the trace lines, the sub-surface can be reconstructed according to each local point cloud. By dividing the point cloud data according to the trace lines first, a plurality of sub-surfaces are generated, the boundaries of the sub-surfaces can be fitted to the trace lines, the key details of the government document are not lost, and the problem of ignoring the details of the trace lines in the traditional whole-surface flattening is solved.

[0112] In a feasible embodiment, the sub-surface is reconstructed by a greedy triangular net reconstruction method, specifically, a seed point is selected in a region with the minimum curvature variance (the mean square error of the principal curvatures of all points in the sub-surface, which reflects the geometric flatness of the sub-surface), and gradually grows to the boundary of the trace line; wherein the growth constraint condition is set as: the curvature change rate difference of adjacent points <0.2, and the included angle with the normal vector of the seed point <15°. For the polygonal region formed by the intersection of the trace lines, Delaunay triangulation is used for subdivision to ensure that the boundary of the sub-surface is completely coincided with the structure line, and to avoid distortion in subsequent flattening.

[0113] In a feasible embodiment, the step of "subjecting each sub-surface to a local affine transformation to obtain a corresponding flattened sub-surface" in step S30 includes steps S301-S302:

[0114] In step S301, after the sub-surface is subjected to a local affine transformation, a flattened sub-surface is obtained.

[0115] Step S302: Optimize the flattened sub-surface using the method of minimizing the energy function to obtain the optimized flattened sub-surface. The energy function includes a normal constraint term and a length constraint term. The energy function is used to measure the degree of distortion of the flattened sub-surface. The first constraint term is used to measure the deviation between the normal vector of the flattened sub-surface and the normal vector of each point of the corresponding sub-surface. The second constraint term is used to measure the deviation of the vector length before and after flattening.

[0116] Specifically, the mathematical expression for the energy function is:

[0117]

[0118] In the formula, As the first constraint term, This is the second constraint term. This value represents the degree of distortion when a subsurface is mapped to a flattened subsurface; the smaller the value, the lower the degree of distortion. For the sub-face ( face Any point within ) For point The unit normal vector in three-dimensional space. The unit normal vector of the flattened sub-surface. Representing one of the sub-faces, This is a weighting coefficient used to balance the influence of the first and second constraint terms. The boundary line of the sub-face. q For The other endpoint of the boundary line is the endpoint. The optimized points are respectively p and points q , The midpoints of the sub-faces p and points q The three-dimensional coordinates Characterized by Euclidean distance, used to measure the deviation of vector length before and after flattening.

[0119] It should be noted that the first constraint term is used to force the unit normal vector of all points within the sub-face in three-dimensional space. The unit normal vector of the flattened sub-surface The included angle is less than a preset angle. In this embodiment, the preset angle is set to 10°. By using cos10°≈0.98, it is ensured that... To prevent the sub-surface from folding or flipping during the flattening process, in this embodiment... The second constraint term is determined by principal component analysis (PCA) of the sub-surface point cloud, specifically the planar vector that best fits the sub-surface. This constraint term controls the pairing of points after flattening. Midpoint of the face The Euclidean distance deviation ensures that the size proportions of semantic elements such as text and seals are consistent.

[0120] Step S40: Extract image modal features from the two-dimensional image and extract geometric modal features from the three-dimensional surface model. Fuse the image modal features and geometric modal features to obtain fused features.

[0121] It should be noted that image modal features include text strokes, seal patterns, texture grayscale, and text lines, while geometric modal features include wrinkle depth, coordinates after sub-surface flattening, and normal vector direction.

[0122] In one feasible implementation, step S40, "fusing image modal features and geometric modal features to obtain fused features," may include steps S401 to S409:

[0123] Step S401: Based on the gating mechanism, image modal features and geometric modal features are fused to obtain fused features.

[0124] It should be noted that, in this embodiment, the gating mechanism is used to dynamically calculate the fusion weight of image modal features and the fusion weight of geometric modal features for each feature dimension, so as to fuse image modal features and geometric modal features to obtain fused features.

[0125] Step S402: Obtain the initial neighborhood point set of each point in the three-dimensional surface model, and calculate the principal curvature of the initial neighborhood point set.

[0126] Specifically, the mathematical expression for step S402 is:

[0127]

[0128] In the formula, These are the two largest eigenvalues ​​of the covariance matrix of the initial neighborhood point set. Let be the principal curvature of the initial neighborhood point set. This principal curvature reflects the degree of dispersion of each point in the initial neighborhood point set in orthogonal directions. The larger the diameter, the more severe the local bending.

[0129] Step S403: Calculate the dynamic neighborhood radius based on the principal curvature of the initial neighborhood point set.

[0130] Specifically, the mathematical expression for step S403 is:

[0131]

[0132] In the formula, For point i The dynamic neighborhood radius, Basic neighborhood radius, This is a learnable weight matrix used to control the degree to which curvature affects the radius.

[0133] Step S404, determining the neighborhood point set according to the dynamic neighborhood radius.

[0134] Step S405, calculating the attention weight according to the point distance and the normal vector angle of each point in the neighborhood point set.

[0135] Specifically, the mathematical expression of step S405 is:

[0136]

[0137] In the formula, is the attention weight of the point pair, is the first attenuation coefficient, used to control the attenuation speed of the distance on the weight, is the second attenuation coefficient, used to control the attenuation speed of the angle on the weight, is the distance of the point pair, is the normal vector angle of the point pair.

[0138] Step S406, based on the attention weight, the feature aggregation output is obtained to obtain the optimized geometric feature.

[0139] Specifically, the mathematical expression of step S406 is:

[0140]

[0141] In the formula, is the feature vector of the aggregated point, represents the nonlinear mapping of the spatial difference between two points, extracting the local geometric feature.

[0142] It should be noted that by dynamically configuring the neighborhood range, the differences between the crease edges and the flat areas can be adapted, and the feature extraction accuracy is improved.

[0143] Step S407, extracting the main folding axis vector of the three-dimensional surface model.

[0144] Specifically, the main folding axis vector is used to represent the overall folding trend of the document, and the main folding axis vector of the three-dimensional surface model is extracted by PCA as part of the position embedding, enhancing the model's perception of the folding structure.

[0145] Step S408, performing curvature Fourier transform on the principal curvature of the initial neighborhood point set to obtain the curvature high-dimensional feature.

[0146] Step S409, after superimposing the main folding axis vector and the curvature high-dimensional feature of the three-dimensional surface model, the geometric modal feature in the fusion feature is enhanced according to the attention mechanism and the optimized geometric feature.

[0147] Specifically, the mathematical expression of the "superimposition of the principal fold axis vector and the curvature high-dimensional feature of the three-dimensional surface model" in step S409 is as follows:

[0148]

[0149] In the formula, is the position encoding vector of the point p , which has a dimension of 512 in this embodiment, is the sine-cosine position encoding, which is used to capture the periodicity of the spatial coordinates, is the Fourier transform of the local curvature, which extracts the frequency characteristics (such as the periodic bending of the folds) of the curvature distribution, is the curvature feature weight matrix, which has a dimension of 512x64 in this embodiment.

[0150] The mathematical expression of "enhancing the geometric modal feature in the fusion feature according to the attention mechanism and the optimized geometric feature" in step S409 is as follows:

[0151]

[0152] In the formula, Q , K , V are the query (Query), key (Key), and value (Value) matrices in the attention mechanism, which are used to capture the association between features;

[0153] is the transpose matrix of the key matrix, which is multiplied by the query matrix to calculate the similarity between them;

[0154] are used to represent the feature dimensions of Q and K , and serve as scaling factors to avoid the numerical value of the operation result being too large, causing the gradient of the softmax function to disappear, and ensuring the stability of the attention calculation;

[0155] is a learnable weight coefficient, which is used to balance the contribution of the geometric distance constraint term in the attention calculation and regulate the degree of influence of geometric information on the attention weight;

[0156] is the geometric distance matrix, which incorporates the spatial geometric distance information between points, and is used to strengthen the attention mechanism's focus on spatial consistency, helping the model better capture the global structure of the document folds (such as fold direction and curling trend);

[0157] ​As an activation function, the attention score is normalized so that the weight values of the output are in the range [0, 1] and the sum is 1, and finally the attention output is obtained by weighted summation of the weights.

[0158] The deep learning-based document wrinkle repair method of the embodiment breaks through the limitation of single geometric features, innovatively reconstructs PointNet++ and Transformer architecture: through a curvature-driven dynamic neighborhood aggregation mechanism and self-adaptive adjustment of the neighborhood range according to the local curvature of the point cloud, and by introducing point pair attention, the accuracy of capturing wrinkle details and the accuracy of capturing global structure (such as curling trend) are improved, and the efficiency of point cloud processing and the feature expression ability are optimized.

[0159] Step S50, based on the fused features, a two-dimensional image is repaired across modalities to obtain a target image.

[0160] Specifically, step S50 can include steps S501-S502:

[0161] Step S501, the image modal feature is corrected using the geometric modal feature, and the geometric modal feature is corrected using the image modal feature to obtain corrected image modal feature and corrected geometric modal feature.

[0162] Specifically, the mathematical expression for correcting the image modal feature using the geometric modal feature to obtain the corrected image modal feature is:

[0163]

[0164] In the formula, is the repaired image modal feature, is the extracted image modal feature, MLP(C) is the feature after nonlinear mapping of the geometric modal feature, and the image modal feature repair process refers to the geometric modal feature through the attention mechanism, avoiding the disconnection between the image modal feature and the geometric modal feature.

[0165] The mathematical expression for correcting the image modal feature using the geometric modal feature to obtain the corrected image modal feature is:

[0166]

[0167] In the formula, is the repaired geometric modal feature, C is the geometric modal feature, and MLP(C) is the feature after nonlinear mapping of the geometric modal feature. The geometric modal feature repair refers to the image modal feature through the attention mechanism, so that the geometric modal feature is more consistent with the real texture feature, and the repair deviation caused by the lack of geometric modal feature constraint in the traditional U-Net++ is compensated.

[0168] Step S502, according to the corrected image modal feature and the corrected geometric modal feature, the two-dimensional image is geometrically repaired, textured and structurally repaired to obtain a target image.

[0169] Specifically, the corrected image modal feature and the corrected geometric modal feature are taken as inputs of the multi-task image processing model, and the two-dimensional image is geometrically repaired, textured and structurally repaired to obtain a target image.

[0170] The total loss function of the multi-task image processing model is:

[0171]

[0172]

[0173]

[0174]

[0175] In the formula, is the total loss function;

[0176] is a geometric loss, used to constrain the three-dimensional coordinate offset;

[0177] is a model-predicted 3D coordinate change, such as the difference between the 3D point coordinates after repair and before repair, reflecting the geometric correction amplitude;

[0178] is a real 3D coordinate change, used to determine the ideal geometric deviation after repair, as a supervision target, wherein the real 3D coordinate change is artificially labeled.

[0179] The geometric loss is used to constrain the accuracy of 3D coordinate repair by (L1 norm), avoiding the 3D structure offset after repair.

[0180] is a physical loss, ensuring visual quality;

[0181] is a model-predicted 2D texture image after repair, such as the document surface texture after repairing the crease, containing details such as text and seal;

[0182] is a real original 2D texture image, using an ideal texture without creases as a supervision target;

[0183] is a structural similarity index, used to measure the visual similarity of two images, and the closer the value is to 1, the more consistent the texture is;

[0184] Physical loss is used to pass At the same time, it constrains pixel-level differences (L1) and visual perception differences (SSIM) to ensure that the texture is natural after restoration (without stretching or blurring).

[0185] This is a structural loss measure used to ensure the continuity of table lines and text rows;

[0186] The structure mask predicted by the model (such as a binary image of table lines or text rows, where 1 indicates the presence of structure and 0 indicates background).

[0187] To predict the gradient (edge ​​variation) of the structure mask (reflecting the edge continuity of table lines and text rows);

[0188] The gradient (edge ​​variation) of the real structure mask (ideal structure edge, as the supervision target);

[0189] Structural loss through (L1 norm) constrains the continuity of structural edges (to prevent broken table lines and misaligned text lines after repair); combined with (Binary cross-entropy), while simultaneously constraining the correctness of the structure's "existence or non-existence".

[0190] The document wrinkle repair method based on deep learning in this embodiment identifies trace lines, then divides the point cloud data into several sub-surfaces based on the trace lines, performs local affine transformations on each sub-surface to obtain corresponding flattened sub-surfaces, stitches together the flattened sub-surfaces to obtain a three-dimensional surface model, extracts image modal features from the two-dimensional image, extracts geometric modal features from the three-dimensional surface model, fuses the image modal features and geometric modal features to obtain fused features, and based on the fused features, performs cross-modal collaborative repair of the two-dimensional image to obtain the target image, thereby improving the repair effect.

[0191] In one feasible implementation, step S10, "acquiring the point cloud data of the document to be processed," includes steps A11 to A14:

[0192] Step A11: Obtain first-view point cloud data and second-view point cloud data.

[0193] It should be noted that first-view point cloud data is different from second-view point cloud data. For example, first-view point cloud data can be frontal scan point cloud data, while second-view point cloud data can be rear scan data.

[0194] Step A12: Based on the point distance constraint and the normal vector gradient consistency constraint, register and stitch together the first-view point cloud data and the second-view point cloud data to obtain fused point cloud data.

[0195] Specifically, the first view point cloud data and the second view point cloud data are registered and spliced according to an error function, and a mathematical expression of the error function is as follows:

[0196]

[0197] In the formula, is the total energy error of ICP registration, and the smaller the value is, the better the alignment effect is; is a point distance constraint term, which is used to minimize the Euclidean distance of corresponding points to achieve global alignment, p is a point in the source point cloud, q is a point in the target point cloud matched with , is the square of the Euclidean distance of p and q ; is the square of the normal vector gradient difference, which is used as a gradient constraint term, is a weight coefficient, which regulates the weight of the gradient constraint term to constrain the normal vector gradient consistency, and avoid geometric mutation in the region where the splicing seam is located, is the normal vector gradient of point p and point q . In step A13, the splicing seam region of the fused point cloud data is located according to the density change in the fused point cloud data.

[0198] It should be noted that in a normal continuous region, the distribution of adjacent points is uniform, and the density change is gentle; while at the splicing seam, due to the incomplete alignment of the point clouds on both sides, the spatial distance of adjacent points will suddenly increase, forming a density fault, and the splicing seam can be located by calculating the point cloud density mutation region.

[0199] In step A14, the RBF interpolation is used to process the splicing seam region to obtain the optimized fused point cloud data.

[0200] In a specific embodiment, the point coordinates in the transition zone are determined by the point clouds on both sides of the splicing seam, the points closer to the left side are more affected by the point cloud on the left side, the points closer to the right side are more affected by the point cloud on the right side, and the middle points balance the weights on both sides, thereby realizing smooth transition from left to right. Therefore, the Gaussian kernel RBF function is used to reconstruct the point cloud coordinates, so that the point coordinates in the transition zone are generated by the weighted point clouds on both sides, and the weight is dynamically adjusted according to the distance from the splicing seam.

[0201] The document wrinkle repairing method based on deep learning in the embodiment realizes seamless fusion of multi-view point clouds by FPFH+ICP registration and RBF interpolation smoothing of the splicing seam.

[0202] In a feasible embodiment, after step S10, the following steps are further included:

[0203] Curvature analysis is performed on the point cloud data to screen out the curved sub-surface, wherein the curvature of the curved sub-surface is annularly distributed.

[0204] Specifically, the three-dimensional point cloud is projected into a polar coordinate space, and a candidate rotation axis is used as a parameter for voting. Specifically, potential rotation axes are generated by traversing the points in the point cloud, the distance variance of the points corresponding to each axis to the axis is calculated, and the points with a variance less than 0.3 mm are regarded as inliers, and the number of inliers is accumulated as a voting score. When the voting threshold exceeds 50% (i.e., more than half of the points meet the cylindrical distribution), the axis is determined as an effective rotation axis.

[0205] According to the growth algorithm, the curved sub-surface is segmented;

[0206] A parameter equation of the rotation axis of the curved sub-surface is established;

[0207] Specifically, the parameter equation of the rotation axis is as follows:

[0208]

[0209] In the formula, r is the curved radius, which is calculated by the average distance of the inliers to the axis, is the intercept of the rotation axis in the xy plane, is the inclination parameter in the z-axis direction, which is optimized by least square fitting of the inlier coordinates, : the three-dimensional coordinates of a point on the curved sub-surface in the 3D point cloud.

[0210] According to the parameter equation of the rotation axis of the curved sub-surface, the curved sub-surface is isometrically developed to obtain a flattened sub-surface.

[0211] Specifically, the 3D cylindrical surface is mapped to a 2D plane, and the mapping formula is , ;

[0212] The specific mapping formula is:

[0213]

[0214] In the formula, is the polar angle, which is calculated by the relative position of the point (x, y) and the intercept (a, b) of the rotation axis, and reflects the position of the point in the circumferential direction of the cylindrical surface, x and y are the xy coordinates of a point on the curved sub-surface in the 3D point cloud, u and v are the 2D plane coordinates (u is the circumferential coordinate and v is the axial coordinate), the isometric mapping of the cylindrical surface to the plane is realized, and z is the coordinate of a point on the curved sub-surface in the 3D point cloud, which is directly taken as the axial coordinate of the 2D plane to ensure the conservation of the axial length.

[0215] It should be noted that for the cylindrical curling area, the equidistant unfolding is realized by rotating shaft determination and parameterized modeling, which solves the problems of texture stretching and structure distortion caused by traditional overall flattening.

[0216] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1.A deep learning-based document crease repair method, characterized by, The method comprises: acquiring a two-dimensional image and point cloud data of a document to be processed, and aligning each pixel point of the two-dimensional image and the point cloud data; calculating a curvature variation rate of a neighborhood of each point in the point cloud data, and calculating a trace line according to the curvature variation rate of the neighborhood of each point; dividing and generating a plurality of sub-surfaces from the point cloud data according to the trace line, performing local affine transformation on each of the sub-surfaces to obtain a corresponding flattened sub-surface, and splicing each of the flattened sub-surfaces to obtain a three-dimensional surface model; extracting image modal features in the two-dimensional image and geometric modal features of the three-dimensional surface model, and fusing the image modal features and the geometric modal features to obtain fused features; based on the fused features, cross-modal collaborative repair of the two-dimensional image is performed to obtain a target image; the calculation of the curvature variation rate of the neighborhood of each point in the point cloud data, and the calculation of the trace line according to the curvature variation rate of the neighborhood of each point, comprises: establishing a neighborhood point set of the neighborhood of each point in the point cloud data; constructing a covariance matrix of the neighborhood point set, and performing eigenvalue decomposition on the covariance matrix to obtain discrete values of each point in the neighborhood point set in three orthogonal directions; according to the discrete values in the three orthogonal directions, the principal curvatures of the neighborhood of each point in the point cloud data are calculated, and the curvature variation rate of the neighborhood of each point in the point cloud data is calculated according to the principal curvatures of the neighborhood of each point in the point cloud data; according to the curvature variation rate of the neighborhood of each point in the point cloud data and a preset curvature variation rate threshold, trace points are calculated, wherein the curvature variation rate of the neighborhood of each trace point is greater than the preset curvature variation rate threshold; linear fitting is performed on each trace point to obtain a trace line. 2.The deep learning-based document crease repair method of claim 1, wherein, The acquisition of the point cloud data of the document to be processed comprises: acquiring a first-view two-dimensional image, a second-view two-dimensional image and TOF depth information of the document to be processed; calculating binocular depth information according to the parallax of the first-view two-dimensional image and the second-view two-dimensional image; calculating noise variances of the binocular depth information and the TOF depth information, and calculating fusion weights of the binocular depth information and the TOF depth information according to the noise variances of the binocular depth information and the TOF depth information; performing Kalman filtering fusion on the binocular depth information and the TOF depth information according to the fusion weights of the binocular depth information and the TOF depth information to obtain fused depth information; converting the fused depth information into point cloud data. 3.The deep learning-based document crease repair method of claim 2, wherein, The calculation of the binocular depth information according to the parallax of the first-view two-dimensional image and the second-view two-dimensional image comprises: collecting the first-view two-dimensional image and the second-view two-dimensional image displaying structured light stripes; calculating first-view structured light phase images and second-view structured light phase images according to the first-view two-dimensional image and the second-view two-dimensional image displaying structured light stripes by Fourier transform method; calculating parallax of each matching point pair of the first-view two-dimensional image and the second-view two-dimensional image according to the first-view grayscale image, the second-view grayscale image, the first-view structured light phase image and the second-view structured light phase image; According to the parallax of each matched point pair of the first-view two-dimensional image and the second-view two-dimensional image, binocular depth information is calculated and obtained. 4.The deep learning-based document crease repair method of claim 1, wherein, The generating of the plurality of sub-surfaces comprises: The sub-surfaces are reconstructed by using a greedy triangulation mesh reconstruction method. 5.The deep learning-based document crease repair method of claim 1, wherein, The local affine transformation of each of the sub-surfaces to obtain a corresponding flattened sub-surface comprises: The local affine transformation of the sub-surfaces to obtain a flattened sub-surface; The flattened sub-surface is optimized by using a method of solving a minimized energy function to obtain an optimized flattened sub-surface, wherein the energy function comprises a first constraint term and a second constraint term, the energy function is used to measure the distortion degree of the flattened sub-surface, the first constraint term is used to measure the deviation of the normal vector of the flattened sub-surface and the normal vector of each point of the corresponding sub-surface, and the second constraint term is used to measure the deviation of the vector length before and after flattening. 6.The deep learning-based document crease repair method of claim 1, wherein The fusion of the image modality feature and the geometry modality feature to obtain a fused feature comprises: The image modality feature and the geometry modality feature are fused based on a gating mechanism to obtain a fused feature; An initial neighbor point set of each point in the three-dimensional surface model is obtained, and principal curvatures of the initial neighbor point set are calculated and obtained; According to the principal curvatures of the initial neighbor point set, a dynamic neighbor radius is calculated; According to the dynamic neighbor radius, a neighbor point set is determined; According to the point distance and the normal vector angle of each point in the neighbor point set, an attention weight is calculated; Based on the attention weight, a feature aggregation output is obtained to obtain an optimized geometry feature; A main folding axis vector of the three-dimensional surface model is extracted; The principal curvatures of the initial neighbor point set are subjected to curvature Fourier transform to obtain a curvature high-dimensional feature; After the main folding axis vector of the three-dimensional surface model and the curvature high-dimensional feature are superimposed, the geometry modality feature in the fused feature is enhanced according to an attention mechanism and an optimized geometry feature. 7.The deep learning-based document crease repair method of claim 1, wherein, The cross-modality collaborative repair of the two-dimensional image based on the fused feature to obtain a target image comprises: The image modality feature is corrected by using the geometry modality feature, and the geometry modality feature is corrected by using the image modality feature to obtain a corrected image modality feature and a corrected geometry modality feature; According to the corrected image modality feature and the corrected geometry modality feature, the two-dimensional image is geometrically repaired, texturally repaired and structurally repaired to obtain a target image. 8.The deep learning based document crease repair method of claim 1, wherein, The obtaining of the point cloud data of the to-be-processed document comprises: First-view point cloud data and second-view point cloud data are obtained; The first-view point cloud data and the second-view point cloud data are registered and spliced based on a point distance constraint and a normal vector gradient consistency constraint to obtain fused point cloud data; According to the density change in the fused point cloud data, a splicing seam area of the fused point cloud data is located; The splicing seam area is processed by using RBF interpolation to obtain optimized fused point cloud data. 9.The deep learning based document crease repair method of claim 1, wherein, Further comprising: The point cloud data is subjected to curvature analysis, and a curled sub-surface is screened out, wherein the curvature of the curled sub-surface is annularly distributed; The curled sub-surface is segmented according to a growth algorithm; A rotation axis parameter equation of the curled sub-surface is established; The curled sub-surface is equidistantly unfolded according to the rotation axis parameter equation of the curled sub-surface to obtain a flattened sub-surface.

Images