A method and apparatus for dynamically displaying the correction process of document images.

Abstract

This invention discloses a document image deformation correction method that dynamically displays the correction process. The input document image is fed into an image edge segmentation model to obtain its edge contours. The input document image is then fed into an image curvature correction model, which outputs a dense displacement field with reversible spatial transformation. The number of frames N for displaying the image deformation correction effect is determined by the standard deviation (std) of the dense displacement field with reversible spatial transformation. Based on N, the dense displacement field with reversible spatial transformation is divided equally. Each frame is obtained by gradually combining the equally divided dense displacement field with the edge contours of the original deformed document image. The calculated N frames are displayed one by one, thus dynamically displaying the entire process of deformation correction of the original deformed document image until it is stretched and flattened. This invention not only stretches and flattens the original deformed document image but also dynamically and intuitively displays the entire stretching process in the form of video and animation.

Description

Technical Field

[0001] This invention relates to a deformation correction method for document images, which can dynamically display the process of flattening curved surfaces. Background Technology

[0002] Electronic documents are easier to archive, edit, sign, and share, and more and more documents are stored and interacted with in electronic form. With the widespread use of high-quality cameras on mobile devices such as smartphones, digital document capture using mobile devices has become very common. Paper documents may be deformed due to bending, curling, or unevenness, and unfavorable shooting conditions or improper shooting processes can easily cause further deformation of the document image. Deformed document images reduce readability, make it difficult to extract information, and increase the difficulty of data augmentation and downstream tasks such as OCR (Optical Character Recognition), layout analysis, and reconstruction. Summary of the Invention

[0003] The technical problem to be solved by this invention is to dynamically and intuitively display the process of stretching deformed document images to flatten them and flattening the original deformed document images from curved surfaces to planes.

[0004] To address the aforementioned technical problems, this invention proposes a document image deformation correction method that dynamically displays the correction process, comprising the following steps: Step S1: Collect deformed document images, annotate key points on the document images, and obtain a stretched and flattened image of the document images using thin-plate spline interpolation (TPS transformation); use the original deformed document images and the stretched and flattened document images as a pair of training data, and multiple pairs of training data constitute a training dataset. Step S2: Perform random perspective transformation, random flipping, and random scaling on the original deformed document images in the training dataset to one of multiple fixed sizes, and correspondingly change the corresponding stretched and flattened document images to obtain an enhanced training dataset. Step S3: Use the enhanced training dataset to train an encoder-decoder type image segmentation network to obtain an image bending correction model with dense displacement field modeling and reversible spatial transformation. Step S4: Train an encoder-decoder type image segmentation network to obtain an image edge segmentation model. Step S5: Send the input document image to the image edge segmentation model to obtain the edge contour of the input document image. Step S6: The input document image is sent to the image warping correction model, which outputs a dense displacement field with reversible spatial transformation. Steps S5 and S6 can be performed either in any order or simultaneously. Step S7: The number of frames N for displaying the image deformation correction effect is determined based on the standard deviation (std) of the dense displacement field with reversible spatial transformation. The dense displacement field with reversible spatial transformation is divided equally into N, and each frame is obtained by gradually combining the equally divided dense displacement field with the edge contour of the original deformed document image. The calculated N frames are displayed one by one, thus dynamically displaying the entire process of deformation correction of the original deformed document image until it is stretched and flattened.

[0005] Further, in step S3, the training method of the image bending correction model is as follows: input the original deformed document image, output a dense displacement field with reversible spatial transformation, and require that the combination of the original deformed document image and the dense displacement field with reversible spatial transformation be as close as possible to the document image after stretching and flattening obtained by the original deformed document image through TPS transformation.

[0006] Furthermore, in step S3, the encoding and decoding of the image bending correction model reduces the width and height of the input image without losing information. Then, it extracts features from the small image and feeds them back to the large image to calculate the loss function. Finally, it downsamples and jointly learns local and global feature regression to obtain a dense displacement field with reversible spatial transformation.

[0007] Furthermore, in step S3, the dense displacement field obtained by the image bending correction model is constrained by the differential homeomorphism property of spatial transformation, including any one of the displacement field first-order gradient constraint, displacement field folding constraint, and displacement field folding penalty constraint.

[0008] Furthermore, in step S3, during the training process, the image bending correction model uses text line and line supervision to calculate the loss on feature maps of multiple different resolutions and backpropagates it, using the weighted sum of the LCC loss function and the folding penalty loss function as the loss function during training; or, the LCC loss function is replaced by the mean absolute error L1, or the mean squared error L2, or the structural similarity index SSIM loss function.

[0009] Further, in step S4, the training method of the image edge segmentation model is as follows: input the original deformed document image, output the edge contour of the original deformed document image, and require the output edge contour to be as close as possible to the edge contour obtained by calculation; the edge contour obtained by calculation is obtained by combining the original deformed document image input by the image bending correction model with the output dense displacement field with reversible spatial transformation to obtain the document image after stretching and flattening the original deformed document image; the stretched and flattened document image is annotated with 4 corner points to obtain 4 straight line contours, and the 4 straight line contours of the stretched and flattened document image are combined with the dense displacement field with reversible spatial transformation.

[0010] Furthermore, in step S4, the image edge cutting model calculates the loss and backpropagates it on feature maps of multiple different resolutions, and uses the weighted sum of Lovasz-softmax and the Intersection over Union (IoU) loss function as the loss function during training.

[0011] Furthermore, in step S7, the larger std is, the larger N is; and vice versa.

[0012] Alternatively, the "dense displacement field" and "displacement field" in steps S3-S7 can be uniformly replaced with "velocity field".

[0013] This invention also proposes a document image deformation correction device that dynamically displays the correction process, including a training data production unit, a training data enhancement unit, a first neural network training unit, a second neural network training unit, an image edge segmentation unit, an image curvature correction unit, and a correction process display unit. The training data production unit collects deformed document images, annotates key points on the document images, and uses thin-plate spline interpolation (TPS) to obtain a stretched and flattened image of the document image. The original deformed document image and the stretched and flattened document image are used as a pair of training data, and multiple pairs of training data constitute a training data set. The training data enhancement unit performs random perspective transformation, random flipping, and random scaling on the original deformed document images in the training data set to one of multiple fixed sizes, and correspondingly changes the corresponding stretched and flattened document image to obtain an enhanced training data set. The first neural network training unit uses the enhanced training data set to train an encoder-decoder type image segmentation network to obtain an image curvature correction model with a dense displacement field modeling with reversible spatial transformation. The second neural network training unit trains an encoder-decoder type image segmentation network to obtain an image edge segmentation model. The image edge segmentation unit sends the input document image to the image edge segmentation model to obtain the edge contour of the input document image. The image curvature correction unit sends the input document image to the image curvature correction model and outputs a dense displacement field with reversible spatial transformation. The correction process display unit determines the number of frames N for displaying the image deformation correction effect based on the standard deviation (std) of the dense displacement field with reversible spatial transformation; it divides the dense displacement field with reversible spatial transformation into equal parts based on N, and gradually combines the equally divided dense displacement field with the edge contour of the original deformed document image to obtain each frame, and displays each frame in sequence, thus showing the entire process of deformation correction of the original deformed document image until it is stretched and flattened.

[0014] The technical effect achieved by this invention is that it not only stretches the original deformed document image to a flat surface, but also dynamically and intuitively displays the entire stretching process in the form of video and animation. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the document image deformation correction method that dynamically displays the correction process, as proposed in this invention.

[0016] Figure 2 This is a schematic diagram of the document image deformation correction device that dynamically displays the correction process, as proposed in this invention.

[0017] The following are the annotations in the figure: 1 is the training data production unit, 2 is the training data augmentation unit, 3 is the first neural network training unit, 4 is the second neural network training unit, 5 is the image edge segmentation unit, 6 is the image curvature correction unit, and 7 is the correction process display unit. Detailed Implementation

[0018] Please see Figure 1 The document image deformation correction method for dynamically displaying the correction process proposed in this invention includes the following steps.

[0019] Step S1: Collect deformed document images, annotate key points on the document images, and use Thin Plate Spline (TPS) interpolation to obtain a stretched and flattened image of the document images. Use the original deformed document images and the stretched and flattened document images as a pair of training data, and multiple pairs of training data constitute a training dataset.

[0020] Step S2: Perform random perspective transformation, random flipping, and random scaling on the original deformed document images in the training dataset to one of several fixed sizes, and correspondingly change the corresponding stretched and flattened document images to obtain the enhanced training dataset. For example, in a pair of training data, if the original deformed document image is flipped horizontally, the corresponding stretched and flattened document image will also be flipped horizontally.

[0021] Step S3: Train an encoder-decoder type image segmentation network, such as the U-Net series convolutional neural network or the MobileNet series convolutional neural network, using an enhanced training dataset to obtain an image bending correction model modeled with a dense displacement field featuring reversible spatial transformation. The training method for this image bending correction model is as follows: input the original deformed document image, preferably a real-world photographed document image, and output a dense displacement field with reversible spatial transformation. The combination of the original deformed document image and the dense displacement field with reversible spatial transformation should be as close as possible to the stretched and flattened document image obtained in step S1. Traditional training methods for neural networks used for image bending correction involve inputting the original deformed document image, typically a document image with a deformed effect created through synthesis, and outputting a dense displacement field, without involving the stretched and flattened document image or reversible spatial transformation. The training method of this invention enhances the robustness of the image bending correction model, providing better stretching and flattening effects for deformed document images in real-world scenarios.

[0022] The image curvature correction model simultaneously decodes and fuses feature layers of multiple sizes from the encoder, enhancing its ability to learn fine-grained visual differences. The encoding and decoding process preferably utilizes a pixel unshuffle layer to reduce the width and height of the input image without information loss, achieving better results at smaller sizes. Subsequently, features are extracted from the small image using multi-scale image features and fed back to the large image to calculate the loss function. Then, downsampling is performed to jointly learn local and global feature regressions to obtain a dense displacement field with reversible spatial transformation. This spatial transformation, applied to the large image after spatial interpolation in the small image, shortens the processing time of the image curvature correction model without sacrificing the resolution of the large image.

[0023] To ensure the reversibility of the dense displacement field obtained by the image bending correction model, this invention utilizes the differential homeomorphism property of spatial transformation to constrain the dense displacement field obtained by the image bending correction model, such as first-order gradient constraint, displacement field folding constraint, and displacement field bending penalty constraint. Preferably, the displacement field bending penalty constraint is used to ensure its reversibility. Each line of text is labeled in each pair of original deformed document images and stretched / flattened document images, referred to as a text line. Each line segment is labeled in each pair of original deformed document images and stretched / flattened document images, referred to as a line. The text lines and lines constitute a feature map. During the training process, the image bending correction model uses a text line and line-supervised method to calculate the loss on multiple feature maps of different resolutions and backpropagates it. The weighted sum of the LCC (Linear Correlation Coefficient) loss function and the bending penalty loss function is used as the loss function during training. The LCC loss function can be replaced by the L1 (mean absolute error, MAE), L2 (mean squared error, MSE), or SSIM (Structural Similarity Index) loss function.

[0024] Step S4: Train an encoder-decoder type image segmentation network, such as the U-Net series of convolutional neural networks, to obtain an image edge segmentation model. The training method for the image edge segmentation model is as follows: input the original deformed document image, output the edge contours (combination of curves) of the original deformed document image, requiring the output edge contours to be as close as possible to the edge contours of the document image before deformation correction obtained through calculation. The calculation method for the "edge contours of the document image before deformation correction obtained through calculation" is as follows: combine the original deformed document image input to the image bending correction model with the dense displacement field with reversible spatial transformation output to obtain the document image after stretching and flattening the original deformed document image. Mark the four corner points of the stretched and flattened document image to obtain four straight line contours. Combine the four straight line contours of the stretched and flattened document image with the dense displacement field with reversible spatial transformation to obtain the edge contours (curved contours) of the document image before deformation correction (i.e., the original deformed image). The image edge cutting model also calculates the loss and backpropagates it on feature maps of multiple different resolutions, using a weighted sum of Lovasz-softmax and IoU (Intersection over Union) loss functions as the loss function during training.

[0025] Step S5: The input document image is fed into the image edge segmentation model to obtain the edge contours of the input document image. Preferably, before feeding the input document image into the image edge segmentation model, this step involves scaling the input document image to a fixed size, such as one of the multiple fixed sizes in the training data of the image edge segmentation model.

[0026] Step S6: Send the input document image to the image curvature correction model and output a dense displacement field (Δx, Δy) with reversible spatial transformation.

[0027] The order of steps S5 and S6 is not strictly limited; they can be performed either before or simultaneously.

[0028] Step S7: Determine the number of frames N for displaying the image deformation correction effect based on the standard deviation (std) of the dense displacement field with reversible spatial transformation. A larger std indicates a greater degree of curvature in the original deformed document image, requiring more frames for a smoother display process. For example, N = 10 when std ≥ 5, N = 5 when 1 ≤ std < 5, and N = 2 when std < 1. Divide the dense displacement field (Δx, Δy) with reversible spatial transformation into (Δx, Δy) / N equal parts based on the number of frames N for displaying the image deformation correction effect. Then, gradually combine the equally divided dense displacement field (Δx, Δy) / N with the edge contour of the original deformed document image to obtain each frame. The first frame is based on the edge contour of the original deformed document image, combined with a densely divided displacement field (Δx, Δy) / N. This first frame is closest to the edge contour of the original deformed document image. Each subsequent frame is based on the edge contour of the document image in the previous frame, again combined with a densely divided displacement field (Δx, Δy) / N. The last frame is closest to the edge contour of the stretched and flattened document image. The display starts with the edge contour of the original deformed document image, then shows the calculated N frames one by one, ending with the edge contour of the stretched and flattened document image. Finally, the entire process of deformation correction and stretching and flattening of the original deformed document image is dynamically and smoothly presented in a video and animation format.

[0029] The “dense displacement field” and “displacement field” in steps S3-S7 can also be uniformly replaced with “velocity field” as a variant implementation of the image bending correction model.

[0030] Please see Figure 2 The document image deformation correction device for dynamically displaying the correction process proposed in this invention includes a training data production unit 1, a training data enhancement unit 2, a first neural network training unit 3, a second neural network training unit 4, an image edge segmentation unit 5, an image curvature correction unit 6, and a correction process display unit 7. Figure 2 The device shown corresponds to Figure 1 The method shown.

[0031] The training data production unit 1 is used to collect deformed document images, annotate key points on the document images, and obtain a stretched and flattened image of the document images using thin-plate spline interpolation transformation. The original deformed document image and the stretched and flattened document image are used as a pair of training data, and multiple pairs of training data constitute a training data set.

[0032] The training data augmentation unit 2 is used to perform random perspective transformation, random flipping, and random scaling to one of multiple fixed sizes on the original deformed document images in the training data set, and accordingly change the corresponding stretched and flattened document images to obtain an augmented training data set.

[0033] The first neural network training unit 3 is used to train an encoder-decoder type image segmentation network using an enhanced training dataset to obtain an image bending correction model with dense displacement field modeling of reversible spatial transformation.

[0034] The second neural network training unit 4 is used to train an encoder-decoder type image segmentation network to obtain an image edge segmentation model.

[0035] The image edge segmentation unit 5 is used to send the input document image to the image edge segmentation model to obtain the edge contour of the input document image.

[0036] The image curvature correction unit 6 is used to send the input document image to the image curvature correction model and output a dense displacement field with reversible spatial transformation.

[0037] The correction process display unit 7 is used to determine the number of image deformation correction effect display frames N based on the standard deviation of the dense displacement field with reversible spatial transformation. The dense displacement field with reversible spatial transformation is divided equally according to the number of image deformation correction effect display frames N. Each frame is obtained by gradually combining the equally divided dense displacement field with the edge contour of the original deformed document image, and each frame is displayed sequentially, thus showing the entire process of deformation correction of the original deformed document image until it is stretched and flattened.

[0038] This invention utilizes a data-driven method to learn the spatial changes of a flattened curved image (represented by a dense displacement field). Based on the invertibility of spatial transformation, the edge contours of the corrected document image are mapped back to the curved contours of the original document image, solving the problem of difficult annotation and acquisition of curved contours. Then, a data-driven method is used to learn the surface contours of the document image. To ensure this invention can be deployed on mobile devices, it reduces the image size to extract features without information loss, then downsamples and jointly learns local and global feature regression to obtain a dense displacement field. The spatial changes are then upsampled and applied to the original image to calculate the loss, ensuring both correction accuracy and fast, real-time operation on mobile devices.

[0039] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A document image deformation correction method that dynamically displays the correction process, characterized in that, Includes the following steps; Step S1: Collect deformed document images, annotate key points on the document images, and use thin plate spline interpolation TPS transformation to obtain the stretched and flattened image of the document images; use the original deformed document images and the stretched and flattened document images as a pair of training data, and multiple pairs of training data constitute a training data set. Step S2: Perform random perspective transformation, random flipping, and random scaling on the original deformed document images in the training dataset to one of multiple fixed sizes, and correspondingly change the corresponding stretched and flattened document images to obtain the enhanced training dataset. Step S3: Train an encoder-decoder type image segmentation network using an enhanced training dataset to obtain an image bending correction model with dense displacement field modeling and reversible spatial transformation; Step S4: Train an encoder-decoder type image segmentation network to obtain an image edge segmentation model; Step S5: The input document image is sent to the image edge segmentation model to obtain the edge contour of the input document image; Step S6: The input document image is sent to the image curvature correction model, and the output is a dense displacement field with reversible spatial transformation; The order of steps S5 and S6 can be arbitrary, or they can be performed simultaneously. Step S7: Determine the number of frames N for displaying the image deformation correction effect based on the standard deviation (std) of the dense displacement field with reversible spatial transformation; divide the dense displacement field with reversible spatial transformation into equal parts based on N, and gradually combine the divided dense displacement field with the edge contour of the original deformed document image to obtain each frame; The calculated N frames are displayed one by one, which dynamically shows the entire process of deformation correction of the original deformed document image until it is stretched and flattened.

2. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S3, the training method of the image bending correction model is as follows: input the original deformed document image, output a dense displacement field with reversible spatial transformation, and require that the combination of the original deformed document image and the dense displacement field with reversible spatial transformation be as close as possible to the document image after stretching and flattening obtained by the original deformed document image through TPS transformation.

3. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S3, the encoding and decoding of the image bending correction model reduces the width and height of the input image without losing information. Then, it extracts features from the small image and feeds them back to the large image to calculate the loss function. Finally, it downsamples and jointly learns local and global feature regression to obtain a dense displacement field with reversible spatial transformation.

4. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S3, the dense displacement field obtained by the image bending correction model is constrained by the differential homeomorphic property of spatial transformation, including any one of the displacement field first-order gradient constraint, displacement field folding constraint, and displacement field folding penalty constraint.

5. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S3, during the training process, the image curvature correction model uses text line and line supervision to calculate the loss on multiple feature maps of different resolutions and backpropagates it. The weighted sum of the LCC loss function and the folding penalty loss function is used as the loss function during training; or, the LCC loss function is replaced by the mean absolute error L1, or the mean squared error L2, or the structural similarity index SSIM loss function.

6. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S4, the training method of the image edge segmentation model is as follows: input the original deformed document image, output the edge contour of the original deformed document image, and require that the output edge contour be as close as possible to the edge contour obtained by calculation; the edge contour obtained by calculation is obtained by combining the original deformed document image input by the image bending correction model with the dense displacement field with reversible spatial transformation output to obtain the document image after stretching and flattening the original deformed document image; the four corner points of the stretched and flattened document image are marked to obtain four straight line contours, and the four straight line contours of the stretched and flattened document image are combined with the dense displacement field with reversible spatial transformation.

7. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S4, the image edge cutting model calculates the loss and backpropagates it on multiple feature maps of different resolutions, and uses the weighted sum of Lovasz-softmax and the Intersection over Union (IoU) loss function as the loss function during training.

8. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In step S7, the larger std is, the larger N is; and vice versa.

9. The document image deformation correction method for dynamically displaying the correction process according to claim 1, characterized in that, In steps S3-S7, "dense displacement field" and "displacement field" are uniformly changed to "velocity field".

10. A document image deformation correction device that dynamically displays the correction process, characterized in that, It includes a training data production unit, a training data augmentation unit, a first neural network training unit, a second neural network training unit, an image edge segmentation unit, an image curvature correction unit, and a correction process display unit; The training data production unit is used to collect deformed document images, annotate key points on the document images, and use thin plate spline interpolation TPS transformation to obtain stretched and flattened images of the document images; the original deformed document images and stretched and flattened document images are used as a pair of training data, and multiple pairs of training data constitute a training data set. The training data augmentation unit is used to perform random perspective transformation, random flipping, and random scaling to one of a number of fixed sizes on the original deformed document images in the training dataset, and accordingly change the corresponding stretched and flattened document images to obtain an augmented training dataset. The first neural network training unit is used to train an encoder-decoder type image segmentation network using an enhanced training dataset to obtain an image bending correction model with dense displacement field modeling of reversible spatial transformation; The second neural network training unit is used to train an encoder-decoder type image segmentation network to obtain an image edge segmentation model; The image edge segmentation unit is used to send the input document image to the image edge segmentation model to obtain the edge contour of the input document image; The image curvature correction unit is used to send the input document image to the image curvature correction model and output a dense displacement field with reversible spatial transformation. The correction process display unit is used to determine the number of frames N for displaying image deformation correction effects based on the standard deviation (std) of the dense displacement field with reversible spatial transformation. The dense displacement field with reversible spatial transformation is divided equally according to N. Each frame is obtained by gradually combining the equally divided dense displacement field with the edge contour of the original deformed document image. Each frame is displayed in sequence, which shows the entire process of deformation correction of the original deformed document image until it is stretched and flattened.

Images