Method and device for accurately fitting a curved boundary to a morphing document image

Abstract

The application discloses a method for accurately fitting a curved boundary of a morphed document image. A foreground picture is superimposed on a background picture after real morphing field transformation to generate a curved document image. The four corner point coordinates of the curved document and the curved mask surrounded by the four curved boundaries are taken as two labels. A deep learning model is trained, which has two branches during learning. The first branch learns to identify the four corner points of the document through the four corner point coordinates of the curved document. The second branch learns to segment the curved contour of the document through the curved mask surrounded by the four curved boundaries of the curved document. The overall boundary contour of the document is extracted from the input document image according to the contour segmentation result of the document. Then, the overall boundary contour of the document is divided into four segments in combination with the four corner points of the document. The segmentation result of the curved document output by the model of the application is more accurate, a light-weight small model can be deployed on a mobile terminal to run, and the extraction of the boundary contour of the curved document is more accurate.

Description

Technical Field

[0001] This application relates to a method for detecting document boundaries in an image. Background Technology

[0002] Document images refer to images whose main content is a document (file), usually obtained by a user photographing or scanning a paper document. Because paper documents may be bent or folded, the document in the image may be deformed, rather than a standard rectangle. Detecting and extracting document boundaries from deformed document images is a technical challenge. Inaccurate boundary detection will negatively impact subsequent document content recognition and other tasks. Summary of the Invention

[0003] The technical problem to be solved by this application is how to accurately detect and extract the boundaries of a document in a deformed document image.

[0004] To address the aforementioned technical problems, this application proposes a method for accurately fitting curved boundaries in deformed document images, comprising the following steps: Step S1: Collect a foreground image, a background image, and a real deformation field; the foreground image is a scanned document image; the background image is a non-document image; each element in the real deformation field is an inverse transformation matrix of a transformation matrix that flattens a curved surface into a plane; the foreground image is transformed by the real deformation field and superimposed on the background image to generate a curved document image, with the coordinates of the four corner points of the curved document and the surface mask formed by the four curved boundaries serving as two annotations. Step S2: Perform data augmentation on the training dataset. Step S3: Train a deep learning model using the augmented training dataset, enabling it to simultaneously identify the four corner points of a document and segment the curved surface contour of the document in a document image; the deep learning model has two branches during learning: the first branch learns to identify the four corner points of the document in the document image using the coordinates of the four corner points of the curved document, and the second branch learns to segment the curved surface contour of the document in the document image using the surface mask formed by the four curved boundaries of the curved document. Step S4: Use the trained deep learning model to obtain the four corner points and contour segmentation results of the document from the input document image. Step S5: Extract the overall boundary contour of the document from the input document image based on the contour segmentation results, and then divide the overall boundary contour of the document into four segments using the four corner points. Step S6: Calculate the curvature of each contour segment of the document, and generate sampling control points for each contour segment based on the curvature magnitude; the larger the curvature, the more control points are generated, and the smaller the curvature, the fewer control points are generated; based on the corner points and control points of each contour segment of the document, use curve fitting technology to fit four curves to represent the four boundaries of the document. Step S7: Calculate the thin-plate spline TPS transformation based on the corner points and control points of each contour curve segment of the document, convert the TPS transformation into a deformation field, and transform the document into a flattened document based on the obtained deformation field.

[0005] Further, in step S1, the coordinates of the four corner points of the document and the outlines of the four straight lines of the document are automatically generated in the foreground image; after the foreground image is transformed by the real deformation field, the planar document is transformed into a curved document, and the coordinates of the four corner points and the outlines of the four straight lines of the planar document are simultaneously transformed by the real deformation field; in the transformed curved document image, the coordinates of the four corner points of the curved document are used as the first annotation, and the curved mask enclosed by the four curved boundaries of the curved document is used as the second annotation.

[0006] Furthermore, in step S2, the curved document image is subjected to one or more of the following processes: random perspective transformation, random cropping, and random flipping. The curved document image is also randomly scaled to one of several different fixed sizes. The two annotations of each curved document image are also processed in the same way.

[0007] Furthermore, in step S3, during training, the deep learning model uses a pixel inverse reconstruction layer to reduce the width and height of the image to extract features, and then uses joint learning of local and global feature regression to obtain the four corner points of the document and the surface contour of the document in the document image.

[0008] Furthermore, in step S3, the deep learning model uses a weighted sum of the L1 loss function, the Lovars loss function, and the cross-union ratio loss function as the training loss function during training.

[0009] Furthermore, in step S3, during training, the deep learning model calculates the loss on feature maps of multiple different resolutions and backpropagates.

[0010] Furthermore, in step S6, based on the corner points at both ends and the control points in the middle of each segment of the document's outline, a curve fitting technique is used to interpolate and connect the corner points and each control point between adjacent corner points and control points, and between adjacent control points, to fit a curve representing a boundary of the document; the curve fitting technique is a parameterized Catmull-Rom curve.

[0011] Furthermore, in step S6, a closed curve representing the edge contour of the document is displayed on the software interface, allowing the user to adjust the position of any control point on the closed curve interactively.

[0012] Furthermore, in step S7, the curve length of each outline of the document is calculated, and the maximum length of the outline curves in the same direction is taken as the width and height of the document after flattening. After the document is transformed into a flattened document, it is scaled to the calculated width and height after flattening.

[0013] This application also proposes a device for accurately fitting curved boundaries to deformed document images, including a training data construction module, an augmentation module, a model training module, a model application module, a boundary segmentation module, a curve fitting module, and a flattening module. The training data construction module collects a foreground image, a background image, and a real deformation field; the foreground image is a scanned document image; the background image is a non-document image; each element in the real deformation field is an inverse transformation matrix of a transformation matrix that flattens a curved surface into a plane; the foreground image is transformed by the real deformation field and superimposed on the background image to generate a curved document image, with the coordinates of the four corner points of the curved document and the surface mask enclosed by the four curved boundaries serving as two labels. The augmentation module is used to augment the training dataset. The model training module trains a deep learning model using the augmented training dataset, enabling it to simultaneously identify the four corner points of a document and segment its surface contours within a document image. The deep learning model has two branches during learning: the first branch learns to identify the four corner points of the document image using the coordinates of these corner points; the second branch learns to segment the document's surface contours using a surface mask formed by the four curved boundaries of the document. The model application module uses the trained deep learning model to obtain the four corner points and contour segmentation results from the input document image. The boundary segmentation module extracts the overall boundary contour of the document from the input document image based on the contour segmentation results, and then divides the overall boundary contour into four segments using the four corner points. The curve fitting module calculates the curvature of each contour segment, generates sampling control points for each segment based on the curvature magnitude, and uses curve fitting techniques to fit four curves representing the four boundaries of the document based on the corner points and control points of each contour segment. The flattening module is used to calculate the thin plate spline TPS transformation based on the corner points and control points of each contour curve of the document, convert the TPS transformation into a deformation field, and transform the document into a flattened document based on the obtained deformation field.

[0014] The technical advantages achieved by this application are: accurate detection and fitting of curved boundaries in deformed document images; more accurate segmentation results for curved document output by the model; the ability to form a lightweight small model for deployment on mobile devices; and more accurate extraction of boundary contours of curved documents. These advantages all contribute to accurately extracting document regions of interest to users and achieving document region correction. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the method for accurately fitting curved boundaries to deformed document images proposed in this application.

[0016] Figure 2 This is a schematic diagram of the segmentation result of a page (curved surface) of a book using a traditional model.

[0017] Figure 3 This is a schematic diagram of the segmentation result of a page (curved surface) of a book using the model of this application.

[0018] Figure 4 This is a schematic diagram showing the result of extracting the boundary contour of a page (curved surface) of a book using existing technology.

[0019] Figure 5 This is a schematic diagram showing the result of extracting the boundary contour of a page (curved surface) of a book in this application.

[0020] Figure 6 This is a schematic diagram of the device for accurately fitting curved boundaries to deformed document images proposed in this application.

[0021] The diagram shows the following labels: Training data construction module 1, Augmentation module 2, Model training module 3, Model application module 4, Boundary segmentation module 5, Curve fitting module 6, Flattening module 7. Detailed Implementation

[0022] Please see Figure 1 The method for accurately fitting curved boundaries to deformed document images proposed in this application includes the following steps.

[0023] Step S1: Collect foreground image, background image, and true deformation field. The foreground image refers to a document image obtained by scanning the document, characterized by minimal deformation and virtually no content outside the document boundaries. The background image refers to a non-document image, which can be an image of a common object used to place documents, such as a desktop background. The true deformation field is a set where each element is the inverse transformation matrix of a transformation matrix that flattens a curved surface into a plane. For example, collect a batch of real-world cases of converting deformed document images into flat, undeformed document images, obtain the transformation matrix in each case that flattens the curved surface (i.e., the deformed document image) into a plane (i.e., the flat, undeformed document image), and use the inverse transformation matrix of this transformation matrix as an element in the true deformation field. Automatically generate the coordinates of the four corner points of the document in the foreground image and the four straight line boundary contours of the document. Substitute a foreground image, transformed by a certain true deformation field, onto a background image to generate a curved document image for training. After the foreground image undergoes the real deformation field transformation, the planar document is transformed into a curved document. The coordinates of the four corner points and the four straight line boundary contours of the planar document are simultaneously transformed by the real deformation field. In the transformed curved document image, the coordinates of the four corner points of the curved document serve as the first annotation of the image, and the curved mask formed by the four curved boundaries of the document serves as the second annotation. Each curved document image and its two annotations serve as training data, and multiple training data constitute the training dataset.

[0024] Step S2: Perform one or more of the following processes on the surface document images in the training dataset: random perspective transformation, random cropping, and random flipping. Then, randomly scale the surface document images to one of several different fixed sizes. Simultaneously, the two labels for each surface document image are also processed using random perspective transformation, random cropping, random flipping, and scaling. The coordinates of the four corner points and the surface mask formed by the four curved boundaries of the processed document serve as the two labels for the processed surface document image. This step performs data augmentation on the training dataset generated in Step S1.

[0025] Step S3: Train a deep learning model using the augmented training dataset to enable it to simultaneously identify the four corner points of a document and segment the surface contours of the document in a document image. The deep learning model may be, for example, a MobileNet series deep learning model, a UNet series deep learning model, or a DeepLab series deep learning model. The deep learning model has two branches during learning: the first branch learns to identify the four corner points of the document in the document image using the coordinates of the four corner points (first annotation); the second branch learns to segment the surface contours of the document in the document image using the surface mask (second annotation) formed by the four curved boundaries of the document.

[0026] Preferably, during training, this step utilizes a pixel unshuffling layer to reduce the image width and height (i.e., reduce the document image size) to extract features. Then, joint learning of local and global feature regression is used to obtain the four corner points of the document and the document surface contour in the document image. This approach avoids loss of image information and reduces the computational resource requirements.

[0027] Preferably, this step uses a weighted sum of the L1 loss function, the Lovasz loss function, and the Intersection over Union (IoU) loss function as the training loss function.

[0028] Preferably, this step involves calculating the loss and backpropagating it on feature maps of multiple different resolutions during training.

[0029] Step S4: Use the trained deep learning model to obtain the four corner points and contour segmentation results of the document from the input document image. If the input is a non-deformable document image, where the document is planar, the deep learning model outputs the four corner points and planar contour segmentation results of the planar document. If the input is a deformable document image, where the document is curved, the deep learning model outputs the four corner points and curved contour segmentation results of the curved document.

[0030] Step S5: Based on the document contour segmentation results, extract the overall boundary contour of the document from the input document image. Then, combine this with the four corner points of the document to divide the overall boundary contour into four segments, each representing one of the four boundaries of the document. Each segment of the document's contour has two corner points at its two ends.

[0031] Step S6: Calculate the curvature of each contour segment of the document, and generate sampling control points for each contour segment based on the curvature magnitude. More control points are generated for segments with greater curvature, and fewer control points are generated for segments with smaller curvature. This results in dense control points in areas of high curvature and sparse control points in areas of low curvature. Traditional methods of randomly generating sampling control points can lead to inaccurate fitting in highly curved areas. This step uses curvature as a basis to generate different numbers of control points, which can better capture the shape characteristics of the curve, thus making the curve fitting more accurate and improving the fitting precision and smoothness. Based on the corner points at both ends and the control points in the middle of each contour segment of the document, curve fitting technology is used to interpolate and connect the corner points and control points between adjacent corner points and control points, and between adjacent control points. Four curves are fitted to represent the four boundaries of the document, and finally, a closed curve that completely displays the edge contour of the document is fitted. The curve fitting technology is, for example, a parametric Catmull-Rom curve. This step, using the four corner points of the document and all given control points, combined with curve fitting techniques, can better capture the shape of complex contours, thus fitting a more accurate closed contour of the document.

[0032] Optionally, after displaying the closed curve representing the document's edge outline on the software interface, users can interactively adjust the position of any control point on the closed curve. This allows users to control the curve's direction more intuitively, and makes it easier for them to control and experience the curve adjustment effects.

[0033] Step S7: Calculate the Thin Plate Spline (TPS) transformation based on the corner points at both ends and the control points in the middle of each contour curve of the document. Convert the TPS transformation into a deformation field. Based on the obtained deformation field, transform the document into a flattened document, which is used as the document after ROI (region of interest) extraction.

[0034] Optionally, this step also calculates the curve length of each outline segment of the document, and takes the maximum length of the outline curves in the same direction (i.e., the two parallel boundaries of the document) as the width and height of the flattened document, respectively. After transforming the document into a flattened document, it is scaled to the calculated flattened width and height.

[0035] Compared with the prior art, this application has the following technological innovations and beneficial effects.

[0036] First, traditional models for segmenting surfaces in images use either surface masks or the four boundaries of the surface as annotations in their training data. The training data constructed in step S1 of this application has two annotations: the surface mask and the coordinates of the four corner points of the surface. Traditional models for segmenting surfaces in images only output the segmented surface contour. The model constructed in step S3 of this application outputs not only the segmented surface contour but also the four corner points of the surface. The model constructed in this application has two branches, each learning two outputs. Through these differences in training data construction, model structure, and output, this application achieves more accurate surface segmentation results.

[0037] Please see Figure 2 This is the segmentation result of a traditional model for a single page (curved surface) of a book, represented by blue lines. Because the book content contains curves, this segmentation result is inaccurate, missing the areas surrounding the two bottom corners.

[0038] Please see Figure 3 This is the segmentation result of the model in this application for a page (curved surface) of a book, represented by the blue line. Because the model in this application adds training and learning to the four corner points of the curved surface, the segmentation result is not affected by the curve contained in the book content, and the segmentation result accurately includes the area surrounding the bottom two corner points.

[0039] As can be seen from the above comparison, this application avoids the defect that the surface segmentation result may miss some corners and surrounding areas of the surface by adding two annotations on the training data, having two branches in the model structure and providing two outputs.

[0040] Second, if the training data only labels the four boundaries of the surface, the model will struggle to learn and fit these boundaries in the image, making it difficult to create a lightweight model suitable for mobile devices. The training data in this application has two labels, which complement each other, improving model performance. This allows for the creation of a lightweight model that can be deployed on mobile devices, significantly accelerating the process while maintaining performance.

[0041] Third, existing methods for extracting the boundary contours of surfaces in images only use the surface segmentation results, which cannot accurately determine the four corner points and may result in corner point offset. The method in step S5 of this application for extracting the boundary contours of surfaces in images uses both the surface segmentation results and the four corner points of the surface, which can obtain more accurate corner points and avoid the defect of corner point offset.

[0042] Please see Figure 4 This is the result of extracting the boundary contour of a page (curved surface) of a book using existing technology. The extraction result is represented by blue lines. It can be observed that the top left corner point of the extracted result is offset compared to the actual corner point.

[0043] Please see Figure 5 This is the result of extracting the boundary contour of a page (curved surface) of a book in this application. The extraction result is represented by blue lines. It can be seen that none of the corner points in the extraction result are offset from the actual corner points.

[0044] Please see Figure 6 The device for accurately fitting curved boundaries of deformed document images proposed in this application includes a training data construction module 1, an augmentation module 2, a model training module 3, a model application module 4, a boundary segmentation module 5, a curve fitting module 6, and a flattening module 7. Figure 6 The device shown corresponds to Figure 1 The method shown.

[0045] The training data construction module 1 is used to collect foreground images, background images, and a real deformation field. The foreground image is a scanned document image. The background image is a non-document image. Each element in the real deformation field is an inverse transformation matrix of a transformation matrix that flattens a surface into a plane. The foreground image is transformed by the real deformation field and then superimposed on the background image to generate a surface document image. The coordinates of the four corner points of the surface document and the surface mask enclosed by the four curved boundaries serve as two annotations.

[0046] The augmentation module 2 is used to augment the training dataset.

[0047] The model training module 3 is used to train a deep learning model using the augmented training dataset, enabling it to simultaneously identify the four corner points of a document and segment the surface contours of the document in a document image. The deep learning model has two branches during learning: the first branch learns to identify the four corner points of the document in the document image using the coordinates of the four corner points of the surface document; the second branch learns to segment the surface contours of the document in the document image using a surface mask formed by the four curved boundaries of the surface document.

[0048] The model application module 4 is used to obtain the four corner points and the outline segmentation results of the document from the input document image using the trained deep learning model.

[0049] The boundary segmentation module 5 is used to extract the overall boundary contour of the document from the input document image based on the document's contour segmentation result, and then divide the overall boundary contour of the document into four segments by combining the four corner points of the document.

[0050] The curve fitting module 6 is used to calculate the curvature of each contour segment of the document and generate sampling control points for each contour segment based on the curvature magnitude. Based on the corner points at both ends and the control points in the middle of each contour segment of the document, curve fitting technology is used to interpolate between adjacent corner points and control points, as well as between adjacent control points, to fit four curves representing the four boundaries of the document.

[0051] The flattening module 7 is used to calculate the thin plate spline TPS transformation based on the corner points at both ends and the control points in the middle of each contour curve of the document, convert the TPS transformation into a deformation field, and transform the document into a flattened document.

[0052] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for accurately fitting curved boundaries to deformed document images, characterized in that, Includes the following steps; Step S1: Collect the foreground image, background image, and real deformation field; the foreground image is a scanned document image; the background image is a non-document image; Each element in the real deformation field is an inverse transformation matrix of a transformation matrix that flattens a curved surface into a plane; the foreground image is transformed by the real deformation field and superimposed on the background image to generate a curved document image, and the coordinates of the four corner points of the curved document and the curved mask enclosed by the four curved boundaries are used as two labels. Step S2: Augment the training dataset; Step S3: Train a deep learning model using the augmented training dataset to enable it to simultaneously identify the four corner points of a document and segment the surface contour of the document in a document image; the deep learning model has two branches during learning. The first branch learns to identify the four corner points of the document in the document image by using the coordinates of the four corner points of the surface document, and the second branch learns to segment the surface contour of the document in the document image by using the surface mask formed by the four curved boundaries of the surface document. Step S4: Use the trained deep learning model to obtain the four corner points and contour segmentation results of the document from the input document image; Step S5: Extract the overall boundary contour of the document from the input document image based on the document contour segmentation result, and then divide the overall boundary contour of the document into four segments by combining the four corner points of the document. Step S6: Calculate the curvature of each contour segment of the document, and generate sampling control points for each contour segment based on the curvature magnitude; the larger the curvature, the more control points are generated, and the smaller the curvature, the fewer control points are generated; based on the corner points and control points of each contour segment of the document, use curve fitting technology to fit four curves to represent the four boundaries of the document. Step S7: Calculate the thin plate spline TPS transformation based on the corner points and control points of each contour curve of the document, convert the TPS transformation into a deformation field, and transform the document into a flattened document based on the obtained deformation field.

2. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S1, the coordinates of the four corner points of the document in the foreground image and the outlines of the four straight lines of the document are automatically generated first. After the foreground image undergoes the real deformation field transformation, the planar document is transformed into a curved document. The coordinates of the four corner points and the outlines of the four straight lines of the planar document are simultaneously transformed by the real deformation field. In the transformed curved document image, the coordinates of the four corner points of the curved document are used as the first annotation, and the curved mask enclosed by the four curved boundaries of the curved document is used as the second annotation.

3. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S2, the curved document image is subjected to one or more of the following processes: random perspective transformation, random cropping, and random flipping. The curved document image is also randomly scaled to one of several different fixed sizes. The two annotations of each curved document image are also processed in the same way.

4. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S3, during training, the deep learning model uses a pixel inverse recombination layer to reduce the width and height of the image to extract features, and then uses joint learning of local and global feature regression to obtain the four corner points of the document and the surface contour of the document in the document image.

5. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S3, the deep learning model uses a weighted sum of the L1 loss function, the Lovars loss function, and the cross-union ratio loss function as the training loss function during training.

6. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S3, the deep learning model calculates the loss and backpropagates it on feature maps of multiple different resolutions during training.

7. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S6, based on the corner points at both ends and the control points in the middle of each segment of the document's outline, a curve fitting technique is used to interpolate and connect the corner points and control points in the middle of adjacent corner points and control points, and in the middle of adjacent control points, to fit a curve representing a boundary of the document; the curve fitting technique is a parametric Catmull-Rom curve.

8. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S6, a closed curve representing the edge contour of the document is displayed on the software interface, allowing the user to adjust the position of any control point on the closed curve interactively.

9. The method for accurately fitting curved boundaries to deformed document images according to claim 1, characterized in that, In step S7, the curve length of each outline of the document is also calculated, and the maximum length of the outline curves in the same direction is taken as the width and height of the document after flattening. After the document is transformed into a flattened document, it is scaled to the calculated width and height after flattening.

10. A device for accurately fitting curved boundaries to deformed document images, characterized in that, It includes modules for training data construction, augmentation, model training, model application, boundary segmentation, curve fitting, and flattening. The training data construction module is used to collect foreground images, background images, and real deformation fields; the foreground images are scanned document images; The background image is a non-document image; Each element in the real deformation field is an inverse transformation matrix of a transformation matrix that flattens a surface into a plane; the foreground image is transformed by the real deformation field and superimposed on the background image to generate a surface document image, and the coordinates of the four corner points of the surface document and the surface mask enclosed by the four curved boundaries are used as two annotations. The augmentation module is used to augment the training dataset. The model training module is used to train a deep learning model using the augmented training dataset, enabling it to simultaneously identify the four corner points of a document and segment the surface contour of the document in a document image. The deep learning model has two branches during learning: the first branch learns to identify the four corner points of the document in the document image using the coordinates of the four corner points of the surface document, and the second branch learns to segment the surface contour of the document in the document image using the surface mask formed by the four curved boundaries of the surface document. The model application module is used to obtain the four corner points and contour segmentation results of the document from the input document image using the trained deep learning model. The boundary segmentation module is used to extract the overall boundary contour of the document from the input document image based on the document's contour segmentation result, and then divide the overall boundary contour of the document into four segments by combining the four corner points of the document. The curve fitting module is used to calculate the curvature of each contour of the document and generate sampling control points for each contour based on the curvature. Based on the corner points and control points of each contour of the document, four curves are fitted using curve fitting technology to represent the four boundaries of the document. The flattening module is used to calculate the thin plate spline TPS transformation based on the corner points and control points of each contour curve of the document, convert the TPS transformation into a deformation field, and transform the document into a flattened document based on the obtained deformation field.

Images