Method, device and storage medium for correcting document images
By detecting edge key points of document image fitting edge lines and performing grid processing, the problem of unsmooth output of deep learning model is solved, smooth correction of document image and uniform distribution of content is achieved, and the accuracy of text recognition is improved. User experience.
Patent Information
- Application Number
- CN202210788200.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-04
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2042-07-04
AI Technical Summary
In the prior art, when deep learning models are used for document image correction, the output correction coordinate mapping matrix is not smooth, resulting in the corrected image not smooth, affecting the accuracy and user experience of document content recognition, and the training data cannot be obtained from the real scene. Acquisition results in poor correction effect of the model in real-life scenarios.
By detecting the edge key points of the document image, fitting and generating edge lines, and grid processing is performed based on the document area surrounded by the edge lines to generate corrected document images, and using perspective transformation to calculate the positions of non-equivalent points to build a rectangular blank image Fill in pixel values to achieve smooth correction of documents.
It realizes smooth correction of surface document images, improves the accuracy and user experience of text recognition, and ensures that the corrected document images are smooth and evenly distributed.
Smart Images

Figure CN115082935B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present invention relate to the field of image processing, and more specifically, embodiments of the present invention relate to a method, a device, and a computer-readable storage medium for correcting a document image. Background Art
[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The descriptions herein may include concepts that may be explored, but not necessarily concepts that have been previously thought of or explored. Therefore, unless otherwise noted herein, the content described in this section is not prior art with respect to the specification and claims of this application, and is not admitted to be prior art by inclusion in this section.
[0003] With the development of artificial intelligence technology, computer vision (CV) related technologies are widely used in various educational and efficiency tool products. Information extraction of paper documents in images is one of the important technologies. However, in real scenes, due to the influence of multiple factors such as shooting equipment, environment, and paper status, the document paper in the document image processed by the computer is prone to various deformations, such as paper tilt, wrinkles, twists, curls, etc., which seriously affects the accuracy of computer algorithms for text recognition and the user's subjective perception of document images. Therefore, how to remove the above-mentioned deformations of the document paper in the input image to output a flat and correct document image is a technical problem that needs to be solved urgently.
[0004] In the prior art, some methods for correcting document images have emerged. For example, in a document correction solution based on deep learning, the deep learning model can be trained so that after receiving a distorted document image, it can output a corrected document image or a coordinate mapping matrix on which the correction depends. The model is trained in a supervised manner and relies on large-scale training data. The training data consists of a large number of sample pairs, each of which can contain two parts: a distorted document image and a corrected coordinate mapping matrix. The training data is usually obtained by generating it through a 3D engine. Summary of the invention
[0005] However, since the correction coordinate mapping matrix output by the deep learning model is often not smooth, the corrected image is not smooth, which still affects the accuracy of content recognition in the document and the user's subjective perception. Furthermore, since the training data used to train the deep learning model cannot be obtained from real scenes, and the distorted document image rendered by the 3D engine is inconsistent with the real world scene, it is easy to cause the deep learning model to overfit the 3D virtual image, resulting in poor correction effect of the deep learning model in real scenes.
[0006] Therefore, in the prior art, this is a very annoying process. For this reason, an improved method for correcting document images is very much needed so that distorted document images in real scenes can be corrected better. In this context, embodiments of the present invention are expected to provide a method, device and computer-readable storage medium for correcting document images.
[0007] In a first aspect of an embodiment of the present invention, a method for correcting a document image is provided, comprising: detecting edge key points of a document in an image to be tested; fitting and generating edge lines of the document based on the distribution trend of the edge key points; and generating a corrected document image based on a document area enclosed by the edge lines.
[0008] In one embodiment of the present invention, fitting and generating edge lines includes: in response to a plurality of edge key points on the same side of the document being distributed in a curve, using a polynomial fitting function to fit and generate corresponding curved edge lines.
[0009] In another embodiment of the present invention, generating a corrected document image based on a document area includes: gridding the document area to generate a grid document image; and constructing a rectangular blank image, and filling the pixel value at each grid coordinate in the grid document image into the corresponding coordinate point in the rectangular blank image to generate the corrected document image.
[0010] In another embodiment of the present invention, the edge lines of the document include two opposite curved edge lines and two opposite straight edge lines, and gridding the document area includes: determining a plurality of equally divided points of the same number on the two curved edge lines; connecting the corresponding equally divided points on the two curved edge lines in the document area to form a plurality of straight lines between the two curved edge lines; determining a plurality of non-equally divided points of the same number on each straight edge line and each straight line; and connecting the corresponding non-equally divided points on the straight edge lines and the straight lines to generate the grid document image.
[0011] In one embodiment of the present invention, determining the same number of multiple non-equally divided points on each straight edge line and each straight line connection includes: using perspective transformation to calculate the length ratio of two adjacent line segments between multiple non-equally divided points on the same straight line; determining the length of each line segment on each straight edge line and each straight line connection according to the length of each straight edge line and each straight line connection, the length ratio and the number of non-equally divided points on the same straight line, so as to determine the positions of each non-equally divided point on each straight edge line and each straight line connection.
[0012] In another embodiment of the present invention, calculating the length ratio using perspective transformation includes: taking the four corner points where two straight edge lines are located as source points and the four corner points of an arbitrary rectangle as target points, and calculating the perspective transformation matrix; taking adjacent first source line segments and second source line segments of equal length on any straight edge line, and using the perspective transformation matrix to perform perspective transformation on the coordinates of the separation point and the two endpoint coordinates between the first source line segment and the second source line segment, respectively, to obtain corresponding first transformed line segments and second transformed line segments; and determining the length ratio according to the ratio of the lengths of the first transformed line segment to the lengths of the second transformed line segment.
[0013] In another embodiment of the present invention, determining the position of each non-equally divided point further comprises: determining the position of the first non-equally divided point on each straight edge line and each straight line connection line according to the relative position relationship between the first transformed line segment and the second transformed line segment.
[0014] In one embodiment of the present invention, the two endpoints are two corner points on the straight edge line; and the separation point is the midpoint of the straight edge line.
[0015] In another embodiment of the present invention, detecting edge key points of a document in an image to be tested includes: detecting the image to be tested using a key point detection model to determine edge key points of the document in the image to be tested.
[0016] In another embodiment of the present invention, before using the key point detection model for detection, the method also includes: determining four sample corner points among the dense contour points of the document in the sample image, and separating the dense contour points into point sets of four sample edges according to the four sample corner points; determining a plurality of equally divided sample points for the curve sample edges at least in the point set of the curve sample edges among the four sample edges; and training the key point detection model using a sample image with four sample corner points and a plurality of equally divided sample points.
[0017] In one embodiment of the present invention, it further includes: determining the sample midpoint of the straight sample edge in the point set of the straight sample edge among the four sample edges; and training the key point detection model using a sample image with four sample corner points, a plurality of equally divided sample points and a sample midpoint.
[0018] In another embodiment of the present invention, before determining the four sample corner points, it also includes: performing a first annotation on the contour points of the document in the sample image; performing a second annotation on the four corner points of the document in the sample image; converting the first annotated contour points into dense contour points; and determining the point in the dense contour points that has the smallest distance to the second annotation as the sample corner point.
[0019] In a second aspect of an embodiment of the present invention, a device for correcting a document image is provided, comprising a processor configured to execute program instructions; and a memory configured to store the program instructions, wherein when the program instructions are executed by the processor, the device executes any one of the methods described in the first aspect of the embodiment of the present invention.
[0020] In a third aspect of the embodiments of the present invention, a computer-readable storage medium is provided, which stores program instructions. When the program instructions are loaded and executed by a processor, the processor executes the method according to any one of the first aspects of the embodiments of the present invention.
[0021] According to the method for correcting a document image according to an embodiment of the present invention, the edge lines of the document can be fitted according to the distribution trend of the edge key points of the document in the image to be tested, so that a corrected document image can be generated according to the document area surrounded by the edge lines. According to such a method, flattening correction of curved document images (or distorted document images) can be achieved, and the smoothness of the corrected document image can be ensured, bringing a better experience to users. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, in which:
[0023] Figure 1 A block diagram of an exemplary device 100 suitable for implementing embodiments of the present invention is shown;
[0024] Figure 2 A flow chart of a method for correcting a document image according to an embodiment of the present invention is schematically shown;
[0025] Figure 3 A flowchart of a method for gridding processing according to an embodiment of the present invention is schematically shown;
[0026] Figure 4 A schematic diagram schematically shows an image to be tested according to an embodiment of the present invention;
[0027] Figure 5 Schematically shows the Figure 4 A schematic diagram of a process of generating a grid document image from a test image as shown in ; and
[0028] Figure 6 The flowchart of the method for performing detection using a key point detection model according to an embodiment of the present invention is schematically shown.
[0029] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0030] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.
[0031] Figure 1 1 is a block diagram of an exemplary device 100 suitable for implementing embodiments of the present invention. Figure 1 As shown, the device 100 may include, for example, a CPU 101, a memory, and its peripheral devices and an external network, wherein the device of the embodiment of the present invention may be used to execute the method for correcting a document image of the present invention.
[0032] Furthermore, if Figure 1 As shown in , the device according to the embodiment of the present invention may include a central processing unit (CPU) 101, a random access memory (RAM) 102, a read-only memory (ROM) 103, and may also include a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, the RAM 102, the ROM 103, the hard disk controller 105, the keyboard controller 106, the serial controller 107, the parallel controller 108, and the display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that Figure 1 The structural block diagram is only for the purpose of illustration, rather than limiting the scope of the present invention. In some cases, some equipment may be added or reduced according to specific circumstances.
[0033] It is known to those skilled in the art that the embodiments of the present invention may be implemented as a device, method or computer program product. Therefore, the present invention may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present invention may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.
[0034] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.
[0035] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0036] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0037] Computer program code for performing the operation of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0038] The following will describe the implementation of the present invention with reference to the flowchart of the method of the embodiment of the present invention and the block diagram of the device (or system). It should be understood that each square frame of the flowchart and / or block diagram and the combination of square frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, thereby producing a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the function / operation specified in the square frame in the flowchart and / or block diagram.
[0039] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.
[0040] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.
[0041] According to an embodiment of the present invention, a method, device and computer-readable storage medium for correcting a document image are provided. In addition, any number of elements in the accompanying drawings is for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning. The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention. SUMMARY OF THE INVENTION
[0043] The inventors have discovered that paper curling is a common problem in real-life scenarios, such as paper curling near the spine of an open book. If only the four corner points of the paper are detected, the machine cannot obtain the shape of the paper edge between the four corner points, so the machine can only assume that the paper is flat during processing, and therefore cannot correct the bent paper.
[0044] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention are described in detail below.
[0045] Application Scenario Overview
[0046] The method for correcting a document image according to an embodiment of the present invention can be applied to various document correction scenarios based on computer vision. Specifically, the method according to an embodiment of the present invention can be applied to various devices that can capture document images, such as smart learning desk lamps, smart phones, computers, scanning pens, and can be combined with text recognition technology to achieve the purpose of identifying the document content in the corrected image. Furthermore, the method of the embodiment of the present invention can be implemented by an application running on a machine. Such an application can be, for example, a document correction APP. The document correction APP can be used to correct various types of document images, such as pictures of books, papers, packaging boxes, posters, etc.
[0047] Exemplary Methods
[0048] In combination with the above application scenarios, Figure 2 The method for correcting a document image according to an exemplary embodiment of the present invention is described. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.
[0049] Figure 2 The flowchart of the method for correcting a document image according to an embodiment of the present invention is schematically shown. Figure 2As shown in , method 200 may include: in step 210, edge key points of a document in the image to be tested may be detected. In some embodiments, the edge key points are points located on the edge of the document. The edge key points may include at least the corner points of the document. The corner points may be understood as the intersection points of two lines, for example, the four vertices of a rectangular document are the four corner points. In other embodiments, the edge key points may also include equal points on each edge, such as a midpoint, a quarter point, a fifth point, a sixth point, etc. In still other embodiments, the edge key points may also include key points at deformation locations on the edge of the document, such as an inflection point on a curved edge of the document. In some embodiments, the edge key points on each edge of the document may include one or more. In some application scenarios, the document may include objects that are prone to deformation, such as books, paper, packaging boxes, file bags, posters, etc., and the image to be tested may be an image containing the document.
[0050] Next, in step 220, the edge line of the document can be fitted based on the distribution trend of the edge key points. In some embodiments, the edge line on each edge can be fitted separately based on the distribution trend of the edge key points on each edge of the document. In one embodiment of the present invention, step 220 may include: in response to the multiple edge key points on the same edge of the document being distributed in a curve, a polynomial fitting function may be used to fit and generate the corresponding curved edge line. Here, the multiple edge key points on the same edge may include two corner points on the same edge and one or more key points between the two corner points. In some embodiments, the document in the image to be tested may have one or more such curved edge lines.
[0051] Compared with the traditional edge detection or linear fitting method, which can only fit multiple edge key points into straight lines but cannot reflect the actual distribution trend characteristics on the document edge, the polynomial fitting function (or polynomial curve fitting function) according to the embodiment of the present invention can select a function with good prediction ability for both known data and unknown data from multiple polynomial functions, so that the fitting curve that best fits the actual shape of the edge key points can be fitted according to the distribution trend of the edge key points. According to such a setting, the curve characteristics of the curve edge line can be retained, which is conducive to machine learning and recognition, and is conducive to the correction of curved surface documents.
[0052] In some other embodiments, step 220 may further include: in response to the multiple edge key points on the same side of the document being distributed in a straight line, a straight line fitting method such as a linear fitting function, a least square method, etc. may be used to fit and generate a corresponding straight edge line. In some other embodiments, for the straight edge of the document, since no shape change occurs, the edge key points detected in step 210 may only include two corner points (or two vertices) on the straight edge, and then step 220 may include: in response to the same side of the document including only two corner key points, the two corner key points may be directly connected to fit and generate a straight edge line. In some embodiments, the document in the image to be tested may have one or more such straight edge lines.
[0053] Furthermore, in step 230, a corrected document image is generated based on the document area enclosed by the edge lines. Generally speaking, since documents in real scenes are usually rectangular, four edge lines can usually be generated in step 220, and the four edge lines are connected by corner key points to enclose the document area to be corrected. The corrected document image can be a horizontal and vertical document image, that is, the deformation of the document in the image to be tested is removed, so as to facilitate subsequent text recognition and other operations, or directly displayed to the user for user reading. In some embodiments, based on the edge lines and the document area, a corrected document image can be generated by perspective transformation.
[0054] In another embodiment of the present invention, step 230 may further include: performing gridding processing on the document area to generate a grid document image; and constructing a rectangular blank image, and filling the pixel value at each grid coordinate in the grid document image into the corresponding coordinate point in the rectangular blank image to generate a corrected document image. The gridding processing can be understood as setting a plurality of cross lines in the document area to divide the document area into a plurality of grids, so that the image including the plurality of grids is a grid document image.
[0055] The rectangular blank image can be a rectangular blank image or a square blank image, and its size can be set as needed. For each coordinate point on the rectangular blank image, the pixel value at the corresponding grid coordinate in the grid document image can be used to fill it, so that a horizontally and vertically corrected document image that is the same as the document content in the image to be tested can be obtained. When the document area is gridded, the denser the grid setting, the clearer the document content in the generated corrected document image, and the more uniform the distribution of each text in the document and the spacing between the texts are, which is conducive to improving the user's subjective perception and the accuracy of text recognition. For corrected document images that are directly generated without gridding, such as corrected document images generated by perspective transformation, although it is possible to flatten the curved document, there may be problems such as uneven distribution or uneven spacing of text in the flattened corrected document image, thereby affecting the user's visual perception and the accuracy of subsequent text recognition.
[0056] Combination of the above Figure 2 The method for correcting a document image according to an embodiment of the present invention is described exemplarily. It can be understood that the method according to the present invention can retain the curve features that may exist in the document edge by fitting the edge line according to the distribution trend of the edge key points, thereby facilitating the flattening of the curved document. It can also be understood that the above description is exemplary and not restrictive. For example, the curve fitting method may not be limited to the above-mentioned polynomial fitting function, and the curve fitting may also be achieved through techniques such as machine learning.
[0057] For example, the generation of the rectified document image based on the document area surrounded by the edge line may not be limited to perspective transformation and grid processing, but may also be generated by other existing or future achievable methods. However, compared with other methods, grid processing will be more conducive to achieving the effect of uniform distribution of text in the flattened rectified document image. Figure 3-Figure 5 A specific implementation method of grid processing is described in detail.
[0058] Figure 3 The flowchart of the method for gridding processing according to an embodiment of the present invention is schematically shown. Figure 2 The embodiment of step 230 described above is a specific form of gridding processing, so the above text combines Figure 2 The description of step 230 may also be applied to the following description of method 300 .
[0059] like Figure 3As shown in , method 300 may include: in step 310, the same number of multiple equally divided points may be determined on the two curved edge lines respectively. In some embodiments, the edge lines of the document may include two opposite curved edge lines and two opposite straight edge lines, that is, the two opposite curved edge lines are a set of opposite sides of the document, and the two opposite straight edge lines are another set of opposite sides of the document. In step 310, multiple equally divided points may be determined on each curved edge line, and the number of equally divided points determined on the two opposite edge lines may be the same. The equally divided points are points that can equally divide the length of the curved edge line. In some other embodiments, assuming that the width of the corrected document image is w d and height is h d , assuming that the position of the curve edge line corresponds to the width of the corrected document image, in order to obtain a dense grid, we can set w on each curve edge line d points to divide the length of each curve edge line into (w d -1) is divided equally, and we get (w d -1) equally divided curve segments.
[0060] Next, in step 320, the corresponding equally divided points on the two curved edge lines may be connected in the document region to form a plurality of straight lines between the two curved edge lines. Since the two curved edge lines have the same number of equally divided points, a plurality of straight lines that do not cross in the document region may be formed by sequentially connecting the corresponding equally divided points on the two curved edge lines.
[0061] Then, the process can proceed to step 330, and the same number of non-uniform points can be determined on each straight edge line and each straight line connection. Specifically, multiple non-uniform points can be set on each straight edge line, and multiple non-uniform points can be set on each straight line connection, and the number of non-uniform points set on each straight edge line and each straight line connection is equal. Non-uniform points are points that divide the length of a straight edge line or a straight line connection into non-uniform parts.
[0062] In other embodiments, the positions of the non-uniform points can be determined according to the shape of the document area. For example, when the document area is in a trapezoidal shape, the closer to the upper base of the trapezoid, the denser the non-uniform points can be set, and the more segmented line segments there are; the closer to the lower base of the trapezoid, the sparser the non-uniform points can be set, and the fewer segmented line segments there are. In some embodiments, multiple non-uniform points can be set in a manner such that the lengths of multiple line segments on the same straight line satisfy the law of geometric progression.
[0063] like Figure 3As further shown in , in some embodiments, step 330 may include: in step 331 (shown in a dotted box), the length ratio of two adjacent line segments between multiple unequal points on the same straight line may be calculated using perspective transformation. The same straight line may include the edge line of the same straight line or the connecting line of the same straight line. In another embodiment of the present invention, calculating the length ratio using perspective transformation may include: taking the four corner points where the edge lines of two straight lines are located as source points and the four corner points of any rectangle as target points, calculating the perspective transformation matrix; taking the adjacent first source line segment and the second source line segment of equal length on any straight edge line, and using the perspective transformation matrix to perform perspective transformation on the coordinates of the separation point and the two endpoint coordinates between the first source line segment and the second source line segment, respectively, to obtain the corresponding first transformed line segment and the second transformed line segment; and determining the length ratio according to the ratio of the length of the first transformed line segment to the length of the second transformed line segment.
[0064] Specifically, assuming that the coordinates of the four corner points where the two straight edge lines are located are marked as P0, P1, P2, and P3 in a clockwise direction starting from the upper left corner, and the four corresponding corner points of any horizontal and vertical rectangle are used as target points, the perspective transformation matrix can be calculated. The process of calculating the perspective transformation matrix can be implemented by, for example, the getPerspectiveTransform function of the OpenCV method.
[0065] Next, taking the straight edge line P0P3 as an example, the midpoint of the straight edge line P0P3 is marked as P lc For the convenience of explanation, the corner points P0 and P3 are the two endpoints of the first source line segment and the second source line segment, and P lc is the dividing point between the first source line segment and the second source line segment, the first source line segment can be represented as P0P lc , the second source segment can be represented as P lc Then, P0, P3, P lc Complement them to form homogeneous vectors (like ), and then multiply each homogeneous vector by the perspective transformation matrix to obtain the same vector as P0, P3, P lc The corresponding transformed coordinates are P′0, P′3, P′ lc , which can be obtained with the first source line segment P0P lc The corresponding first transformed line segment P′0P′ lc , and get the second source line segment P lc The second transformed line segment P' corresponding to P3 lc P′3. The perspective transformation process can be implemented by using, for example, the warpPerspective function of the OpenCV method.
[0066] Further, according to the first transformed line segment P′0P′ lcand the second transformed line segment P′ lc The ratio of the lengths of P′3 can determine the length ratio of two adjacent line segments between the unequally divided points, for example, calculated using the following formula 1:
[0067]
[0068] Where k represents the length ratio, P′0P′ lc represents the first transformed line segment, P′ lc P′3 represents the second transformed line segment. It can be understood that the above description is exemplary rather than restrictive. For example, the two endpoints of the first source line segment and the second source line segment are not limited to being two corner points on the straight edge line, and the separation point is not limited to being the midpoint of the straight edge line. The two endpoints can be selected from any two points on the straight edge line as needed, as long as the separation point is set at the center of the two endpoints. For another example, the first source line segment and the second source line segment are not limited to being selected from the straight edge line P0P3, but can also be selected from the straight edge line P1P2 as needed, or can also be selected from any straight line.
[0069] Then, after determining the length ratio, the process can proceed to step 332 (shown in the dotted box), and the length of each line segment on each straight edge line and each straight line connection can be determined based on the length of each straight edge line and each straight line connection, the length ratio and the number of non-equally divided points on the same straight line to determine the position of each straight edge line and each non-equally divided point on each straight line. The number of non-equally divided points on the same straight line can be set as needed. In some embodiments, the number of non-equally divided points on the same straight line can be determined based on the size of the corrected document image. For example, assuming that the position of the straight edge line corresponds to the height of the corrected document image, in order to obtain a dense grid, h can be set on each straight edge line and each straight line connection. d unequally divided points to divide the length of each straight line by (h d -1) is not equally divided, and we get (h d -1) non-uniformly divided straight line segments.
[0070] In some other embodiments, the length of the first line segment on the same straight line can be calculated using the geometric progression summation formula, and then the length of each line segment on the same straight line can be determined, so that the position of each non-uniform point on the corresponding straight line can be determined according to the length of each line segment. Taking the determination of the length of the first line segment on the straight edge line P0P3 as an example, the length of the first line segment can be calculated using the following formula 2:
[0071]
[0072] Among them, d can represent the length of the first line segment, k can represent the length ratio, and hd It can represent the height of the corrected document image, and P0P3 can represent the length of the straight line segment to be unequally divided (such as any straight line edge line or any straight line connection line). It can be understood that when the straight line edge line corresponds to the width of the corrected document image, h in Formula 2 is d Can be replaced by correcting the width of the document image.
[0073] Furthermore, if Figure 3 As shown in FIG. 3 , in step 340, the corresponding non-uniform points on the straight edge lines and the straight line lines can be connected to generate a grid document image. By setting the same number of non-uniform points on each straight edge line and each straight line line, and by sequentially connecting the corresponding non-uniform points on each straight edge line and the straight line line, a plurality of curved lines that do not cross in the document area can be formed. These curved lines and the straight line lines formed in step 320 together constitute the grid document image.
[0074] For ease of understanding, the following will be Figure 4 Take the image to be tested as an example, and combine it with Figure 5 The gridding process shown is explained. Figure 4 A schematic diagram schematically shows an image to be tested according to an embodiment of the present invention. Figure 5 Schematically shows the Figure 4 Schematic diagram of the process of generating a grid document image from the image to be tested shown in .
[0075] like Figure 4 As shown in FIG, the image to be tested 400 is an image containing a document 401, wherein the document 401 is an open book, and the paper near the spine of the book is curled. By detecting the image to be tested 400, a plurality of edge key points 402 of the document 401 can be obtained. Based on the plurality of edge key points 402, a plurality of edge lines of the document 401 can be fitted and generated.
[0076] like Figure 5 As shown in FIG. 4( a ), the plurality of edge key points 402 may include four corner points P0, P1, P2, and P3. The plurality of edge key points 402 are distributed on two opposite sides of the document 401 and are distributed in a curved line. Figure 5 As shown in FIG. 5( b ), according to the distribution trend of the plurality of edge points 402 , two relative curved edge lines 501 and 502 can be fitted and generated.
[0077] The process can then proceed to Figure 5 As shown in Figure (c), by Figure 5The corresponding corner points on the two curved edge lines 501 and 502 shown in FIG. (b) are connected, that is, the corner points P0 and P3 are connected to generate a straight edge line P0P3, and the corner points P1 and P2 are connected to generate a straight edge line P1P2, thereby forming a document area 503. Then, the midpoint P lc , and / or determine the midpoint P on the straight edge line P1P2 rc .
[0078] Furthermore, if Figure 5 As shown in Figure (d), the perspective transformation can be used to calculate the corner points P0, P1, P2, P3 and the midpoint P lc , P rc The corresponding transformed points are P′0, P′1, P′2, P′3, P′ lc ,P′ rc . Assume that the first source segment P0P is selected lc and the second source segment P lc P3, then P0 and P3 are the first source line segment P0P lc and the second source segment P lc The two endpoints of P3, the midpoint P lc is the first source segment P0P lc and the second source segment P lc The separation point of P3, after transformation, the first transformed line segment is P′0P′ lc and the second transformed line segment is P′ lc P′3. Assume that the first source line segment P1P is selected rc and the second source segment P rc P2, then P1 and P2 are the first source line segment P1P rc and the second source segment P rc The two endpoints of P2, the midpoint P rc is the first source line segment P1P rc and the second source segment P rc The separation point of P2, after transformation, the first transformed line segment is P′1P′ rc and the second transformed line segment is P′ rc P′2.
[0079] Take the first transformed line segment as P′1P′ rc and the second transformed line segment is P′ rc Taking P′2 as an example, the length ratio k can be calculated using the following formula 3:
[0080]
[0081] In another embodiment of the present invention, determining the position of each non-equally divided point may further include: determining the position of the first non-equally divided point on each straight edge line and each straight line connection line according to the relative position relationship between the first transformed line segment and the second transformed line segment. In some embodiments, the relative position relationship between the first non-equally divided point and the second non-equally divided point on each straight edge line and each straight line connection line may be the same as the relative position relationship between the first transformed line segment and the second transformed line segment. For example, the first transformed line segment is located above the second transformed line segment, and the first non-equally divided point may be located above the second non-equally divided point.
[0082] like Figure 5 As shown in Figure (d), the first transformed line segment P′1P′ rc and the second transformed line segment P′ rc The relative position relationship of P′2 is that the second transformed line segment P′ rc P′2 is located on the first transformed line segment P′1P′ rc In the positive direction of the y-axis, that is, the second transformed line segment is P′ rc P′2 is located on the first transformed line segment P′1P′ rc In the figure below, if Figure 5 The first non-uniform division point 505 on each straight edge line and each straight line connection line shown in FIG. (e) is the first position point for non-uniform division operation starting from the upper corner point P1 in the figure. After the first non-uniform division point 505 is determined, other non-uniform division points 504 can be determined in sequence according to the length ratio.
[0083] Furthermore, if Figure 5 As shown in FIG. (f), a grid document image 507 can be generated by connecting the corresponding equally divided points on two relative curved edge lines P0P1 and P3P2, and connecting each straight edge line (such as P0P3 and P1P2 in the figure) and the corresponding non-equally divided points on each straight line 506.
[0084] Combination of the above Figure 3-Figure 5 The gridding process according to the embodiment of the present invention is described in detail. It can be understood that what is shown in the figure is exemplary and not restrictive. For example, the equally divided points on the edge line of the curve may not be limited to Figure 5 The six points shown in Figure (f) are not limited to the straight edge lines and straight line connecting lines. Figure 5 The six non-uniformly divided points shown in FIG. (f) can be set as needed, and the number of equally divided points and the number of non-uniformly divided points will not be described here. In addition, since the curved edge line in the embodiment of the present invention is fitted according to the distribution trend of the edge key points, and the adjacent grid points in the generated grid document image are smoothly transitioned, it is conducive to outputting a uniform and smooth corrected document image.
[0085] Figure 6 The flowchart of the method for detecting using the key point detection model according to an embodiment of the present invention is schematically shown. Figure 2 A specific embodiment of the method 200 described above, therefore, Figure 2 The description of method 200 may also be applicable to the following description of method 600 .
[0086] like Figure 6 As shown in , method 600 may include: in step 610, edge key points of the document in the image to be tested may be detected. In some embodiments, step 610 may include step 611 (shown in a dotted box), in which a key point detection model may be used to detect the image to be tested to determine the edge key points of the document in the image to be tested. The key point detection model may adopt any existing or future achievable model structure. By inputting the image to be tested into the key point detection model, the edge key points of the document in the image to be tested may be output.
[0087] In another embodiment of the present invention, before using the key point detection model for detection, the method 600 may further include steps 640 to 660 to train the key point detection model before detecting the image to be detected. Figure 6 As shown in , in step 640 (shown by the dotted box), four sample corner points may be determined from the dense contour points of the document in the sample image, and the dense contour points may be separated into four sample edge point sets according to the four sample corner points. In some embodiments, the dense contour points may include a plurality of closely distributed points located on the edge of the document in the sample image, and the distance between each two adjacent points may be less than or equal to 1 pixel. The sample image may be derived from an image of a curved document collected in a real scene.
[0088] In some embodiments, the four corner points of the document in the sample image may be secondly marked, and the point in the dense contour points with the smallest distance from the second markup may be determined as the sample corner point. The second markup may be implemented by manual marking or machine marking. In some scenarios, a corner point of the second markup may coincide with a point in the dense contour points, that is, the distance between the coincident point and the second markup is zero, and the coincident point may be determined as the sample corner point. In other scenarios, the corner point of the second markup does not coincide with any point in the dense contour points, and the point with the smallest distance (e.g., Euclidean distance) from the second markup may be found in the dense contour points and determined as the sample corner point.
[0089] In other embodiments, performing the second annotation may include: performing Gaussian blur processing on the sample image, and then using, for example, the Shi-Tomasi algorithm or the Harris algorithm to perform corner point detection on the document in the sample image. After the initial corner points are detected, the initial corner points that are too close can be filtered out by limiting the corner point spacing and other rules, and then the four initial corner points farthest from the center point of the document are selected from the filtered initial corner points as the four corner points of the second annotation. In some scenarios, if there are less than four initial corner points after filtering, the sample image can be discarded.
[0090] In another embodiment of the present invention, determining four sample corner points in step 640 may also include: first marking contour points of the document in the sample image; second marking four corner points of the document in the sample image; converting the contour points of the first marking into dense contour points; and determining the points with the smallest distance from the second marking in the dense contour points as sample corner points. The first marking may be implemented by manual marking or machine marking. Compared with dense contour points, the contour points of the first marking are sparse. Converting the contour points of the first marking into dense contour points may be implemented, for example, by interpolation or conversion tools in OpenCV methods.
[0091] After four sample corner points are determined in the dense contour points, segmentation can be performed based on the four sample corner points so that the dense contour points are separated into four point sets respectively distributed on four sample edges (eg, top, bottom, left, and right) of the document.
[0092] Next, in step 650 (shown by a dotted line frame), a plurality of equally divided sample points of the curve sample edge may be determined from at least the point set of the curve sample edge among the four sample edges. For example, a plurality of equally divided sample points may be determined by performing an average division on the point set of the curve sample edge, for example, five equally divided sample points or six equally divided sample points of the curve sample edge may be determined.
[0093] Then, the process can proceed to step 660 (shown in the dotted box), and the key point detection model can be trained using a sample image with four sample corner points and multiple equally divided sample points. The trained key point detection model can detect the four corner key points in the document of the image to be tested and the equally divided key points on the curve edge line.
[0094] In one embodiment of the present invention, in step 650, the sample midpoint of the straight sample edge can also be determined from the point set of the straight sample edge in the four sample edges; and in step 660, the key point detection model can be trained using a sample image with four sample corner points, a plurality of equally divided sample points and a sample midpoint. The key point detection model trained using such training data can detect four corner key points, equally divided key points on the curve edge line and midpoint key points on the straight edge line in the document of the image to be tested.
[0095] By using the above-mentioned training data to train the key point detection model, supervised training of the key point detection model can be achieved, and the problem of difficulty and low accuracy in directly manually marking key points or equally dividing points on curves can be solved. The above-mentioned training data construction method according to an embodiment of the present invention also has the advantages of simple scheme, easy implementation and low labeling cost. Furthermore, in some embodiments, by adding the sample midpoints of the straight line sample edges to the training data, more comprehensive supervision information can be provided for the training of the key point detection model, and practice has shown that this can significantly improve the accuracy of key point detection.
[0096] like Figure 6 As further shown in FIG. 6 , after detecting the edge key points of the document, method 600 may continue to perform step 620 and step 630. In step 620, the edge line of the document may be fitted based on the distribution trend of the edge key points. In step 630, a corrected document image may be generated based on the document area surrounded by the edge line. Step 620 and step 630 have been described in conjunction with the above. Figure 2 Step 220 and step 230 are described in detail and will not be repeated here.
[0097] Through the above description of the solution for correcting document images in the embodiment of the present invention and its multiple embodiments, those skilled in the art can understand that the present invention generates the edge line of the document by fitting according to the distribution trend of the edge key points. For curved documents, a smooth curved edge line can be generated, thereby generating a smooth corrected document image, which has a better human subjective perception and is more conducive to the recognition and extraction of document content by technologies such as OCR. In some embodiments, by gridding the document area to generate a grid document image, the text in the corrected document image can be evenly distributed, which is conducive to subsequent operations such as text recognition and extraction.
[0098] In addition, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.
[0099] The use of the verbs "comprise", "include" and their conjugations mentioned in the application documents does not exclude the presence of elements or steps other than those recorded in the application documents. The article "a" or "an" before an element does not exclude the presence of a plurality of such elements.
[0100] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and this division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.
Claims
1. A method for correcting a document image, comprising: Detect edge key points of the document in the image to be tested; According to the distribution trend of the edge key points, the edge line of the document is generated by fitting; as well as generating a corrected document image based on the document area enclosed by the edge lines; The step of detecting edge key points of a document in an image to be tested comprises: Use the key point detection model to detect the image to be tested to determine the edge key points of the document in the image to be tested; Before using the key point detection model for detection, the method further includes: Determine four sample corner points from the dense contour points of the document in the sample image, and separate the dense contour points into four sample edge point sets according to the four sample corner points; Determine a plurality of equally divided sample points for the curve sample edge from at least a set of points of the curve sample edge among the four sample edges; and The key point detection model is trained using a sample image with four sample corner points and a plurality of equally divided sample points.
2. The method according to claim 1, wherein fitting to generate edge lines comprises: In response to the fact that a plurality of edge key points on the same side of the document are distributed in a curve, a polynomial fitting function is used to fit and generate a corresponding curve edge line.
3. The method according to claim 1 or 2, wherein generating a corrected document image based on the document area comprises: Performing gridding processing on the document area to generate a grid document image; as well as A rectangular blank image is constructed, and the pixel value at each grid coordinate in the grid document image is filled into the corresponding coordinate point in the rectangular blank image to generate the corrected document image.
4. The method according to claim 3, wherein the edge lines of the document include two opposite curved edge lines and two opposite straight edge lines, and Meshing a document area includes: Determine a plurality of equally divided points of the same number on the edge lines of the two curves; Connecting corresponding equally divided points on two curved edge lines in the document region to form a plurality of straight lines between the two curved edge lines; Determine the same number of unequally divided points on each straight edge line and each straight line; and The corresponding non-uniformly divided points on the straight edge lines and the straight line connecting lines are connected to generate the grid document image.
5. The method according to claim 4, wherein determining the same number of multiple non-uniformly divided points on each straight edge line and each straight connecting line comprises: Use perspective transformation to calculate the length ratio of two adjacent line segments between multiple unequal points on the same straight line; According to the length of each straight edge line and each straight line connection line, the length ratio and the number of non-equally divided points on the same straight line, the length of each straight edge line and each line segment on each straight line connection line is determined to determine the position of each non-equally divided point on each straight edge line and each straight line connection line.
6. The method according to claim 5, wherein calculating the length ratio using perspective transformation comprises: Take the four corner points of the two straight line edges as source points and the four corner points of any rectangle as target points to calculate the perspective transformation matrix; Take a first source line segment and a second source line segment that are adjacent and equal in length on any straight edge line, and use the perspective transformation matrix to perform perspective transformation on the coordinates of the separation point and the two endpoints between the first source line segment and the second source line segment, respectively, to obtain a corresponding first transformed line segment and a second transformed line segment; as well as The length ratio is determined according to a ratio of the lengths of the first transformed line segment to the lengths of the second transformed line segment.
7. The method according to claim 6, wherein determining the positions of the non-uniformly divided points further comprises: According to the relative position relationship between the first transformed line segment and the second transformed line segment, the position of each straight line edge line and the first non-uniformly divided point on each straight line connection line is determined.
8. The method according to claim 6 or 7, wherein The two endpoints are two corner points on the straight edge line; and The separation point is the midpoint of the straight edge line.
9. The method according to claim 1, further comprising: Determine the sample midpoint of the straight line sample edge in the point set of the four sample edges; as well as The key point detection model is trained using a sample image with four sample corner points, a plurality of equally divided sample points and a sample midpoint.
10. The method according to claim 1 or 9, wherein before determining the four sample corner points, it further comprises: Performing a first annotation on the contour points of the document in the sample image; Perform a second annotation on the four corner points of the document in the sample image; Convert the first annotated contour points into dense contour points; as well as The point with the smallest distance from the second annotation in the dense contour point is determined as the sample corner point.
11. A device for correcting a document image, comprising: a processor configured to execute program instructions; A memory configured to store the program instructions, which, when executed by the processor, causes the device to perform the method according to any one of claims 1 to 10.
12. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, causes the processor to execute the method according to any one of claims 1 to 10.
