Document image extraction method and apparatus, and storage medium and electronic device

Abstract

The present application relates to the technical field of image processing. Disclosed are a document image extraction method and apparatus, and a storage medium and an electronic device. The method comprises: performing detection on an image to be processed, so as to obtain a document area and vertex coordinates; detecting recommended document edges; on the basis of the vertex coordinates, calculating center point coordinates; detecting straight lines in the document area; and on the basis of the straight lines, the center point coordinates and the recommended document edges, correcting the vertex coordinates. The present application can improve an image effect of an extracted document image.

Description

Document image extraction methods, apparatus, storage media and electronic devices

[0001] This application claims priority to Chinese Patent Application No. 202410805249.3, filed on June 20, 2024, entitled “Document Image Extraction Method, Apparatus, Storage Medium and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, specifically to a document image extraction method, apparatus, storage medium, and electronic device. Background Technology

[0003] Document image extraction is a task that extracts and outputs document images containing text from an image to be processed. Currently, among the relevant technologies for document image extraction, there are schemes that use neural networks to detect document images and output document images based on the detection results. Technical issues

[0004] Since the vertices of a document region typically do not possess obvious statistical characteristics, the method of fitting and regressing vertex coordinates using a neural network is difficult to achieve the required accuracy, resulting in poor image quality of the output document image. Technical solutions

[0005] This application provides a document image extraction scheme that can effectively improve the image quality of document images extracted from images to be processed.

[0006] The embodiments of this application provide the following technical solutions:

[0007] According to one embodiment of this application, a document image extraction method includes: performing target detection on an image to be processed to obtain a document region corresponding to a document in the image to be processed and vertex coordinates of recommended document vertices; performing edge detection on the document region to obtain recommended document edges of the document; calculating the center point coordinates of the document based on the vertex coordinates of the recommended document vertices; performing line detection on the document region to obtain straight lines in the document region; and correcting the vertex coordinates based on the straight lines, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates, wherein the corrected vertex coordinates are used to output a document image.

[0008] In some embodiments of this application, the step of correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates includes: performing similarity analysis on the straight line in the document region and each of the recommended document edges to obtain straight lines similar to each of the recommended document edges; filtering the straight lines similar to each of the recommended document edges to obtain corrected lines corresponding to each of the recommended document edges; and correcting the vertex coordinates according to the intersection coordinates of the intersection points of each of the corrected lines to obtain corrected vertex coordinates.

[0009] In some embodiments of this application, the step of filtering the straight lines similar to the edges of each recommended document to obtain the correction lines corresponding to the edges of each recommended document includes: calculating the sum of the distances between the straight lines similar to the edges of each recommended document and the vertices of the recommended documents corresponding to the edges of each recommended document; filtering the straight lines similar to the edges of each recommended document based on the sum of the distances corresponding to each straight line to obtain the correction lines corresponding to the edges of each recommended document.

[0010] In some embodiments of this application, the step of filtering from the straight lines with similar edges of each recommended document based on the sum of the distances corresponding to each straight line to obtain the correction line corresponding to the edge of each recommended document includes: filtering from the straight lines with similar edges of each recommended document the straight line corresponding to the smallest sum of distances as the correction line corresponding to the edge of each recommended document.

[0011] In some embodiments of this application, after correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain corrected vertex coordinates, the method further includes: extracting the coordinate region formed by the corrected vertex coordinates from the document region to obtain an initial image; and outputting the document image based on the initial image.

[0012] In some embodiments of this application, the step of outputting the document image based on the initial image includes: determining the corresponding output size based on the initial image; and performing a perspective transformation on the initial image based on the corrected vertex coordinates and the output size to obtain the document image.

[0013] In some embodiments of this application, the step of performing edge detection on the document region to obtain the recommended document edges of the document includes: performing noise reduction processing on the document region to obtain a noise-reduced region; and performing edge detection on the noise-reduced region to obtain the recommended document edges of the document.

[0014] In some embodiments of this application, the step of performing object detection on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertex includes: using an object detection network to detect whether the image to be processed contains a document; if so, using the object detection network to mark the document region corresponding to the document in the image to be processed; and using the object detection network to perform document vertex localization on the image to be processed to obtain the vertex coordinates of the recommended document vertex.

[0015] In some embodiments of this application, the target detection network is trained as follows: multiple training images containing sample documents are acquired, and standard document regions and standard document vertices of the sample documents are labeled in each training image; the target detection network to be trained is used to perform document recognition, document region labeling, and vertex localization on each training image to obtain the predicted document regions and predicted document vertices corresponding to each training image; the target detection network to be trained is adjusted according to the predicted document regions and predicted document vertices corresponding to each training image and the standard document regions and standard document vertices until the end of training conditions are met, thereby obtaining the trained target detection network.

[0016] According to one embodiment of this application, a document image extraction apparatus includes: a target detection module for performing target detection on an image to be processed, obtaining a document region corresponding to a document in the image to be processed and vertex coordinates of recommended document vertices; an edge detection module for performing edge detection on the document region, obtaining recommended document edges of the document; a coordinate calculation module for calculating the center point coordinates of the document based on the vertex coordinates of the recommended document vertices; a line detection module for performing line detection on the document region, obtaining lines in the document region; and a coordinate correction module for correcting the vertex coordinates based on the lines, the center point coordinates, and the recommended document edges, obtaining corrected vertex coordinates, wherein the corrected vertex coordinates are used to output a document image.

[0017] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the methods described in the embodiments of this application.

[0018] According to another embodiment of this application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0019] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in the embodiments of this application. Beneficial effects

[0020] In this embodiment, target detection is performed on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices; edge detection is performed on the document region to obtain the recommended document edges; the center point coordinates of the document are calculated based on the vertex coordinates of the recommended document vertices; line detection is performed on the document region to obtain the lines in the document region; the vertex coordinates are corrected based on the lines, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates, which are used to output the document image.

[0021] In this manner, the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices are obtained through object detection. Then, the edges of the recommended document are obtained through edge detection, and the center point coordinates are calculated based on the vertex coordinates of the recommended document vertices. Contained straight lines are detected. By correcting the vertex coordinates of the recommended document vertices based on the straight lines, center point coordinates, and recommended document edges, highly accurate corrected vertex coordinates can be obtained. Based on the corrected vertex coordinates, a document image with very good image quality can be output. Therefore, the method of this application can effectively improve the image quality of the document image extracted from the image to be processed. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 shows a flowchart of a document image extraction method according to an embodiment of this application.

[0024] Figure 2 shows a flowchart of target detection according to an embodiment of this application.

[0025] Figure 3 shows a flowchart of edge detection according to an embodiment of this application.

[0026] Figure 4 shows a block diagram of a document image extraction apparatus according to an embodiment of this application.

[0027] Figure 5 shows a block diagram of an electronic device according to an embodiment of this application.

[0028] Implementation methods of this application

[0029] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0030] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0031] For example, the document image extraction method provided in this disclosure includes a series of steps, but the document image extraction method provided in this disclosure is not limited to the steps described. Similarly, the document image extraction device provided in this disclosure includes a series of units, but the device provided in this disclosure is not limited to the units explicitly described, and may also include units that need to be set up for obtaining relevant information or processing based on information.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0033] Figure 1 schematically illustrates a flowchart of a document image extraction method according to an embodiment of this application. The subject executing this document image extraction method can be any device or server with processing capabilities, such as a television, computer, mobile phone, smartwatch, and home appliance, etc., and a server such as a physical server, cloud server, etc.

[0034] As shown in Figure 1, the document image extraction method may include steps S110 to S150.

[0035] Step S110: Perform target detection on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices.

[0036] Step S120: Perform edge detection on the document region to obtain the recommended document edges of the document;

[0037] Step S130: Calculate the coordinates of the center point of the document based on the vertex coordinates of the recommended document vertices.

[0038] Step S140: Perform line detection on the document region to obtain the straight lines in the document region.

[0039] Step S150: Based on the straight line, the center point coordinates, and the recommended document edge, the vertex coordinates are corrected to obtain corrected vertex coordinates, which are used to output the document image.

[0040] When performing object detection on the image to be processed, if the image contains text (i.e., a document), the document region where the document is located and the vertex coordinates of the recommended document vertices in the image can be detected.

[0041] The document region is the area containing the document, while the document image is typically the smallest region containing the image, which is usually located within the document region. The recommended document vertices are the vertices recommended by the document image obtained through object detection. The vertices of the document region are usually different from the recommended document vertices. However, in some special cases, when the document region obtained by object detection is the same as the final required document image, the vertices of the document region may be the same as the recommended document vertices.

[0042] Edge detection of a document region yields recommended document edges in the image to be processed. Recommended document edges are the edges of the recommended document image. For example, there may be four recommended document edges: top, bottom, left, and right edges. Edge detection can be performed using edge detection models or by connecting pixels.

[0043] Based on the geometric principles of center points, the center point coordinates of a document can be calculated from the vertex coordinates of the recommended document vertices. These center point coordinates can also be understood as the recommended center point coordinates. For example, if there are four recommended document vertices, connecting two opposite recommended document vertices creates two intersecting lines; the intersection of these two lines can be used as the center point.

[0044] Line detection in a document region can detect multiple straight lines within that region, such as the bottom and top edges of a line of text. These lines can be in any direction.

[0045] The aforementioned straight lines, center point coordinates, and recommended document edges can further reflect the characteristics of the document from multiple perspectives. By correcting the vertex coordinates of the recommended document vertices based on the straight lines, center point coordinates, and recommended document edges, we can obtain corrected vertex coordinates with higher accuracy than before the correction.

[0046] Based on the corrected vertex coordinates (which may include the corrected vertex coordinates of the top-left recommended document vertex, bottom-left recommended document vertex, top-right recommended document vertex, and bottom-right recommended document vertex), an initial image containing the document can be accurately acquired from the image to be processed. Based on this initial image, a document image with good document presentation effect can be further output.

[0047] In this manner, based on steps S110 to S150, the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices are obtained through object detection. Then, the edges of the recommended document are obtained through edge detection, and the center point coordinates are calculated based on the vertex coordinates of the recommended document vertices. Contained straight lines are detected. By correcting the vertex coordinates of the recommended document vertices based on the straight lines, center point coordinates, and recommended document edges, highly accurate corrected vertex coordinates can be obtained. Based on the corrected vertex coordinates, a document image with very good image quality can be output. This effectively improves the image quality of the document image extracted from the image to be processed.

[0048] The following describes further optional embodiments of the steps performed during document image extraction in the embodiment shown in Figure 1.

[0049] In one embodiment, referring to FIG2, the step of performing target detection on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices may include:

[0050] Step S210: Use an object detection network to detect whether the image to be processed contains a document; Step S220: If so, use the object detection network to mark the document region corresponding to the document in the image to be processed; Step S230: Use the object detection network to locate the document vertices in the image to be processed to obtain the vertex coordinates of the recommended document vertices.

[0051] Typically, object detection networks only perform object detection, meaning they usually only detect and output document regions. In this embodiment of the application, the output of the object detection network is extended; the extended output is the vertex coordinates of recommended document vertices. By performing document recognition, document region detection, and document vertex localization using the object detection network, the network can output the detected document regions and the vertex coordinates of recommended document vertices.

[0052] As shown in the background, the statistical characteristics of document vertex features are usually not obvious. Although the vertex coordinates of the recommended document vertices output by the object detection network may not be entirely accurate, they are still closer to the final target coordinates than other existing methods. In subsequent steps, the vertex coordinates of the recommended document vertices can be corrected to obtain more accurate corrected vertex coordinates.

[0053] Furthermore, the object detection network can be implemented using a smaller YOLO neural network, or any neural network model that meets the requirements.

[0054] Furthermore, the target detection network can be specifically trained in the following manner:

[0055] Multiple training images containing sample documents are acquired, and the standard document regions and standard document vertices of the sample documents in each training image are labeled. The target detection network to be trained is used to perform document recognition, document region labeling, and vertex localization on each training image to obtain the predicted document regions and predicted document vertices corresponding to each training image. The target detection network to be trained is adjusted according to the predicted document regions and predicted document vertices corresponding to each training image and the standard document regions and standard document vertices until the end of training conditions are met, and the trained target detection network is obtained.

[0056] After selecting the target detection network to be trained, multiple training images containing sample documents are collected. The standard document regions and standard document vertices of the sample documents in each training image are labeled. The standard document regions and standard document vertices are the labels.

[0057] By using the target detection network to be trained, document recognition, document region labeling, and vertex localization are performed on each training image, the "predicted document region and predicted document vertex corresponding to each training image" output by the target detection network to be trained can be obtained.

[0058] Based on the "predicted document region and predicted document vertex" and "standard document region and standard document vertex" corresponding to each training image, the loss can be calculated using a predetermined loss function. The network parameters of the target detection network to be trained can be adjusted according to the loss until the end of training conditions are met (e.g., learning reaches a predetermined number of times or the detection accuracy reaches a predetermined requirement). The trained target detection network can then be used for document recognition, document region labeling, and vertex localization.

[0059] In one embodiment, the step of performing edge detection on the document region to obtain the recommended document edges of the document may include: directly performing edge detection on the document region to obtain the recommended document edges of the document.

[0060] Furthermore, in one embodiment, referring to FIG3, the step of performing edge detection on the document region to obtain the recommended document edges of the document may include: step S310, performing noise reduction processing on the document region to obtain a noise-reduced region; step S320, performing edge detection on the noise-reduced region to obtain the recommended document edges of the document.

[0061] The document region can be denoised by filtering, blurring and other methods to obtain a denoised region. Further edge detection of the denoised region can yield more accurate recommended document edges.

[0062] In one embodiment, correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates may include:

[0063] The straight lines in the document region are compared with the edges of each of the recommended documents to obtain straight lines similar to the edges of each of the recommended documents; the straight lines similar to the edges of each of the recommended documents are filtered to obtain correction lines corresponding to the edges of each of the recommended documents; the vertex coordinates are corrected according to the intersection coordinates of the intersection points of each of the correction lines to obtain corrected vertex coordinates.

[0064] By performing similarity analysis between the straight lines in the document area and the edges of each recommended document, lines similar to the edges of each recommended document are obtained. For example, a document area typically includes multiple straight lines. If the top edge 1, bottom edge 2, left edge 3, and right edge 4 of a recommended document can each be a side of a rectangle, then performing similarity analysis between each of these straight lines in the document area and the edges of each recommended document will yield lines similar to edge 1, edge 2, edge 3, and edge 4.

[0065] Furthermore, by filtering the lines similar to the edges of each recommended document, correction lines corresponding to the edges of each recommended document can be obtained. For example, from the lines similar to edge 1 of recommended document, a line can be selected as correction line 1 corresponding to edge 1; from the lines similar to edge 2 of recommended document, a line can be selected as correction line 2 corresponding to edge 2, and so on.

[0066] Furthermore, based on the intersection coordinates of each correction line, the vertex coordinates are corrected to obtain the corrected vertex coordinates. For example, if four correction lines 1, 2, 3, and 4 are obtained, then these four lines can have four intersection points (top left, bottom left, top right, and bottom right); the recommended document vertices can also be four (top left, bottom left, top right, and bottom right). Based on the intersection coordinates, the vertex coordinates of recommended document vertices in the same location can be corrected to obtain the corrected vertex coordinates. For example, the vertex coordinates of the top left recommended document vertex can be corrected based on the intersection coordinates of the top left intersection point to obtain the corrected vertex coordinates of the top left recommended document vertex.

[0067] Based on the intersection coordinates of each intersection point, the vertex coordinates of recommended document vertices in the same orientation can be corrected. The corrected vertex coordinates can be obtained by using the intersection coordinates of each intersection point as the corrected vertex coordinates of recommended document vertices in the same orientation. For example, the intersection coordinates of the top left corner intersection point can be used as the corrected vertex coordinates of the top left corner recommended document vertex.

[0068] The coordinates of the recommended document vertices in the same location can be corrected based on the coordinates of the intersection points. The corrected vertex coordinates can be obtained by: calculating the average of the coordinates of the intersection points and the vertex coordinates of the recommended document vertices in the same location, and using the average as the corrected vertex coordinates of the recommended document vertices. For example, the average of the coordinates of the intersection point at the top left corner and the vertex coordinates of the top left corner recommended document vertex can be calculated and used as the corrected vertex coordinates of the top left corner recommended document vertex.

[0069] In one embodiment, performing similarity analysis between straight lines in the document region and the edges of each of the recommended documents to obtain straight lines similar to the edges of each of the recommended documents may include: detecting straight lines from the straight lines in the document region whose angle with the edges of each of the recommended documents is less than a predetermined degree, to obtain angularly similar straight lines corresponding to the edges of each of the recommended documents; and determining straight lines from the angularly similar straight lines corresponding to the edges of each of the recommended documents whose first distance and second distance are less than a predetermined distance, to obtain straight lines similar to the edges of each of the recommended documents, wherein the first distance is the distance between the edge of the recommended document and the center point, and the second distance is the distance between the angularly similar straight line and the center point.

[0070] From the straight lines in the document area, detect the straight lines whose angle with the edge of each recommended document is less than a predetermined degree, and obtain the angle-similar straight lines corresponding to the edge of each recommended document. For example, if the predetermined degree is 5 degrees, if the angle between straight line 1 and straight line 2 and edge 1 of recommended document is less than or equal to 5 degrees, then straight line 1, straight line 2, etc. are angle-similar straight lines corresponding to edge 1 of recommended document. Similarly, the angle-similar straight lines corresponding to the edges of other recommended documents can be obtained.

[0071] Furthermore, from the angularly similar lines corresponding to the edges of each recommended document, lines whose first and second distances are less than a predetermined distance are identified, thus obtaining lines similar to the edges of each recommended document. For example, if the predetermined distance is 2, the second distance between line 1 and the center point is 30, the second distance between line 2 and the center point is 25, and the first distance between recommended document edge 1 and the center point is 29, then line 1 is a line similar to recommended document edge 1. Similarly, lines similar to the edges of other recommended documents can be obtained. Specifically, this distance can be the spatial distance between pixels (e.g., the Euclidean distance between pixels).

[0072] In one embodiment, the step of filtering straight lines similar to the edges of each of the recommended documents to obtain correction lines corresponding to the edges of each of the recommended documents may include:

[0073] Calculate the sum of the distances between the lines that are similar to the edges of each of the recommended documents and the vertices of the recommended documents corresponding to the edges of each of the recommended documents; from the lines that are similar to the edges of each of the recommended documents, filter them according to the sum of the distances corresponding to each line to obtain the correction lines corresponding to the edges of each of the recommended documents.

[0074] Calculate the sum of the distances between lines similar to the edges of recommended documents and the corresponding vertices of recommended documents. For example, for the top-side recommended document edge 1, if the corresponding vertices of recommended document edge 1 can include the top-left and top-right recommended document vertices, and if lines similar to recommended document edge 1 include line 1, line 3, etc., first, calculate the distance 1-1 between line 1 and the top-left recommended document vertex, and calculate the distance 1-2 between line 1 and the top-right recommended document vertex. Then, the sum of the distances between line 1 (similar to recommended document edge 1) and the corresponding vertices of recommended document edge 1 is equal to the sum of distance 1-1 and distance 1-2. Similarly, the sum of the distances between line 3 (similar to recommended document edge 1) and the corresponding vertices of recommended document edge 1 is equal to the sum of distance 3-1 and distance 3-2.

[0075] From the lines with similar edges of the recommended documents, a correction line is obtained for each recommended document edge by filtering based on the sum of the distances between the corresponding lines. For example, from the lines with similar edges of recommended document 1, a line is selected as the correction line for recommended document edge 1 by filtering based on the sum of the distances between the corresponding lines.

[0076] Furthermore, in one embodiment, the step of filtering from the straight lines with similar edges of each of the recommended documents based on the sum of the distances corresponding to each straight line to obtain the correction line corresponding to the edge of each of the recommended documents may include: filtering from the straight lines with similar edges of each of the recommended documents the straight line corresponding to the smallest sum of distances, and using it as the correction line corresponding to the edge of each of the recommended documents.

[0077] For example, among lines 1 and 3 that are similar to edge 1 of the recommended document, the minimum sum of distances is the sum of distances corresponding to line 1, which is "the sum of distance 1-1 and distance 1-2". Therefore, line 1 is used as the correction line corresponding to edge 1 of the recommended document. Similarly, correction lines corresponding to other edges of the recommended documents can be selected.

[0078] In one embodiment, after correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain corrected vertex coordinates, the method further includes: extracting the coordinate region formed by the corrected vertex coordinates from the document region to obtain an initial image; and outputting the document image based on the initial image.

[0079] The corrected vertex coordinates can include the corrected vertex coordinates of the top-left, bottom-left, top-right, and bottom-right recommended document vertices. The coordinate region formed by connecting the corrected vertex coordinates of the top-left, bottom-left, top-right, and bottom-right recommended document vertices is the smallest document region where the document is located. The initial image is obtained by first viewing the image of this coordinate region.

[0080] Furthermore, the final document image can be output based on the initial image. For example, in some embodiments, the initial image can be output as the document image.

[0081] In one embodiment, outputting the document image based on the initial image includes: determining the corresponding output size based on the initial image; and performing a perspective transformation on the initial image based on the corrected vertex coordinates and the output size to obtain the document image.

[0082] The aspect ratio of the initial image can be calculated based on the corrected vertex coordinates. Furthermore, the matching output size can be retrieved from the predicted size table; this output size is the size of the document image. Then, a perspective transformation is performed based on the corrected vertex coordinates and the output size to obtain a document image with a better display effect. For example, a perspective transformation matrix can be calculated based on the corrected vertex coordinates and the output size, and then this matrix can be used to perform a perspective transformation on the initial image to obtain the desired document image.

[0083] To facilitate better implementation of the document image extraction method provided in the embodiments of this application, this application also provides a document image extraction device based on the above-described document image extraction method. The meanings of the terms used are the same as in the document image extraction method described above, and specific implementation details can be found in the descriptions in the method embodiments. Figure 4 shows a block diagram of a document image extraction device according to an embodiment of this application.

[0084] As shown in Figure 4, the document image extraction device 400 may include: a target detection module 410, which can be used to perform target detection on the image to be processed, and obtain the document region corresponding to the document and the vertex coordinates of the recommended document vertex in the image to be processed; an edge detection module 420, which can be used to perform edge detection on the document region, and obtain the recommended document edge of the document; a coordinate calculation module 430, which can be used to calculate the center point coordinate of the document based on the vertex coordinate of the recommended document vertex; a line detection module 440, which can be used to perform line detection on the document region, and obtain the line in the document region; and a coordinate correction module 450, which can be used to correct the vertex coordinates based on the line, the center point coordinate, and the recommended document edge, and obtain corrected vertex coordinates, which are used to output the document image.

[0085] In some embodiments of this application, the coordinate correction module 450 can be used to: perform similarity analysis between the straight lines in the document area and the edges of each of the recommended documents to obtain straight lines similar to the edges of each of the recommended documents; perform filtering processing on the straight lines similar to the edges of each of the recommended documents to obtain correction lines corresponding to the edges of each of the recommended documents; and correct the vertex coordinates according to the intersection coordinates of the intersection points of each of the correction lines to obtain corrected vertex coordinates.

[0086] In some embodiments of this application, the coordinate correction module 450 can be used to: calculate the sum of distances between straight lines similar to the edges of each recommended document and the vertices of the recommended documents corresponding to the edges of each recommended document; and filter from the straight lines similar to the edges of each recommended document according to the sum of distances corresponding to each straight line to obtain the correction lines corresponding to the edges of each recommended document.

[0087] In some embodiments of this application, the coordinate correction module 450 can be used to: select the line corresponding to the smallest sum of distances from the lines with similar edges of each of the recommended documents, and use it as the correction line corresponding to the edge of each of the recommended documents.

[0088] In some embodiments of this application, after correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain corrected vertex coordinates, the device further includes an image output module, configured to: extract the coordinate region formed by the corrected vertex coordinates from the document region to obtain an initial image; and output the document image based on the initial image.

[0089] In some embodiments of this application, the image output module is configured to: determine the corresponding output size based on the initial image; and perform perspective transformation on the initial image based on the corrected vertex coordinates and the output size to obtain the document image.

[0090] In some embodiments of this application, the edge detection module is used to: perform noise reduction processing on the document region to obtain a noise-reduced region; and perform edge detection on the noise-reduced region to obtain the recommended document edges of the document.

[0091] In some embodiments of this application, the object detection module is configured to: use an object detection network to detect whether the image to be processed contains a document; if so, use the object detection network to mark the document region corresponding to the document in the image to be processed; use the object detection network to perform document vertex localization on the image to be processed to obtain the vertex coordinates of the recommended document vertex.

[0092] In some embodiments of this application, the apparatus further includes a training module, configured to: acquire multiple training images containing sample documents, wherein each training image is labeled with a standard document region and a standard document vertex of the sample document; employ a target detection network to be trained to perform document recognition, document region labeling, and vertex localization on each training image to obtain a predicted document region and a predicted document vertex corresponding to each training image; and adjust the target detection network to be trained based on the predicted document region and predicted document vertex corresponding to each training image and the standard document region and standard document vertex, until the end of training conditions are met, thereby obtaining the trained target detection network.

[0093] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0094] Furthermore, this application also provides an electronic device, as shown in FIG5. FIG5 shows a block diagram of an electronic device according to an embodiment of this application, specifically:

[0095] The electronic device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that the electronic device structure shown in FIG. 5 does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0096] The processor 501 is the control center of the electronic device. It connects to various parts of the computer device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 502, and by calling data stored in the memory 502, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and application programs, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0097] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0098] The electronic device also includes a power supply 503 that supplies power to various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0099] The electronic device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0100] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the electronic device loads the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application. For example, the processor 501 can perform the following steps:

[0101] The process involves: performing object detection on the image to be processed to obtain the document region corresponding to the document and the vertex coordinates of the recommended document vertices in the image to be processed; performing edge detection on the document region to obtain the recommended document edges; calculating the center point coordinates of the document based on the vertex coordinates of the recommended document vertices; performing line detection on the document region to obtain the lines in the document region; and correcting the vertex coordinates based on the lines, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates, which are used to output the document image.

[0102] In some embodiments of this application, the step of correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates includes: performing similarity analysis on the straight line in the document region and each of the recommended document edges to obtain straight lines similar to each of the recommended document edges; filtering the straight lines similar to each of the recommended document edges to obtain corrected lines corresponding to each of the recommended document edges; and correcting the vertex coordinates according to the intersection coordinates of the intersection points of each of the corrected lines to obtain corrected vertex coordinates.

[0103] In some embodiments of this application, the step of filtering the straight lines similar to the edges of each recommended document to obtain the correction lines corresponding to the edges of each recommended document includes: calculating the sum of the distances between the straight lines similar to the edges of each recommended document and the vertices of the recommended documents corresponding to the edges of each recommended document; filtering the straight lines similar to the edges of each recommended document based on the sum of the distances corresponding to each straight line to obtain the correction lines corresponding to the edges of each recommended document.

[0104] In some embodiments of this application, the step of filtering from the straight lines with similar edges of each recommended document based on the sum of the distances corresponding to each straight line to obtain the correction line corresponding to the edge of each recommended document includes: filtering from the straight lines with similar edges of each recommended document the straight line corresponding to the smallest sum of distances as the correction line corresponding to the edge of each recommended document.

[0105] In some embodiments of this application, after correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain corrected vertex coordinates, the method further includes: extracting the coordinate region formed by the corrected vertex coordinates from the document region to obtain an initial image; and outputting the document image based on the initial image.

[0106] In some embodiments of this application, the step of outputting the document image based on the initial image includes: determining the corresponding output size based on the initial image; and performing a perspective transformation on the initial image based on the corrected vertex coordinates and the output size to obtain the document image.

[0107] In some embodiments of this application, the step of performing edge detection on the document region to obtain the recommended document edges of the document includes: performing noise reduction processing on the document region to obtain a noise-reduced region; and performing edge detection on the noise-reduced region to obtain the recommended document edges of the document.

[0108] In some embodiments of this application, the step of performing object detection on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertex includes: using an object detection network to detect whether the image to be processed contains a document; if so, using the object detection network to mark the document region corresponding to the document in the image to be processed; and using the object detection network to perform document vertex localization on the image to be processed to obtain the vertex coordinates of the recommended document vertex.

[0109] In some embodiments of this application, the target detection network is trained as follows: multiple training images containing sample documents are acquired, and standard document regions and standard document vertices of the sample documents are labeled in each training image; the target detection network to be trained is used to perform document recognition, document region labeling, and vertex localization on each training image to obtain the predicted document regions and predicted document vertices corresponding to each training image; the target detection network to be trained is adjusted according to the predicted document regions and predicted document vertices corresponding to each training image and the standard document regions and standard document vertices until the end of training conditions are met, thereby obtaining the trained target detection network.

[0110] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0111] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0112] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0113] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0114] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0115] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. A document image extraction method, wherein, include: Target detection is performed on the image to be processed to obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices; Edge detection is performed on the document region to obtain the recommended document edges; The coordinates of the center point of the document are calculated based on the vertex coordinates of the recommended document's vertices. Line detection is performed on the document region to obtain the straight lines in the document region; The vertex coordinates are corrected based on the straight line, the center point coordinates, and the edge of the recommended document to obtain corrected vertex coordinates, which are used to output the document image.

2. The method according to claim 1, wherein, The step of correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edges to obtain corrected vertex coordinates includes: The straight lines in the document region are compared with the edges of each of the recommended documents to obtain the straight lines that are similar to the edges of each of the recommended documents. The straight lines that are similar to the edges of each of the recommended documents are filtered to obtain the correction lines corresponding to the edges of each of the recommended documents. Based on the intersection coordinates of the intersection points of the aforementioned correction lines, the vertex coordinates are corrected to obtain the corrected vertex coordinates.

3. The method according to claim 2, wherein, The step of filtering straight lines similar to the edges of each of the recommended documents to obtain correction lines corresponding to the edges of each of the recommended documents includes: Calculate the sum of the distances between the lines that are similar to the edges of each of the recommended documents and the vertices of the recommended documents corresponding to the edges of each of the recommended documents; From the straight lines with similar edges of the recommended documents, the correction lines corresponding to the edges of the recommended documents are obtained by filtering according to the sum of the distances corresponding to the straight lines.

4. The method according to claim 3, wherein, The step of filtering from straight lines with similar edges of each of the recommended documents based on the sum of the distances corresponding to each straight line to obtain the correction line corresponding to the edge of each of the recommended documents includes: The line with the smallest sum of distances among the lines with similar edges of the recommended documents is selected as the correction line for the edges of the recommended documents.

5. The method according to claim 1, wherein, After correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain the corrected vertex coordinates, the method further includes: The coordinate region formed by the corrected vertex coordinates is extracted from the document region to obtain the initial image; The document image is output based on the initial image.

6. The method according to claim 5, wherein, The step of outputting the document image based on the initial image includes: Determine the corresponding output size based on the initial image; The initial image is subjected to perspective transformation based on the corrected vertex coordinates and the output size to obtain the document image.

7. The method according to claim 1, wherein, The step of performing edge detection on the document region to obtain the recommended document edges includes: The document region is subjected to noise reduction processing to obtain the noise-reduced region; Edge detection is performed on the noise reduction region to obtain the recommended document edges of the document.

8. The method according to claim 1, wherein, The process of performing object detection on the image to be processed, obtaining the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertices, includes: An object detection network is used to detect whether the image to be processed contains a document; If so, the target detection network is used to mark the document region corresponding to the document in the image to be processed; The object detection network is used to locate document vertices in the image to be processed, and the vertex coordinates of the recommended document vertices are obtained.

9. The method according to claim 8, wherein, The target detection network was trained in the following manner: Obtain multiple training images containing sample documents, and label the standard document regions and standard document vertices of the sample documents in each training image; A target detection network to be trained is used to perform document recognition, document region labeling and vertex localization on each training image to obtain the predicted document region and predicted document vertex corresponding to each training image. Based on the predicted document regions and vertices corresponding to each training image, and the standard document regions and vertices, the target detection network to be trained is adjusted until the end of training is met, thus obtaining the trained target detection network.

10. The method according to claim 2, wherein, The step of correcting the vertex coordinates based on the intersection coordinates of the intersection points of each of the correction lines to obtain the corrected vertex coordinates includes: The coordinates of each intersection point are used as the corrected vertex coordinates of the recommended document vertex located at the same position as each intersection point.

11. The method according to claim 2, wherein, The step of correcting the vertex coordinates based on the intersection coordinates of the intersection points of each of the correction lines to obtain the corrected vertex coordinates includes: Calculate the average of the intersection coordinates of each intersection point and the vertex coordinates of the recommended document vertices that are in the same orientation as each intersection point, and use the average value corresponding to each intersection point as the corrected vertex coordinates of the recommended document vertices that are in the same orientation as each intersection point.

12. The method according to claim 2, wherein, The step of performing similarity analysis between the straight lines in the document region and the edges of each of the recommended documents to obtain straight lines similar to the edges of each of the recommended documents includes: From the straight lines in the document area, detect the straight lines whose angle with the edge of each of the recommended documents is less than a predetermined degree, and obtain the angle-similar straight lines corresponding to the edge of each of the recommended documents; From the angle-similar lines corresponding to the edges of each of the recommended documents, determine the lines whose first distance and second distance are less than a predetermined distance, and obtain the lines that are similar to the edges of each of the recommended documents. The first distance is the distance between the edge of the recommended document and the center point, and the second distance is the distance between the angle-similar line and the center point.

13. The method according to claim 5, wherein, The step of outputting the document image based on the initial image includes: outputting the initial image as the document image.

14. A document image extraction device, wherein, include: The object detection module is used to perform object detection on the image to be processed, and obtain the document region corresponding to the document in the image to be processed and the vertex coordinates of the recommended document vertex. An edge detection module is used to perform edge detection on the document region to obtain the recommended document edges of the document; The coordinate calculation module is used to calculate the coordinates of the center point of the document based on the vertex coordinates of the recommended document's vertices; A line detection module is used to perform line detection on the document region to obtain the lines in the document region; The coordinate correction module is used to correct the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain corrected vertex coordinates, which are used to output the document image.

15. The apparatus according to claim 14, wherein, The coordinate correction module is used to: perform similarity analysis between the straight lines in the document area and the edges of each of the recommended documents to obtain straight lines similar to the edges of each of the recommended documents; filter the straight lines similar to the edges of each of the recommended documents to obtain correction lines corresponding to the edges of each of the recommended documents; and correct the vertex coordinates according to the intersection coordinates of the intersection points of each of the correction lines to obtain corrected vertex coordinates.

16. The apparatus according to claim 15, wherein, The coordinate correction module is used to: calculate the sum of the distances between the straight lines similar to the edges of each recommended document and the vertices of the recommended documents corresponding to the edges of each recommended document; and filter the straight lines similar to the edges of each recommended document according to the sum of the distances corresponding to each straight line to obtain the correction lines corresponding to the edges of each recommended document.

17. The apparatus according to claim 16, wherein, The coordinate correction module is used to: select the line with the smallest sum of distances from the lines with similar edges of each of the recommended documents, and use it as the correction line corresponding to the edge of each of the recommended documents.

18. The apparatus according to claim 14, wherein, After correcting the vertex coordinates based on the straight line, the center point coordinates, and the recommended document edge to obtain the corrected vertex coordinates, the device further includes an image output module, used to: extract the coordinate region formed by the corrected vertex coordinates from the document region to obtain an initial image; and output the document image based on the initial image.

19. A storage medium, wherein, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the method described in any one of claims 1 to 13.

20. An electronic device, wherein, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to perform the method described in any one of claims 1 to 13.

Images