Method and apparatus for recognizing content of pdf scan

Abstract

The application discloses a PDF scan content recognition method and device, which comprises the following steps: (1) converting a PDF scan into a picture; (2) rectifying the direction of the picture by using the variance change of the sum of horizontal pixel points; (3) performing layout analysis based on a preset first model to recognize a seal, a table, a directory title and a text in the picture; (4) removing the recognized seal in the picture by using a preset second model; (5) finding the outer contour of the table and the inner cell contour, positioning the cell position, extracting the table text and filling the table text into the corresponding cell to complete the table content recognition; and (6) generating a hierarchical directory structure by using a preset third model to recognize the directory title and the text, and completing the content recognition. The application has the advantages of faster speed and higher recognition accuracy.

Description

Technical Field

[0001] This invention relates to image processing technology, and more particularly to a method and apparatus for recognizing the content of PDF scanned documents. Background Technology

[0002] Optical Character Recognition (OCR) technology has made great strides over the past few decades. However, OCR technology is limited to image text recognition and cannot recognize the original structural content of a scanned document, such as headers, footers, tables, and figures. Therefore, content recognition in PDF scans remains an application area that urgently needs research and breakthroughs.

[0003] A complete PDF scan typically needs to consider the following layout elements: headers, footers, tables, figures, and text. In addition, formal PDF scans often include official seals, and a small number also contain handwritten text. Current layout analysis technology can generally distinguish headers, footers, tables, table titles, figures, figure titles, handwritten text, seals, and other layout elements. However, it has shortcomings in areas such as orientation correction, table extraction, seal removal, paragraph recognition, and document table of contents structure recognition, specifically as follows:

[0004] 1) Orientation Correction. Current orientation correction techniques mainly include two methods: traditional image processing methods and deep learning methods. The former primarily uses a rectangle to frame the image and then determines the rotation angle of the rectangle to infer the image's tilt angle. This method is suitable for images where the pixel shapes are roughly rectangular, but not for PDF scans that are mainly text-based. The latter, using deep learning technology, has significant advantages in orientation correction capabilities, but due to the large size of deep learning models, it requires substantial hardware resources and computation time.

[0005] 2) Table Extraction. Current table extraction technologies generally meet the requirements when the scanned document is clear and the table lines are not distorted. However, if the table is distorted and the scan clarity is low, erroneous extraction is likely to occur.

[0006] 3) Seal removal. Current OCR recognition technology often fails to consider the situation where text is covered by a seal, resulting in a sharp drop in recognition accuracy.

[0007] 4) Paragraph Recognition. Similar to stamp removal, current layout analysis techniques fail to consider paragraph information in text, resulting in OCR recognition being unable to output paragraph information. Paragraph information is crucial for some downstream document mining tasks.

[0008] 5) Document Directory Structure Recognition. Document directory structure recognition is helpful for downstream document mining tasks, such as extracting contract clauses from scanned contracts. If the document directory structure is not integrated, it will be difficult to extract relevant contract clauses because some clauses themselves do not contain key clause elements; only the clause titles contain clause element information. However, current PDF content recognition technologies generally do not consider document directory structure recognition, resulting in incomplete recognition content and inaccurate application. Summary of the Invention

[0009] Purpose of the invention: This invention addresses the problems existing in the prior art by providing a method and apparatus for recognizing the content of PDF scanned documents that is faster and has a higher recognition accuracy.

[0010] Technical solution: The PDF scan content recognition method of the present invention includes the following steps:

[0011] (1) Convert PDF scans to images;

[0012] (2) Correct the orientation of the image by utilizing the variance of the sum of horizontal pixels;

[0013] (3) Based on the preset first model, perform layout analysis to identify the seals, tables, table of contents titles and text in the image;

[0014] (4) Use the preset second model to remove the stamps identified in the image;

[0015] (5) By finding and locating the cell position through the outer contour and inner cell contour of the table, extract the text in the table and fill it into the corresponding cell to complete the table content recognition.

[0016] (6) The recognized directory titles and text are used to generate a hierarchical directory structure, thus completing the content recognition.

[0017] Furthermore, step (2) specifically includes:

[0018] (2.1) Process the image into a binary image;

[0019] (2.2) Determine whether the width of the binary image is less than its height. If not, rotate the binary image by 90 degrees and execute (2.3). If yes, execute (2.3) directly.

[0020] (2.3) Use OCR to determine whether the character can be recognized. If not, rotate 180 degrees and execute (2.4). If yes, execute (2.4) directly.

[0021] (2.4) Rotate the binary image using a rotation angle less than a preset threshold;

[0022] (2.5) Determine whether a horizontal straight line can be detected through the pixels of the binary image; if so, output the current binary image as the corrected image directly; otherwise, execute (2.6).

[0023] (2.6) Calculate the sum of the number of pixels in each horizontal line from top to bottom of the image, and calculate the variance of the horizontal pixels;

[0024] (2.7) Determine whether the variance of the horizontal pixels has increased compared to the last time. If yes, increase the rotation angle by the preset step size and return to execute (2.4). If no, execute (2.8).

[0025] (2.8) Accumulate the number of times the variance of the horizontal pixels does not increase, and determine whether the number of times has reached the threshold. If yes, execute (2.9); otherwise, increase the rotation angle according to the preset step size and return to execute (2.4).

[0026] (2.9) Output the binary image with the largest variance as the final corrected image.

[0027] Furthermore, in step (3), the first preset model is specifically a deep convolutional neural network model, which is obtained through training. The samples used during training are: several original PDF scans and annotated PDF scans with manual annotations of stamps, tables, table of contents titles and text.

[0028] Furthermore, in step (4), the second preset model is specifically an encoder-decoder architecture, which is obtained through training. The samples used during training are: several images containing seals and corresponding images without seals.

[0029] Furthermore, step (5) specifically includes:

[0030] (5.1) Extract the table area from the image obtained by page layout analysis;

[0031] (5.2) Perform binarization and pixel dilation on the captured table area image;

[0032] (5.3) Find the largest outline region of the image after processing in step (5.2) and delete the pixels outside the largest outline region;

[0033] (5.4) Obtain the connected contour regions in the image after processing in step (5.3), and determine the top contour with the most sub-contours and the corresponding sub-contours based on these connected contours.

[0034] (5.5) Determine the cell position of the sub-outline based on its position and corresponding positional relationship;

[0035] (5.6) Use OCR to extract the text content within the sub-contours and fill the text content into the corresponding cell positions to complete the table content recognition.

[0036] Furthermore, the third preset model in step (6) is specifically a deep learning model, which is trained to generate a hierarchical directory structure containing document titles, hierarchical headings, and text elements from the identified directory titles and body text through a sequence labeling task. The deep learning model is specifically a CNN+BILSTM+CRF model, which simultaneously extracts character representations and word embedding representations from each sentence to ultimately generate the elements of the hierarchical directory structure. The character features are obtained through a character representation extraction model, and the word embedding features are obtained through preprocessing and real-time generation. The character representation extraction model is specifically a CNN network including several convolutional layers and one max-pooling layer. The preprocessing steps for the word embedding features include: a. using jieba segmentation to construct a primary vocabulary based on the training corpus; b. expanding the primary vocabulary based on the pre-trained word vector vocabulary and storing the expanded vocabulary and corresponding word vectors; the training corpus is manually annotated document directory-level data, and the pre-trained word vectors are publicly available word vectors trained based on Baidu Encyclopedia; the real-time generation steps include: a. using jieba segmentation to segment sentences from the training data obtained from the annotated corpus in a batch; b. finding the corresponding word vector representation for each segmented word based on the stored vocabulary and converting it into that word vector.

[0037] The PDF scan content recognition device of the present invention includes a processor and a computer program stored in a memory and executable on the processor. When the processor executes the program, it implements the above-described method.

[0038] Beneficial effects: Compared with the prior art, the significant advantages of this invention are: it is faster, has higher recognition accuracy, and can better serve downstream tasks of analyzing and mining PDF scanned documents. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating an embodiment of the PDF scan content recognition method provided by the present invention;

[0040] Figure 2 This is a flowchart of the image correction process of the present invention;

[0041] Figure 3 This is a structural diagram of the second preset model for removing the seal according to the present invention;

[0042] Figure 4 This is a flowchart of the table extraction process of this invention;

[0043] Figure 5 This is a flowchart illustrating the generation process of the hierarchical directory structure of this invention;

[0044] Figure 6 This is a structural diagram of the third preset model of the present invention;

[0045] Figure 7 This is a structural diagram of the character representation extraction model;

[0046] Figure 8 This is a schematic diagram of an embodiment of the PDF scan content recognition device provided by the present invention. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Example 1

[0049] This embodiment provides a method for recognizing the content of a PDF scanned document, such as... Figure 1 As shown, it includes the following steps:

[0050] (1) Convert PDF scans to images. One page of PDF corresponds to one image; convert all PDF scans into an image set.

[0051] (2) Use the variance of the sum of horizontal pixels to correct the orientation of the image.

[0052] In specific implementation, such as Figure 2 As shown, the specific steps for direction correction include:

[0053] (2.1) Process the image into a binary image;

[0054] (2.2) Determine whether the width of the binary image is less than its height. If not, rotate the binary image by 90 degrees and execute (2.3). If yes, execute (2.3) directly.

[0055] (2.3) Use OCR to determine whether the character can be recognized. If not, rotate 180 degrees and execute (2.4). If yes, execute (2.4) directly.

[0056] (2.4) Rotate the binary image using a rotation angle less than a preset threshold;

[0057] (2.5) Determine whether a horizontal straight line can be detected through the pixels of the binary image; if so, output the current binary image as the corrected image directly; otherwise, execute (2.6).

[0058] (2.6) Calculate the sum of the number of pixels in each horizontal line from top to bottom of the image, and calculate the variance of the horizontal pixels;

[0059] (2.7) Determine whether the variance of the horizontal pixels has increased compared to the last time. If yes, increase the rotation angle by the preset step size and return to execute (2.4). If no, execute (2.8).

[0060] (2.8) Accumulate the number of times the variance of the horizontal pixels does not increase, and determine whether the number of times has reached the threshold. If yes, execute (2.9); otherwise, increase the rotation angle according to the preset step size and return to execute (2.4).

[0061] (2.9) Output the binary image with the largest variance as the final corrected image.

[0062] The preset threshold for the rotation angle can be set to ±3 degrees, the step size can be set to 0.1 degrees, and the cumulative number of times the variance of the horizontal pixels does not increase can be set to 7.

[0063] (3) Based on the preset first model, perform layout analysis to identify the seals, tables, table titles and text in the image.

[0064] Specifically, the first preset model is a deep convolutional neural network model, obtained through training. For example, it can be a deep convolutional network model formed by using YOLOv3 as the base model and replacing the original DarkNet-53 backbone with ResNet50-vd. The samples used for training this model are: several original PDF scans, and annotated PDF scans of the original PDF scans after manual annotation of stamps, tables, table of contents titles, and body text (which may also include headers, footers, paragraph text, figures, figure titles, handwritten text, etc.). The training method is a commonly used technique in this field and will not be elaborated further.

[0065] (4) Use the preset second model to remove the stamps identified in the image.

[0066] The second preset model is specifically an encoder-decoder architecture, implemented using the self-attention Transformer framework, such as... Figure 3 As shown, the training process uses image pairs consisting of images containing stamps and corresponding images without stamps. The stamps are manually labeled, and the image pairs are identical except for the stamps. A second pre-defined model is trained using these image pairs, with the images containing stamps as input and the images without stamps as output. Then, the trained second pre-defined model predicts the corresponding stamp-free images for the images containing stamps. Finally, based on the stamp location analyzed in the previous step, the image region containing the stamp is obtained, and the stamp-free image predicted by the second pre-defined model replaces this region, thus achieving stamp removal.

[0067] (5) By finding and locating the cell position through the outer contour and inner cell contour of the table, extract the text in the table and fill it into the corresponding cell to complete the table content recognition.

[0068] In specific implementation, such as Figure 4 As shown, the specific steps for table content recognition include:

[0069] (5.1) Extract the table area from the image obtained by page layout analysis;

[0070] (5.2) Perform binarization and pixel dilation on the captured table area image;

[0071] (5.3) Use the OpenCV library to find the largest contour region of the image after processing in step (5.2) and delete the pixels outside the largest contour region;

[0072] (5.4) Use the OpenCV library to obtain the connected contour regions in the image after processing in step (5.3). Based on these connected contours, determine the top contour with the most sub-contours and the corresponding sub-contours.

[0073] (5.5) Determine the cell position of the sub-outline based on its position and corresponding positional relationship;

[0074] (5.6) Use OCR to extract the text content within the sub-contours and fill the text content into the corresponding cell positions to complete the table content recognition.

[0075] (6) The recognized directory titles and text are used to generate a hierarchical directory structure, thus completing the content recognition.

[0076] The third preset model is specifically a deep learning model, obtained through training. The generation method is as follows: Figure 5 As shown, based on the text content identified in the previous step, a deep learning model is used to determine the document's table of contents structure: document title, first-level headings, second-level headings, third-level headings, body text, and fourth-level and above headings (referred to as negative examples in model training).

[0077] like Figure 6 As shown, in practical implementation, the deep learning model can be set as a CNN+BILSTM+CRF model, and the intelligent document directory structure recognition task can be modeled as a sequence labeling task, simultaneously extracting character representations and word embedding representations from each sentence. The character features are obtained through a character representation extraction model, and the word embedding features are obtained through preprocessing and real-time generation. Figure 7As shown, the character representation extraction model is specifically a CNN network comprising several convolutional layers and one max-pooling layer. The preprocessing steps for the word embedding features include: a. using jieba segmentation to construct a primary vocabulary based on the training corpus; b. expanding the primary vocabulary based on the pre-trained word vector vocabulary and storing the expanded vocabulary and corresponding word vectors; the training corpus is manually annotated document directory-level data, and the pre-trained word vectors are publicly available word vectors trained based on Baidu Encyclopedia; the real-time generation steps include: a. segmenting sentences from a batch of training data obtained from the annotated corpus using jieba segmentation; b. finding the corresponding word vector representation for each segmented word based on the stored vocabulary and converting it into that word vector. To avoid the problem of excessively long sentences, a sentence is treated as a "word," and the maximum length of the "word" is limited (i.e., the number of characters in a single line of a sentence). This approach avoids the difficulty of representing long sentences, allowing the label of the sentence to be predicted directly from the partial content representation of the long sentence. By combining the input original text with the truncated text, the problem of incomplete original text during prediction can be avoided. Figure 6 In this code, LSTM stands for Long Short-Term Memory Neural Network, and CRF stands for Conditional Random Field Neural Network. The CRY tag represents the main text, the T tag represents the document title, the T1 tag represents a first-level heading, the T2 tag represents a second-level heading, the T3 tag represents a third-level heading, and the O tag represents a negative example (headings at level four and above are treated as negative examples). Finally, the code outputs the corresponding text content, figures, figure titles, tables, and table titles identified from the PDF scan, according to the document's table of contents hierarchy. The results of this PDF scan content recognition can be provided to downstream document mining tasks via HTTP.

[0078] Example 2

[0079] Figure 8 This is a schematic diagram of the structure of a device provided by an embodiment of the present invention. The embodiment of the present invention provides services for the implementation of the method in the above embodiment one of the present invention, and can be configured with the computing device in the above embodiment one. Figure 8 A block diagram of an exemplary device 12 suitable for implementing embodiments of the present invention is shown. Figure 8 The device 12 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0080] like Figure 8 As shown, device 12 is represented as a general-purpose computing device. Components of device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and a bus 18 connecting different system components, including system memory 28 and processing unit 16.

[0081] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0082] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0083] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 8 Not shown; usually referred to as a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0084] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.

[0085] Device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with device 12, and / or with any device that enables device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 8 As shown, network adapter 20 communicates with other modules of device 12 via bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0086] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the method provided in Embodiment 1 of the present invention.

[0087] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method of PDF scan content recognition, characterized by The method comprises the following steps: (1) converting a PDF scan into a picture; (2) rectifying the direction of the picture by using the variance of the sum of horizontal pixels; (3) performing layout analysis based on a preset first model to identify a seal, a table, a directory title and a text in the picture; (4) removing the identified seal in the picture by using a preset second model; (5) finding and positioning a cell position by searching for an outer contour of the table and an inner cell contour, extracting table text and filling the table text into the corresponding cell to complete table content recognition; (6) generating a hierarchical directory structure by using a preset third model to generate the identified directory title and text, and completing content recognition; Step (2) specifically comprises: (2.1) processing the picture into a binary image; (2.2) judging whether the width of the binary image is less than the height, if not, rotating the binary image by 90 degrees and executing (2.3), if yes, directly executing (2.3); (2.3) judging whether the character can be recognized by using OCR, if not, rotating the binary image by 180 degrees and executing (2.4), if yes, directly executing (2.4); (2.4) rotating the binary image by a rotation angle less than a preset threshold; (2.5) judging whether a horizontal straight line can be detected by using the binary image pixels; if yes, directly outputting the current binary image as a rectified picture; otherwise, executing (2.6); (2.6) calculating the sum of the number of horizontal pixels from top to bottom of the picture, and calculating the horizontal pixel variance; (2.7) judging whether the horizontal pixel variance is larger than the last time, if yes, increasing the rotation angle by a preset step and returning to execute (2.4), if not, executing (2.8); (2.8) accumulating the number of times that the horizontal pixel variance does not increase, judging whether the number of times reaches a threshold, if yes, executing (2.9), if not, increasing the rotation angle by a preset step and returning to execute (2.4); (2.9) outputting the binary image with the largest variance as the final rectified picture.

2. The method of claim 1, wherein: The preset first model in step (3) is a deep convolutional neural network model, which is obtained by training, and the samples used in the training are: a plurality of original PDF scans and a plurality of labeled PDF scans obtained by manually labeling seals, tables, directory titles and texts in the original PDF scans.

3. The method of claim 1, wherein: The preset second model in step (4) is an encoder-decoder architecture, which is obtained by training, and the samples used in the training are: a plurality of pictures containing seals and a plurality of corresponding pictures without seals.

4. The method of claim 1, wherein: Step (5) specifically comprises: (5.1) cutting the area of the table in the picture identified by the layout analysis; (5.2) performing binaryzation and pixel dilation processing on the cut table area picture; (5.3) finding the maximum contour area of the picture processed in step (5.2) and deleting the pixels outside the maximum contour area; (5.4) obtaining the connected contour areas in the picture processed in step (5.3), and judging the top contour with the most sub-contours and the corresponding sub-contours according to the connected contours; (5.5) judging the cell position where the sub-contour is located according to the position of the sub-contour and the corresponding positional relationship; (5.6) The text content in the sub-contour is extracted by using OCR, and the text content is filled into the corresponding cell position to complete the table content recognition.

5. The method of claim 1, wherein: The preset third model in step (6) is a deep learning model obtained by training, and the recognized directory title and body are generated into a hierarchical directory structure containing a document title, a hierarchical title and a body element through a sequence labeling task.

6. The method of claim 5, wherein: The deep learning model is specifically a CNN+BILSTM+CRF model, which simultaneously extracts character features and word embedding features from each sentence to finally generate elements of the hierarchical directory structure. The character features are obtained by a character representation extraction model, and the word embedding features are obtained by preprocessing and real-time generation.

7. The method of claim 6, wherein: The character representation extraction model is specifically a CNN network including a plurality of convolution layers and a layer of maximum pooling layers.

8. The method of claim 6, wherein: The preprocessing step of the word embedding features includes: a. using jieba to segment and construct a primary word table according to the training corpus; b. expanding the primary word table according to the pre-trained word vector word table, and storing the expanded word table and the corresponding word vector; the training corpus is manually annotated document directory hierarchical data, and the pre-trained word vector is a word vector trained based on Baidu Encyclopedia on the Internet. The real-time generation step includes: a. using jieba to segment the sentence in a batch of training data processed from the annotated corpus; b. finding the corresponding word vector representation according to the stored word table, and converting it into the word vector.

9. A PDF scan content recognition apparatus comprising a processor and a computer program stored on a memory and executable on the processor, characterized in that: The processor implements the method of any one of claims 1-8 when executing the program.

Images