PDF file analysis method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a PDF file analysis method and system, electronic equipment and a storage medium, and belongs to the field of document analysis. The method comprises the steps that a PDF file to be analyzed is read, the PDF file is recognized as a file in a markdown format, and a first file containing PDF content is obtained and recorded as markdown content; identifying the PDF file in an open source ocr mode to obtain a second file containing PDF content, and recording the second file as ocrcontent; according to the position of the label, performing character verification on the Markdown content, and restoring the verified file into a file in a Markdown format according to the position of the label; and extracting a title and content of the file. According to the method, the PDF file is subjected to format conversion, so that the file is easier to edit, and finally, the structure and format information in the original PDF file is reserved, so that the format of the converted Markdown file is closer to that of the original file.

Description

Technical Field

[0001] The present invention relates to the technical field of document parsing, and in particular to a PDF file parsing method, system, electronic device and storage medium. Background Art

[0002] PDF files have good format stability and content readability, and are widely used in file transmission, web page publishing and other fields.

[0003] However, most PDF files are generated based on scanned files for format compatibility and security reasons and are not editable. When you need to edit the file content, you often need to convert it into a document in other formats. The most common way is to convert it into a Word document. There are many applications for this method, such as Adobe Acrobat, Quick PDF Converter, etc.

[0004] Although the existing design can perform format conversion / content recognition on PDF files, the format of the converted files is relatively simple, almost all of which are converted to Word. Different types of files - such as Word, PPT, Excel pictures - may all generate PDF formats. If all PDF files are converted to Word format, on the one hand, it may cause inconsistent content and inconvenience in editing, and on the other hand, it may also cause content loss, segmentation errors, and text recognition errors. Considering the needs of editing, any of the above problems will make the conversion effect lose its value. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a PDF file parsing method, system, electronic device and storage medium, which are used to completely or at least partially solve the technical problems existing in the above-mentioned prior art.

[0006] In order to achieve the above object, an embodiment of the present invention provides a PDF file parsing method, comprising:

[0007] Read the PDF file to be parsed, and identify the PDF file to be parsed as a file in markdown format;

[0008] Record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content;

[0009] Use the open source OCR method to recognize the PDF file, obtain the second file containing the PDF content, and record it as ocr_content;

[0010] Use ocr_content to check markdown_content, and restore the text-checked file to a markdown file based on the position of the recorded markdown format tags;

[0011] Extract the title and content of the file after text verification and restore it to the markdown format to generate title-content pairs.

[0012] Optionally, before recording the position of the markdown format tag, obtaining the first file containing the PDF content, and recording it as markdown_content, the PDF file parsing method further includes:

[0013] Extract the content you want to extract from the markdown file and delete the special symbols and irrelevant content in the part.

[0014] Optionally, use ocr_content to perform text verification on markdown_content, and restore the text-verified file to a markdown format file according to the recorded markdown format tag positions, including:

[0015] According to the same rules, split the ocr_content document and markdown_content document into sentences;

[0016] For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, the character recognition is considered correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the character is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the OCR recognition result according to the position.

[0017] Optionally, extract the title and content of the file restored to the markdown format after the text verification, and generate the title-content pair, including:

[0018] The required title is obtained by using a python script to identify the position and number of characters representing the title in the markdown format, where different levels of title tags in the markdown format are represented by different numbers of title characters;

[0019] Determine the paragraphs after each title and obtain the content corresponding to each title;

[0020] Based on each title and the content corresponding to each title, a title-content pair is generated.

[0021] On the other hand, the present invention also provides a PDF file parsing system, comprising:

[0022] A reading unit, used for reading a PDF file to be parsed, and identifying the PDF file to be parsed as a file in markdown format;

[0023] The first recording unit is used to record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content;

[0024] The second recording unit is used to recognize the PDF file by using the open source OCR method to obtain a second file containing PDF content, and record it as ocr_content;

[0025] The verification unit is used to perform text verification on markdown_content using ocr_content, and restore the text-verified file to a file in markdown format according to the position of the recorded markdown format tag;

[0026] The extraction unit is used to extract the title and content of the file restored to the markdown format after the text verification, and generate a title-content pair.

[0027] Optionally, the verification unit is specifically used for:

[0028] According to the same rules, split the ocr_content document and markdown_content document into sentences;

[0029] For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, the character recognition is considered correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the character is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the OCR recognition result according to the position.

[0030] Optionally, the extraction unit is specifically used for:

[0031] The required title is obtained by using a python script to identify the position and number of characters representing the title in the markdown format, where different levels of title tags in the markdown format are represented by different numbers of title characters;

[0032] Determine the paragraphs after each title and obtain the content corresponding to each title;

[0033] Based on each title and the content corresponding to each title, a title-content pair is generated.

[0034] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned PDF file parsing method when executing the program.

[0035] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned PDF file parsing method when executed by a processor.

[0036] Through the above technical solution, PDF files are converted to Markdown format, making document content easier to edit, share and version control. The text in PDF files is recognized by OCR technology, which avoids garbled text or recognition errors caused by fonts, encoding and other issues in traditional PDF parsing, ensuring the accuracy of text content. In the process of verifying and restoring the text content recognized by OCR back to Markdown format, the structure and format information in the original PDF document is retained, making the converted Markdown file closer to the original document in format, and the generated title-content pairs make the document content easier to search, extract and reuse.

[0037] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings:

[0039] Figure 1 It is an implementation flow chart of a PDF file parsing method provided by an embodiment of the present invention;

[0040] Figure 2 The present invention is a schematic diagram of the structure of a PDF file parsing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The specific implementation of the embodiment of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the embodiment of the present invention, and is not used to limit the embodiment of the present invention.

[0042] See also Figure 1 The implementation flow chart of a PDF file parsing method provided by an embodiment of the present invention includes the following execution steps:

[0043] Step 100: Read the PDF file to be parsed, and identify the PDF file to be parsed as a file in markdown format.

[0044] Specifically, the PDF file is identified as a file in markdown format through open source models such as marker and pix2text, which is recorded as marker_markdown.

[0045] Step 101: Record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content.

[0046] In some implementations, before executing step 101, the PDF file parsing method process further includes: intercepting the portion of the markdown format file where the content to be extracted is located, and deleting special symbols and irrelevant content in the portion.

[0047] For example, some documents contain XML format tags, which are only for display convenience and do not contain specific content. In addition, some documents contain more content, and the content to be extracted is concentrated in the upper half of the document. In order to facilitate subsequent processing, the upper half of the document can be extracted separately in preprocessing. After processing, the positions of all markdown format tags are recorded. After recording, all markdown format tags are deleted to obtain a file containing only PDF content, which is recorded as markdown_content.

[0048] Step 102: Use the open source OCR method to identify the PDF file, obtain a second file containing PDF content, and record it as ocr_content.

[0049] Specifically, the PDF file is recognized using open source OCR technology or model to obtain another file containing the PDF content, which is recorded as ocr_content.

[0050] More specifically, use pdf2image or other tools to convert PDF files into image files (such as PNG, JPEG, etc.) to convert the text content in the PDF into an image format that can be processed by the OCR tool. During the conversion process, you can set the image resolution, color mode and other parameters as needed. Preprocess the converted image, such as binarization, denoising, and tilt correction, to improve the accuracy of OCR recognition. Call the OCR tool: Use the selected OCR tool to recognize the preprocessed image. You can call the OCR tool through the command line. The OCR tool will recognize the text in the image and return it as a string. Save the recognized text to a new file, which is ocr_content. Organize the recognized text, such as removing extra spaces and line breaks. If necessary, you can also format the text, such as adding paragraph marks and titles. Save the sorted text to the ocr_content file. You can choose to save it as a plain text file (.txt), Markdown file (.md) or other file formats. Compare the original PDF file and the ocr_content file to verify the accuracy of OCR recognition. Use text comparison tools or manual inspection to verify. Proofread and modify the recognition errors or omissions to ensure the accuracy of the ocr_content file.

[0051] Step 103: Use ocr_content to perform text verification on markdown_content, and restore the text-verified file to a file in markdown format according to the recorded positions of markdown format tags.

[0052] Specifically, when executing step 103, the following steps may be specifically performed:

[0053] S1030: According to the same rule, split the ocr_content document and the markdown_content document into sentences.

[0054] It should be noted that segmentation is performed according to the same rules, for example, according to symbols representing the end of a sentence, such as a period and a line break.

[0055] S1031: For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, it is considered that the character recognition is correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the character is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the recognition result of OCR according to the position.

[0056] Specifically, a prompt to ask the big model questions is constructed based on the sentence and the recognition results of the two methods. The construction method is: "If you are an excellent text recognition expert, please judge which of the following two recognition methods is correct based on the text provided. Text: the sentence, the nth character to be judged is replaced with a special symbol. Among them, the OCR recognition result of the nth character is: OCR recognition result, and the model recognition result is: model recognition result". Determine which recognition method is correct based on the answer of the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, modify the corresponding character to the OCR recognition result according to the position.

[0057] Step 104: extract the title and content of the file after text verification and restore it to the markdown format, and generate a title-content pair.

[0058] Specifically, when executing step 104, the following steps may be specifically performed:

[0059] S1040: The required title is obtained by identifying the position and number of characters representing the title in the markdown format through a python script, wherein title tags of different levels in the markdown format are represented by different numbers of title characters.

[0060] In some implementations, different levels of tags in the markdown format are marked with different numbers of # signs, for example, # indicates a first-level title, ## indicates a second-level title, and so on, up to a sixth-level title. The position and number of # signs can be identified by a python script to obtain the desired title, and the paragraph after the title is the content corresponding to the title.

[0061] S1041: Determine the paragraphs after each title and obtain the content corresponding to each title.

[0062] S1042: Generate a title-content pair based on each title and the content corresponding to each title.

[0063] Through the above technical solution, PDF files are converted to Markdown format, making document content easier to edit, share and version control. The text in PDF files is recognized by OCR technology, which avoids garbled text or recognition errors caused by fonts, encoding and other issues in traditional PDF parsing, ensuring the accuracy of text content. In the process of verifying and restoring the text content recognized by OCR back to Markdown format, the structure and format information in the original PDF document is retained, making the converted Markdown file closer to the original document in format, and the generated title-content pairs make the document content easier to search, extract and reuse.

[0064] See also Figure 2 FIG. 1 is a schematic diagram of a PDF file parsing system provided by an embodiment of the present invention, comprising:

[0065] The reading unit 20 is used to read the PDF file to be parsed and identify the PDF file to be parsed as a file in markdown format;

[0066] The first recording unit 21 is used to record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content;

[0067] The second recording unit 22 is used to recognize the PDF file by using the open source OCR method to obtain a second file containing PDF content, and record it as ocr_content;

[0068] The verification unit 23 is used to perform text verification on the markdown_content by using ocr_content, and restore the text-verified file to a file in the markdown format according to the position of the recorded markdown format tag;

[0069] The extraction unit 24 is used to extract the title and content of the file restored to the markdown format after the text verification, and generate a title-content pair.

[0070] In some implementations, the verification unit is specifically used to:

[0071] According to the same rules, split the ocr_content document and markdown_content document into sentences;

[0072] For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, it is considered that the character recognition is correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the word is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the OCR recognition result according to the position.

[0073] In some embodiments, the extraction unit is specifically used for:

[0074] The required title is obtained by using a python script to identify the position and number of characters representing the title in the markdown format, where different levels of title tags in the markdown format are represented by different numbers of title characters;

[0075] Determine the paragraphs after each title and obtain the content corresponding to each title;

[0076] Based on each title and the content corresponding to each title, a title-content pair is generated.

[0077] On the other hand, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the PDF file parsing method described in any one of the above embodiments when executing the program.

[0078] On the other hand, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the PDF file parsing method described in any one of the above embodiments are implemented.

[0079] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0080] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0085] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0087] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A PDF file parsing method, characterized in that: include: Read the PDF file to be parsed, and identify the PDF file to be parsed as a file in markdown format; Record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content; Use the open source OCR method to recognize the PDF file, obtain the second file containing the PDF content, and record it as ocr_content; Use ocr_content to check markdown_content, and restore the text-checked file to a markdown file based on the position of the recorded markdown format tags; Extract the title and content of the file after text verification and restore it to the markdown format to generate title-content pairs.

2. The PDF file parsing method according to claim 1, characterized in that: Before recording the position of the markdown format tag, obtaining the first file containing the PDF content, and recording it as markdown_content, the PDF file parsing method further includes: Extract the content you want to extract from the markdown file and delete the special symbols and irrelevant content in the part.

3. The PDF file parsing method according to claim 1, characterized in that: Use ocr_content to verify the markdown_content, and restore the verified file to a markdown file according to the position of the recorded markdown format tags, including: According to the same rules, split the ocr_content document and markdown_content document into sentences; For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, the character recognition is considered correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the character is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the OCR recognition result according to the position.

4. The PDF file parsing method according to claim 1, characterized in that: Extract the title and content of the file after text verification and restore it to the markdown format, and generate the title-content pair, including: The required title is obtained by using a python script to identify the position and number of characters representing the title in the markdown format, where different levels of title tags in the markdown format are represented by different numbers of title characters; Determine the paragraphs after each title and obtain the content corresponding to each title; Based on each title and the content corresponding to each title, a title-content pair is generated.

5. A PDF file parsing system, characterized in that: include: A reading unit, used for reading a PDF file to be parsed, and identifying the PDF file to be parsed as a file in markdown format; The first recording unit is used to record the position of the markdown format tag, obtain the first file containing the PDF content, and record it as markdown_content; The second recording unit is used to recognize the PDF file by using the open source OCR method to obtain a second file containing PDF content, and record it as ocr_content; The verification unit is used to perform text verification on markdown_content using ocr_content, and restore the text-verified file to a file in markdown format according to the position of the recorded markdown format tag; The extraction unit is used to extract the title and content of the file restored to the markdown format after the text verification, and generate a title-content pair.

6. The PDF file parsing system according to claim 5, characterized in that: The verification unit is specifically used for: According to the same rules, split the ocr_content document and markdown_content document into sentences; For each sentence, starting from the first word of each sentence, the characters recognized by OCR and the characters recognized by marker are compared in sequence. If the characters recognized by the two are the same, the character recognition is considered correct; if the characters recognized by OCR are different from the characters recognized by marker, the sentence containing the character is input into the big model, and the characters recognized by both are input into the big model. If the marker recognition is correct, continue to judge the next character. If the marker recognition is wrong, the corresponding character is modified to the OCR recognition result according to the position.

7. The PDF file parsing system according to claim 5, characterized in that: The extraction unit is specifically used for: The required title is obtained by using a python script to identify the position and number of characters representing the title in the markdown format, where different levels of title tags in the markdown format are represented by different numbers of title characters; Determine the paragraphs after each title and obtain the content corresponding to each title; Based on each title and the content corresponding to each title, a title-content pair is generated.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the PDF file parsing method according to any one of claims 1 to 4 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the PDF file parsing method according to any one of claims 1 to 4 are implemented.

Images