Multi-model based pdf resolution method, electronic device, and storage medium

Abstract

The application provides a multi-model-based PDF analysis method, an electronic device and a storage medium. The method comprises the following steps: acquiring all element types that may appear in a complex PDF by using a rule extraction method, a layout analysis method, a table identification method, a formula detection method, a formula recognition method, a multi-modal large model chart2text method and a multi-modal large model paragraph sorting method. The application can realize complete structured analysis of the complex PDF and support scanning and non-scanning.

Description

Technical Field

[0001] This invention relates to the field of document processing, and in particular to a multi-model-based PDF parsing method, electronic device, and storage medium. Background Technology

[0002] PDF (Portable Document Format) has become one of the standard formats for electronic documents. Its main advantage lies in its ability to effectively maintain document layout and formatting, ensuring consistent display across different operating systems and devices. With the rapid development of information technology, PDF files are increasingly used in various academic, commercial, and legal documents. Therefore, research and application of PDF parsing have gradually attracted the attention of academia and industry. The widespread use of PDF means that a large amount of important information is encapsulated in this format, mainly including text, images, tables, charts, formulas, etc. However, the structural complexity of PDF makes it difficult for traditional text extraction methods to efficiently extract all the information. Therefore, developing efficient PDF parsing technology has become necessary, which involves not only the extraction of text and images but also the reconstruction and semantic understanding of structured information. Currently, there are three main approaches to PDF parsing. The first is rule-based parsing. This type of method relies on specific rules and patterns in the PDF file structure for parsing. It extracts content by parsing the object structure of the PDF (such as text boxes, images, tables, etc.). This method has the advantages of simplicity and efficiency, but it can only effectively extract text and images from PDFs, and its accuracy is very low when extracting other types of data. The second is deep learning-based parsing. With the development of deep learning technology, deep learning-based PDF parsing has gradually gained attention. One approach involves training layout analysis models, table recognition, and OCR (Optical Character Recognition) models to identify the structure and content of documents. This method has advantages in image recognition and text classification, capable of acquiring different types of information and reconstructing structured information. However, some information is lost during the PDF-to-image conversion process, and due to insufficient model accuracy and robustness, some parsing errors occur, such as misspellings and omissions. A third approach is based on multimodal large-scale models. For PDF files with rich content, each page is converted into an image, and a multimodal model parses the images to obtain a comprehensive analysis of the page's information. While this method has good semantic understanding capabilities, the extracted information is incomplete when the input is a whole page image, and the output may have some visual illusions when the input is a complex layout. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] According to a first aspect of the present invention, a multi-model-based PDF parsing method is provided, the method comprising the following steps:

[0005] S100: Split the obtained preset PDF file into m single pages, and convert each single page into an image to obtain m page images; set the single page counter i = 1.

[0006] S200: If i ≤ m, determine whether the i-th single page is a scanned page based on the PDF file parsing tool. If yes, obtain the text and corresponding text bounding box in the i-th single page based on the OCR model, and execute S700. If not, use the PDF file parsing tool to extract the text and corresponding text bounding box, as well as the image and corresponding image bounding box in the i-th single page. If the image is extracted, execute S300; otherwise, execute S700. If i > m, execute S1000. The value of i is from 1 to m, and the initial value is 1.

[0007] S300: Based on the chart identification model, determine whether the extracted image of the i-th single page belongs to a chart image or a non-chart image. If it is a non-chart image, obtain the text and corresponding text bounding box in the image of the i-th single page based on the OCR model, and execute S700. If it is a chart image, obtain the natural language description in the image of the i-th single page based on the multimodal large model, and obtain the text and text bounding box of the image of the i-th single page; execute S700.

[0008] S400, input the i-th page image into the layout analysis model to extract all element categories and corresponding element category bounding boxes contained in the i-th page image; execute S500.

[0009] S500, if the table category is included in all element categories contained in the i-th page image, the table image contained in the i-th page image is identified using the improved table recognition model to obtain the corresponding table structure and the corresponding table structure bounding box; execute S700.

[0010] S600, input the i-th page image into the formula detection model. If a formula is detected, extract the formula image based on the bounding box of the detected formula, and use the formula recognition model to recognize the image of the detected formula to obtain the corresponding formula text, thus obtaining the detected formula text and the formula bounding box; execute S700.

[0011] S700: Based on the bounding boxes corresponding to all text descriptions and element category bounding boxes corresponding to all element categories in the i-th single page obtained from S100 to S600, classify all text descriptions in the i-th single page to obtain the category corresponding to each text description; execute S800.

[0012] S800, perform post-processing on the categorized text description of the i-th single page to obtain the post-processed text description of the i-th single page; wherein, the post-processing includes text deduplication, paragraph merging, and table text and table structure merging on the categorized text description; execute S900.

[0013] S900: Sort the paragraphs in the text description of the i-th single page after post-processing using a multimodal large model, and finally parse the i-th single page; set i = i + 1, and execute S200.

[0014] S1000, merge the parsing results of m single pages according to the page order to obtain the parsing result of the preset PDF file.

[0015] According to a second aspect of the present invention, an electronic device is provided, including a processor and a memory; the processor executes the steps of the method described in the first aspect of the present invention by invoking a program or instructions stored in the memory.

[0016] According to a second aspect of the present invention, a computer-readable storage medium is provided that stores a program or instructions that cause a computer to perform the steps of the method described in the first aspect of the present invention.

[0017] The present invention has at least the following beneficial effects:

[0018] The PDF parsing method provided in this invention can effectively parse complex PDF files containing tables, images, charts, formulas, at least two columns, and at least two languages.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating the multi-model-based PDF parsing method provided in this embodiment of the invention;

[0022] Figure 2 Here is an example chart image. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0025] It should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. A process can be terminated when its operation is complete, but it may also have additional steps not included in the figures. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0026] This invention provides a multi-model-based PDF parsing method, such as... Figure 1 As shown, the method may include the following steps:

[0027] S100: Split the obtained preset PDF file into m single pages, and convert each single page into an image to obtain m page images; set the single page counter i = 1. m > 1.

[0028] In this embodiment of the invention, the preset PDF file includes tables, images, charts, formulas, at least two columns of text, and at least two languages. The preset PDF file can be a file entered by the user.

[0029] S200: If i ≤ m, determine whether the i-th single page is a scanned page based on the PDF file parsing tool. If yes, obtain the text and corresponding text bounding box in the i-th single page based on the OCR model, and execute S700. If not, use the PDF file parsing tool to extract the text and corresponding text bounding box, as well as the image and corresponding image bounding box in the i-th single page. If an image is extracted, execute S300; otherwise, execute S700. The value of i ranges from 1 to m, and the initial value is 1. If i > m, execute S1000.

[0030] In this embodiment of the invention, the PDF parsing tool can be an existing PDF parsing tool, such as the pymupdf API. The PDF parsing tool will extract all text and images from the PDF completely and correctly based on the rules.

[0031] Because PDF parsing based on deep learning models is limited by model accuracy and robustness, errors can occur in the parsing results, such as misspellings and omissions. This invention uses rules to determine whether a page is a scanned page. Non-scanned pages do not use the OCR model to extract text. Only scanned pages and images within the page need to use the OCR model to extract text. This not only enables complete parsing of both scanned and non-scanned PDFs, but also improves the accuracy of text extraction for non-scanned documents and speeds up the text extraction process.

[0032] In this embodiment of the invention, the parsed bounding box can be represented by the coordinates of its four vertices. The vertex coordinates can be coordinates in the page coordinate system, where the origin of the page coordinate system is the top-left corner of the page, the x-axis is the horizontal direction of the page, and the y-axis is the vertical direction of the page.

[0033] In this embodiment of the invention, the text extracted by the PDF file parsing tool and the text extracted by the OCR model are processed line by line, that is, each line of text has a bounding box.

[0034] S300: Based on the chart identification model, determine whether the extracted image of the i-th single page belongs to a chart image or a non-chart image. If it is a non-chart image, obtain the text and corresponding text bounding box in the image of the i-th single page based on the OCR model, and execute S700. If it is a chart image, obtain the natural language description in the image of the i-th single page based on the multimodal large model, and obtain the text and text bounding box of the image of the i-th single page; execute S700.

[0035] In this embodiment of the invention, the chart identification model can be an existing chart identification model. The multimodal large model can be an existing multimodal large model, such as the chart2text model.

[0036] In this embodiment of the invention, the charts include stacked bar charts, complex line charts, scatter plots, pie charts, regular line charts, grouped bar charts, regular bar charts, three-dimensional bar charts, and hybrid charts, etc.

[0037] In this embodiment of the invention, a multimodal large language model can be instructed to obtain natural language descriptions from images via a predefined command statement. For example, the command statement could be "Please describe the content of this chart in detail, paying attention to completeness and accuracy," and the chart to be described could be... Figure 2As shown. Based on the received instructions and charts, the large language model will obtain a natural language description of the corresponding chart image. For example, it could be: "The chart shows the average math scores of the two classes. Specifically: - Class 2-1's average math score is 96.5. - Class 2-2's average math score is 98. The vertical axis of the chart represents the average math score, and the horizontal axis represents the class."

[0038] S400, input the i-th page image into the layout analysis model to extract all element categories and corresponding element category bounding boxes contained in the i-th page image; execute S500.

[0039] In this embodiment of the invention, the layout analysis model can be an existing layout analysis model. The element categories may include headings, body paragraphs, tables, table headings, images, image headings, charts, chart headings, formulas, headers and footers, footers, annotations, etc.

[0040] S500, if the table category is included in all element categories contained in the i-th page image, the table image contained in the i-th page image is identified using the improved table recognition model to obtain the corresponding table structure and the corresponding table structure bounding box; execute S700.

[0041] Existing table recognition models are trained on public datasets and cannot correctly recognize complex table images. To address this, this invention utilizes an improved table recognition model to improve table recognition performance.

[0042] In this embodiment of the invention, the improved table recognition model may include a conventional table recognition model and a text detection model. The conventional table recognition model includes a backbone network, a neck network, and a head network connected in sequence. The inputs to the backbone network and the text detection model are both input tables, and the input to the neck network is a concatenation result obtained by concatenating the outputs of the backbone network and the text detection model.

[0043] In this embodiment of the invention, the training dataset for the improved table recognition model includes a public dataset and a specified dataset. The public dataset can be, for example, the PubTabNet dataset, and the tables in the specified dataset include tables with merged cells, tables without borders or with incomplete borders, and tables whose structures do not meet the set table structure requirements. The set table structure requirements are requirements that characterize the table structure specifications and can be existing table structure requirements.

[0044] In this embodiment of the invention, since the concatenation result obtained by concatenating the output of the backbone network and the output of the text detection model is used as the input of the neck network, it can provide information on the text bounding boxes in the table image for the recognition of table bounding boxes, thereby improving the recognition accuracy of table bounding boxes.

[0045] In this embodiment of the invention, the text detection model can be an existing text detection model. During the training process of the improved table recognition model, the parameters of the text detection model do not participate in gradient backpropagation and parameter updates. During the training process of the improved table recognition model, a combination of cross-entropy loss and L1 loss can be used to train the model. The model's parameters can be updated using the Adam optimizer.

[0046] As will be known to those skilled in the art, the table structure can be an existing table structure.

[0047] The table structure obtained in S500 does not include the text in the table.

[0048] S600, input the i-th page image into the formula detection model. If a formula is detected, extract the formula image based on the bounding box of the detected formula, and use the formula recognition model to recognize the image of the detected formula to obtain the corresponding formula text, thus obtaining the detected formula text and the formula bounding box; execute S700.

[0049] In this embodiment of the invention, the formula detection model can be an existing formula detection model.

[0050] S700: Based on the bounding boxes corresponding to all text descriptions and element category bounding boxes corresponding to all element categories in the i-th single page obtained from S100 to S600, classify all text descriptions in the i-th single page to obtain the category corresponding to each text description; execute S800.

[0051] Furthermore, in S700, if (I ht / U ht If k ≥ k0, determine that the category described by the h-th text description is the category of the t-th element. ht U represents the number of pixels corresponding to the intersection of the bounding box of the h-th text description and the bounding box of the t-th element category. ht Let g(i) be the number of pixels corresponding to the union of the bounding box of the h-th text description and the bounding box of the t-th element category. The value of h is from 1 to g(i), and the value of t is from 1 to z(i). g(i) is the number of text descriptions contained in the i-th single page obtained based on S100 to S600, z(i) is the number of element categories contained in the i-th single page obtained based on S100 to S600, and k0 is a preset coefficient.

[0052] In this embodiment of the invention, the intersection of the bounding box of the h-th text description and the bounding box of the t-th element category is the set of coordinates formed by the common pixel coordinates among all pixel coordinates corresponding to the bounding box of the h-th text description and all pixel coordinates corresponding to the bounding box of the t-th element category. The union of the bounding box of the h-th text description and the bounding box of the t-th element category is the set of coordinates formed by combining all pixel coordinates corresponding to the bounding box of the h-th text description and all pixel coordinates corresponding to the bounding box of the t-th element category. In this embodiment of the invention, 0 < k0 < 1, which can be an empirical value, for example, k0 = 0.6.

[0053] S800, perform post-processing on the categorized text description of the i-th single page to obtain the post-processed text description of the i-th single page; wherein, the post-processing includes text deduplication, paragraph merging, and table text and table structure merging on the categorized text description; execute S900.

[0054] Furthermore, the S800 specifically includes:

[0055] S810, merge the text descriptions with the same bounding box in the classified text descriptions of the i-th single page to remove duplicate text descriptions, and use them as the first text description of the i-th single page.

[0056] S820, merge the paragraphs in the first text description of the i-th single page to obtain the first text description with merged paragraphs, which is used as the second text description of the i-th single page.

[0057] S830, fill the table text in the second text description of the i-th single page into the corresponding table structure to obtain the post-processed text description of the i-th single page.

[0058] Furthermore, the S820 may specifically include:

[0059] S821, sort all the bounding boxes of the text descriptions on the i-th single page according to the top-left corner coordinates from top to bottom and from left to right to obtain f(i) sorted bounding boxes; set the bounding box counter j = 1.

[0060] S822, if j≤f(i), obtain the overlap length IL of the j-th bounding box and the (j+1)-th bounding box in the x-axis direction of the page coordinate system. j(j+1) and merged length UL j(j+1) And the y-axis spacing d between the j-th bounding box and the (j+1)-th bounding box in the page coordinate system. j(j+1) If (IL) j(j+1) / UL j(j+1) )≥k, and d j(j+1)<c×△d, execute S823; otherwise, set j=j+1 and execute S822; k is a preset coefficient, c is a preset value greater than 1, △d=max(d j d j+1 ), where d j Let d be the height of the text within the j-th bounding box. j+1 Δd represents the height of the text within the (j+1)th bounding box, meaning Δd is the maximum height between the text heights within the j-th and (j+1)-th bounding boxes.

[0061] In this embodiment of the invention, c = 2, 0 < k < 1, which can be empirical values, such as k = 0.8.

[0062] In an embodiment of the present invention, IL j(j+1) The following condition must be met: If x j+1 start >x j end IL j(j+1) =Null, if x j+1 start ≤x j end IL j(j+1) =(x j end -x j+1 start ), Null represents empty, x j+1 start Let x be the x-coordinate of the top-left or bottom-left corner of the (j+1)th bounding box. j end It is the x-coordinate of the top-right or bottom-right corner of the j-th bounding box.

[0063] In an embodiment of the present invention, UL j(j+1) UL j(j+1) =(x j+1 end -x j end ), where x j+1 end It is the x-coordinate of the upper right or lower right corner of the (j+1)th bounding box.

[0064] In an embodiment of the present invention, d j(j+1) =(y j end -y j+1 start ), y j+1 start The y-coordinate of the (j+1)th bounding box is either the top-left or bottom-left corner. j endIt is the y-coordinate of the top-right or bottom-right corner of the j-th bounding box.

[0065] S823, if the last punctuation mark in the j-th bounding box is a marker indicating the end of a paragraph, merge the j-th bounding box and the (j+1)-th bounding box; otherwise, do not merge the j-th bounding box and the (j+1)-th bounding box; set j = j+1, and execute S822.

[0066] In this embodiment of the invention, the symbol indicating the end of a paragraph can be a suitable symbol for indicating the end of a paragraph, such as a period.

[0067] Furthermore, S821 specifically includes:

[0068] S8210: Obtain the top-left corner coordinates of the bounding boxes of all text descriptions on the i-th single page, obtain the top-left corner coordinate set, and sort the top-left corner coordinates in the top-left corner coordinate set in ascending order of y-coordinate values ​​to obtain the sorted top-left corner coordinate set.

[0069] S8211, Place the bounding box corresponding to the first top-left corner coordinate in the sorted top-left corner coordinate set at the corresponding position in the page coordinate system, and set the coordinate counter r = 2.

[0070] S8212, if r≤x(i), execute S8213, otherwise execute S8214, where x(i) is the number of coordinates in the sorted top-left corner coordinate set of the i-th single page.

[0071] S8213, If the y-coordinate value of the r-th top-left corner in the sorted top-left corner coordinate set is less than the y-coordinate value of the (r-1)-th top-left corner, in the page coordinate system, place the bounding box corresponding to the r-th top-left corner at the corresponding position below the bounding box corresponding to the (r-1)-th top-left corner. If the y-coordinate value of the r-th top-left corner in the sorted top-left corner coordinate set is equal to the y-coordinate value of the (r-1)-th top-left corner, in the page coordinate system, place the bounding box corresponding to the r-th top-left corner at the corresponding position to the right of the bounding box corresponding to the (r-1)-th top-left corner. Set r = r + 1, execute S8212; S8214, exit the current control program, and obtain the sorted f(i) bounding boxes.

[0072] S900: Sort the paragraphs in the text description of the i-th single page after post-processing using a multimodal large model, and finally parse the i-th single page; set i = i + 1, and execute S200.

[0073] In this embodiment of the invention, a multimodal large model can be instructed to sort the paragraphs in the text description of the i-th single page after post-processing by setting an instruction statement. For example, the instruction statement is "[parsed result (text segment 1, text segment 2...)] The above are the text segments parsed from the PDF. The sorting of the text segments above may be incorrect. Please read all the text segments above carefully and sort them correctly according to the semantics of the text. Please output the sorting result in the form of a list."

[0074] S1000, merge the parsing results of m single pages according to the page order to obtain the parsing result of the preset PDF file. The parsing result includes all categories of text paragraph information, image information, etc. of the PDF file, as well as the corresponding categories and bounding boxes of the text paragraph information and image information.

[0075] The multi-model-based PDF parsing method provided in this invention integrates the advantages of current mainstream parsing methods, including rule extraction, layout analysis, table recognition, formula detection, formula recognition, multimodal large model chart2text, and multimodal large model paragraph sorting. It can obtain all element types that may appear in complex PDFs, realize the complete structured parsing of complex PDFs, and support both scanned and non-scanned documents. In addition, in the parsing of non-scanned PDFs, it also has the advantages of comprehensive parsing types, accurate sorting, accurate and complete text, fast extraction speed, and enhanced chart parsing capabilities and optimized table recognition capabilities, making the recognition of complex tables more accurate.

[0076] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in this invention.

[0077] This invention also provides a computer-readable storage medium storing computer-executable instructions for performing the methods described in this invention.

[0078] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0079] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A multi-model based PDF resolving method, characterized in that, The method comprises the following steps: S100, split the obtained preset PDF file by page to obtain corresponding m single pages, and convert each single page into an image to obtain m page images; set a single page counter i = 1; S200, if i <= m, determine whether the i th single page belongs to a scanned page according to a PDF file parsing tool, if yes, obtain the text in the i th single page and the corresponding text bounding box based on an OCR model, and execute S700; if no, extract the text in the i th single page and the corresponding text bounding box and the image and the corresponding image bounding box by using the PDF file parsing tool, if the image is extracted, execute S300, otherwise, execute S700; if i > m, execute S1000; the value of i is 1 to m, and the initial value is 1; S300, determine whether the extracted image of the i th single page belongs to a chart image or a non-chart image based on a chart identification model, if it is a non-chart image, obtain the text in the image of the i th single page and the corresponding text bounding box based on an OCR model, and execute S700; if it is a chart image, obtain the natural language description in the image of the i th single page based on a multi-modal large model to obtain the text and the text bounding box of the image of the i th single page; execute S700; S400, input the i th page image into a layout analysis model to extract all element categories contained in the i th page image and the corresponding element category bounding box; execute S500; S500, if the i th page image contains a table category among all element categories contained in the i th page image, identify the table image contained in the i th page image by using an improved table recognition model to obtain the corresponding table structure and the corresponding table structure bounding box; execute S700; S600, input the i th page image into a formula detection model, if a formula is detected, intercept the formula image according to the detected formula bounding box, and identify the image of the detected formula by using a formula recognition model to obtain the corresponding formula text, and obtain the detected formula text and the formula bounding box; execute S700; S700, classify all text descriptions in the i th single page based on the bounding boxes of all text descriptions contained in the i th single page obtained in S100 to S600 and the element category bounding boxes of all element categories to obtain the corresponding category of each text description; execute S800; S800, post-process the classified text descriptions of the i th single page to obtain the post-processed text descriptions of the i th single page; wherein the post-processing comprises text deduplication processing, paragraph merging processing, and table text and table structure merging processing on the classified text descriptions; execute S900; S900, sort the paragraphs in the post-processed text descriptions of the i th single page by using a multi-modal large model, and finally obtain the analysis result of the i th single page; set i = i + 1, and execute S200; S1000, merge the analysis results of the m single pages according to the page order to obtain the analysis result of the preset PDF file.

2. The method of claim 1, wherein, The preset PDF file package table, image, chart, formula, at least two columns of layout and at least two languages of text, and the element category include title, text paragraph, table, table title, image, image title, chart, chart title, formula, header and footer, and annotation.

3. The method of claim 1, wherein, The improved table recognition model comprises a regular table recognition model and a text detection model, the regular table recognition model comprises a backbone network, a neck network and a head network connected in sequence, and the input of the backbone network and the text detection model is a table, and the input of the neck network is a splicing result obtained by splicing the output of the backbone network and the output of the text detection model.

4. The method of claim 3, wherein, The training data set of the improved table recognition model comprises a public data set and a specified data set, and the table in the specified data set comprises a table with a merged cell, a table without a border or an incomplete border, and a table with a table structure not meeting a set table structure requirement.

5. The method of claim 1, wherein, S800 specifically comprises: S810, the text descriptions with the same bounding box in the classified text description of the i-th single page are merged to remove the repeated text descriptions as the first text description of the i-th single page; S820, the paragraphs in the first text description of the i-th single page are merged to obtain the first text description with the merged paragraphs as the second text description of the i-th single page; S830, the table text in the second text description of the i-th single page is filled into the corresponding table structure to obtain the text description of the i-th single page after post-processing.

6. The method of claim 5, wherein, S820 specifically comprises: S821, the bounding boxes of all the text descriptions of the i-th single page are sorted in the order from top to bottom and from left to right according to the upper left corner coordinates to obtain the sorted f(i) bounding boxes, and a bounding box counter j is set to 1; S822, if j≤f(i), respectively, the jth bounding box and the j+1th bounding box in the page coordinate system in the x-axis direction of the length of the overlap IL j(j+1) and the length of the merger UL j(j+1) , and the jth bounding box and the j+1th bounding box in the page coordinate system in the y-axis direction of the interval d j(j+1) , if (IL j(j+1) / UL j(j+1) )≥k, and d j(j+1) <c×△d, execute S823, otherwise, set j=j+1, execute S822; k is a preset coefficient, c is a preset value greater than 1, △d=max(d j , d j+1 ), wherein d j is the height of the text in the jth bounding box, and d j+1 is the height of the text in the j+1th bounding box; wherein the origin of the page coordinate system is the upper left corner of the page, the x-axis direction is the horizontal direction of the page, and the y-axis direction is the vertical direction of the page; S823, if the last punctuation symbol in the j-th bounding box is a symbol identifier representing the end of a paragraph, the j-th bounding box and the j+1-th bounding box are merged, otherwise, the j-th bounding box and the j+1-th bounding box are not merged, j is set to j+1, and S822 is executed.

7. The method of claim 6, wherein, S821 specifically comprises: S8210, the upper left corner coordinates of the bounding boxes of all the text descriptions of the i-th single page are obtained to obtain an upper left corner coordinate set, and the upper left corner coordinates in the upper left corner coordinate set are sorted in the order from small to large according to the y coordinate values to obtain a sorted upper left corner coordinate set; S8211, the bounding box corresponding to the first upper left corner coordinate in the sorted upper left corner coordinate set is placed at the corresponding position of the page coordinate system, and a coordinate counter r is set to 2; S8212, if r≤x(i), S8213 is executed, otherwise, S8214 is executed, and x(i) is the number of coordinates in the sorted upper left corner coordinate set of the i-th single page; S8213, the bounding box corresponding to the r-th upper left corner coordinate in the sorted upper left corner coordinate set is placed at the corresponding position of the page coordinate system, r is set to r+1, and S8212 is executed; S8214, the bounding box corresponding to the r-th upper left corner coordinate in the sorted upper left corner coordinate set is placed at the corresponding position of the page coordinate system, r is set to r+1, and S8212 is executed. S8213, if the y coordinate value of the rth top-left corner coordinate in the sorted top-left corner coordinate set is smaller than the y coordinate value of the r-1th top-left corner coordinate, place the bounding box corresponding to the rth top-left corner coordinate below the bounding box corresponding to the r-1th top-left corner coordinate in the page coordinate system, if the y coordinate value of the rth top-left corner coordinate in the sorted top-left corner coordinate set is equal to the y coordinate value of the r-1th top-left corner coordinate, place the bounding box corresponding to the rth top-left corner coordinate right to the bounding box corresponding to the r-1th top-left corner coordinate in the page coordinate system; set r=r+1, execute S8212; S8214, exit the current control procedure, and obtain the sorted f(i) bounding boxes.

8. The method of claim 1, wherein, In S700, if (I ht / U ht ) ≥ k0, it is determined that the category described by the hth text description is the tth element category, I ht is the number of pixel points corresponding to the intersection of the bounding box of the hth text description and the bounding box of the tth element category, U ht is the number of pixel points corresponding to the union of the bounding box of the hth text description and the bounding box of the tth element category, h takes a value from 1 to g(i), t takes a value from 1 to z(i), g(i) is the number of text descriptions contained in the ith single page obtained based on S100 to S600, z(i) is the number of element categories contained in the ith single page obtained based on S100 to S600, and k0 is a preset coefficient.

9. An electronic device, comprising: comprising a processor and a memory; the processor is configured to execute the steps of the method according to any one of claims 1 to 8 by invoking the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that, the computer readable storage medium is configured to store the program or instruction, which causes the computer to execute the steps of the method according to any one of claims 1 to 8.

Images