PDF analysis method based on multiple models, electronic equipment and storage medium

Abstract

The invention provides a PDF (Portable Document Format) analysis method based on multiple models, electronic equipment and a storage medium. The method comprises the following steps: acquiring all element types possibly appearing in a complex PDF by utilizing methods such as rule extraction, layout analysis, table identification, formula detection, formula identification, multi-modal large model chart2text, multi-modal large model paragraph sorting and the like. According to the method, complete structured analysis of the complex PDF can be realized, and scanning pieces and non-scanning pieces can be supported.

Description

Technical Field

[0001] The present invention relates to the field of document processing, and in particular to a PDF parsing method, electronic equipment and storage medium based on multiple models. Background Art

[0002] PDF (Portable Document Format) has become one of the standard formats of electronic documents. Its main advantage is that it can effectively maintain the layout and format of documents so that they can be displayed consistently on different operating systems and devices. With the rapid development of information technology, PDF files are increasingly used in various academic, commercial and legal documents. Therefore, the research and application of PDF parsing has gradually attracted the attention of academia and industry. The widespread use of PDF has resulted in a large amount of important information being encapsulated in this format, which mainly includes text, images, tables, charts, formulas and other contents. However, the structural complexity of PDF makes it difficult for traditional text extraction methods to efficiently obtain all the information in it. Therefore, it is necessary to develop efficient PDF parsing technology, which not only involves the extraction of text and images, but also includes the reconstruction and semantic understanding of structured information. There are currently three main routes for PDF parsing. The first is rule-based parsing methods. This type of method relies on specific rules and patterns of the PDF file structure for parsing. By parsing the object structure of PDF (such as text boxes, images, tables, etc.), the content is extracted. This method has the advantages of simple implementation and high efficiency, but it can only effectively extract text and images from PDF, and the accuracy is very low when extracting other types of data; the second is a parsing method based on deep learning. With the development of deep learning technology, PDF parsing based on deep learning has gradually received attention. By training layout analysis models, table recognition and OCR (Optical Character Recognition) models, the structure and content in the document can be identified. This method has certain advantages in image recognition and text classification. It can obtain different types of information and reconstruct structured information. However, some information will be lost in the process of converting PDF to images, and due to the lack of model accuracy and robustness, some parsing errors will occur, such as typos and missing words. The third method: parsing method based on multimodal large models. For PDF files with rich content, each page of the PDF is converted into an image, and the image is parsed by a multimodal model to obtain a comprehensive analysis of the information on this page. Although this method has good understanding ability at the semantic level, when the input is a whole page image, the extracted information is incomplete and when the input is a complex layout, the output may have a certain illusion. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to a first aspect of the present invention, a PDF parsing method based on multiple models is provided, the method comprising the following steps:

[0005] S100, splitting the acquired preset PDF file into pages to obtain corresponding m single pages, and converting each single page into an image to obtain m page images; setting a single page counter i=1.

[0006] S200, if i≤m, determine whether the i-th single page belongs to the scanned page according to the PDF file parsing tool, if yes, obtain the text and the corresponding text bounding box in the i-th single page based on the OCR model, and execute S700; if not, use the PDF file parsing tool to extract the text and the corresponding text bounding box in the i-th single page as well as the image and the corresponding image bounding box, if the image is extracted, execute S300, otherwise, execute S700; if i>m, execute S1000; the value of i ranges from 1 to m, and the initial value is 1.

[0007] S300, based on the chart identification model, determine whether the extracted image of the i-th single page belongs to a chart image or a non-chart image. If it is a non-chart image, obtain the text and the corresponding text bounding box in the image of the i-th single page based on the OCR model, and execute S700; if it is a chart image, obtain the natural language description in the image of the i-th single page based on the multimodal large model, and obtain the text and text bounding box of the image of the i-th single page; execute S700.

[0008] S400, inputting the i-th page image into the layout analysis model to extract all element categories and corresponding element category bounding boxes contained in the i-th page image; executing S500.

[0009] S500, if all element categories contained in the i-th page image include a table category, use the improved table recognition model to recognize the table image contained in the i-th page image, and obtain the corresponding table structure and the corresponding table structure bounding box; execute S700.

[0010] S600, input the i-th page image into the formula detection model. If a formula is detected, the formula image is captured according to the detected formula bounding box, and the detected formula image is recognized through the formula recognition model to obtain the corresponding formula text, thereby obtaining the detected formula text and the formula bounding box; and executing S700.

[0011] S700, based on the bounding boxes corresponding to all text descriptions contained in the i-th single page obtained from S100 to S600 and the element category bounding boxes corresponding to all element categories, classify all text descriptions in the i-th single page to obtain the category corresponding to each text description; execute S800.

[0012] S800, post-processing the classified text description of the ith single page to obtain the post-processed text description of the ith single page; wherein the post-processing includes deduplication processing of the classified text description, paragraph merging processing, and merging processing of table text and table structure; executing S900.

[0013] S900, sort the paragraphs in the text description of the i-th single page after post-processing through the multimodal large model, and finally obtain the parsing result of the i-th single page; set i=i+1, and execute S200.

[0014] S1000, merging the parsing results of m single pages according to the order of the pages to obtain the parsing result of the preset PDF file.

[0015] According to a second aspect of the present invention, there is provided an electronic device comprising a processor and a memory; the processor is used to execute the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.

[0016] According to a second aspect of the present invention, there is provided a computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method according to the first aspect of the present invention.

[0017] The present invention has at least the following beneficial effects:

[0018] The PDF parsing method provided by the embodiment of the present invention can effectively parse a complex PDF file containing tables, images, charts, formulas, at least two columns and texts in at least two languages.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a PDF parsing method based on multiple models provided in an embodiment of the present invention;

[0022] Figure 2 is a sample chart image. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0025] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0026] The embodiment of the present invention provides a PDF parsing method based on multiple models, such as Figure 1 As shown, the method may include the following steps:

[0027] S100, split the acquired preset PDF file into pages to obtain corresponding m single pages, and convert each single page into an image to obtain m page images; set a single page counter i=1. m>1.

[0028] In an embodiment of the present invention, the preset PDF file includes a table, an image, a chart, a formula, at least two columns and text in at least two languages. The preset PDF file may be a file input by a user.

[0029] S200, if i≤m, determine whether the i-th single page belongs to the scanned page according to the PDF file parsing tool, if yes, obtain the text and the corresponding text bounding box in the i-th single page based on the OCR model, and execute S700; if not, use the PDF file parsing tool to extract the text and the corresponding text bounding box in the i-th single page as well as the image and the corresponding image bounding box, if the image is extracted, execute S300, otherwise, execute S700; the value of i ranges from 1 to m, and the initial value is 1; if i>m, execute S1000.

[0030] In an embodiment of the present invention, the PDF parsing tool may be an existing PDF parsing tool, such as pymupdf api. The PDF parsing tool will completely and correctly extract all texts and images in the PDF based on rules.

[0031] Since PDF parsing based on deep learning models is limited by model accuracy and robustness, errors may occur in the parsing results, and problems such as typos and missing characters may easily occur. The present invention uses rules to determine whether a page is a scanned page. Non-scanned pages do not obtain text through the OCR model. Only scanned pages and images in the pages need to obtain text through the OCR model. This not only can achieve complete parsing of scanned and non-scanned PDFs, but also can improve the accuracy of text extraction from non-scanned documents and speed up text extraction.

[0032] In an embodiment of the present invention, the parsed bounding box can be represented by the coordinates of the four vertices of the bounding box. The vertex coordinates can be coordinates in a page coordinate system, where the origin of the page coordinate system is the upper left corner of the page, the x-axis direction is the horizontal direction of the page, and the y-axis direction is the vertical direction of the page.

[0033] In the embodiment of the present invention, the text extracted by the PDF file parsing tool and the OCR model extracts text by line, that is, each line of text has a bounding box.

[0034] S300, based on the chart identification model, determine whether the extracted image of the i-th single page belongs to a chart image or a non-chart image. If it is a non-chart image, obtain the text and the corresponding text bounding box in the image of the i-th single page based on the OCR model, and execute S700; if it is a chart image, obtain the natural language description in the image of the i-th single page based on the multimodal large model, and obtain the text and text bounding box of the image of the i-th single page; execute S700.

[0035] In the embodiment of the present invention, the chart identification model may be an existing chart identification model. The multimodal large model may be an existing multimodal large model, for example, a chart2text model.

[0036] In the embodiment of the present invention, the charts include stacked bar charts, complex line charts, scatter charts, pie charts, conventional line charts, grouped bar charts, conventional bar charts, three-dimensional bar charts, and mixed charts.

[0037] In the embodiment of the present invention, the multimodal large language model can be instructed to obtain the natural language description in the image through a set instruction statement. For example, the instruction statement can be "please describe the content of this chart in detail, paying attention to completeness and accuracy", and the chart to be described can be Figure 2As shown. Based on the received instruction sentence and chart, the large language model will obtain the natural language description of the corresponding chart image, for example, "The chart shows the average math scores of the two classes. Specifically: \n\n-The average math score of Class 1, Grade 2 is 96.5 points. \n-The average math score of Class 2, Grade 2 is 98 points. \n\nThe vertical axis of the chart represents the average math score, and the horizontal axis represents the class."

[0038] S400, inputting the i-th page image into the layout analysis model to extract all element categories and corresponding element category bounding boxes contained in the i-th page image; executing S500.

[0039] In the embodiment of the present invention, the layout analysis model may be an existing layout analysis model. The element categories may include titles, body paragraphs, tables, table titles, images, image titles, charts, chart titles, formulas, headers and footers, comments, etc.

[0040] S500, if all element categories contained in the i-th page image include a table category, use the improved table recognition model to recognize the table image contained in the i-th page image, and obtain the corresponding table structure and the corresponding table structure bounding box; execute S700.

[0041] The existing table recognition model is obtained by training with a public data set and cannot correctly recognize complex table images. In view of this situation, the embodiment of the present invention uses an improved table recognition model to recognize tables, which can improve the table recognition effect.

[0042] In an embodiment of the present invention, the improved table recognition model may include a conventional table recognition model and a text detection model, and the conventional table recognition model includes a backbone network, a neck network and a head network connected in sequence, wherein the inputs of the backbone network and the text detection model are both input tables, and the input of the neck network is the splicing result obtained by splicing the output of the backbone network and the output of the text detection model.

[0043] In an embodiment of the present invention, the training data set of the improved table recognition model includes a public data set and a specified data set. The public data set may be, for example, a pubtabnet data set, and the tables in the specified data set include tables with merged cells, tables without borders or incomplete borders, and tables whose table structures do not meet the set table structure requirements. The set table structure requirements are requirements that characterize the table structure specifications, and may be existing table structure requirements.

[0044] In an embodiment of the present invention, since the splicing result obtained by splicing the output of the backbone network and the output of the text detection model is used as the input of the neck network, the information of the text boundary box in the table image can be provided for the recognition of the table boundary box, thereby improving the recognition accuracy of the table boundary box.

[0045] In an embodiment of the present invention, the text detection model may be an existing text detection model. In the training process of the improved table recognition model, the parameters of the text detection model do not participate in the gradient backpropagation and parameter update. In the training process of the improved table recognition model, a combination of a cross entropy loss function and an L1 loss function may be used to train the model. The model may update the parameters of the model through an Adam optimizer.

[0046] Those skilled in the art know that the table structure may be an existing table structure.

[0047] The table structure obtained in S500 does not include the text in the table.

[0048] S600, input the i-th page image into the formula detection model. If a formula is detected, the formula image is captured according to the detected formula bounding box, and the detected formula image is recognized through the formula recognition model to obtain the corresponding formula text, thereby obtaining the detected formula text and the formula bounding box; and executing S700.

[0049] In the embodiment of the present invention, the formula detection model may be an existing formula detection model.

[0050] S700, based on the bounding boxes corresponding to all text descriptions contained in the i-th single page obtained from S100 to S600 and the element category bounding boxes corresponding to all element categories, classify all text descriptions in the i-th single page to obtain the category corresponding to each text description; execute S800.

[0051] Further, in S700, if (I ht / U ht )≥k0, determine the category described by the h-th text description as the t-th element category, I ht is the number of pixels corresponding to the intersection of the bounding box of the hth text description and the bounding box of the tth element category, U ht is the number of pixels corresponding to the union of the bounding box of the h-th text description and the bounding box of the t-th element category, h ranges from 1 to g(i), t ranges from 1 to z(i), g(i) is the number of text descriptions contained in the i-th single page obtained based on S100 to S600, z(i) is the number of element categories contained in the i-th single page obtained based on S100 to S600, and k0 is a preset coefficient.

[0052] In an embodiment of the present invention, the intersection of the bounding box of the hth text description and the bounding box of the tth element category is a coordinate set formed by the same pixel coordinates among all pixel coordinates corresponding to the bounding box of the hth text description and all pixel coordinates corresponding to the bounding box of the tth element category, and the union of the bounding box of the hth text description and the bounding box of the tth element category is a coordinate set formed by the combination of all pixel coordinates corresponding to the bounding box of the hth text description and all pixel coordinates corresponding to the bounding box of the tth element category. In an embodiment of the present invention, 0<k0<1 can be an empirical value, for example, k0=0.6.

[0053] S800, post-processing the classified text description of the ith single page to obtain the post-processed text description of the ith single page; wherein the post-processing includes deduplication processing of the classified text description, paragraph merging processing, and merging processing of table text and table structure; executing S900.

[0054] Furthermore, S800 specifically includes:

[0055] S810: Merge text descriptions with the same bounding box in the classified text descriptions of the i-th single page to remove duplicate text descriptions, and use the merged text descriptions as the first text description of the i-th single page.

[0056] S820: Merge the paragraphs in the first text description of the i-th single page to obtain the first text description with the merged paragraphs as the second text description of the i-th single page.

[0057] S830, filling the table text in the second text description of the i-th single page into the corresponding table structure to obtain the post-processed text description of the i-th single page.

[0058] Further, S820 may specifically include:

[0059] S821, sort the bounding boxes of all text descriptions of the i-th single page in order from top to bottom and from left to right according to the coordinates of the upper left corners to obtain sorted f(i) bounding boxes; set the bounding box counter j=1.

[0060] S822, if j≤f(i), obtain the overlapping length IL of the jth bounding box and the j+1th bounding box in the x-axis direction in the page coordinate system respectively j(j+1) and combined length UL j(j+1) , and the distance d between the jth bounding box and the j+1th bounding box in the y-axis direction in the page coordinate system j(j+1) , if (IL j(j+1) / UL j(j+1) )≥k, and d j(j+1)<c×△d, execute S823, otherwise, set j=j+1 and execute S822; k is a preset coefficient, c is a preset value greater than 1, △d=max(d j , d j+1 ), where d j is the height of the text in the jth bounding box, d j+1 is the height of the text in the j+1th bounding box, that is, △d is the maximum height of the text in the jth bounding box and the j+1th bounding box.

[0061] In the embodiment of the present invention, c=2. , 0<k<1, which may be an empirical value, for example, k=0.8.

[0062] In the embodiment of the present invention, IL j(j+1) The following conditions are met: If x j+1 start >x j end , I.L. j(j+1) = Null, if x j+1 start ≤x j end , I.L. j(j+1) =(x j end -x j+1 start ), Null means empty, x j+1 start is the x coordinate of the upper left corner or lower left corner of the j+1th bounding box, x j end is the x coordinate of the upper right corner or lower right corner of the jth bounding box.

[0063] In the embodiment of the present invention, UL j(j+1) UL j(j+1) =(x j+1 end -x j end ), where x j+1 end It is the x coordinate of the upper right corner or lower right corner of the j+1th bounding box.

[0064] In the embodiment of the present invention, d j(j+1) =(y j end -y j+1 start ), y j+1 start is the y coordinate of the upper left corner or lower left corner of the j+1th bounding box, y j endis the y coordinate of the upper right corner or lower right corner of the jth bounding box.

[0065] S823, if the last punctuation mark in the jth bounding box is a symbol indicating the end of a paragraph, merge the jth bounding box and the j+1th bounding box, otherwise, do not merge the jth bounding box and the j+1th bounding box; set j=j+1 and execute S822.

[0066] In the embodiment of the present invention, the symbol indicating the end of a paragraph may be a symbol suitable for marking the end of a paragraph, such as a period.

[0067] Furthermore, S821 specifically includes:

[0068] S8210, obtaining the upper left corner coordinates of the bounding boxes of all text descriptions of the i-th single page, obtaining an upper left corner coordinate set, and sorting the upper left corner coordinates in the upper left corner coordinate set in ascending order of y coordinate values ​​to obtain a sorted upper left corner coordinate set.

[0069] S8211, placing the bounding box corresponding to the first upper left corner coordinate in the sorted upper left corner coordinate set at the corresponding position of the page coordinate system, and setting the coordinate counter r=2.

[0070] S8212, if r≤x(i), execute S8213, otherwise, execute S8214, where x(i) is the number of coordinates in the sorted upper left corner coordinate set of the i-th single page.

[0071] S8213, if the y-coordinate value of the r-th upper left corner coordinate in the sorted upper left corner coordinate set is less than the y-coordinate value of the r-1-th upper left corner coordinate, in the page coordinate system, place the bounding box corresponding to the r-th upper left corner coordinate at the corresponding position below the bounding box corresponding to the r-1-th upper left corner coordinate; if the y-coordinate value of the r-th upper left corner coordinate in the sorted upper left corner coordinate set is equal to the y-coordinate value of the r-1-th upper left corner coordinate, in the page coordinate system, place the bounding box corresponding to the r-th upper left corner coordinate at the corresponding position to the right of the bounding box corresponding to the r-1-th upper left corner coordinate; set r=r+1, execute S8212; S8214, exit the current control program, and obtain the sorted f(i) bounding boxes.

[0072] S900, sort the paragraphs in the text description of the i-th single page after post-processing through the multimodal large model, and finally obtain the parsing result of the i-th single page; set i=i+1, and execute S200.

[0073] In an embodiment of the present invention, a command statement can be set to instruct the multimodal large model to sort the paragraphs in the text description of the i-th single page after post-processing. For example, the command statement is "[parsed result (text segment 1, text segment 2...)] The above is the text segment obtained by parsing the PDF. The sorting of the above text segments may be wrong. Please read all the above text segments carefully and sort them correctly according to the semantics of the text. Please output the sorting results in the form of a list."

[0074] S1000, merging the parsing results of m single pages in the order of the pages to obtain the parsing result of the preset PDF file, the parsing result including all categories of text paragraph information, image information, etc. of the PDF file and corresponding categories and bounding boxes of the text paragraph information and image information.

[0075] The multi-model based PDF parsing method provided in the embodiment of the present invention integrates the advantages of the current mainstream parsing methods, including rule extraction, layout analysis, table recognition, formula detection, formula recognition, multimodal large model chart2text, multimodal large model paragraph sorting and other methods, which can obtain all element types that may appear in complex PDFs, realize complete structured parsing of complex PDFs, and support scanned and non-scanned documents; in addition, in the parsing of non-scanned PDFs, it has the advantages of comprehensive parsing types, accurate sorting, accurate and complete text, fast extraction speed, as well as increased chart parsing capabilities and optimized table recognition capabilities, making the recognition of complex tables more accurate.

[0076] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0077] The embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0078] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0079] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A PDF parsing method based on multiple models, characterized in that: The method comprises the following steps: S100, splitting the acquired preset PDF file into pages to obtain corresponding m single pages, and converting each single page into an image to obtain m page images; setting a single page counter i=1; S200, if i≤m, determine whether the i-th single page belongs to the scanned page according to the PDF file parsing tool, if yes, obtain the text and the corresponding text bounding box in the i-th single page based on the OCR model, and execute S700; if no, extract the text and the corresponding text bounding box in the i-th single page and the image and the corresponding image bounding box using the PDF file parsing tool, if the image is extracted, execute S300, otherwise, execute S700; if i>m, execute S1000; the value of i ranges from 1 to m, and the initial value is 1; S300, judging whether the image of the i-th single page extracted is a chart image or a non-chart image based on the chart identification model, and if it is a non-chart image, obtaining the text and the corresponding text bounding box in the image of the i-th single page based on the OCR model, and executing S700; If it is a chart image, obtain the natural language description in the image of the i-th single page based on the multimodal large model, and obtain the text and text bounding box of the image of the i-th single page; execute S700; S400, inputting the i-th page image into the layout analysis model to extract all element categories and corresponding element category bounding boxes contained in the i-th page image; executing S500; S500, if all element categories included in the i-th page image include a table category, use the improved table recognition model to recognize the table image included in the i-th page image, and obtain the corresponding table structure and the corresponding table structure bounding box; execute S700; S600, inputting the i-th page image into the formula detection model, if a formula is detected, intercepting the formula image according to the detected formula bounding box, and recognizing the detected formula image through the formula recognition model, obtaining the corresponding formula text, and obtaining the detected formula text and the formula bounding box; executing S700; S700, based on the bounding boxes corresponding to all text descriptions contained in the i-th single page and the element category bounding boxes corresponding to all element categories obtained in S100 to S600, classify all text descriptions in the i-th single page to obtain a category corresponding to each text description; execute S800; S800, post-processing the classified text description of the i-th single page to obtain a post-processed text description of the i-th single page; wherein the post-processing includes performing text deduplication processing, paragraph merging processing, and table text and table structure merging processing on the classified text description; executing S900; S900, sorting the paragraphs in the text description of the i-th single page after post-processing by using the multimodal large model, and finally obtaining the parsing result of the i-th single page; Set i=i+1 and execute S200; S1000, merging the parsing results of m single pages according to the order of the pages to obtain the parsing result of the preset PDF file.

2. The method according to claim 1, characterized in that The preset PDF file package includes tables, images, charts, formulas, at least two columns and texts in at least two languages, and the element categories include titles, body paragraphs, tables, table titles, images, image titles, charts, chart titles, formulas, headers and footers, and comments.

3. The method according to claim 1, characterized in that The improved table recognition model includes a conventional table recognition model and a text detection model, wherein the conventional table recognition model includes a backbone network, a neck network and a head network connected in sequence, wherein the inputs of the backbone network and the text detection model are both tables, and the input of the neck network is a splicing result obtained by splicing the output of the backbone network and the output of the text detection model.

4. The method according to claim 3, characterized in that The training data set of the improved table recognition model includes a public data set and a specified data set. The tables in the specified data set include tables with merged cells, tables without borders or incomplete borders, and tables whose table structures do not meet the set table structure requirements.

5. The method according to claim 1, characterized in that S800 specifically includes: S810, merging text descriptions with the same bounding box in the classified text descriptions of the i-th single page to remove duplicate text descriptions, and taking the result as the first text description of the i-th single page; S820, merging the paragraphs in the first text description of the i-th single page to obtain the first text description with the merged paragraphs as the second text description of the i-th single page; S830, filling the table text in the second text description of the i-th single page into the corresponding table structure to obtain the post-processed text description of the i-th single page.

6. The method according to claim 5, characterized in that S820 specifically includes: S821, sorting the bounding boxes of all text descriptions of the i-th single page in order from top to bottom and from left to right according to the coordinates of the upper left corners to obtain sorted f(i) bounding boxes; setting the bounding box counter j=1; S822, if j≤f(i), obtain the overlapping length IL of the jth bounding box and the j+1th bounding box in the x-axis direction in the page coordinate system respectively j(j+1) and combined length UL j(j+1) , and the distance d between the jth bounding box and the j+1th bounding box in the y-axis direction in the page coordinate system j(j+1) , if (IL j(j+1) / UL j(j+1) )≥k, and d j(j+1) <c×△d, execute S823, otherwise, set j=j+1 and execute S822; k is a preset coefficient, c is a preset value greater than 1, △d=max(d j , d j+1 ), where d j is the height of the text in the jth bounding box, d j+1 The height of the text in the j+1th bounding box is; where the origin of the page coordinate system is the upper left corner of the page, the x-axis direction is the horizontal direction of the page, and the y-axis direction is the vertical direction of the page; S823, if the last punctuation mark in the jth bounding box is a symbol indicating the end of a paragraph, merge the jth bounding box and the j+1th bounding box, otherwise, do not merge the jth bounding box and the j+1th bounding box; set j=j+1 and execute S822.

7. The method according to claim 6, characterized in that S821 specifically includes: S8210, obtaining the upper left corner coordinates of the bounding boxes of all text descriptions of the i-th single page to obtain an upper left corner coordinate set, and sorting the upper left corner coordinates in the upper left corner coordinate set in ascending order of y coordinate values ​​to obtain a sorted upper left corner coordinate set; S8211, placing the bounding box corresponding to the first upper left corner coordinate in the sorted upper left corner coordinate set at the corresponding position of the page coordinate system, and setting the coordinate counter r=2; S8212, if r≤x(i), execute S8213, otherwise, execute S8214, where x(i) is the number of coordinates in the sorted upper left corner coordinate set of the i-th single page; S8213, if the y-coordinate value of the r-th upper left corner coordinate in the sorted upper left corner coordinate set is less than the y-coordinate value of the r-1-th upper left corner coordinate, in the page coordinate system, place the bounding box corresponding to the r-th upper left corner coordinate at the corresponding position below the bounding box corresponding to the r-1-th upper left corner coordinate; if the y-coordinate value of the r-th upper left corner coordinate in the sorted upper left corner coordinate set is equal to the y-coordinate value of the r-1-th upper left corner coordinate, in the page coordinate system, place the bounding box corresponding to the r-th upper left corner coordinate at the corresponding position to the right of the bounding box corresponding to the r-1-th upper left corner coordinate; set r=r+1, execute S8212; S8214, exit the current control program, and obtain the sorted f(i) bounding boxes.

8. The method according to claim 1, characterized in that: In S700, if (I ht / U ht )≥k0, determine the category described by the h-th text description as the t-th element category, I ht is the number of pixels corresponding to the intersection of the bounding box of the hth text description and the bounding box of the tth element category, U ht is the number of pixels corresponding to the union of the bounding box of the h-th text description and the bounding box of the t-th element category, h ranges from 1 to g(i), t ranges from 1 to z(i), g(i) is the number of text descriptions contained in the i-th single page obtained based on S100 to S600, z(i) is the number of element categories contained in the i-th single page obtained based on S100 to S600, and k0 is a preset coefficient.

9. An electronic device, characterized in that: including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a program or an instruction, wherein the program or the instruction enables a computer to execute the steps of the method according to any one of claims 1 to 8.

Images