A method for table extraction and reconstruction of PDF documents used for model training, and a readable storage medium.

Abstract

This invention provides a method for table extraction and reconstruction from PDF documents used for model training, along with a readable storage medium, belonging to the field of document data processing technology. The method includes: parsing the document to identify and locate text and table areas; performing structural analysis on the tables in the table areas; converting the analyzed tables into HTML table code; traversing the document in reading order, adding the text content of the text areas to the output sequence; for the table areas, first generating a unique position marker for each table and inserting it sequentially into the original position of each table in the document in the output sequence, then recording the HTML table code corresponding to each table sequentially in a table mapping table associated with the position marker; and outputting a mixed sequence containing text content and all position markers, as well as a table mapping table associated with all position markers. This invention effectively improves the accuracy of table recognition and reconstruction in documents.

Claims

1. A method for table extraction and reconstruction from PDF documents used for model training, characterized in that, The table extraction and reconstruction method includes: The target PDF document is parsed to identify and locate text and table areas; Perform structural parsing on the tables in the aforementioned table area; Convert the parsed table into HTML table code; Traverse the target PDF document in the order of reading. For text areas, append their text content to the output sequence; For table areas, a unique position marker is first generated for each table and then sequentially inserted into the original position of each table in the target PDF document in the output sequence. Specifically, this includes: based on the target PDF document, establishing a Y-coordinate cursor in the mixed sequence to represent the position of the text stream traversing the target PDF document; when encountering the bounding box of a table area in the target PDF document, recording the value of the top Y-coordinate of the table and finding the same Y-coordinate cursor value in the mixed sequence, inserting the position marker of the table into the position corresponding to the value of the Y-coordinate cursor; then, sequentially recording the HTML table code corresponding to each table in a table mapping table associated with the position marker, wherein the table mapping table includes a set of table numbers, table structure information, and table text information that associate the position markers with the complete HTML table code; Output a mixed sequence containing text content and all table position markers, as well as a table mapping table associated with all position markers.

2. The table extraction and reconstruction method according to claim 1, characterized in that, The text area includes a header row, text blocks, and table headers; The table area includes a header and a body. The table header includes row headers and / or list headers. The list header indicates the category of each column cell, and the row header indicates the category of each row cell.

3. The table extraction and reconstruction method according to claim 1, characterized in that, The specific steps of performing structural parsing on the tables in the table area include: Distinguish between table headers and table bodies, parse merged row cells and record their row spanning attributes, and / or parse merged column cells and record their column spanning attributes.

4. The table extraction and reconstruction method according to claim 3, characterized in that, The cross-row attribute includes the cross-row identifier, the number of merged rows in the cell, and the content recorded in the merged rows. The cross-column attribute includes the cross-column identifier, the number of columns to be merged, and the content recorded in the merged columns.

5. The table extraction and reconstruction method according to claim 4, characterized in that, The specific methods for distinguishing between the table header and the table body include: A cell is considered part of the table header if it is located in the top row and / or the first column from the left. When the font style of the text in a cell is different from the font style of the text in the table body, the cell is determined to belong to the table header, or A cell is considered to belong to the header if its color differs from that of a multi-row table and / or multi-column table.

6. The table extraction and reconstruction method according to claim 5, characterized in that, The font style includes any one or any combination of font, glyph, font size, and font color.

7. The table extraction and reconstruction method according to claim 3, characterized in that, The cell type is identified by analyzing the cell's boundary lines and / or coverage area, where the type is any one of independent cell, merged row cell, merged column cell, and merged row and column cell; Record the position information of the starting row, ending row, starting column, and ending column of each cell to determine the values ​​to be assigned for the number of rows to be merged and the number of columns to be merged.

8. The table extraction and reconstruction method according to any one of claims 1-7, characterized in that, The specific steps of converting the parsed table into HTML table code include: Use labels to mark header cells. Use labels to mark table body cells. Use rowspan to restore the structure of merged cells in a row. Use colspan to restore the structure of merged cells in a column.

9. The table extraction and reconstruction method according to any one of claims 2-7, characterized in that, When the table header is determined to be a list header, the list header of the parsed table structure is converted into HTML format from left to right, and / or When the table header is determined to be a row header, the row headers of the parsed table structure are converted into HTML format from top to bottom. When the cells in the row header and / or the column header are merged into rows and / or columns, the merged rows and / or columns are identified, and the number and content of the merged cells are recorded.

10. The table extraction and reconstruction method according to claim 9, characterized in that, When it is determined that there are merged rows in the list header, the total number of rows in the list header is determined based on the position of the merged cells and the number of merged cells. Based on the total number of rows in the list header, convert the list header line by line from left to right into HTML format; and / or When it is determined that there are column merges in the row header, the total number of columns in the row header is determined based on the position of the merged cells and the number of merged cells. Based on the total number of columns in the row header, the row header is converted into HTML format column by column from top to bottom.

11. The table extraction and reconstruction method according to claim 10, characterized in that, When it is determined that there are multiple merged rows in the list header and at least two of the merged rows in the multiple merged rows are not the same, the total number of rows in the list header is determined based on the position of the merged row of each of the at least two merged rows. and / or When it is determined that there are multiple merged columns in the row header and at least two of the merged columns are not the same, the total number of columns in the row header is determined based on the position of the merged column of each of the at least two merged columns.

12. The table extraction and reconstruction method according to claim 9, characterized in that, When converting the table body into HTML table code, the table body is converted into HTML format line by line in a top-to-bottom order, and for each line of the table body, it is converted into HTML format line by line in a left-to-right order. When it is determined that there is a row merging in the corresponding transformation row, the first row of the corresponding transformation row is converted to HTML format sequentially from left to right until the rightmost column of the corresponding transformation row is reached. The remaining lines below the first line are converted to HTML format line by line from top to bottom, and each line is converted from left to right. and / or When it is determined that there is column merging in the corresponding transformation row, the merged columns in the corresponding transformation row are marked, and the number of cells merged and the cell content are recorded.

13. The table extraction and reconstruction method according to claim 9, characterized in that, The location markers are generated using an auto-incrementing numbering method or a UUID method. The location marker is a semantically meaningful ID identifier. The ID identifier format is as follows:<table_placeholder_id="TABLE_X"> , where X is a unique identifier.

14. The table extraction and reconstruction method according to claim 1, characterized in that, The model used for training includes any one of the following: large language model, multimodal model, knowledge base model, compliance review model, and pre-trained language model.

15. The table extraction and reconstruction method according to any one of claims 10-14, characterized in that, The table extraction and reconstruction method described above is used to parse the target PDF document and input HTML table code into the model for any of the following application scenarios: DRAM test report analysis, semiconductor equipment maintenance log analysis, industrial manufacturing and supply chain management, and structured analysis of healthcare reports.

16. A readable storage medium, characterized in that, The readable storage medium stores a computer-readable program or instructions, which, when executed by a processor, implement the table extraction and reconstruction method according to any one of claims 1-15.

Images