Method for extracting and aligning multi-format document table data based on large language model

Abstract

The present invention discloses a method for extracting and aligning multi-format document table data based on a large language model, including extracting tables from editable documents / non-editable documents to obtain tables in HTML format; analyzing the tables in HTML format based on a large language model to screen out useful tables; and aligning the screened useful tables according to set standard fields using a large language model. The present invention transforms the table data alignment standardization problem based on artificial rules into the data extraction and alignment standardization problem based on semantic recognition of a large language model, and can effectively handle overlong tables, cross-page tables, and table formatting problems in multiple types of documents, extract complete tables, and implement the storage of standardized fields of tables.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for extracting and aligning multi-format document table data based on a large language model. Background Art

[0002] With the development of information technology, enterprises and organizations have accumulated a large amount of document data, which includes both structured data (such as Excel tables) and unstructured data (such as PDF reports, Word documents, etc.). These documents often contain key information, especially data presented in tabular form. However, how to effectively extract useful information from these multi-type documents and align the data to convert it into standard structured data that can be analyzed and used (the goal of data alignment is to establish a unified data field naming rule and format. Regardless of which document the original table comes from or what different terms are used, the system can recognize various synonyms or similar expressions of the field and convert them into consistent standard names, thereby ensuring the uniformity and comparability of the data) has always been a challenge in the field of information processing.

[0003] There is no full-process processing method for processing multiple types of document tables. For table data alignment, a common method is to use expert prior knowledge to formulate data alignment rules, but its limitation is that the rules need to be manually formulated and maintained, which is very time-consuming and labor-intensive. The present invention proposes a full-process multi-format document table extraction and LLM-Agent-based table data alignment technology, aiming to solve these problems and provide an automated and intelligent table processing solution. Summary of the invention

[0004] The purpose of the present invention is to provide a method for extracting and aligning multi-format document table data based on a large language model, so as to solve the technical problem of the difficulty in extracting and aligning existing document table data.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] The method for extracting and aligning multi-format document table data based on a large language model includes the following steps:

[0007] S1. Extracting a table from an editable document / non-editable document to obtain a table in HTML format;

[0008] S2. Analyze the tables in HTML format based on the large language model and filter out useful tables;

[0009] S3. Use a large language model to align the selected useful tables according to the set standard fields.

[0010] Furthermore, the editable documents include Word documents, Excel documents and non-scanned PDF files, and the table extraction of the editable documents includes: using Spire.Doc and DocX to extract the table from Word documents, using Xlrd, OpenpyXL and Pandas to extract the table from Excel documents, and using Spire.PDF and pdfplumber to extract the table from non-scanned PDF files to obtain a table in HTML format;

[0011] The non-editable document includes a scanned PDF file, and the table extraction of the non-editable document includes: SA1, using a pre-trained Layout model to perform layout analysis on the scanned PDF file to obtain coordinate information of the table image in the file; SA2, based on the obtained coordinate information, cutting out the table image from the file; SA3, inputting all the cut table images into a table recognition model to obtain a table in HTML format.

[0012] Furthermore, the table extraction from the editable document / non-editable document includes the recognition and merging of cross-page tables, text translation of HTML tables, and table header recognition.

[0013] Furthermore, the identification and merging of cross-page tables include the following operations: SB1, creating a temporary storage space; SB2, traversing each table of each page of the non-scanned PDF file or the scanned PDF file: SB21, if the table is not a page end table, directly output the current table; if the table is a page end table, save the current page end table in the temporary storage space, and traverse all tables on the next page; SB22, if the first table on the next page is a page header table and there is a table in the temporary storage space, and the number of columns of the page header table is consistent with the number of columns and format of the last table in the temporary storage space, save the current page header table in the temporary storage space, if the first table is also a page end table, continue to process the next page, if the first table is not a page header table, merge all tables in the temporary storage space in order and output them, and clear the temporary storage space, the output table is a complete cross-page table; SB23, repeat SB21 and SB22 until all cross-page tables are identified and merged.

[0014] Furthermore, the judgment of whether the table is at the beginning or the end of a page includes the following operations: obtaining the page information of the non-scanned version of the PDF file and the layout information of the table coordinates from pdfplumber, or obtaining the page information of the scanned version of the PDF file and the layout information of the table coordinates from the Layout model; judging whether the table is at the beginning or the end of the page based on the layout information and the prior threshold.

[0015] Furthermore, the text translation of the HTML table includes the following operations: SC1, loading an editable document / non-editable document; SC2, traversing each page of the editable document / non-editable document: SC21, loading each chapter of the current page, traversing all tables in each chapter; SC22, for each table in the chapter, initializing an HTML table string and creating a table tag; SC23, traversing each column of the table, adding a row tag for each row of the table, traversing each row of the table, traversing all cells of the current row inside the row tag, removing extra spaces in the cells, wrapping the processed cell content in a table cell tag inside the row tag and adding it to the HTML table string, when all cells in a row are processed, closing the row tag, and when all rows are processed, closing the table tag.

[0016] Furthermore, the header recognition includes the following operations: SD1, input the first 20 rows of data of the table and the preset first prompt into the large language model; the large language model performs semantic understanding on the first 20 rows of data input according to the first prompt, identifies the row number where the header is located and obtains the index of the end position of the header; SD1, determines and marks the starting position and end position of the header according to the index of the end position of the header, and extracts the header; SD2, selects the segmentation length, and divides the remaining table except the header into multiple sub-tables according to the segmentation length; and adds the header to the head of each sub-table.

[0017] Furthermore, the S2 includes: S21, inputting the HTML format table and the preset second Prompt into the large language model, the large language model checks the data in the table item by item according to the second Prompt, parses the table title, the relationship between rows and columns, and the coherence and consistency of numerical or text entries, and outputs the parsing results; S22, based on the parsing results, filtering out tables that do not meet the preset data quality standards or relevance to obtain useful tables.

[0018] Furthermore, the S3 includes: S31, inputting the useful table and the preset third prompt into the large language model, for each useful table, the large language model identifies each field in the table according to the third prompt, and converts different fields with synonymous or similar expressions into the same set standard field; S32, aligning multiple useful tables processed by S31 according to the standard fields.

[0019] Furthermore, the S32 includes: S321, filling the data corresponding to the same standard field in each useful table into the current standard field, generating a new table, and completing the alignment of the standard fields.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] The present invention transforms the problem of table data alignment standardization based on manual rules into a data extraction and alignment standardization problem based on semantic recognition of a large language model. It can effectively handle overlong tables, cross-page tables and table formatting problems in multiple types of documents, extract complete tables, and implement the storage of standardized fields of tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0023] Terminology explanation:

[0024] Xlrd‌‌ is a Python library for reading Excel files;

[0025] OpenpyXL‌‌ is a Python library for reading and writing Excel files;

[0026] Pandas is a tool based on NumPy;

[0027] Spire.PDF‌ is a professional PDF component that can independently create, write, edit, manipulate and read PDF files, supporting multiple programming languages ​​and platforms;

[0028] Spire.Doc is a MS Word component used to process Word documents;

[0029] pdfplumber is a Python library for processing PDF files;

[0030] The ‌Layout model‌ is a deep learning model specifically designed to process document layout and text information;

[0031] The MuTabNet model is a deep learning model based on tabular data;

[0032] The ‌ConvStem model‌ is a model that combines the convolutional neural network (CNN) and the Transformer structure;

[0033] The SLANet model is a lightweight table recognition model;

[0034] Prompt is a technology based on artificial intelligence (AI) instructions that guide the output of language models through clear and specific instructions.

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] In the description of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0037] Currently, open source tools such as Pandas, spire.doc and Spire.PDF have become key methods for processing and extracting table text information in Excel, Word and document-style PDF. These tools can extract table text information by reading the metadata of the document, but they do not retain the format information of the table. In addition, the presence of very long table text will affect the contextual understanding ability of the large language model, thereby reducing the processing effect. For this reason, the present invention provides a multi-format document table data extraction and alignment method based on a large language model.

[0038] like Figure 1 As shown, the multi-format document table data extraction and alignment method based on a large language model includes the following steps:

[0039] S1. Extracting a table from an editable document / non-editable document to obtain a table in HTML format;

[0040] S2. Analyze the tables in HTML format based on the large language model and filter out useful tables;

[0041] S3. Use a large language model to align the selected useful tables according to the set standard fields.

[0042] The present invention transforms the traditional table standardization problem based on manual rules into a table semantic analysis problem based on a large language model (LLM). By utilizing the powerful processing capabilities of the large model, the problems of table filtering, data alignment and standardization are gradually solved, thereby achieving efficient and accurate processing and standardized storage of table information in various documents.

[0043] The present invention optimizes the table extraction process for multiple types of documents (including editable documents and non-editable documents). The specific process is as follows:

[0044] The editable documents include Word documents (.doc, .docx format), Excel documents (.xls, .xlsx format) and non-scanned PDF files. Among them, the table extraction of editable documents includes: using Spire.Doc and DocX to extract tables from Word documents, using Xlrd, OpenpyXL and Pandas to extract tables from Excel documents, and using Spire.PDF and pdfplumber to extract tables from non-scanned PDF files to obtain tables in HTML format.

[0045] The non-editable document includes a scanned PDF file, and the table extraction of the non-editable document includes: SA1, using a pre-trained Layout model to perform layout analysis on the scanned PDF file to obtain coordinate information of the table image in the file; SA2, based on the obtained coordinate information, cutting out the table image from the file; SA3, inputting all the cut table images into a table recognition model to obtain a table in HTML format.

[0046] Preferably, the Layout model is a PicoDet_LCNet_x1_0_FGD_Layout_CDLa model or a PicoDet_LCNet_x1_0_FGD_Layout model, and the table recognition model is a MuTabNet model, a ConvStem model or a SLANet model.

[0047] Preferably, the table extraction from the editable document / non-editable document includes the recognition and merging of cross-page tables, text translation of HTML tables, and recognition of table headers.

[0048] Since cross-page tables have page continuity and format consistency, it means that you only need to determine whether the table on the previous page is at the bottom of the page, whether the table on the next page is at the top of the page, and whether the number of columns and formats of the previous and next tables are consistent. Merging cross-page tables can avoid data loss caused by loss of semantic information.

[0049] Preferably, the identification and merging of cross-page tables specifically include the following operations: SB1, create a temporary storage space; SB2, traverse each table of each page of the non-scanned PDF file or the scanned PDF file: SB21, if the table is not a page end table, directly output the current table; if the table is a page end table, save the current page end table in the temporary storage space, and traverse all tables on the next page; SB22, if the first table on the next page is a page header table and there is a table in the temporary storage space, and the number of columns of the page header table is consistent with the number of columns and format of the last table in the temporary storage space, save the current page header table in the temporary storage space, if the first table is also a page end table, continue to process the next page, if the first table is not a page header table, merge all tables in the temporary storage space in order and output them, and clear the temporary storage space, the output table is a complete cross-page table; SB23, repeat SB21 and SB22 until all cross-page tables are identified and merged.

[0050] The temporary storage space created is used to store tables that may need to be merged across pages. That is, during the merging process, the table at the end of each page is gradually saved and tried to be merged with the table at the beginning of the next page. There are two types of tables on a page, the first type is an independent table, and the second type is a cross-page table. If there is a table at the end of the previous page, it may be a cross-page table, and if the table on the next page is at the beginning of the page and the number of table columns and format are consistent with the end table of the previous page, then the two tables are considered to be cross-page, and the current head table is saved in a temporary table and merged with the last table of the temporary table, and then the merged table is output and the table in the temporary table is cleared; if the table on the next page is not only the head table, but a full-page table, that is, a table occupies one page, then the merged table is not output and the table in the temporary table is not cleared, but the next page is processed continuously to determine whether the first table on the next page is at the beginning of the page and the number of table columns and format are consistent with the number of columns and format of the last table of the temporary table. If so, the current head table and the last table of the temporary table are combined, and the merged table is output and the table in the temporary table is cleared; otherwise, the previously merged table is directly output and the table in the temporary table is cleared. If the first table is not the header table, clear the tables in the temporary table; if the last table on the next page is the footer table, save the current footer table in the temporary table; repeat the operation until all cross-page tables are identified and merged. The temporary table does not save the data of all cross-page tables at once, but dynamically saves and merges the parts that may belong to the same cross-page table when processing each page. This method can effectively process cross-page tables while maintaining memory efficiency and avoiding loading too much unnecessary data at one time.

[0051] For judging whether the table is at the beginning or the end of a page, preferably, pdfplumber obtains the page information of the non-scanned PDF file and the layout information of the table coordinates, or obtains the page information of the scanned PDF file and the layout information of the table coordinates from the Layout model; judging whether the table is at the beginning or the end of a page based on the layout information and the prior threshold, which is specifically expressed as follows:

[0052]

[0053]

[0054] in, Indicates that you are at the top of the page. Indicates that you are at the end of the page. is the current page; It is a page Middle The bounding box of the table; and Respectively represent the table The top and bottom coordinate; Indicates the height of the current page; and They are the prior thresholds used to determine whether the table is at the beginning and end of the page.

[0055] Since the traditional table extraction formula cannot directly output the table format, the present invention loads the document and reads the table, gradually extracts the cell information in the table, integrates the information into the HTML frame, and finally generates an HTML table with a complete table format.

[0056] Preferably, the text translation of the HTML table includes the following operations: SC1, loading an editable document / non-editable document, for example, using tools such as Spire.Doc to load; SC2, traversing each page of the editable document / non-editable document: SC21, loading each section (Section) of the current page, a document may have multiple sections, each section may contain different content such as text, pictures or tables, traversing all tables (Tables) in each section; SC22, for each table in the section, initializing an HTML table string, creating a table tag, the table tag is an HTML with a border element( Label); SC23, traverse each column of the table, and for each row of the table, add a row label ( Tag), traverse each row of the table, traverse all cells in the current row inside the row tag, remove extra spaces in the cells, and wrap the processed cell content in the table cell tag inside the row tag ( Tags) and added to the HTML table string. When all cells in a row are processed, the row tag is closed, and when all rows are processed, the table tag is closed.

[0057] When processing tables in documents, traditional table extraction methods often ignore the importance of headers, which may lead to misunderstandings or errors in the subsequent data analysis process. Especially for large tables, the information of the header may be distributed on the first page, and there is the possibility of complex headers (complex headers usually involve multi-level nesting, merged cells, cross-row / cross-column, etc.), as shown in Table 1, which means that it is impossible to directly assume that the first row of the table is the header, otherwise the semantic information of the table itself will be incomplete. Traditional methods have difficulty in correctly associating headers with corresponding cell data. In addition, large language models (LLMs) face limitations on the number of input tokens when processing tabular data, which limits the model's ability to process long tables. The present invention proposes a table header recognition method that solves the above problems.

[0058] Table 1 Example of a table with a complex header

[0059]

[0060] Preferably, the table header recognition includes the following operations: SD1, input the first 20 rows of data of the table and the preset first prompt into the large language model; the large language model performs semantic understanding on the first 20 rows of data input according to the first prompt, identifies the row number where the header is located and obtains the index of the end position of the header; SD1, determines and marks the starting position and end position of the header according to the index of the end position of the header, and extracts the header; SD2, selects the segmentation length, and divides the remaining table except the header into multiple sub-tables according to the segmentation length; adds the header to the head of each sub-table.

[0061] The present invention is based on the use of a large language model to convert the problem of using rule matching to find the header into a large language model semantic understanding problem. According to known priors, the number of header rows usually does not exceed 20 rows. For this reason, the first 20 rows of an ultra-long table in HTML format are input into the large language model. The large language model is allowed to perform semantic understanding based on the first 20 rows of input data, identify the number of rows where the header is located, and obtain the index of the end position of the header. Then, the starting position and end position of the header are determined and marked according to the index of the end position of the header, and the header is extracted. Finally, the table is reasonably segmented according to the header to ensure that each segmented table contains a complete header, and the segmentation length is selected, for example, the segmentation length is 40 rows or 80 rows, and the remaining table except the header is segmented into multiple sub-tables according to the segmentation length; the header is added to the head of each sub-table.

[0062] The large language model performs semantic understanding on the first 20 rows of input data according to the preset first prompt, identifies the row number where the header is located, and obtains the index of the end position of the header. The following is a detailed description in a specific scenario:

[0063] # Role

[0064] Role X is a senior HTML file source code analyst who needs to analyze the given HTML file and understand the data content in it.

[0065] # Task

[0066] Analyze the table represented by the following HTML and identify the header content and data content.

[0067] The entire HTML is a table defined by a table tag, where the tbody tag contains multiple tr tags, each of which contains multiple td cell data.

[0068] The first n lines of this HTML form the table header and define the table data model. The remaining data is the data content of the table.

[0069] Role X needs to determine which HTML table tr rows describe the header information.

[0070] 1. The table header may be a tr data or multiple tr data.

[0071] 2. Flexible judgment is required based on the specific structure of the table and the completeness of the column information.

[0072] The remaining tr tags outside the table header are the specific data of the table, and the data format of each tr row is the same.

[0073] Role X needs to analyze the table header from tr row to tr row, and output the attribute field list of the parsed table header.

[0074] # Output

[0075] Make sure to output a json in the following format without any extraneous content:

[0076] {

[0077] "data_model": ["<attribute 1>","<attribute 2>",...]

[0078] }.

[0079] After extracting tables in HTML format from documents, directly storing them in the database may face two major problems: (1) there are useless tables in the table set (i.e., tables that do not meet data requirements); (2) the table data is complexly distributed and the format is inconsistent. To this end, the present invention analyzes tables in HTML format based on a large language model to filter out useful tables; then uses the large language model to standardize the fields of the filtered useful tables, and then aligns them according to the set standard fields.

[0080] Preferably, S2 includes: S21, inputting a table in HTML format and a preset second prompt into a large language model, the large language model checking the data in the table item by item according to the second prompt, parsing the table title, the relationship between rows and columns, and the coherence and consistency of numerical or text entries, and outputting the parsing results; S22, filtering out tables that do not meet the preset data quality standards or relevance according to the parsing results to obtain useful tables.

[0081] The large language model can parse the table according to the preset second prompt. During the training process, the large language model has learned how to associate different parts of the text and infer the logical relationship and dependency between them. In this way, the large language model can help users effectively screen out high-quality and highly relevant data tables. Then, based on the parsing results, tables that do not meet the preset data quality standards or relevance are eliminated. This process helps to improve the efficiency of subsequent data processing and analysis, retain useful and relevant tables, reduce the impact of noise data, and ensure the accuracy and reliability of the results. For example, the preset second prompt is to analyze the table containing data items and prices. The large language model parses whether the input table is a table of data items and prices, and then filters out tables that are not data items and prices based on the parsing results.

[0082] Among them, the large language model can parse the table according to the preset second prompt and check the data in the table item by item. The following is a detailed description in a specific scenario:

[0083] # Role Settings

[0084] Role X is a table analyst. Role X's job is to analyze table types based on the table's content.

[0085] # Task Settings

[0086] Please ask Role X to summarize the table given in HTML format and analyze whether there is information such as material quantity, amount, model, etc. in the table.

[0087] There are the following types of tabular data, including [material, hire, other], which are defined as follows:

[0088] Meterial) bill of materials table data, including multiple types of materials and corresponding prices.

[0089] hire) employment list data, including information such as the type and function of hired workers.

[0090] Other) Other tables, table data not included in the above categories, if not included.

[0091] # Output requirements

[0092] The output must be a json in the following format

[0093] {

[0094] "caption": <table summary, within 50 words, a simple caption description of the table>,

[0095] "header": <Whether the table contains a header, bool type>,

[0096] "category": ,

[0097] "material_num": <material type quantity>,

[0098] "contain_price": <whether to include material price>

[0099] }.

[0100] Preferably, the S3 includes: S31, inputting the useful table and the preset third prompt into the large language model, for each useful table, the large language model identifies each field in the table according to the third prompt, and converts different fields with synonymous or similar expressions into the same set standard field; S32, aligning the multiple useful tables processed by S31 according to the standard fields.

[0101] The useful tables that have been screened are input into the large language model. These tables have been preliminarily cleaned and verified to ensure the purity and applicability of the data set. Then, all fields in the table are standardized using the powerful processing capabilities of the large language model. This process aims to unify the naming rules and formats of the fields. No matter what document the original table comes from or what terminology is used, the LLM model can identify synonyms or similar expressions of the fields through analysis and convert them into unified standard fields to ensure data consistency and comparability.

[0102] Preferably, a standard field database can be constructed, which contains all the set standard fields. For example, for material price information, standard fields such as item name, branch name, unit, price, unit price including tax, unit price excluding tax, and tax rate are set. The table fields obtained in the document may be item name, price / unit, tax rate, total, total price, etc. The table fields do not match the standard fields of the database, which will make it impossible to align the fields. Therefore, it is necessary to rely on the semantic understanding of the table fields by the language model, and convert different fields with synonymous or similar expressions into the same set standard field, such as converting the item name into the item name, to ensure the consistency and comparability of the data. After completing the field standardization, it will further ensure that the table data from the same or different documents can be aligned according to the set standard fields. In addition to setting the third prompt, other prompts can also be set. For example, for price / unit, it needs to be divided into two fields: price and unit; for total and total price, the unit price needs to be calculated, and the unit price = total price / total.

[0103] More preferably, the S32 includes: S321, filling the data corresponding to the same standard field in each useful table into the current standard field, generating a new table, and completing the alignment of the standard fields. Through the alignment operation, even if the original table structures are different, seamless data docking can be achieved, which is convenient for centralized management and analysis, thereby improving the overall data processing efficiency and quality, and also laying a foundation for subsequent data integration, processing and database storage.

[0104] The large language model identifies each field in the table based on the third prompt and converts different fields with synonymous or similar expressions into the same standard field. The following is a detailed description in a specific scenario:

[0105] # Role Settings

[0106] Role X is a table analyst. Role X's job is to extract the required table data based on the content of the table.

[0107] # Task Settings

[0108] Please ask role X to extract the following types of data based on the given HTML form. The name of one type of data is a standard field set:

[0109] product_name: material name, indicating the name of the goods;

[0110] product_model: material model, indicating the specification model of the goods;

[0111] brand_name: material brand, indicating the brand of the goods;

[0112] unit_name: material unit of measurement;

[0113] tax_rate: tax rate, material tax ratio;

[0114] unit_price_with_tax: the price of the material after tax, which means the unit price and has nothing to do with the quantity of the material;

[0115] unit_price_without_tax: the price of the material before tax, which means the unit price and has nothing to do with the quantity of the material;

[0116] total_price_with_tax: the price of the material after tax, indicating the total price / total price, which is related to the quantity of the material;

[0117] total_price_without_tax: the price of the material before tax, indicating the total price / total price, which is related to the quantity of the material;

[0118] product_amount: the quantity of the material;

[0119] Some tables only contain unit prices, while others only contain total prices. You need to be careful to distinguish them.

[0120] If a field does not exist, role X sets the default value of the field to an empty string.

[0121] # Output requirements

[0122] The output must be a json file with the following format (assuming the table has 2 records, the actual number of lists depends on the actual content of the table)

[0123] [{

[0124] "product_name": "<Material Name 1>",

[0125] "product_model": "<Material Model 1>",

[0126] "brand_name": "<Material Brand 1>",

[0127] "unit_name": "<Material Unit 1>",

[0128] "tax_rate": "<Tax rate 1>",

[0129] "total_price_with_tax": "<total price after material tax 1>",

[0130] "total_price_without_tax": "<Total price before tax of material 1>",

[0131] "unit_price_with_tax": "<Unit price after tax of material 1>",

[0132] "unit_price_without_tax": "<Unit price before tax of material 1>",

[0133] "product_amount": "<Quantity of material 1>"

[0134] },

[0135] {

[0136] "product_name": "<Name of material 2>",

[0137] "product_model": "<Model of material 2>",

[0138] "brand_name": "<Brand of material 2>",

[0139] "unit_name": "<Unit of measurement of material 2>",

[0140] "tax_rate": "<Tax rate 2>",

[0141] "total_price_with_tax": "<Total price after tax of material 2>",

[0142] "total_price_without_tax": "<Total price before tax of material 2>",

[0143] "unit_price_with_tax": "<Unit price after tax of material 2>",

[0144] "unit_price_without_tax": "<Unit price before tax of material 2>",

[0145] "product_amount": "<Quantity of material 2>"

[0146] }

[0147] 。

[0148] Finally, it should be noted that the above embodiments are only preferred embodiments of the present invention to illustrate the technical solutions of the present invention, rather than limiting them, and certainly not limiting the patent scope of the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. In other words, any changes or modifications made to the main design concept and spirit of the present invention that have no substantive significance, and the technical problems they solve are still consistent with the present invention, should be included in the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention in other related technical fields is also included in the patent protection scope of the present invention.

Claims

1. A method for extracting and aligning multi-format document table data based on a large language model, characterized in that: The following steps are involved: S1. Extracting a table from an editable document / non-editable document to obtain a table in HTML format; S2. Analyze the tables in HTML format based on the large language model and filter out useful tables; S3, using a large language model to align the selected useful tables according to the set standard fields; The S2 includes: S21, inputting the table in HTML format and the preset second prompt into the large language model, the large language model checking the data in the table item by item according to the second prompt, parsing the table title, the relationship between rows and columns, and the coherence and consistency of numerical or text entries, and outputting the parsing result; S22, filtering out the tables that do not meet the preset data quality standards or relevance according to the parsing result, and obtaining a useful table; The S3 includes: S31, inputting the useful form and the preset third prompt into the large language model. For each useful form, the large language model will identify each field in the form and convert different fields with synonymous or similar expressions into the same set standard field; S32, aligning multiple useful forms processed by S31 according to the standard fields.

2. The method for extracting and aligning multi-format document table data based on a large language model according to claim 1, characterized in that: The editable documents include Word documents, Excel documents and non-scanned PDF files. The table extraction of the editable documents includes: using Spire.Doc and DocX to extract the table from Word documents, using Xlrd, OpenpyXL and Pandas to extract the table from Excel documents, and using Spire.PDF and pdfplumber to extract the table from non-scanned PDF files to obtain a table in HTML format; The non-editable document includes a scanned PDF file, and the table extraction of the non-editable document includes: SA1, converting each page of the scanned PDF file into a picture; SA2, using a pre-trained Layout model to perform layout analysis on the picture to obtain coordinate information of the table in the picture; SA3, based on the obtained coordinate information, cutting out a table image from the picture; SA4, inputting all the cut table images into a table recognition model to obtain a table in HTML format.

3. The method for extracting and aligning multi-format document table data based on a large language model according to claim 1, characterized in that: The table extraction from the editable document / non-editable document includes the recognition and merging of cross-page tables, text translation of HTML tables, and table header recognition.

4. The method for extracting and aligning multi-format document table data based on a large language model according to claim 3 is characterized in that: The identification and merging of cross-page tables include the following operations: SB1, creating a temporary storage space; SB2, traversing each table of each page of the non-scanned PDF file or the scanned PDF file: SB21, if the table is not a page end table, directly output the current table; if the table is a page end table, save the current page end table in the temporary storage space, and traverse all tables on the next page; SB22, if the first table on the next page is a page header table and there is a table in the temporary storage space, and the number of columns of the page header table is consistent with the number of columns and format of the last table in the temporary storage space, save the current page header table in the temporary storage space, if the first table is also a page end table, continue to process the next page, if the first table is not a page header table, merge all tables in the temporary storage space in order and output them, and clear the temporary storage space, the output table is a complete cross-page table; SB23, repeat SB21 and SB22 until all cross-page tables are identified and merged.

5. The method for extracting and aligning multi-format document table data based on a large language model according to claim 4 is characterized in that: The judgment of whether the table is at the beginning or the end of a page includes the following operations: obtaining the page information of the non-scanned version of the PDF file and the layout information of the table coordinates from pdfplumber, or obtaining the page information of the scanned version of the PDF file and the layout information of the table coordinates from the Layout model; judging whether the table is at the beginning or the end of the page based on the layout information and the prior threshold.

6. The method for extracting and aligning multi-format document table data based on a large language model according to claim 3, characterized in that: The text translation of the HTML table includes the following operations: SC1, loading an editable document / non-editable document; SC2, traversing each page of the editable document / non-editable document: SC21, loading each chapter of the current page, traversing all tables in each chapter; SC22, for each table in the chapter, initializing an HTML table string and creating a table tag; SC23, traversing each column of the table, adding a row tag for each row of the table, traversing each row of the table, traversing all cells of the current row inside the row tag, removing extra spaces in the cells, wrapping the processed cell content in a table cell tag inside the row tag and adding it to the HTML table string, when all cells in a row are processed, closing the row tag, and when all rows are processed, closing the table tag.

7. The method for extracting and aligning multi-format document table data based on a large language model according to claim 3, characterized in that: The table header recognition includes the following operations: SD1, inputting the first 20 rows of data in the table and the preset first prompt into the large language model; The large language model performs semantic understanding on the first 20 rows of input data according to the first prompt, identifies the row number where the table header is located and obtains the index of the end position of the table header; SD1, determines and marks the starting and ending positions of the table header according to the index of the end position of the table header, and extracts the table header; SD2, selects the segmentation length, and divides the remaining table except the table header into multiple sub-tables according to the segmentation length; adds the table header to the head of each sub-table.

8. The method for extracting and aligning multi-format document table data based on a large language model according to claim 1, characterized in that: The S32 includes: S321, filling the data corresponding to the same standard field in each useful table into the current standard field, generating a new table, and completing the alignment of the standard fields.

Images