Method for identifying cross-page table in document

Abstract

The invention discloses a method for identifying a cross-page table in a document, which comprises the following steps of: detecting a table region in a document page picture, and cutting the table region to obtain a table picture; performing table structure identification on the cut table picture; setting the bottom of the first page as a first table and the top of the second page as a second table; the first page and the second page are front and back adjacent pages; and if the header of the first table and the header of the second table are completely the same, or the last line of the first table and the first line of the second table have the same table structure and the cell content of each same column belongs to the same entity category, judging that the first table and the second table are cross-page tables. According to the invention, cross-page table identification can be realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method for recognizing cross-page tables in a document. Background Technology

[0002] In the multimodal understanding capabilities of large models, such as Large Language Models (LLMs), document (e.g., PDF documents) comprehension and learning, or document comprehension dialogue, is a very important area. For document comprehension, the first important step is to convert all the information within the document into plain text format, such as HTML5 or Markdown.

[0003] In PDF document comprehension, table understanding is a crucial task, especially for tables spanning multiple pages. Because these tables are located on different pages, current technologies typically identify them as separate tables on different pages and store them in different data chunks. This affects the LLM's ability to answer factual questions based on tables. Furthermore, the inability to accurately extract information from cross-page tables during LLM training also impacts its ability to answer similar questions. Summary of the Invention

[0004] The technical problem solved by this invention is to provide a method for identifying cross-page tables in documents. This method can identify the relationship between tables on adjacent pages, determine whether tables on different pages are cross-page tables, and merge their structures if so. This solves the problem that LLM (Limited Language Management) systems could not effectively read and understand cross-page tables in the past, thus improving the accuracy of LLM in understanding and interpreting documents.

[0005] Firstly, this application provides a method for recognizing cross-page tables in a document, including:

[0006] S1. Table position detection: Detect the table area within the document page image, crop it out, and obtain the table image.

[0007] S2. Table Structure Recognition; Perform table structure recognition on the cropped table image;

[0008] S3, Detection of tables spanning multiple pages;

[0009] S3.1 Structural Judgment

[0010] Let the bottom of the first page be the first table and the top of the second page be the second table; the first page and the second page are adjacent pages.

[0011] Determine if the second table has a header;

[0012] If the second table has a header, then determine whether the headers of the first table and the second table are exactly the same; if they are exactly the same, then determine that the first table and the second table are cross-page tables; otherwise, determine that the first table and the second table are non-cross-page tables.

[0013] If the second table does not have a header, determine whether the last row of the first table and the first row of the second table have the same table structure; if they do not have the same table structure, determine that the first table and the second table are not cross-page tables; otherwise, go to step S3.2.

[0014] S3.2, Semantic Relationship Judgment;

[0015] Using a trained Named Entity Recognition (NER) model, named entity recognition is performed on the cell content of each identical column in the last row of the first table and the first row of the second table, and the entity category is determined. If all the cell content of the identical columns belongs to the same entity category, then the first table and the second table are determined to be cross-page tables.

[0016] In one possible implementation of the first aspect, the method further includes:

[0017] S4. Table Merging: Merges information from tables spanning multiple pages to obtain a merged table.

[0018] 3. The method for recognizing cross-page tables in documents according to claim 2, characterized in that the method further includes:

[0019] S5. Table format conversion;

[0020] Convert the merged table information into HTML format to generate standard HTML table tags. 、 、 、 The structure is ) where tr represents a row span, td represents a cell, and th represents a table header, which is obtained from the table structure identification.

[0021] In one possible implementation of the first aspect, the method further includes:

[0022] If the first and second tables are determined to be cross-page tables, and the entire second page consists of the second table without a header, and the top of the third page is the third table; and the second and third pages are adjacent pages; then the header of the first table is considered as the header of the second table, and thus it is determined whether the second and third tables are cross-page tables.

[0023] In one possible implementation of the first aspect, the entity categories include: long text, short text, cardinal number, date, event, facility, geopolitical entity, language name, law / bill, other location, currency amount, ethnic / religious or political group, ordinal number, organization or company name, percentage, personal name, product name, quantity, time, and work.

[0024] Secondly, this application provides a system for recognizing cross-page tables in a document, including: a table position detection module, a table structure recognition module, and a cross-page table judgment module;

[0025] The table position detection module is used to detect the table area in the document page image and crop it out to obtain a table image;

[0026] The table structure recognition module is used to recognize the table structure of the cropped table image;

[0027] The cross-page table judgment module is used for structure judgment, including:

[0028] Let the bottom of the first page be the first table and the top of the second page be the second table; the first page and the second page are adjacent pages.

[0029] Determine if the second table has a header;

[0030] If the second table has a header, then determine whether the headers of the first table and the second table are exactly the same; if they are exactly the same, then determine that the first table and the second table are cross-page tables; otherwise, determine that the first table and the second table are non-cross-page tables.

[0031] If the second table does not have a header, determine whether the last row of the first table and the first row of the second table have the same table structure; if they do not have the same table structure, determine that the first table and the second table are not cross-page tables; otherwise, go to step S3.2.

[0032] The cross-page table judgment module is also used for semantic relationship judgment;

[0033] Using a trained Named Entity Recognition (NER) model, named entity recognition is performed on the cell content of each identical column in the last row of the first table and the first row of the second table, and the entity category is determined. If all the cell content of the identical columns belongs to the same entity category, then the first table and the second table are determined to be cross-page tables.

[0034] In one possible implementation of the second aspect, the system further includes:

[0035] The table merging module is used to merge information from tables spanning multiple pages to obtain merged table information;

[0036] In one possible implementation of the second aspect, the system further includes:

[0037] The table format conversion module is used to convert merged table information into HTML format, generating standard HTML table tags. 、 、 、 The structure is ) where tr represents a row span, td represents a cell, and th represents a table header, which is obtained from the table structure identification.

[0038] In one possible implementation of the second aspect, the cross-page table determination module is further configured to:

[0039] If the first and second tables are determined to be cross-page tables, and the entire second page consists of the second table without a header, and the top of the third page is the third table; and the second and third pages are adjacent pages; then the header of the first table is considered as the header of the second table, and thus it is determined whether the second and third tables are cross-page tables.

[0040] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0041] The memory is used to store computer programs;

[0042] The processor is used to invoke the computer program to execute the method described above.

[0043] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed on an electronic device, causes the electronic device to perform the method described above.

[0044] Fifthly, this application provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0045] The specific implementation methods of the second to fifth aspects of this application can refer to the implementation methods of the first aspect, and will not be elaborated here.

[0046] Beneficial effects:

[0047] This application identifies the relationships between tables on adjacent pages through table structure recognition and cell named entity semantic relationship recognition, determining whether tables on different pages are cross-page tables and merging them if so. This solves the problem of LLM (Limited Language Management) systems being unable to effectively read and understand cross-page tables, improving the accuracy of LLM document comprehension. Based on this application, large models can be used to parse cross-page tables in PDF documents with excellent results. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating one embodiment of the present application. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0050] It should be noted that terms such as "first" and "second" in the specification, claims, and accompanying drawings of this application are used merely to distinguish one entity or operation from another, and do not necessarily require or imply a specific relationship or order between these entities or operations. It should be understood that the words "first" and "second" do not limit the quantity or order of execution, and that "first" and "second" are not necessarily different. It should be understood that such data used can be interchanged where appropriate. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements explicitly listed, but also other elements not explicitly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0051] The following will refer to Figure 1 A specific implementation method according to this application is described.

[0052] like Figure 1 The diagram shown is a flowchart of an embodiment of this application.

[0053] This application provides a method for recognizing cross-page tables in a document, including:

[0054] S1, Table position detection;

[0055] A table detection model was trained using YOLOv5 to detect table regions in document page images and then cropped out to obtain table images.

[0056] S2, Table structure recognition;

[0057] Perform table structure recognition on the cropped table image;

[0058] The algorithm for table structure recognition can utilize existing technology, referring to the table structure recognition method described in the patent application document CN202310724082.3 entitled "A Method, Apparatus, Device and Storage Medium for Text Recognition", to recognize the table structure.

[0059] S3, Detection of tables spanning multiple pages;

[0060] S3.1 Structural Judgment

[0061] Let the bottom of the first page be the first table and the top of the second page be the second table; the first page and the second page are adjacent pages.

[0062] Determine if the second table has a header;

[0063] If the second table has a header, then determine whether the headers of the first table and the second table are exactly the same; if they are exactly the same, then determine that the first table and the second table are cross-page tables; otherwise, determine that the first table and the second table are non-cross-page tables.

[0064] The presence or absence of a header in the second table is the simplest and most common type of table spanning multiple pages.

[0065] A cross-page table means that although two tables are on different pages, they are actually the same table.

[0066] If the second table does not have a header, determine whether the last row of the first table and the first row of the second table have the same table structure; if they do not have the same table structure, determine that the first table and the second table are not cross-page tables; otherwise, go to step S3.2.

[0067] These tables share the same table structure, such as having the same number of columns.

[0068] S3.2, Semantic Relationship Judgment;

[0069] Using a trained Named Entity Recognition (NER) model, named entity recognition (semantic relationship judgment) is performed on the cell content of each identical column in the last row of the first table and the first row of the second table, and the entity category is determined. If the cell content of all identical columns belongs to the same entity category, then the first table and the second table are determined to be cross-page tables.

[0070] In some embodiments, the Ner model is obtained by pre-training and fine-tuning BERT on a large amount of data.

[0071] In some embodiments, the entity category includes:

[0072] LONGTEXT: Long text, specifically refers to text or paragraphs with relatively long content (e.g., "At 4:30 a.m., the crabapple blossoms are still awake..."). The length of the text can also be used to help determine its length.

[0073] SHORTTEXT: Short text, specifically referring to shorter text that is not included in the Ner entity below. (e.g., "chef", "rose", "cell phone")

[0074] CARDINAL: Cardinal number, a number representing a quantity (e.g., "five", "200");

[0075] DATE: Date (e.g., "September 19, 2024", "yesterday");

[0076] EVENT: event, major event, or holiday (e.g., "Olympic Games", "Christmas");

[0077] FAC: Facilities, such as buildings, airports, bridges, etc. (e.g., "Great Wall", "Tokyo Airport");

[0078] GPE: Geopolitical Entity (country, etc.) (e.g., "China");

[0079] LANGUAGE: Language name (e.g., "Chinese", "English");

[0080] LAW: Laws, Acts (e.g., "Consumer Rights Protection Act");

[0081] LOC: Other locations (such as mountains, rivers) (e.g., "Himalayan Mountains", "Yellow River");

[0082] MONEY: Currency amount (e.g., "100 yuan", "5 US dollars");

[0083] NORP: Ethnic, religious, or political groups (e.g., "Han nationality", "Buddhist Association of China");

[0084] ORDINAL: Ordinal number, a number indicating sequence (e.g., "first", "second");

[0085] ORG: Organization or company name (e.g., "Google", "United Nations");

[0086] PERCENT: Percentage (e.g., "50%", "80%");

[0087] PERSON: A person's name (e.g., "Zhang San", "Alice");

[0088] PRODUCT: Product name (e.g., "iPhone", "Toyota");

[0089] QUANTITY: quantity, referring to a specific unit of measurement (e.g., "10 kilograms", "3 kilometers");

[0090] TIME: time, specific time (e.g., "3 p.m.", "8 p.m.");

[0091] WORK_OF_ART: Works, including books, movies, paintings, etc. (e.g., "Starry Night", "Titanic");

[0092] The above entity categories can basically include all table content entity types.

[0093] For example, the first table is:

[0094] Li Si 4415211232 135xxxxx Shenzhen, Guangdong Province Backend engineer

[0095] The second table is as follows:

[0096] Zhang San 4415211231 135xxxxX Shanwei City, Guangdong Province Algorithm Engineer

[0097] The first table above has 5 columns, and the second table also has 5 columns. Based on the above steps, it can be determined that they have the same table structure.

[0098] The entity category information corresponding to the first table above is: PERSON, CARDINAL, CARDINAL, LOC, SHORTTEXT;

[0099] The entity category information corresponding to the second table above is: PERSON, CARDINAL, CARDINAL, LOC, SHORTTEXT;

[0100] Therefore, the first table and the second table are determined to have the same table structure through the above steps, and the cell content of all the same columns belongs to the same entity category. Thus, the first table and the second table are determined to be cross-page tables.

[0101] In some embodiments, the above method further includes:

[0102] S4. Merge tables;

[0103] Merge the information from tables spanning multiple pages to obtain the merged table information;

[0104] You can convert the first table and the second table into two different DataFrames, and then add the contents of the second DataFrame directly to the first DataFrame.

[0105] A DataFrame is a data structure in Python's Pandas library. A DataFrame is a two-dimensional table data structure, consisting of rows and columns of data.

[0106] In some embodiments, the above method further includes:

[0107] S5. Table format conversion;

[0108] Convert the merged table information into HTML format to generate standard HTML table tags. 、 、 、 The structure is as follows: tr represents a row span, td represents a cell, and th represents the table header, which is determined by the table structure.

[0109] After being identified and pieced together, the table can be fed into a large model, where it can be read and used for corresponding understanding and learning or document comprehension dialogue, or for training the large model.

[0110] In some embodiments, if the first table and the second table are determined to be cross-page tables, and the entire second page consists of the second table without a header, and the top of the third page is the third table; and the second and third pages are adjacent pages; then the header of the first table is considered the header of the second table, thereby determining whether the second and third tables are cross-page tables. The specific method is as follows:

[0111] If the third table does not have a header, then it is determined whether the last row of the second table and the first row of the third table have the same table structure. If they do not have the same table structure, then the second and third tables are determined to be non-page-crossing tables. Otherwise, semantic relationship judgment is performed. That is, using the trained named entity recognition model, named entity recognition is performed on the content of each cell in the same column in the last row of the second table and the first row of the third table, and the entity category is determined. If the content of all cells in the same column belongs to the same entity category, then the second and third tables are determined to be page-crossing tables.

[0112] Therefore, it is possible to identify and merge tables that span multiple consecutive pages in a document.

[0113] This application also provides an electronic device, including: a memory and a processor;

[0114] The memory is used to store computer programs;

[0115] The processor is used to invoke the computer program to execute the method described above.

[0116] This application also provides a computer-readable storage medium storing a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0117] This application also provides a computer program product, including a computer program that, when run on an electronic device, causes the electronic device to perform the method described above.

[0118] This application also provides specific implementations of a system, electronic device, computer-readable storage medium, and computer program product. These specific implementations can be referred to in the above-described methods and will not be repeated here.

[0119] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0120] The above description of the embodiments of this application is only a part of the embodiments of this application, and is used to enable those skilled in the art to implement or use the content of this application, and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for recognizing cross-page tables in a document, characterized in that, include: S1, Table position detection; The table area within the document page image is detected and cropped out to obtain the table image; S2, Table structure recognition; Perform table structure recognition on the cropped table image; S3, Detection of tables spanning multiple pages; S3.1 Structural Judgment Let the bottom of the first page be the first table and the top of the second page be the second table; the first page and the second page are adjacent pages. Determine if the second table has a header; If the second table has a header, then determine whether the headers of the first table and the second table are exactly the same; If they are exactly the same, then the first table and the second table are determined to be cross-page tables; otherwise, the first table and the second table are determined to be non-cross-page tables. If the second table does not have a header, determine whether the last row of the first table and the first row of the second table have the same table structure; if they do not have the same table structure, determine that the first table and the second table are not cross-page tables; otherwise, go to step S3.

2. S3.2, Semantic Relationship Judgment; Using a trained Named Entity Recognition (NER) model, named entity recognition is performed on the cell content of each identical column in the last row of the first table and the first row of the second table, and the entity category is determined. If all the cell content of the identical columns belongs to the same entity category, then the first table and the second table are determined to be cross-page tables.

2. The method for recognizing cross-page tables in documents according to claim 1, characterized in that, The method further includes: S4. Table Merging: Merges information from tables spanning multiple pages to obtain a merged table.

3. The method for recognizing cross-page tables in documents according to claim 2, characterized in that, The method further includes: S5. Table format conversion; Convert the merged table information into HTML format to generate standard HTML table tags. 、 、 、 The structure is ) where tr represents a row span, td represents a cell, and th represents a table header, which is obtained from the table structure identification.

4. The method for recognizing cross-page tables in documents according to claim 1, characterized in that, The method further includes: If the first and second tables are determined to be cross-page tables, and the entire second page consists of the second table without a header, and the top of the third page is the third table; and the second and third pages are adjacent pages; then the header of the first table is considered as the header of the second table, and thus it is determined whether the second and third tables are cross-page tables.

5. A system for recognizing cross-page tables in documents, characterized in that, include: Table position detection module, table structure recognition module, and cross-page table judgment module; The table position detection module is used to detect the table area in the document page image and crop it out to obtain a table image; The table structure recognition module is used to recognize the table structure of the cropped table image; The cross-page table judgment module is used for structure judgment, including: Let the bottom of the first page be the first table and the top of the second page be the second table; the first page and the second page are adjacent pages. Determine if the second table has a header; If the second table has a header, then determine whether the headers of the first table and the second table are exactly the same; if they are exactly the same, then determine that the first table and the second table are cross-page tables; otherwise, determine that the first table and the second table are non-cross-page tables. If the second table does not have a header, determine whether the last row of the first table and the first row of the second table have the same table structure; if they do not have the same table structure, determine that the first table and the second table are not cross-page tables; otherwise, go to step S3.

2. The cross-page table judgment module is also used for semantic relationship judgment; Using a trained Named Entity Recognition (NER) model, named entity recognition is performed on the cell content of each identical column in the last row of the first table and the first row of the second table, and the entity category is determined. If all the cell content of the identical columns belongs to the same entity category, then the first table and the second table are determined to be cross-page tables.

6. The document cross-page table recognition system according to claim 5, characterized in that, The system also includes: The table merging module is used to merge information from tables spanning multiple pages to obtain a merged table.

7. The method for recognizing cross-page tables in documents according to claim 6, characterized in that, The system also includes: The table format conversion module is used to convert merged table information into HTML format, generating standard HTML table tags. 、 、 、 The structure is ) where tr represents a row span, td represents a cell, and th represents a table header, which is obtained from the table structure identification.

8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to perform the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 4.

Images