Cross-page table parsing method and system based on visual and semantic analysis

Abstract

The application belongs to the technical field of artificial intelligence and multi-modal data processing, and provides a cross-page table analysis method and system based on visual and semantic analysis; the method comprises the following steps: S1, a target detection model and a neural network model are used to identify a table boundary box and row and column lines; S2, visual feature extraction is performed on the identified table boundary box and row and column lines, and semantic feature extraction is performed between cells; S3, a knowledge graph is constructed according to the extracted visual features and semantic features, and it is judged whether cross-page table header inheritance is performed; S4, for the table with cross-page table header inheritance, the geometric and semantic association of cross-page cells is established, and the table is reconstructed to obtain a reconstructed table; S5, data consistency verification is performed on the reconstructed table; and S6, the verified overall table is parsed and output to an interactive interface. The application can improve the accuracy of table fuzzy boundary scene detection and cross-page table recognition.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and multimodal data processing technology, specifically relating to a method and system for parsing cross-page tables based on visual and semantic analysis. Background Technology

[0002] In existing document parsing solutions, there are generally two ways to handle tables. One is traditional OCR technology, which can only recognize single-page content and cannot capture the visual continuity of tables at page breaks; at the same time, recognizing and parsing the document content from left to right also causes the original positional information of the table to disappear. The other method relies on rule matching or simple deep learning models, which has difficulty handling irregular table layouts, such as nested tables and merged cells. Summary of the Invention

[0003] This invention aims to address the technical problems existing in the prior art by providing a cross-page table parsing method based on visual and semantic analysis, which can improve the accuracy of table fuzzy boundary scene detection and cross-page table recognition.

[0004] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0005] A cross-page table parsing method based on visual and semantic analysis includes the following steps:

[0006] S1. Use the object detection model to locate the candidate table region in the document, and use the neural network model to detect the fuzzy boundary of the candidate table region to identify the table bounding box and row and column lines.

[0007] S2. Visual features are extracted from the identified table bounding boxes and row and column lines, semantic features between cells are extracted, and the table header is further identified.

[0008] S3. Construct a knowledge graph based on the extracted visual and semantic features, and determine whether to perform cross-page header inheritance;

[0009] S4. For tables that inherit headers across pages, establish geometric and semantic relationships between cells across pages and reconstruct the table to obtain the reconstructed table.

[0010] S5. Perform data consistency verification on the reconstructed table;

[0011] S6. Parse the validated table and output it to the interactive interface.

[0012] Preferably, step S1 specifically includes:

[0013] Object detection stage: The YOLO v10 object detection model is used to locate candidate table regions in the document and output the bounding box coordinates with confidence scores.

[0014] Layout analysis stage: The candidate table region image slices are input into the Transformer model. Through the multi-head self-attention mechanism in the Transformer model, the internal structure of the candidate table is analyzed to obtain the table's row and column lines.

[0015] Preferably, a boundary enhancement attention module is obtained by designing a multi-head self-attention mechanism to detect the fuzzy boundaries of the candidate table region and process the fuzzy boundaries based on the detection results.

[0016] Preferably, the visual feature extraction in step S2 specifically involves: obtaining the output bounding box and row and column lines from step S1, and performing visual feature extraction, whereby the visual features include the cell coordinates of the table, border continuity, and line thickness.

[0017] Preferably, the semantic feature extraction in step S2 specifically involves: extracting text entities from the table and capturing contextual semantic information through a pre-trained large language model, and extracting the semantic features of cells in the table.

[0018] Preferably, the semantic features of a cell include contextual semantic information, the key of the table header, the value corresponding to the key of the table header, and the relationship between cells.

[0019] Preferably, step S2 further includes recognizing the visual features of the table and the semantic features between cells through a cross-attention mechanism, using the visual features and semantic features to jointly determine whether a cell is a table header, and recognizing and marking the table header.

[0020] Preferably, step S3 specifically involves: constructing a knowledge graph based on the relationship between the table header and each cell obtained in step S2. The node attributes of the knowledge graph include entity type, cell position coordinates, semantic vector of each cell content, consistency of table header content, and page continuity. Based on the knowledge graph, it is determined whether to perform cross-page table header inheritance. When it is determined to be yes, the table header of the previous page is automatically inherited.

[0021] Preferably, the knowledge graph includes nodes and lines. Nodes are cells, and lines represent the relationships between cells. Each node is marked with the cell's category. Based on the category, position coordinates, semantic vector, and page continuity of the nodes in the knowledge graph, a comprehensive judgment is made on whether to perform cross-page header inheritance.

[0022] Preferably, in step S4, a triple attention mechanism is constructed to model the geometric and semantic association of cross-page cells by constructing geometric similarity, semantic similarity and layout continuity. Using the results of the triple attention mechanism, a dynamic programming algorithm is used to optimize the table structure, automatically merge broken cells and reconstruct the table hierarchy.

[0023] Preferably, in step S5, the table reconstructed by the rule engine is used for data consistency verification, which specifically includes:

[0024] Numeric type validation: Perform numeric type validation on the KEY and corresponding data value VALUE of the same table header;

[0025] Logical relationship verification: Perform logical relationship verification on the KEY and corresponding data value VALUE of the same table header.

[0026] Preferably, step S6 further includes correcting the output table based on user feedback, specifically including:

[0027] Users can annotate the table correction areas in the interactive interface, and use the annotated data to incrementally train the large language model, thereby optimizing the knowledge graph and cross-page recognition results.

[0028] This invention provides a cross-page table parsing system based on visual and semantic analysis, including a processor capable of executing a computer program, which, when executed, implements the aforementioned cross-page table parsing method based on visual and semantic analysis.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] This invention discloses a method for parsing cross-page tables based on visual and semantic analysis. Through hybrid detection using YOLO v10 combined with the Transformer model, visual and semantic feature alignment, knowledge graph reasoning, and a dynamic optimization engine, it achieves end-to-end parsing of cross-page tables in complex documents. Compared to existing technologies, this invention significantly improves the accuracy of detecting blurred table boundaries and recognizing cross-page tables. Furthermore, this invention provides interactive iterative optimization capabilities, enhancing the practicality of the technology and possessing broad guiding and promotional significance. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating step S1 of an embodiment of the present invention;

[0032] Figure 2 This is a table obtained after the two-stage identification in step S1 of an embodiment of the present invention;

[0033] Figure 3 This is a flowchart illustrating step S2 of an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram illustrating step S2 of a table processing according to an embodiment of the present invention;

[0035] Figure 5 This is a flowchart illustrating step S3 of an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram illustrating step S3 of a table processing according to an embodiment of the present invention;

[0037] Figure 7 This is a flowchart illustrating step S4 of an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram illustrating step S4 of a table processing according to an embodiment of the present invention;

[0039] Figure 9 This is a flowchart illustrating step S5 of an embodiment of the present invention;

[0040] Figure 10 This is a flowchart illustrating step S6 of an embodiment of the present invention;

[0041] Figure 11 This is an example of a table before user correction in step S6 of an embodiment of the present invention;

[0042] Figure 12 Here is an example of a user-corrected table from step S6 of an embodiment of the present invention;

[0043] Figure 13 This is a flowchart of a cross-page table parsing method based on visual and semantic analysis, according to an embodiment of the present invention. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] Combination Figures 1-13 As shown, this embodiment of the invention provides a method for parsing cross-page tables based on visual and semantic analysis, including the following steps:

[0046] S1. Use the object detection model to locate the candidate table region in the document, and use the neural network model to detect the fuzzy boundary of the candidate table region to identify the table bounding box and row and column lines.

[0047] S2. Visual features are extracted from the identified table bounding boxes and row and column lines, semantic features between cells are extracted, and the table header is further identified.

[0048] S3. Construct a knowledge graph based on the extracted visual and semantic features, and determine whether to perform cross-page header inheritance;

[0049] S4. For tables that inherit headers across pages, establish geometric and semantic relationships between cells across pages and reconstruct the table to obtain the reconstructed table.

[0050] S5. Perform data consistency verification on the reconstructed table;

[0051] S6. Parse the validated table and output it to the interactive interface.

[0052] According to a specific embodiment of the present invention, step S1 specifically includes:

[0053] Object detection stage: The YOLO v10 object detection model is used to locate candidate table regions in the document and output the bounding box coordinates with confidence scores.

[0054] Layout analysis stage: The candidate table region image slices are input into the Transformer model. Through the multi-head self-attention mechanism in the Transformer model, the internal structure of the candidate table is analyzed to obtain the table's row and column lines.

[0055] According to a specific embodiment of the present invention, a boundary enhancement attention module is obtained by designing a multi-head self-attention mechanism to detect the fuzzy boundaries of the candidate table region and process the fuzzy boundaries according to the detection results.

[0056] According to a specific embodiment of the present invention, the visual feature extraction in step S2 specifically involves: obtaining the output bounding box and row and column lines in step S1, and performing visual feature extraction, wherein the visual features include the cell coordinates of the table, the continuity of the border, and the thickness of the lines.

[0057] According to a specific embodiment of the present invention, the semantic feature extraction in step S2 is specifically as follows: by using a pre-trained large language model, extracting text entities in the table and capturing contextual semantic information, and extracting semantic features of cells in the table.

[0058] According to a specific embodiment of the present invention, the semantic features of a cell include contextual semantic information, the key of the table header, the value VALUE corresponding to the key of the table header, and the association relationship between cells.

[0059] According to a specific embodiment of the present invention, step S2 further includes recognizing the visual features of the table and the semantic features between cells through a cross-attention mechanism, using the visual features and semantic features to jointly determine whether a cell is a table header, and recognizing and marking the table header.

[0060] According to a specific embodiment of the present invention, step S3 specifically involves: constructing a knowledge graph based on the relationship between the table header and each cell obtained in step S2. The node attributes of the knowledge graph include entity type, cell position coordinates, semantic vector of each cell content, consistency of table header content, and page continuity. Based on the knowledge graph, it is determined whether to perform cross-page table header inheritance. When it is determined to be yes, the table header of the previous page is automatically inherited.

[0061] According to a specific embodiment of the present invention, the knowledge graph includes nodes and lines, where nodes are cells and lines represent the relationships between cells. Each node is marked with the cell category. Based on the category, position coordinates, semantic vector, and page continuity of the nodes in the knowledge graph, a comprehensive judgment is made on whether to perform cross-page header inheritance.

[0062] According to a specific embodiment of the present invention, in step S4, a triple attention mechanism of geometric similarity, semantic similarity and layout continuity of cross-page cells is constructed to model the geometric and semantic association of cross-page cells. Using the results of the triple attention mechanism, a dynamic programming algorithm is used to optimize the table structure, automatically merge broken cells and reconstruct the table hierarchy.

[0063] According to a specific embodiment of the present invention, step S5, which establishes a rule engine to reconstruct the table and performs data consistency verification, specifically includes:

[0064] Numeric type validation: Perform numeric type validation on the KEY and corresponding data value VALUE of the same table header;

[0065] Logical relationship verification: Perform logical relationship verification on the KEY and corresponding data value VALUE of the same table header.

[0066] According to a specific embodiment of the present invention, step S6 further includes correcting the output overall table based on user feedback, specifically including:

[0067] Users can annotate the table correction areas in the interactive interface, and use the annotated data to incrementally train the large language model, thereby optimizing the knowledge graph and cross-page recognition results.

[0068] This invention provides a cross-page table parsing system based on visual and semantic analysis, including a processor capable of executing a computer program, which, when executed, implements the aforementioned cross-page table parsing method based on visual and semantic analysis.

[0069] Example 1

[0070] This invention provides a method for parsing cross-page tables based on visual and semantic analysis, including the following steps:

[0071] S1. Utilize an object detection model to locate candidate table regions in the document, and employ a neural network model to detect the fuzzy boundaries of these candidate table regions, identifying the table bounding boxes and row and column lines; for example... Figures 1-2 As shown;

[0072] This step uses a two-stage hybrid architecture of YOLO v10 + Transformer to perform preliminary parsing of the input document, identify the overall layout information of text, images, and tables, and obtain the position of the tables.

[0073] Object detection stage: The YOLO v10 object detection model is used to locate candidate table regions in the document and output the bounding box coordinates with confidence scores.

[0074] Layout analysis stage: The candidate table region image slices are input into the Transformer model. Through the multi-head self-attention mechanism in the Transformer model, the internal structure of the candidate table is analyzed to obtain the table's row and column lines.

[0075] S2. Visual features are extracted from the identified table bounding boxes and row and column lines, semantic features between cells are extracted, and the table header is further identified.

[0076] The visual feature extraction in step S2 specifically involves: obtaining the output bounding box and row and column lines from step S1, and extracting visual features, including the cell coordinates of the table, border continuity, and line thickness.

[0077] The semantic feature extraction in step S2 is as follows: using a pre-trained large language model, extract text entities from the table and capture contextual semantic information, and extract semantic features from the cells in the table.

[0078] The semantic features of a cell include contextual semantic information, the header key KEY, the value VALUE corresponding to the header key KEY, and the relationships between cells.

[0079] Step S2 also includes recognizing the visual features and semantic features between cells of the table through a cross-attention mechanism, jointly determining whether a cell is a table header using visual and semantic features, and identifying and labeling the table header. Visual geometric features (coordinates, lines) and semantic understanding features (text entities, context) are jointly modeled to improve the robustness of cross-page tables. For example... Figures 3-4 As shown.

[0080] Visual features are similar to screenshots, determined by the top, bottom, left, and right positions. Semantic features include the relationship between the table header and the corresponding cell, i.e., key and value. If the two are consistent, then the cell is determined to be the table header.

[0081] Each table category has a threshold. After comparing against the threshold, both visual and semantic features will have a result. Threshold setting needs to consider three aspects: 1. Ensure that the difference between the results of visual and semantic features is not too large, i.e., the variance cannot be too large; 2. The average values ​​of visual and semantic features cannot be too small; 3. Special cases.

[0082] S3. Construct a knowledge graph based on the extracted visual and semantic features, and determine whether to perform cross-page header inheritance; for example... Figures 5-6 As shown.

[0083] Step S3 specifically involves: constructing a knowledge graph based on the table header and the relationships between cells obtained in step S2. The node attributes of the knowledge graph include entity type, cell position coordinates, semantic vector of each cell content, consistency of table header content, and page continuity. Based on the knowledge graph, it is determined whether to perform cross-page table header inheritance. If the determination is yes, the table header of the previous page is automatically inherited.

[0084] A knowledge graph consists of nodes and lines. Nodes are cells, and lines represent the relationships between cells. Knowledge graph nodes contain attributes such as cell category (KEY / VALUE / RELATION, key / corresponding value / relationship), location coordinates, semantic vector, header content consistency, and page continuity. Based on the category, location coordinates, semantic vector, and page continuity of the nodes in the knowledge graph, a comprehensive judgment is made on whether to perform cross-page header inheritance.

[0085] S4. For tables with header inheritance that spans multiple pages, establish geometric and semantic relationships between the spanning cells and reconstruct the table to obtain the reconstructed table; such as Figures 7-8 As shown.

[0086] In step S4, a triple attention mechanism is constructed to model the geometric and semantic associations of cross-page cells by constructing geometric similarity, semantic similarity, and layout continuity. Using the results of the triple attention mechanism, a dynamic programming algorithm is used to optimize the table structure, automatically merge broken cells, and reconstruct the table hierarchy.

[0087] S5. Perform data consistency checks on the reconstructed tables; such as... Figure 9 As shown.

[0088] Establish a rules engine to verify the data consistency of the overall table after association, thus ensuring the accuracy of parsing. This includes:

[0089] Numeric type validation: Perform numeric type validation on the KEY and corresponding data value VALUE of the same table header, such as numeric type or date type;

[0090] Logical relationship validation: Perform logical relationship validation on the KEY and corresponding data value VALUE of the same header, such as matching amount with quantity unit.

[0091] S6. Parse the validated table and output it to the interactive interface.

[0092] Step S6 also includes making corrections to the overall output table based on user feedback, such as... Figures 10-12 As shown, it specifically includes:

[0093] Users can annotate areas in the table within the interactive interface to correct errors. This corrected data is then used for incremental training of the large language model, optimizing the knowledge graph and cross-page recognition results. This serves two purposes: firstly, to improve the accuracy of the parsing; and secondly, to incrementally update the knowledge graph and model parameters with user feedback data, enabling iterative model optimization.

[0094] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-page table parsing method based on visual and semantic analysis, characterized in that, Includes the following steps: S1. Use the object detection model to locate the candidate table region in the document, and use the neural network model to detect the fuzzy boundary of the candidate table region to identify the table bounding box and row and column lines. S2. Visual features are extracted from the identified table bounding boxes and row and column lines, semantic features between cells are extracted, and the table header is further identified. S3. Construct a knowledge graph based on the extracted visual and semantic features, and determine whether to perform cross-page header inheritance; S4. For tables that inherit headers across pages, establish geometric and semantic relationships between cells across pages and reconstruct the table to obtain the reconstructed table. S5. Perform data consistency verification on the reconstructed table; S6. Parse the validated table and output it to the interactive interface; Specifically, step S3 involves constructing a knowledge graph based on the relationship between the table header and each cell obtained in step S2. The node attributes of the knowledge graph include entity type, cell position coordinates, semantic vector of each cell content, consistency of table header content, and page continuity. Based on the knowledge graph, it is determined whether to perform cross-page table header inheritance. If the determination is yes, the table header of the previous page is automatically inherited. The knowledge graph consists of nodes and lines. Nodes are cells, and lines represent the relationships between cells. Each node is marked with the cell's category. Based on the category, position coordinates, semantic vector, and page continuity of the nodes in the knowledge graph, a comprehensive judgment is made on whether to perform cross-page header inheritance.

2. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, Step S1 specifically includes: Object detection stage: The YOLO v10 object detection model is used to locate candidate table regions in the document and output the bounding box coordinates with confidence scores. Layout analysis stage: The candidate table region image slices are input into the Transformer model. Through the multi-head self-attention mechanism in the Transformer model, the internal structure of the candidate table is analyzed to obtain the table's row and column lines.

3. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, The visual feature extraction in step S2 specifically involves: obtaining the output bounding box and row and column lines from step S1, and extracting visual features, including the cell coordinates of the table, border continuity, and line thickness.

4. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, The semantic feature extraction in step S2 is as follows: using a pre-trained large language model, extract text entities from the table and capture contextual semantic information, and extract semantic features from the cells in the table.

5. The cross-page table parsing method based on visual and semantic analysis according to claim 4, characterized in that, The semantic features of a cell include contextual semantic information, the header key KEY, the value VALUE corresponding to the header key KEY, and the relationships between cells.

6. The cross-page table parsing method based on visual and semantic analysis according to claim 5, characterized in that, Step S2 also includes recognizing the visual features of the table and the semantic features between cells through a cross-attention mechanism, using visual features and semantic features to jointly determine whether a cell is a table header, and recognizing and marking the table header.

7. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, In step S4, a triple attention mechanism is constructed to model the geometric and semantic associations of cross-page cells by constructing geometric similarity, semantic similarity, and layout continuity. Using the results of the triple attention mechanism, a dynamic programming algorithm is used to optimize the table structure, automatically merge broken cells, and reconstruct the table hierarchy.

8. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, Step S5 involves establishing a rule engine to reconstruct the table and performing data consistency verification, specifically including: Numeric type validation: Perform numeric type validation on the KEY and corresponding data value VALUE of the same table header; Logical relationship verification: Perform logical relationship verification on the KEY and corresponding data value VALUE of the same table header.

9. The cross-page table parsing method based on visual and semantic analysis according to claim 1, characterized in that, Step S6 also includes making user feedback corrections to the overall output table, specifically including: Users can annotate the table correction areas in the interactive interface, and use the annotated data to incrementally train the large language model, thereby optimizing the knowledge graph and cross-page recognition results.

10. A cross-page table parsing system based on visual and semantic analysis, characterized in that, The system includes a processor capable of executing a computer program that, when executed, implements the cross-page table parsing method based on visual and semantic analysis as described in any one of claims 1-9.

Images