Complex table intelligent analysis method based on multi-modal fusion and semantic analysis

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a complex table intelligent analysis method and system based on multi-modal fusion and semantic analysis. The method comprises the following steps: S1, performing preliminary analysis on a document based on a multi-modal fusion technology, identifying position information of a table area, a text and an image, and outputting the position information of the table area; s2, reconstructing the table structure based on the topological graph structure and the geometrical relationship of the table to obtain a reconstructed table; s3, identifying the content of the reconstructed table by utilizing machine vision, and correcting the identified content of the table and the position information of the table area in combination with semantics to obtain all the content of table analysis; and S4, outputting the analyzed table content according to a required format. According to the method, the complex table can be analyzed and corrected.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a complex table intelligent analysis method based on multi-modal fusion and semantic analysis. BACKGROUND

[0002] In the existing document analysis scheme, there are generally two processing methods for encountering a table. One is to ignore the existence of the table and directly recognize and analyze the content of the document from left to right. This case will cause the original position information of the document table to disappear, and because the reading order from left to right cannot reflect the relationship between the table cells, the semantic information of the document is also lost. The other is to identify the table structure according to M*N, which is poor in recognition for continuous tables across pages, and low in recognition accuracy for non-M*N regular tables, and even cannot recognize in some cases. N to identify the table structure, which is poor in recognition for continuous tables across pages, and low in recognition accuracy for non-M*N regular tables, and even cannot recognize in some cases. SUMMARY

[0003] The application aims at the technical problems in the prior art, and provides a complex table intelligent analysis method based on multi-modal fusion and semantic analysis, which can analyze and correct the multi-modal complex table, and improves the correctness of the table.

[0004] To achieve the above technical purposes, the application provides a complex table intelligent analysis method based on multi-modal fusion and semantic analysis, comprising the following steps: S1, performing preliminary analysis on the document based on a multi-modal fusion technology to identify the position information of the table region, text and image, and output the position information of the table region; S2, reconstructing the table structure based on the topological graph structure and geometric relationship of the table to obtain a reconstructed table; S3, recognizing the content of the reconstructed table by using machine vision, correcting the recognized table content and the position information of the table region in combination with semantics to obtain all the contents of the table analysis; S4, outputting the analyzed table content in a required format.

[0005] Preferably, in step S1, the multi-modal fusion technology uses a deep learning network to perform preliminary analysis on the document.

[0006] Preferably, the deep learning network is a fusion network of CNN and Transformer.

[0007] Preferably, in step S1, the output table region position information includes the boundary box of the table and the coordinates of the initial cells.

[0008] Preferably, step S2 reconstructs the irregular table, specifically comprising the following steps: S201, construct a topological graph structure of the table; S202, process the cells across rows and / or columns in the topological graph structure of the table, and correct the topological graph structure; S203, supplement the part missing the boundary line in the topological graph structure of the table, and output the reconstructed table.

[0009] Preferably, in step S201, a graph model of the table is constructed by taking the table cells as nodes and the merging and adjacent edge relationship as edges.

[0010] Preferably, in step S202, the cells across rows and / or columns in the topological graph structure of the table are processed based on a spatial clustering algorithm, and the topological graph structure is corrected in combination with the diagonal line detection in the table content.

[0011] Preferably, in step S203, a virtual dividing line is supplemented to the part missing the boundary line in the topological graph structure of the table, so as to obtain the reconstructed table.

[0012] Preferably, in step S3, the case that the key and the value are in the same cell is identified and corrected by means of semantics, and the consistency of the semantic content of the cell is verified by means of text features.

[0013] Preferably, in step S3, the attribution relationship between the content of the cell and the table header is corrected by means of semantic understanding.

[0014] Preferably, in step S4, the output table format includes one of json, Excel and XML.

[0015] The application also provides a complex table intelligent analysis system based on multi-modal fusion and semantic analysis, comprising a processor, the processor can execute a computer program, the computer program can realize the complex table intelligent analysis method based on multi-modal fusion and semantic analysis when executed.

[0016] Compared with the prior art, the application has the following beneficial effects: 1. After visual detection, the application corrects the table by means of semantics to obtain an accurately analyzed table; by jointly modeling the text semantics and the table position information, the robustness of table recognition is improved.

[0017] 2. The application preliminarily analyzes the document by means of multi-modal fusion technology, identifies the position information of the table area, text and image, and outputs the position information of the table area, so as to facilitate the subsequent.

[0018] 3. The present application gradually and continuously corrects the table obtained by preliminary analysis through mechanisms such as reconstruction of the topological relationship of irregular tables (there are cross-row / column cases, and there are key and value and diagonal and other special cases in the same cell), semantic-level repair, etc. Finally, the correctness of table analysis is greatly improved, which has wide guiding significance and popularization significance. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The red box part is the table obtained by preliminary analysis of an embodiment of the present application; Figure 2 The example diagram before table structure reconstruction of an embodiment of the present application; Figure 3 The example diagram after table structure reconstruction of an embodiment of the present application; Figure 4 The example diagram before table text content repair of an embodiment of the present application; Figure 5 The example diagram after table text content repair of an embodiment of the present application; Figure 6 The flowchart of the complex table intelligent analysis method based on multi-modal fusion and semantic analysis of an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0021] As shown in Figures 1-6 The present application provides a complex table intelligent analysis method based on multi-modal fusion and semantic analysis, which comprises the following steps: S1, performing preliminary analysis on the document based on multi-modal fusion technology, identifying the position information of the table area, text and image, and outputting the position information of the table area; S2, reconstructing the table structure based on the topological graph structure and geometric relationship of the table, and obtaining the reconstructed table; S3, identifying the content of the reconstructed table by machine vision, combining semantics to correct the identified content of the table and the position information of the table area, and obtaining the complete content of the table analysis; S4, outputting the analyzed table content in the required format.

[0022] In an embodiment of the present application, in step S1, the multi-modal fusion technology utilizes a deep learning network to perform preliminary analysis on the document.

[0023] In an embodiment of the present application, the deep learning network is a fusion network of CNN and Transformer.

[0024] In an embodiment of the present application, in step S1, the output table region position information includes a boundary box of the table and coordinates of initial cells.

[0025] In an embodiment of the present application, step S2 reconstructs the irregular table, specifically including the following steps: S201, constructing a topological graph structure of the table; S202, processing cells across rows and / or columns in the topological graph structure of the table, and correcting the topological graph structure; S203, supplementing parts missing boundary lines in the topological graph structure of the table, and outputting the reconstructed table.

[0026] In an embodiment of the present application, step S201 constructs a graph model of the table, taking table cells as nodes and merging and adjacent edge relationships as edges.

[0027] In an embodiment of the present application, step S202 processes cells across rows and / or columns in the topological graph structure of the table based on a spatial clustering algorithm, and corrects the topological graph structure in combination with diagonal line detection in the table content.

[0028] In an embodiment of the present application, step S203 supplements a virtual dividing line to parts missing boundary lines in the topological graph structure of the table, to obtain the reconstructed table.

[0029] In an embodiment of the present application, in step S3, the case where the key and the value are in the same cell is corrected by means of semantics, and the consistency of the cell content semantics is verified by text features.

[0030] In an embodiment of the present application, in step S3, the attribution relationship between the content of the cell and the table header is corrected by means of semantic understanding.

[0031] In an embodiment of the present application, in step S4, the output table format includes one of json, Excel, and XML.

[0032] The present invention also provides a complex table intelligent parsing system based on multimodal fusion and semantic analysis, including a processor, the processor being able to execute a computer program, the computer program being able to implement the above-mentioned complex table intelligent parsing method based on multimodal fusion and semantic analysis when executed.

[0033] Example 1 This invention provides an intelligent parsing method for complex tables based on multimodal fusion and semantic analysis, comprising the following steps: (1) Preliminary analysis based on multimodal fusion technology to identify table regions. Multimodal fusion technology uses deep learning models (such as CNN+Transformer fusion network) to perform preliminary analysis of the document, identifying the overall layout information of text, images, and tables, such as... Figure 1 As shown. Output the position information of the table area, such as the coordinates of the table's bounding box and the initial cell. (2) Reconstruct irregular tables based on their topological structure and geometric relationships.

[0034] First, we need to build a graph structure model of the table cells, where nodes are cells and edges represent merge / adjacent relationships; Then, based on the spatial clustering algorithm, cross-row / column cells are processed, and the graph structure is corrected by combining diagonal line detection; Spatial clustering is essentially about "automatically organizing location-based data," and the specific processing method is as follows: First, based on the cell's inherent position information (including the visual position information of the layout document and the position information of the flow document), we determine that cells spanning multiple rows are similar in position and then initially group them together. Second, based on the semantic information of the text within the cell, further determine whether the semantics are coherent and group them together.

[0035] The method for detecting and correcting diagonal lines is as follows: First, determine the existence of the diagonal line based on visual positional information; Second, for cases with diagonal lines, cells with diagonal lines are initially defined as "double headers," meaning they are both horizontal and vertical headers. Third, based on semantic information, re-verify the semantic relationship between the horizontal and vertical headers and their corresponding sub-cells. If the semantics are consistent, it can be determined that the header is a double header; if the semantics are inconsistent, further judgment is made by examining the relationship between the cell and its surrounding cells, as well as some special cases (such as whether it is a printing error, whether it is a page crease, etc.).

[0036] Finally, for tables with missing boundary lines, add virtual dividing lines. Output the reconstructed table structure.

[0037] The original table before reconstruction is shown in Figure 2 The table after reconstruction is shown in Figure 3 .

[0038] (3) The content of the table is recognized by machine vision. Any machine vision algorithm in the prior art can be used as long as it can recognize text. The specific algorithm is not limited in the present application. Then, the reconstructed table is corrected and repaired in combination with semantics. The recognition of the content and structure of the table by machine vision may have recognition errors. For example, the key and value in a cell such as "total price 100 yuan" may be recognized as a value. At this time, the semantic consistency of the cell content needs to be verified by text features.

[0039] For example, the ownership relationship between the cell content and the table header also needs to be checked by semantic understanding.

[0040] The recognized table content and position are corrected by semantics for a new round, and finally the entire content of the table analysis is obtained.

[0041] For example, another reconstructed table is shown in Figure 4 The table obtained after correction according to semantics is shown in Figure 5 .

[0042] (4) Finally, the information recognized and analyzed is output in the required format, such as outputting json, converting to Excel or XML, etc.

[0043] The above only describes the embodiments of the present application and does not limit the present application. Any modification, equivalent replacement and improvement within the scope of the present application should be included in the protection scope of the present application.

Claims

1. A method for intelligent parsing of complex tables based on multimodal fusion and semantic analysis, characterized in that: Includes the following steps: S1. Based on multimodal fusion technology, perform preliminary analysis of the document, identify the position information of table areas, text, and images, and output the position information of table areas; S2. Based on the topological graph structure and geometric relationships of the table, the table structure is reconstructed to obtain the reconstructed table; S3. Use machine vision to recognize the reconstructed table content, and combine semantics to correct the recognized table content and the position information of the table area to obtain the complete content of the table parsing. S4. Output the parsed table content in the required format.

2. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, In step S1, the multimodal fusion technology uses a deep learning network to perform preliminary parsing of the document.

3. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 2, characterized in that, The deep learning network is a fusion of CNN and Transformer.

4. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, In step S1, the output table area location information includes the table's bounding box and the coordinates of the initial cell.

5. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, Step S2 involves reconstructing the irregular table, specifically including the following steps: S201, Construct the topological graph structure of the table; S202, Process cells that span rows and / or columns in the topology of the table and modify the topology structure; S203, fills in the missing boundary lines in the topological graph structure of the table, and outputs the reconstructed table.

6. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 5, characterized in that, Step S201: Construct a graph model of the table using table cells as nodes and merge and adjacent relationships as edges.

7. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 5, characterized in that, Step S202: Process the cells spanning rows and / or columns in the topological graph structure of the table based on the spatial clustering algorithm, and correct the topological graph structure by combining the detection of diagonal lines in the table content.

8. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 5, characterized in that, Step S203: For the parts of the table's topological graph structure that lack boundary lines, add virtual dividing lines to obtain the reconstructed table.

9. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, In step S3, the recognition results are corrected by using semantics when the key and value are in the same cell, and the consistency of the cell content semantics is verified by using text features.

10. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, In step S3, the relationship between the cell content and the table header is corrected using semantic understanding.

11. The intelligent parsing method for complex tables based on multimodal fusion and semantic analysis according to claim 1, characterized in that, In step S4, the output table format includes one of JSON, Excel, or XML.

12. A complex table intelligent parsing system based on multimodal fusion and semantic analysis, characterized in that, The system includes a processor capable of executing a computer program, which, when executed, implements the intelligent parsing method for complex tables based on multimodal fusion and semantic analysis as described in any one of claims 1-11.

Images