Industrial table data recognition method and program product for large model knowledge enhancement

Abstract

The present invention discloses an industrial complex table image processing method and program product for large model knowledge enhancement, firstly identifying and locating the table area in the industrial file, extracting the data block containing only the table content; then extracting the horizontal and vertical frame lines of the table border, inferring the coordinates of all cells, and restoring the table structure; then matching the identified content with the cell coordinates to form data in the "cell content + coordinate" format; then splitting the merged cells in the complex table into the smallest cells for textual representation; then using the large model to classify the cell content in the textual table, distinguishing it into key type or value type, and storing it in a vector database; finally, realizing question and answer with the table data content in the vector database through vector similarity matching and retrieval enhancement generation technology. The present invention effectively improves the efficiency and accuracy of the large model in processing and understanding industrial complex table data.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer science technology, and relates to large model knowledge enhancement and table data recognition technology, and specifically to a data recognition method and program product for industrial complex tables for large model knowledge enhancement. Background Art

[0002] At present, Large Language Models (LLMs) technology has been relatively mature in processing pure text information. With the help of natural language processing technology, LLMs can directly use text data for training and reasoning. However, in the industrial field, knowledge files contain not only text information, but also a large number of complex data forms such as tables. The existing large model domain knowledge enhancement technology still has many challenges and shortcomings in processing complex industrial tables.

[0003] Unlike text information that can be directly used by large models, table information contains not only data content but also complex table structures. Therefore, how to retrieve table information in large model domain knowledge enhancement is a major difficulty in the current industry. Industrial complex tables usually have multi-level headers and a large number of merged cells, with complex associations between cells. When querying table content, it is usually necessary to query headers at different levels multiple times to locate the table value to be queried, which is not conducive to the extraction and use of table information.

[0004] In order to enable the big model to learn the table information in the domain knowledge document, the industry generally extracts the specific table content according to the template, converts it into text question-answer pairs and participates in the big model training. However, this type of method is not generalizable and needs to replace the template when processing tables of different formats, which makes the operation cumbersome and inefficient. At the same time, most of the data sets for big model table question-answering tasks are data sets that distinguish between table headers and table contents. The big model can answer based on more table information. However, in the process of extracting table information, there is no process that can distinguish cell categories. And the text tables in these data sets usually retain the structural content of the table, which means that the big model still needs to understand the table structure before answering based on the table content.

[0005] In summary, existing large model knowledge enhancement technologies face many challenges when processing complex industrial tables, including the complexity of table structure, the diversity of table content, and the lack of generalization of existing methods. How to achieve efficient retrieval and utilization of table information in large model domain knowledge enhancement is a technical problem that needs to be solved urgently. Summary of the invention

[0006] (I) Purpose of the invention

[0007] In view of the above-mentioned defects and deficiencies in the prior art when processing complex industrial tables, the present invention proposes a data recognition method and program product for industrial tables for large model knowledge enhancement. The complex table data in the domain file is converted into text format data that is easy for the large model to understand through natural language processing technology, so that the large model can use these text data to better answer domain-related questions, and improve the accuracy of table structure recognition by adopting technical means such as table frame optimization and extension. By applying a fine-grained table content representation method, the large model's ability to understand the table content is improved while retaining the table structure information, thereby achieving efficient processing and utilization of complex industrial tables and significantly improving the performance of the large model in domain knowledge enhancement.

[0008] (II) Technical solution

[0009] In order to achieve the purpose of the invention and solve the technical problems, the present invention adopts the following technical solutions:

[0010] The first object of the present invention is to provide a data recognition method for industrial tables for large model knowledge enhancement, which is used to extract and represent table content data from industrial files containing tables to improve the understanding and utilization of these data by the large model. The method comprises the following steps:

[0011] SS1. Industrial table data acquisition: Use file parsing tools to identify and locate table areas from input industrial files, and extract data blocks containing only table content from file pages;

[0012] SS2. Industrial table structure acquisition: Use line detection algorithm to extract all horizontal and vertical frame lines of the table border, use the intersection of the horizontal and vertical frame lines of the table border to infer the coordinates of all cells, confirm the position and boundary of each cell, and restore the structure of the table;

[0013] SS3. Industrial table content acquisition: According to the cell coordinates obtained in step SS2, and by identifying the table content in the table data block, the identified table content is matched and stored according to the cell coordinates to form combined data in the "cell content + coordinate" format;

[0014] SS4. Textualization of industrial table data: By extending the horizontal and vertical borders in the table to the table boundary, the merged cells in the original complex table are split into the smallest cells in the table, and the table data is represented in text form in the form of the smallest cells;

[0015] SS5. Fine-grained representation of industrial table data: Use a large model to classify the content of each cell in the text table to distinguish whether the cell content belongs to the key type or value type, represent the table data in the form of fine-grained key-value pairs, and store them in the vector database;

[0016] SS6. Industrial table question-answering retrieval enhancement: When a user asks a question about the table content, the question content is converted into a vector format. Through vector similarity matching and combined with RAG retrieval enhancement generation technology, question-answering with fine-grained table data content in the vector database is achieved.

[0017] A second object of the present invention is to provide a computer program product comprising computer instructions, wherein the computer instructions instruct a computing device to execute the above-mentioned data recognition method for industrial tables for large model knowledge enhancement.

[0018] (III) Technical Effect

[0019] Compared with the prior art, the data recognition method and program product for industrial tables with large model knowledge enhancement of the present invention have the following beneficial and significant technical effects:

[0020] (1) The present invention represents complex table data in text form and uses fine-grained key-value pairs to represent table data, so that the large model can understand and process the table content more accurately, thereby improving the accuracy of knowledge questions and answers. In addition, the present invention vectorizes the table content and combines it with RAG retrieval enhancement generation technology to achieve efficient vector similarity matching, which can quickly and accurately retrieve relevant content from the vector database, thereby improving the efficiency and accuracy of user questions and answers.

[0021] (2) The present invention uses a line detection algorithm and the intersection of the horizontal and vertical lines of the table border to infer the coordinates of all cells, which significantly improves the accuracy of recovering the table structure and ensures that the position and border of each cell are correctly identified. At the same time, the present invention stores the classified fine-grained table content into a vector database, providing a solid foundation for subsequent data retrieval and knowledge questions and answers, and improving the utilization and value of the data. In addition, the present invention does not rely on a specific table format, has strong generalization capabilities, and can process industrial complex table data in various formats, which improves the scope of application and practicality of the method.

[0022] (3) The present invention proposes and applies a fine-grained table content representation method. Compared with conventional serialized tables, it does not have table structure information, which is more conducive to the understanding of the table content by the large model; at the same time, in the retrieval enhancement generation step, if there is too much table data, the fine-grained table content can be segmented according to the maximum limit of text block segmentation without losing the table content data. If it is serialized table content, the table structure will be destroyed due to text block segmentation, thereby losing table information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the industrial table data recognition method for large model knowledge enhancement of the present invention.

[0024] Figure 2 is based on Figure 1 Specific flow chart of knowledge question-answering retrieval enhancement.

[0025] Figure 3 It is a schematic diagram of an industrial complex table data recognition process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention, and are mainly used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. In order to further understand the structure, features and purposes of the present invention, the following description is now provided in conjunction with the drawings. The implementations described in the drawings are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention.

[0027] As a specific example, Figure 1 , 3 As shown, the data recognition method for industrial tables for large model knowledge enhancement of the present invention is used to extract and represent table content data from industrial files containing tables to improve the understanding and utilization of these data by the large model, including industrial table acquisition, industrial table structure acquisition, industrial table content acquisition, industrial table data texting, industrial table data fine-grained representation, industrial table question-answering retrieval enhancement and other aspects. The main relevant steps of the data recognition method for industrial tables during implementation are as follows:

[0028] SS1. Industrial table data acquisition

[0029] The file parsing tool is used to identify and locate the table area from the input industrial file, and the data block containing only the table content is extracted from the file page.

[0030] More specifically, when actually executing this step, first, the text material containing industrial tables can be converted into a file format suitable for processing, and then the industrial tables in the file can be identified and located by using file parsing combined with layout analysis technology. Finally, based on the boundary position of the table in the file, the data block containing only the table content can be extracted from the file page.

[0031] As a more preferred example, when executing this step, for example, the following sub-steps may be included:

[0032] SS11. Convert text materials containing industrial tabular data into file formats suitable for processing;

[0033] SS12. Use file parsing and layout analysis technology such as img2table library to identify and locate the table in the file, and obtain the extraction result BBox table bounding box, which contains the horizontal and vertical coordinates of the upper left corner and the lower right corner of the table position, to locate the position of the table in the file;

[0034] SS13. After obtaining the bounding box and bounding coordinate position of the table in the file, extract the data block containing only the table content from the file page.

[0035] SS2. Industrial table structure acquisition

[0036] Use the line detection algorithm to extract all horizontal and vertical border lines of the table border, and use the intersection of the horizontal and vertical border lines of the table border to infer the coordinates of all cells, confirm the position and boundaries of each cell, and restore the structure of the table.

[0037] More specifically, when actually executing this step, you can use a line detection algorithm to obtain the horizontal and vertical frame lines in the table data, extend the two ends of each horizontal and vertical frame line to a certain extent, highlight the intersection of the horizontal and vertical lines, and then merge the horizontal and vertical frame lines, take the intersection of the merged lines, obtain the table data containing the intersection of the horizontal and vertical lines of the table, normalize the intersection, and finally use heuristic rules to obtain the table structure for the processed intersection and obtain the coordinates of each cell.

[0038] As a more preferred example, when executing this step, for example, the following sub-steps may be included:

[0039] SS21. Perform grayscale and binarization processing on the acquired table data, so that it becomes binary table data whose pixel values ​​only contain two values ​​of 0 and 255;

[0040] SS22. Then use a line detection algorithm, such as the horizontal and vertical line recognition algorithm in the cv2 library, to extract the horizontal and vertical lines contained in the binary table data, and optimize and extend the horizontal and vertical lines. Then, merge the horizontal and vertical lines and take the intersection to obtain the intersection point of the horizontal and vertical lines.

[0041] SS2.3 After obtaining all the intersections of the horizontal and vertical lines in sub-step SS22, since the intersections of the horizontal and vertical lines are actually composed of multiple pixels, the number of pixels obtained here is actually far more than the number of pixels required to restore the table. In order to restore the table structure, these pixels need to be normalized, and the intersection value algorithm is used to normalize the pixels with similar horizontal and vertical coordinates to the same pixel. If the obtained pixel points include [10, 11, 12, 35, 36, 88, 89, 90], then 10, 11, and 12 are normalized to the same pixel, 35 and 36 are normalized to the same pixel, and 88, 89, and 90 are normalized to the same pixel.

[0042] SS24. After obtaining the normalized intersections of the table in step SS2.3, these intersections can be used to restore the structure of each cell in the table. Traverse all intersections in order. Since all cells in a table are adjacent, all intersections can be traversed according to adjacent intersections. In the process of confirming each cell, first start from the upper left corner of each cell and query the adjacent upper right corner for horizontal lines. If the sum of all the values ​​between the two points is greater than or equal to the value of a horizontal line, it proves that there is an actual connecting horizontal line between the two points. Similarly, traverse whether the upper right corner and the lower right corner are connected by a horizontal line, whether the lower right corner and the lower left corner are connected by a horizontal line, and whether the lower left corner and the upper left corner are connected by a horizontal line. The closed loop is completed. If the four adjacent points are all connected by horizontal lines, it proves that the four points form a cell. If there is no connection line between two points during the process of confirming the cell, such as the upper right corner point and the lower right corner point, continue to traverse to the right to find the upper right corner point and the next point on the right, and so on, until a complete four-point rectangle is formed. After obtaining a table cell, the horizontal and vertical coordinates of the upper left corner and lower right corner of the cell are stored to describe each cell.

[0043] SS3. Industrial table content acquisition

[0044] According to the cell coordinates obtained in step SS2, and by identifying the table content in the table data block, the identified table content is matched and stored according to the cell coordinates to form combined data in the "cell content + coordinates" format.

[0045] As a preferred example, when executing this step, for example, the following sub-steps may be included:

[0046] SS31. Use character recognition technology to perform character recognition on table data, and match the recognized characters with the cell coordinates according to the character coordinates to form the same object.

[0047] SS32. Figure 3 The cell content containing the word "large" is: "['large',[55,122,440,827]]", the first element is the coordinates of the cell, and the second element list contains the horizontal and vertical coordinates of the upper left and lower right corners of the cell, which are used to confirm the cell position.

[0048] SS4. Textualization of industrial table data

[0049] By extending the horizontal and vertical borders in the table to the table boundary, the merged cells in the original complex table are split into the smallest cells in the table, and the table data is textually represented in the form of the smallest cells.

[0050] As a preferred example, when executing this step, for example, the following sub-steps may be included:

[0051] SS41. Extend each horizontal and vertical line in the table to the border of the table, so that a complex table can be transformed into a simple two-dimensional table;

[0052] SS42. By using multiple simple two-dimensional tables to merge cells, a complex cell with multiple levels of horizontal and vertical merging is formed. By extending the horizontal and vertical lines of the table, a complex cell with multiple levels of horizontal and vertical merging is split into multiple identical two-dimensional tables, such as Figure 3 The following is a partial text table:

[0053] |Large|Large|Small|Small|

[0054] |Class I|Class II|Class I|Class II|

[0055] It can be seen that in the long-distance structure, the cells containing the words "large" and "small" occupy two consecutive cells respectively, while "category one" and "category two" only occupy one cell under their category.

[0056] SS5. Fine-grained representation of industrial tabular data

[0057] The large model is used to classify the content of each cell in the textual table to distinguish whether the cell content belongs to the key type or the value type. The table data is represented in the form of fine-grained key-value pairs and stored in the vector database.

[0058] As a preferred example, when executing this step, for example, the following sub-steps may be included:

[0059] SS51. Through the large model instruction engineering, the large model's understanding of plain text tables is used to classify the contents of each cell. The rules contained in the instruction engineering are mainly composed of heuristic rules, and the final classification types include key (Key) and value (Value), which can also be regarded as "header" and "content". The classified cells can be more accurately represented by "key" and "value", such as ['large', [55, 122, 440, 827], Key], it can be known that the cell with the word "large" is of the "Key" type, that is, the header. ['±2.0', [189, 273, 634, 827], Value], it can be known that the cell where "±2.0" is located is of the "Value" type, that is, the content.

[0060] SS52. After obtaining the cell with category characteristics, all the table headers (including secondary and tertiary headers) can be traversed. Generally, a pair of complete row headers and column headers can only point to one content cell. With each content cell as the output target, the content of a complete table can be obtained by matching and traversing the headers of a simple two-dimensional table to obtain the content of the "header 1-header 2-content" format, such as Figure 3 The following two examples are "Large Class I Fuselage Category I Zone Basic ±1.5, Large Class I Fuselage Category I Zone Local ±2.5 (15%)". It can be seen that the key-value pair has integrated the content of the header and the content into an accurate statement, which means that there is no need to find the horizontal and vertical headers and then obtain the required values ​​through table mapping. It is equivalent to converting a complex table into a table structure that can be represented by characters. At this point, the complex structure of the table information has been reduced to a simple flat sequence structure. The data is highly interpretable. It is no different from other text data in the original file.

[0061] SS53. Vectorize and store classified fine-grained table contents.

[0062] SS6. Enhanced industrial table question and answer retrieval

[0063] Taking questions related to the table content as input, the questions are processed into the form of vector data, and through vector similarity matching combined with RAG retrieval enhancement generation technology, question and answering with the table content in the vector database is realized.

[0064] In a preferred embodiment, when performing this step, Figure 2 As shown, the following sub-steps may be included:

[0065] SS61. Vector similarity retrieval, after obtaining the user's question, convert the user's question content into a vector format, and then match it with the text blocks stored in the vector database. If the matched text block has a high degree of similarity with the user's question, the text block is extracted.

[0066] SS62. Retrieval enhancement generation, based on the relevant sorting algorithm contained in the RAG retrieval enhancement generation technology, selects several text blocks with high matching degrees and assigns them to the big model in the form of input. The big model then answers the user's questions based on the content.

[0067] Through the above embodiments, the purpose of the present invention is fully and effectively achieved. Those skilled in the art can understand that the present invention includes but is not limited to the contents described in the drawings and the above specific embodiments. Although the present invention has been described with respect to the most practical and preferred embodiments currently considered, it should be understood that the present invention is not limited to the disclosed embodiments, and any modification that does not deviate from the functional and structural principles of the present invention will be included in the scope of the claims.

Claims

1. A data recognition method for industrial tables for large model knowledge enhancement, characterized in that: The data identification method comprises at least the following steps when implemented: SS1. Industrial table data acquisition: Use file parsing tools to identify and locate table areas from input industrial files, and extract data blocks containing only table content from file pages; SS2. Industrial table structure acquisition: Use the line detection algorithm to extract all horizontal and vertical frame lines of the table border, use the intersection of the horizontal and vertical frame lines of the table border to infer the coordinates of all cells, confirm the position and boundaries of each cell, and restore the structure of the table. Industrial table structure acquisition includes: SS21. Perform grayscale and binarization processing on the acquired table data, so that it becomes binary table data whose pixel values ​​only contain two values: 0 and 255; SS22. Use a line detection algorithm to identify the obtained binary table data, and obtain all horizontal and vertical frame lines contained in the table data; SS23. Perform pixel traversal on the binary table data. When continuous horizontal / vertical frame line pixels are detected, add appropriate amount of pixels at both ends to optimize and extend both ends of the horizontal and vertical frame lines. SS24. Based on the extended horizontal and vertical frame lines, the horizontal frame line and the vertical frame line are merged to obtain the intersection point of the horizontal frame line and the vertical frame line, and the intersection point is normalized; SS25. After obtaining the normalized intersection points, use these intersection points to restore the structure of each cell in the table, traverse all intersection points in order, and confirm the position and boundaries of each cell; And among them, In sub-step SS24, an intersection value algorithm is used to normalize and merge intersections with similar horizontal and vertical coordinate positions into the same intersection; when sub-step SS25 restores the table structure, all intersections are traversed in order according to the heuristic rule to determine whether four points that can form a rectangle are a cell. If they can be formed, the coordinate information contained in the four intersections is stored. After the determination is completed, the determination continues from the next group of adjacent intersections in a clockwise direction; when all the coordinates of a row are determined, the determination process of the next row is entered until all the intersections are traversed, and the obtained coordinate information of all cells is the table structure; SS3. Industrial table content acquisition: According to the cell coordinates obtained in step SS2, by identifying the table content in the table data block, the identified table content is matched and stored according to the cell coordinates to form combined data in the "cell content + coordinate" format, wherein the industrial table content acquisition includes: SS31. Use character recognition technology to perform character recognition on table data blocks, and match the recognized characters with cell coordinates according to the character coordinates; SS32. Merge the recognized text content with the corresponding cell coordinates to form combined data in the format of "cell content + coordinates"; SS4. Textualization of industrial table data: By extending the horizontal and vertical borders in the table to the table boundary, the merged cells in the original complex table are split into the smallest cells in the table, and the table data is represented in text form in the form of the smallest cells. The textualization of industrial table data includes: SS41. Extend the horizontal and vertical lines of table data to maximize the extension of the horizontal and vertical borders in the table, split a large number of merged cells in a complex table into the smallest cells, convert the complex table into a simple two-dimensional table, and obtain the structure of the simple two-dimensional table based on the intersection of the horizontal and vertical lines of the table; SS42. According to the cell structure obtained in sub-step SS41, traverse each cell of the simple two-dimensional table. If a single cell is included in or belongs to a merged cell in the original table, assign it the content contained in the merged cell in the original table; SS5. Fine-grained representation of industrial table data: Use the big model to classify the content of each cell in the text table to distinguish whether the cell content belongs to the key type or value type, represent the table data in the form of fine-grained key-value pairs, and store them in the vector database. The fine-grained representation of industrial table data includes: SS51. Using instruction engineering, the content of each cell is divided into "key" and "value" through a large model, and the "key" and "value" are regarded as "header" and "content"; SS52. After obtaining the cells with category characteristics, all "header" cells in the text table are traversed and processed, and the table content information is extracted by key-value matching to obtain the content in the form of "header 1-header 2-content", forming a simple flat sequence structure after dimensionality reduction; SS53. The obtained content in the form of "header 1-header 2-content" is vectorized and stored in the vector database together with the text in the original file; SS6. Industrial table question-answering retrieval enhancement: Taking the relevant field questions containing table content as input, the questions are processed into the form of vector data, and through vector similarity matching and combined with RAG retrieval enhancement generation technology, question-answering with the table content in the vector database is realized.

2. The data recognition method for industrial tables for large model knowledge enhancement according to claim 1 is characterized in that: In the above step SS1, the industrial table data acquisition includes: SS11. Convert text materials containing industrial tabular data into a file format suitable for processing; SS12. Use file parsing combined with layout analysis technology to identify and locate industrial tables in the file, and obtain the bounding box of the table location and its boundary coordinate information in the file; SS13. Extract a data block containing only the table content from the file page according to the bounding box and its bounding coordinate information of the obtained table in the file.

3. The data recognition method for industrial tables for large model knowledge enhancement according to claim 1 is characterized in that: In the above step SS6, the table question and answer retrieval enhancement includes: SS61. When a user asks a question about the table content, the question content is converted into vector format, and the vector database is used as the data source of RAG to perform vector similarity matching with the question asked by the user. Based on the similarity matching results, multiple related text blocks are returned to the large model; SS62. The large model combines the text block content returned by vector similarity retrieval to return more accurate and highly interpretable answers.

4. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions instruct a computing device to execute the data recognition method for industrial tables for large model knowledge enhancement according to any one of claims 1 to 2.

Images