Information extraction index construction system for complex table

Abstract

The invention discloses an information extraction index construction system for a complex table. The information extraction index construction system comprises an offline processing module and an online processing module, wherein the offline processing module is configured to execute table analysis, header identification, table splitting, multi-dimensional index construction and index vectorization storage; the online processing module is configured to execute user query keyword semantic vectorization and multi-dimensional index recall filtering, and a large model combines context information and table block content to generate answers; header identification adopts a self-adaptive rule extension and depth method; the multi-dimensional index construction comprises a coarse-grained index and a fine-grained index. The invention provides an information extraction index construction system for a complex table, which fundamentally solves the problems of structural analysis and semantic alignment through a self-adaptive header identification and multi-dimensional index technology.

Description

Technical Field

[0001] This invention relates to the field of information construction technology, and more specifically, to an information extraction index construction system for complex tables. Background Technology

[0002] With the deepening of e-government construction, the government sector has accumulated massive amounts of complex tabular data, covering various types such as statistical reports, fiscal budgets, and population censuses. These tables generally exhibit structural diversity (e.g., cross-row and cross-column headers, multi-level headers) and inconsistent header positions. For example, in hierarchical tables, headers may span multiple rows and columns, and the nesting relationship between data rows and headers is complex. Against this backdrop, achieving unified governance of various types of tabular data and building a vector knowledge base suitable for enterprise internal retrieval augmented generation (RAG) systems has become a significant challenge. Current common practices often lack refined analysis and decomposition of complex table structures, directly importing the raw table data into vector databases. This results in significant information redundancy in the database, with tables interfering with each other, severely impacting the accuracy of retrieval and reducing the credibility and usability of large models in intelligent question answering and data analysis. Summary of the Invention

[0003] This invention overcomes the shortcomings of existing technologies and provides an information extraction index building system for complex tables that fundamentally solves the problems of structural parsing and semantic alignment through adaptive header recognition and multi-dimensional indexing technology.

[0004] The technical solution of the present invention is as follows:

[0005] An information extraction and index building system for complex tables includes an offline processing module and an online processing module.

[0006] The offline processing module configuration includes performing table parsing, header recognition, table splitting, multi-dimensional index construction, and index vectorization into the database;

[0007] The online processing module configuration includes performing semantic vectorization of user query keywords, multi-dimensional index recall filtering, and generating answers by combining contextual information and table block content with a large model.

[0008] The header recognition employs an adaptive rule expansion and depth approach, including predefined rule template judgment and an improved Transformer deep learning model for handling complex government tables with cross-row and cross-column structures and variable header rows and columns. The improved Transformer deep learning model adds absolute row position encoding to the Transformer position encoding module, concatenates it with relative position encoding, and then connects it with the input word vectors to obtain row and column position features from the global vector. The concatenated vector is then input into a multi-layer Transformer encoder to obtain the table context embedding vector. A fully connected MLP is added after the Transformer encoder as a decoder to output the optimal header classification result.

[0009] The multi-dimensional index construction method includes two levels: coarse-grained index and fine-grained index. The coarse-grained index is built based on table relevance and covers information such as table title, table header, and non-numerical columns. The fine-grained index is built based on content relevance and generates natural language description summaries of table block content by introducing a large model, thereby effectively reducing the gap between user queries and table semantics.

[0010] Furthermore, the table is parsed as an input Excel document, and the content is read and parsed through a table parser. The data is normalized to string type, null value replacement is performed, and meaningless information is removed. Finally, a Markdown format text table is output.

[0011] Furthermore, the predefined rule template judgment, when the table conforms to the predefined rule template, uses regular expressions to split the table into multiple rows, creates a table header list, and adds rows to the table header list according to the table data characteristics; the table data characteristics include: the number of cells in the current row exceeds 50% of the total number of cells; or the content of the cells in the current row and the previous or next row meets one of the following conditions: (1) the current row element is not empty and the next row element is empty; (2) the current row element is empty and the next row element is not empty; (3) the current row element is not empty and the previous row element is empty; (4) the current row element is empty and the previous row element is not empty;

[0012] The predefined rule template is applicable to hierarchical table structures and uses regular expressions to process multiple table headers.

[0013] Furthermore, the calculation formula for absolute row position encoding is added to the Transformer position encoding module as follows:

[0014]

[0015] Where A is the row position of the cell vector (A=(1,2,...,n), n is the total number of rows in the table), d modelis the word vector dimension, i is the word vector position, and pos is the position of the word in the sentence.

[0016] Furthermore, the table splitting includes defining a block character accumulator, traversing from the data row position index obtained in the table header recognition stage; accumulating the current data row into the current block character, and determining whether the length of the current block character is greater than the specified character limit; if it is greater, the block is added to the block list; otherwise, traversal continues, and the final block result is output; wherein, the block splitting process is applicable to long table processing, ensuring that the number of characters in each table block does not exceed the limit.

[0017] Furthermore, in the construction of multi-dimensional indexes, the coarse-grained index extracts the metadata structure information such as the title and cross-list headers for each table block and merges them into a unified keyword vector to support full-text retrieval and preliminary recall based on specific terms.

[0018] Fine-grained indexes target the specific content of table blocks and use a large language model to generate natural language description summaries that simulate the user's natural expression to enhance the understanding of query semantics.

[0019] Among them, coarse-grained indexes are used to quickly recall multiple related table blocks from large-scale data, while fine-grained indexes are used to perform semantic similarity calculations and filtering on the recall results.

[0020] Furthermore, the indexes are vectorized and stored in the database. The BGE vector model is used to convert the index list of table block elements into vectors; the vectors are then inserted into the Milvus vector library to complete the offline processing flow.

[0021] Furthermore, the online processing module performs semantic vectorization processing on the user's query keywords, generating corresponding sparse vectors and dense vectors respectively;

[0022] In the multi-dimensional index recall and filtering stage, sparse vectors are first used to recall table blocks from the coarse-grained index, and the number of recalls can be flexibly expanded; then, dense vectors and fine-grained indexes are used to calculate similarity, and irrelevant table blocks are filtered based on a preset threshold.

[0023] Finally, the large language model combines the filtered table block context information with prompt words to generate the final answer.

[0024] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the system content as described above.

[0025] A computer-readable storage medium storing a computer program that, when executed, implements the system content as described above.

[0026] The advantages of this invention compared to the prior art are:

[0027] This invention addresses the challenges of structural diversity (such as cross-row and cross-list headers, multi-level headers, etc.) and the unfixed header position prevalent in government affairs forms. It proposes a method combining adaptive rule expansion and deep learning, and introduces a multi-dimensional index construction mechanism. Refined table parsing, structural decomposition, and index construction are core to improving the accuracy of relevant table retrieval and are key steps in achieving efficient retrieval, intelligent question answering, and data analysis of massive government affairs forms. This solution achieves high-precision header positioning by integrating adaptive rule templates with an improved Transformer model. To address the issue that government affairs forms typically contain thousands of rows of data, and directly inputting them into large models can easily exceed the context length limit, this solution employs a dynamic table decomposition strategy combined with multi-dimensional index construction to ensure that each data chunk retains key structural information while meeting model input requirements. Traditional methods, lacking refined processing of table data, often result in low accuracy in relevant table knowledge retrieval, leading to illusions in large models (such as incorrectly associating "company name" with "revenue"). Therefore, this solution effectively ensures the accuracy and reliability of the input context through a two-stage online processing flow and a multi-level filtering mechanism.

[0028] This solution effectively addresses the diversity of government affairs table structures and header misalignment issues through adaptive header recognition, multi-dimensional indexing, and dynamic block partitioning mechanisms, achieving significant improvements in header recognition accuracy, question-and-answer reliability, and processing efficiency. Specifically, this method greatly enhances header recognition capabilities, significantly reduces the error rate during question-and-answer processing, and effectively improves overall processing speed. Simultaneously, this mechanism helps reduce the illusion phenomenon caused by incomplete contextual information in large models, providing a more reliable and efficient technical foundation for intelligent question answering in government affairs. This solution has been validated on real-world government affairs datasets, demonstrating good generalization and scalability. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0030] Figure 2 This is a schematic diagram of the header recognition framework of the present invention;

[0031] Figure 3 This is a schematic diagram of the improved Transformer deep learning model structure of the present invention;

[0032] Figure 4 Example diagram for a predefined rule template;

[0033] Figure 5 This is an example diagram of a hierarchical table. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below, wherein the same or similar reference numerals denote the same or similar elements or elements with similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0035] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0036] The numbering of steps mentioned in the various embodiments is merely for descriptive convenience and does not imply a sequential relationship. Different steps in various specific embodiments can be combined in different orders to achieve the inventive objective of this invention.

[0037] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0038] like Figures 1 to 3 As shown, an information extraction and index building system for complex tables includes an offline processing module and an online processing module, such as... Figure 1 As shown, the general process of the offline processing module and the online processing module is as follows:

[0039] The offline processing module configuration includes performing table parsing, table header recognition, table splitting, multi-dimensional index construction, and index vectorization and storage, which are executed sequentially.

[0040] The online processing module configuration includes performing semantic vectorization of user query keywords, multi-dimensional index recall filtering, and generating answers by combining contextual information and table block content with a large model.

[0041] The process begins with table parsing. This involves inputting an Excel document, reading and parsing its content using a table parser, performing operations such as data normalization to string type, null value replacement, and removing meaningless information, ultimately outputting a Markdown formatted text table.

[0042] Next, header recognition is performed. The header recognition adopts an adaptive rule expansion and depth method, including predefined rule template judgment and improved Transformer deep learning model recognition, to handle the situation of cross-row and cross-column and non-fixed header rows and columns in government forms.

[0043] That is, the header recognition will determine whether the Markdown table obtained by table parsing belongs to a predefined rule template. If it does, the table will be split according to the predefined rule template. Otherwise, the header position will be identified according to the model.

[0044] Specific examples can be derived from predefined rule templates. Figure 4 As shown.

[0045] The predefined rule template header recognition uses regular expressions to split it into multiple rows, obtain the dividing line, and create a header list; then, starting from the current table data position index, it determines whether to add the row to the header list based on the table data characteristics. The specific header characteristics are: 1. The number of cells in the current row exceeds 50% of the total number of cells; 2. Under the premise of the first list of cell elements: (1) The current row element is not empty, and the next row element is empty; (2) The current row element is empty, and the next row element is not empty; (3) The current row element is not empty, and the previous row element is empty; (4) The current row element is empty, and the previous row element is not empty. After traversing all the table data rows, the final data row position index and multi-row header list are determined.

[0046] Irregular tables suffer from issues such as headers spanning multiple rows and columns, and inconsistent header row and column positions. Using regular templates for header recognition is insufficient for various complex scenarios. To address the issue of inconsistent header positions, an improved Transformer deep learning model is needed. This involves adding absolute row position encoding to the Transformer position encoding module, concatenating it with relative position encoding, and then linking it to the input word vectors. This allows the global vector to acquire row and column position features. The concatenated vector is then input into a multi-layer Transformer encoder to obtain the table context embedding vector. Finally, a fully connected MLP is added after the Transformer encoder as a decoder to output the optimal header classification result.

[0047] The improved Transformer deep learning model is as follows: Figure 2 As shown, the structure includes a table position encoding layer, a stack of N repeating Transformer layers, and an MLP layer. Each Transformer layer consists of a multi-head attention layer and a feedforward neural network layer, with summation and row normalization steps surrounding each layer. The Transformer's multi-head attention mechanism and relative position encoding are suitable for processing tabular data that does not emphasize row cell position.

[0048] Traditional table row position encoding uses a relative position encoding layer that alternates between sine and cosine functions to construct a matrix with the same dimension as the input vector. This matrix is ​​then added to the input vector to obtain the input for the multi-head attention layer. However, the attention encoding layer used in the Transformer model cannot capture the order of input words, but the position vector allows the model to learn the relative positions of table cells. Therefore, the relevant information can be obtained by adjusting the encoding layer according to the following formula:

[0049]

[0050]

[0051] Where PE is a two-dimensional matrix with the same dimension as the input vector; pos is the position of the word in the sentence. is the dimension of the word vector; i is the position of the word vector.

[0052] The table positional encoding achieves overall table structure localization by adding absolute positional information to the positional encoding of each word vector. This addresses the issue of relative positional encoding failing to capture cross-row positions and reducing recognition accuracy when columns serve as headers within a table. For each row of the table, there is an absolute position. This absolute position is multiplied by a sine and cosine structure matrix of the same dimension as the word vectors to obtain a table positional encoding containing overall positional information, which replaces the table row positional encoding. The specific calculation formula is as follows:

[0053]

[0054] Where: A is the row position of the cell vector, A=(1,2,…,n), and n is the total number of rows in the table.

[0055] Using a self-built dataset of government and statistical tables, the table data was categorized into rule-based and irregular table data. Irregular table data included tables with cross-row and cross-column headers, multi-level headers, and tables with variable numbers of rows and columns. Examples of such data are shown below.

[0056] Table cell matrix (3x4 example)

[0057] table_cells = [

[0058] ["", "2020", "2021", "2022"],

[0059] ["Income", "500", "600", "700"],

[0060] ["Expenditure", "300", "400", "450"] ]

[0062] Header label matrix (0: non-header, 1: cross-row header, 2: cross-list header)

[0063] header_labels = [

[0064] [0, 2, 2, 2], # Line 0: spans the list header "Year (2020, 2021, 2022)"

[0065] [1, 0, 0, 0], # Row 1: Cross-row header "Income"

[0066] [1, 0, 0, 0] # Line 2: Cross-line header "Expenditure"

[0067] ].

[0068] The hyperparameter settings for the model are shown in Table 1 below:

[0069] Table 1

[0070]

[0071] The hyperparameter values ​​in Table 1 were used to train the model. After training, the model weights with the lowest loss were selected as the baseline model. The accuracy of header recognition was evaluated on the table samples in the test set. The specific calculation formula is as follows:

[0072]

[0073] Here, TP represents the number of correctly identified header cells (including those spanning rows / columns), FP represents the number of non-header cells misidentified as header cells, and FN represents the number of true header cells missed. This accuracy assessment method covers two dimensions: 1. Position sensitivity: cells must simultaneously satisfy both row / column position and header type correct. 2. Distinguishing between two different types: spanning row headers and spanning column headers.

[0074] Then, the table is split, which includes defining a block character accumulator to count the cumulative number of characters in the current table data row. Traversal begins from the data row position index obtained in the header identification stage; the current data row is added to the current block character, and it is determined whether the length of the current block character exceeds a specified character limit; if it does, the block is added to the block list; otherwise, traversal continues, and the final block result is output. This block splitting process is suitable for long table processing, ensuring that the number of characters in each table block does not exceed the limit.

[0075] Next, multi-dimensional index construction was carried out. For most tables containing a lot of numerical statistics, the structural information of the table is even more important. In order to better match the semantic gap between user questions and the tables expected to be retrieved, the index was constructed from two perspectives: table relevance and content relevance.

[0076] The multi-dimensional index construction includes coarse-grained indexes and fine-grained indexes. The coarse-grained index is based on table relevance, while the fine-grained index is based on content relevance (including using a large model to generate natural language description summaries) to narrow the gap between user queries and table semantics.

[0077] In essence, the coarse-grained index performs the aforementioned identification and segmentation process for each long table, preserving the metadata structure information of each segmented chunk: table title, table header, and non-numeric columns. This information is then merged into a single entity to construct a keyword vector for subsequent retrieval. When user queries involve fields from a single table or multiple tables, keyword matching is performed first, searching for results based on specific terms—essentially a full-text search—ultimately retrieving chunk content from multiple tables. This coarse-grained index is implemented for the initial retrieval of multiple table chunks.

[0078] Coarse-grained indexing extracts metadata structure information such as table title and cross-row headers for each table block and merges them into an overall keyword vector for full-text retrieval and recall based on specific terms.

[0079] Fine-grained indexing uses a large model to generate natural language summaries for the specific content of each table block. These summaries mimic user language descriptions to understand the semantics of the query. Fine-grained indexing is used in applications involving similarity calculation and filtering.

[0080] This refers to fine-grained indexing. For each pre-defined chunk, including table titles, headers, and all cell values, a large model generates a natural language summary of each chunk based on the table structure and cell content. This summary closely resembles the user's language and leverages semantic context to understand the meaning behind the user's query. During the multi-dimensional indexing recall and filtering phase, since coarse-grained indexing retrieves a large number of table chunks, this content needs to be filtered. Similarity calculations are performed between the user's query semantic vector and the fine-grained index, and a threshold is set to filter out table chunks less relevant to the user's query. Ultimately, the most relevant table chunks remain as contextual reference information for the large model's question answering.

[0081] Finally, the indexes are vectorized and stored in the database. The BGE vector model is used to convert the list of indexes for table chunk elements into vectors. These vectors are then inserted into the Milvus vector library, completing the offline processing workflow. In other words, the list of indexes for table chunk elements, obtained through multi-dimensional indexing, is vectorized using the BGE vector model and inserted into the Milvus vector library, thus completing the entire process of table parsing, header recognition, splitting, multi-dimensional index construction, and vectorized storage.

[0082] The online processing module processes the following:

[0083] First, semantic vectorization of user query keywords is performed, generating both sparse and dense vectors. Next, multi-dimensional indexing and filtering involves first using the sparse vectors and a coarse-grained index for recall, allowing for scalable recall of table blocks; then, the dense vectors and a fine-grained index are used to calculate similarity, filtering irrelevant table blocks based on a threshold. Finally, the large-scale model combines contextual information and table block content to generate the answer, combining the filtered table block context and prompts to produce the final response.

[0084] The following example of complex table header recognition, combined with the example of a hierarchical table, illustrates the detailed process of header recognition, table splitting, and multidimensional index construction.

[0085] An example of the hierarchical table identified by the header is as follows: Figure 5 As shown.

[0086] The header recognition process is as follows:

[0087] First, the current table data is compared against a predefined rule-based table template. Since the table does not belong to any of the predefined templates, a trained header recognition model is used to identify cross-row and cross-list headers. After the header recognition model outputs the table cell classification results, a post-processing result parsing function is used to obtain the following results:

[0088] {

[0089] "Cross-list header keyword list":

[0090] [Revenue and expenditure by region according to qualification level, unit: 100 million yuan, region, total, Level 1, Level 2, Level 3 and below, 2022, 2023, ...]

[0091] "Cross-list header keyword position list":

[0092] [(1,1),(1,-1),(2,1),(2,2),(2,4),(2,8),(2,12),(3,2),(3,3),......],

[0093] "Cross-row header keyword list":

[0094] [City 1, City 2, City 3, Income, Expenditure, Total, ...]

[0095] "List of Keyword Positions in Cross-Row Headers":

[0096] [(7,1), (8,1), (9,1), (5,2), (6,2), (5,1), ...]

[0097] }

[0098] The splitting phase begins by calculating the position of the largest row index across the list header and then splitting the table content into segments. Initially, the chunking result is an empty list, the current chunk list accumulator is empty, and the character length is 0. To ensure the data row remains a cohesive whole and avoid information loss, the above data row is divided into multiple chunk lists, as follows: [

[0100] “Chunk1”: “City 1|x|x|x|x|x|x|x|x|\n...\nCity 3|x|x|x|x|x|x|x|x|”,

[0101] “Chunk2”: “City 4|x|x|x|x|x|x|x|x|\n...\nCity 6|x|x|x|x|x|x|x|x|”, ......

[0103] "ChunkN": "City n-3|x|x|x|x|x|x|x|x|\n...\nCity n|x|x|x|x|x|x|x|x|x| ]

[0105] Multidimensional index construction is performed. Table relevance (fine-grained indexing) is implemented by combining the metadata structure information of each split chunk—table title, cross-column headers, and cross-row headers—into a single keyword vector for subsequent retrieval. The specific metadata structure is as follows:

[0106] {

[0107] Table Title: "Revenue and Expenditure by Qualification Level for Each Region"

[0108] "Header Keywords": "Unit: RMB 100 million, Region, Total, Level 1, Level 2, Level 3 and below, 2022, 2023, City 1, City 2, ......"

[0109] }

[0110] Content relevance (fine-grained indexing) consists of two parts: 1. For each of the above-mentioned segmented chunks, an index is built using cross-list headers and table row content to represent the specific content of the chunk; 2. A natural language summary of each chunk is generated using a larger model. This summary is closer to the user's language description and can leverage semantic context to understand the meaning behind the user's query. The specific fine-grained index structure is as follows:

[0111] {

[0112] Table title: "",

[0113] "Fine-grained Index 1": "Regions by Qualification Level: xxx Revenue and Expenditure\nUnit: 100 Million Yuan\nRegion| |2022 Total|2023 Total|2022 Level 1|2023 Level 1|2022 Level 2|2023 Level 2|2022 Level 3 and Below|2023 Level 3 and Below\nCity 1 Revenue|x|x|x|x|x|x|x|x|\nCity 1 Expenditure|x|x|x|x|x|x|x|x|x|\n",

[0114] "Fine-grained index 2": "Table title: xxx, table header description (field explanation, unit, etc.), table row content overview"

[0115] }

[0116] In the multi-dimensional index recall and filtering stage, since coarse-grained index recall will recall a large amount of table chunk content, this content needs to be filtered. The semantic vector of the user query is used to calculate the similarity with the fine-grained index, and a threshold is set to filter out table chunks that are not very relevant to the user query. Finally, the most relevant table chunks are left as contextual reference information for the large model question answering.

[0117] For the online phase, suppose the user's question is: "Please compare the income of City 1 and City 2 in 2023 for Level 1 qualifications." The system will first extract keywords from the question, including "City 1," "City 2," "2023," "Level 1," and "income," and construct a query vector based on these keywords to initially recall a list of relevant table fragments. Subsequently, based on the initial selection of a small number of table fragments, the system further employs a semantic vector filtering method to encode the user's question into a query vector and calculate its similarity with the summary vectors of each table fragment, thereby achieving precise filtering and recall of the most relevant content fragments. Based on the recalled table content fragments, the system finally combines the user's question to complete a comprehensive analysis and generate an answer.

[0118] In one embodiment, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the system content provided in the above embodiments.

[0119] In one embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the system content provided in the above embodiments.

[0120] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The several embodiments described in this application are quite specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and refinements should also be considered within the scope of protection of this invention. Therefore, the scope of protection of this patent application should be determined by the appended claims.

Claims

1. A system for constructing an information extraction index for complex tables, characterized in that, Includes offline processing modules and online processing modules: The offline processing module configuration includes performing table parsing, header recognition, table splitting, multi-dimensional index construction, and index vectorization into the database; The online processing module configuration includes performing semantic vectorization of user query keywords, multi-dimensional index recall filtering, and generating answers by combining contextual information and table block content with a large model. The header recognition employs an adaptive rule expansion and depth approach, including predefined rule template judgment and an improved Transformer deep learning model for handling cross-row and cross-column issues and non-fixed header rows and columns in government forms. The improved Transformer deep learning model adds absolute row position encoding to the Transformer position encoding module, concatenates it with relative position encoding, and then connects it with the input word vectors to obtain row and column position features in the global vector. The concatenated vector is then input into a multi-layer Transformer encoder to obtain the table context embedding vector. A fully connected MLP is added after the Transformer encoder as a decoder to output the optimal header classification result. The multi-dimensional index construction includes coarse-grained indexes and fine-grained indexes. The coarse-grained index is based on table relevance, and the fine-grained index is based on content relevance, in order to narrow the gap between user queries and table semantics. Table relevance includes table titles, headers, and non-numeric columns, while content relevance includes generating natural language description summaries using a large model.

2. The information extraction index construction system for complex tables according to claim 1, characterized in that, The table is parsed as an input Excel document. The table parser reads and parses the content, performs data normalization to string type, replaces empty values, removes meaningless information, and finally outputs a Markdown formatted text table.

3. The information extraction and indexing system for complex tables according to claim 1, characterized in that, The predefined rule template determines that when the table conforms to the predefined rule template, regular expressions are used to split the table into multiple rows, a header list is created, and rows are added to the header list according to the table data characteristics. The table data characteristics include: the number of cells in the current row exceeds 50% of the total number of cells; or the content of the cells in the current row and the previous or next row meets one of the following conditions: (1) the current row element is not empty and the next row element is empty; (2) the current row element is empty and the next row element is not empty; (3) the current row element is not empty and the previous row element is empty; (4) the current row element is empty and the previous row element is not empty. The predefined rule template is applicable to hierarchical table structures and uses regular expressions to process multiple table headers.

4. The information extraction index construction system for complex tables according to claim 1, characterized in that, The formula for calculating absolute row position encoding is added to the Transformer position encoding module as follows: Where A is the row position of the cell vector (A=(1,2,...,n), n is the total number of rows in the table), d model is the word vector dimension, i is the word vector position, and pos is the position of the word in the sentence.

5. The information extraction index construction system for complex tables according to claim 1, characterized in that, The table splitting process includes defining a block character accumulator, traversing from the data row position index obtained in the table header recognition stage; accumulating the current data row into the current block character, and determining whether the length of the current block character is greater than the specified character limit; if it is greater, then adding the block to the block list. Otherwise, continue iterating until the block result is output. The block division process is suitable for long table processing, ensuring that the number of characters in each table block does not exceed the limit.

6. The information extraction index construction system for complex tables according to claim 1, characterized in that, The coarse-grained index constructed by the multi-dimensional index extracts metadata structure information for each table block and merges it into an overall keyword vector for full-text retrieval and recall based on specific terms. Fine-grained indexing uses a large model to generate natural language description summaries for the specific content of each table block. These summaries simulate user language descriptions to understand query semantics. The coarse-grained index is used to initially recall multiple table blocks, while the fine-grained index is used for similarity calculation and filtering.

7. The information extraction index construction system for complex tables according to claim 1, characterized in that, The index vectorization and storage process involves using the BGE vector model to convert the index list of table block elements into a vector; then inserting the vector into the Milvus vector library to complete the offline processing flow.

8. The information extraction index construction system for complex tables according to claim 1, characterized in that, The online processing module performs semantic vectorization of user query keywords, generating sparse vectors and dense vectors respectively. The multi-dimensional index recall filtering first uses sparse vectors and coarse-grained indexes for recall, and the number of table blocks recalled can be expanded; then, dense vectors and fine-grained indexes are used to calculate similarity, and irrelevant table blocks are filtered based on thresholds. The large model combines contextual information and table block content to generate the answer, which is the final response generated by combining the filtered table block context and prompt words.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the system content as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed, it implements the system content as described in claims 1 to 8.

Images