Intelligent table data question and answer method based on multi-dimensional coordinate index

Abstract

The invention discloses a multi-dimensional coordinate index-based table data intelligent question and answer method. The method comprises the step of constructing a multi-dimensional coordinate index system, a three-layer vector database, an intelligent question analysis and routing system, an intelligent retrieval and context construction system and an answer generation and output system. Receiving a natural language question input by a user; performing problem type judgment and selecting a vector database; carrying out parallel retrieval and constructing a structured context; and generating answers, returning data coordinates and retrieving statistical information. According to the method, the defects of traditional table data retrieval are overcome, cell-level accurate positioning is achieved by constructing the multi-dimensional coordinate index, the retrieval precision and efficiency are improved, the accuracy of context understanding is improved, and the query time cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an intelligent question-answering method for tabular data based on multidimensional coordinate indexing. Background Technology

[0002] With the development of enterprise informatization, tabular data (such as Excel and CSV) has become an important carrier for storing structured information. Users often need to ask questions in natural language to obtain specific information from the tables, such as "Which department does Zhang San work in?" or "What is the total amount of a certain project?". Traditional methods rely on manual searching or simple keyword matching, which are inefficient and cannot understand semantic relationships.

[0003] Traditional tabular data queries rely on fixed field names and database query statements, which can be quite challenging for non-technical users. With the development of artificial intelligence, users expect to be able to ask questions directly in natural language, much like conversing with a human, and for the system to automatically parse the question, locate the data, and return the answer. The key challenge lies in enabling machines to accurately understand tables with complex structures such as multi-level headers and merged cells, and in building efficient indexes to support rapid data retrieval.

[0004] Existing technologies include table-based question-answering systems based on vector retrieval or large language models, but these systems have the following drawbacks: 1. Low retrieval accuracy: Relying solely on semantic similarity, it cannot accurately locate specific table cells; 2. Weak understanding of context: Lack of understanding of table structure, such as the relationship between rows, columns, and coordinates; 3. Does not support complex queries: such as cross-row / column aggregation, wildcard queries, and range expansion; 4. Poor scalability: It is difficult to adapt to complex scenarios such as multi-table joins and time-series analysis. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide an intelligent question-answering method for tabular data based on multidimensional coordinate index. This method overcomes the defects of traditional tabular data retrieval, and achieves precise cell-level positioning by constructing a multidimensional coordinate index, thereby improving retrieval accuracy and efficiency, as well as the accuracy of contextual understanding, and reducing query time costs.

[0006] To address the aforementioned technical issues, this invention provides an intelligent question-answering method for tabular data based on multidimensional coordinate indexing. This method includes constructing a multidimensional coordinate indexing system, a three-layer vector database, an intelligent question analysis and routing system, an intelligent retrieval and context construction system, and an answer generation and output system. It receives natural language questions input by the user, inputs these questions into the intelligent question analysis and routing system to determine the question type, and dynamically selects the vector database type for retrieval. Based on the question type, the intelligent retrieval and context construction system retrieves relevant cell, row, and column information in parallel within the multidimensional coordinate indexing system. It intelligently expands adjacent data areas based on the retrieved data coordinates to construct a structured context. Finally, it inputs the constructed structured context and the user-input natural language question into the answer generation and output system, uses a large language model to generate a natural language answer, and returns relevant data coordinates and retrieval statistics.

[0007] Furthermore, the multidimensional coordinate indexing system is established based on the coordinates of table cells.

[0008] Furthermore, the multidimensional coordinate indexing system generates unique coordinates for each cell in the table, establishes a bidirectional mapping between cell coordinates and values, and supports wildcard queries.

[0009] Furthermore, the three-layer vector database includes a cell-level vector library, a row-level vector library, and a column-level vector library.

[0010] Furthermore, the cell-level vector library stores the coordinates, column name, and value of each cell; the row-level vector library stores the concatenated text of the entire row of data; and the column-level vector library stores column statistics, including non-empty numbers, unique values, and example values.

[0011] Furthermore, the intelligent problem analysis and routing system determines the problem type of the natural language problem input by the user by identifying keywords, and then uses aggregation, entity, relation and coordinate query to determine the problem type after identification.

[0012] Furthermore, based on the question type, the intelligent retrieval and context construction system, after constructing a structured context, will re-analyze the question type and perform supplementary retrieval if the structured context is incomplete.

[0013] Furthermore, the answer generation and output system employs large language models such as DeepSeek or Ollam; generates accurate and concise natural language answers; and returns relevant data coordinates and retrieval statistics.

[0014] Furthermore, if a user's natural language question involves multiple tables, the intelligent question analysis and routing system expands the retrieval of related tables and updates the structured context through the intelligent retrieval and context building system.

[0015] Furthermore, the intelligent problem analysis and routing system supports time-series data analysis, extracts time-series related features, including timestamp coordinates, through a three-layer vector database, adjusts the retrieval range of the three-layer vector database according to the time-series features, and outputs time-series analysis results. Because this invention employs the aforementioned technical solution for intelligent question answering of tabular data based on multidimensional coordinate indexing, the method includes constructing a multidimensional coordinate indexing system, a three-layer vector database, an intelligent question analysis and routing system, an intelligent retrieval and context construction system, and an answer generation and output system; receiving natural language questions input by the user; determining the question type and selecting a vector database; performing parallel retrieval and constructing a structured context; generating answers and returning data coordinates and retrieval statistics. This method overcomes the shortcomings of traditional tabular data retrieval, achieving precise cell-level positioning through the construction of a multidimensional coordinate index, improving retrieval accuracy and efficiency, as well as the accuracy of context understanding, and reducing query time costs. Attached Figure Description

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments: Figure 1 This is a block diagram illustrating the principle of the intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to the present invention. Detailed Implementation

[0017] Implementation, for example Figure 1 As shown, the present invention's intelligent question-answering method for tabular data based on multidimensional coordinate indexing includes constructing a multidimensional coordinate indexing system, a three-layer vector database, an intelligent question analysis and routing system, an intelligent retrieval and context construction system, and an answer generation and output system. It receives natural language questions input by the user, inputs these questions into the intelligent question analysis and routing system to determine the question type, and dynamically selects the type of vector database to retrieve. Based on the question type, the intelligent retrieval and context construction system retrieves relevant cell, row, and column information in parallel within the multidimensional coordinate indexing system, intelligently expands adjacent area data based on the retrieved data coordinates, and constructs a structured context. The constructed structured context and the user's natural language question are input into the answer generation and output system, which uses a large language model to generate a natural language answer and returns relevant data coordinates and retrieval statistics.

[0018] Preferably, the multidimensional coordinate indexing system is built based on the coordinates of table cells. This process achieves precise positioning by converting the rows and columns of the table into a high-dimensional coordinate space. The specific implementation process is as follows: First, the table structure is parsed to extract the row and column numbers; then, a multidimensional index tree is constructed using the KD-Tree algorithm, where each node represents a coordinate partition. Data points are organized by recursively dividing the high-dimensional space (such as row and column dimensions) to achieve a query complexity of O(log n); finally, Locality Sensitive Hashing (LSH) is integrated to handle approximate queries and supports fuzzy matching. The parameter settings include hash function k=8 and hash table number L=10 to balance query accuracy and speed.

[0019] Preferably, the multidimensional coordinate indexing system generates unique coordinates for each cell in the table, establishes a bidirectional mapping between cell coordinates and values, and supports wildcard queries. Specifically, a unique coordinate is generated for each cell in the table, in the format "R3C4". This process is achieved by string concatenation: coordinate = "R" + str(row number) + "C" + str(column number), and a hash function (such as MurmurHash) is used to calculate the unique identifier of the coordinate to avoid collisions. A bidirectional mapping between "coordinate → value" and "value → coordinate" is established. This process uses two hash tables: one forward mapping (coordinate as key, value as value) and another inverted index (value as key, coordinate list as value). Batch insertion operations ensure an average access time of O(1). Wildcard queries are supported, such as "R3,*", "*C4", "R3-5,C2-4". This process parses wildcards with regular expressions, converts them into range queries, and then uses the R-Tree algorithm to traverse the index tree, extract the matching coordinate set, and achieve efficient region scanning. The R-Tree algorithm parameters include a minimum number of nodes m=2 and a maximum number of nodes M=4 to optimize tree balance.

[0020] Preferably, the three-layer vector database includes a cell-level vector library, a row-level vector library, and a column-level vector library. This process uses high-dimensional vectors to represent table elements to achieve similarity search. Specifically, the process involves: compressing the high-dimensional vectors using the Product Quantization (PQ) algorithm to reduce storage space; PQ parameters include the number of subquantizers M=8 and the number of bits per subspace k=8; then, applying a Hierarchical Navigable Small World (HNSW) graph structure for indexing to support fast approximate nearest neighbor (ANN) queries; HNSW parameters include the number of levels M=16 and efConstruction=200 to improve construction and query efficiency.

[0021] Preferably, the cell-level vector library stores the coordinates, column name, and value of each cell. This process first uses the BERT embedding model to generate a 768-dimensional vector representation: vector = Embed(column name + ": " + value), with embedding parameters including batch_size=32 and max_length=512. Then, the FAISS library's IVF (Inverted File) index partitions the vector space to achieve subquantization search. IVF parameters include nlist=1024 and nprobe=16. The row-level vector library stores the concatenated text of the entire row of data. This process generates a row string through text concatenation: row text = " ".join(column name i + ": " + value i for i in row), then embeds it as a vector, and uses LSH projection to a low-dimensional space to accelerate the query. LSH parameters include hash function k=10 and projection dimension d=128. The column-level vector library stores column statistics, including the number of non-empty values, unique values, and sample values. This process calculates statistics: non-empty values ​​= len(column) - The function `sum(isnull(value))` is used to calculate the unique value, which is then set to `len(set(column value))`. The example value is then set to `column value[:3]`. This is then vectorized into a statistical vector and searched using HNSW multi-level graph navigation. The statistical vector has a dimension of 64.

[0022] Preferably, the intelligent question analysis and routing system determines the question type of the natural language question input by the user by including keyword recognition. After recognition, the natural language question is classified using aggregation, entity, relation, and coordinate queries. Keyword recognition includes terms such as "total," "who," and "which row," to determine the question type. This process uses the table-specific Transformer of the TAPAS model to encode the question and keywords, and calculates an attention score threshold of 0.8 for classification. TAPAS parameters include a learning rate of 1e-5, batch size of 16, and epoch of 10. The system dynamically selects the type of vector database to retrieve. This process maps the classification results to the vector database: for example, aggregation queries are routed to column-level vector databases. Routing decisions are made through a rule engine (such as a decision tree), with a decision tree depth of 5 and a branch threshold of 0.7.

[0023] Preferably, based on the question type, after constructing a structured context, if the structured context is incomplete, the intelligent retrieval and context construction system re-analyzes the question type and performs supplementary retrieval. The system intelligently expands neighboring region data based on the retrieved coordinates. The process of constructing the structured context uses Breadth-First Search (BFS) to expand the neighborhood from the core coordinates: traversing from (Ri, Cj) to (Ri±k, Cj±k), where k is initially 1 and dynamically adjusted to cover relevant data. The queue size is limited to 100 to avoid overflow. If the structured context is incomplete and supplementary retrieval is performed, the context coverage rate (e.g., keyword matching rate <90%) is checked. If it is low, the routing system iterates, with an upper limit of 3 iterations.

[0024] Preferably, the answer generation and output system employs a large language model including DeepSeek or Ollam; it generates accurate and concise natural language answers; and returns relevant data coordinates and retrieval statistics. This process constructs a prompt template: prompt = "Question: " + Question + "\nContext: " + Structured Context, with a prompt length limited to 2048 tokens; the process of generating accurate and concise natural language answers uses beam search decoding with a width of 4 to ensure generation quality, a temperature parameter of 0.7, and top_p = 0.9; the process of returning relevant data coordinates and retrieval statistics is formatted as a JSON object, including a list of coordinates and statistics (such as retrieval time and number of matches).

[0025] Preferably, if the user's natural language question involves multiple tables, the intelligent question analysis and routing system expands the retrieval of related tables and updates the structured context through the intelligent retrieval and context building system. This process calculates the Jaccard similarity threshold of 0.9 to match tables by sharing column names, then merges the context vectors and uses matrix multiplication to fuse the multi-table data, with a matrix dimension of 768x768.

[0026] Preferably, the intelligent problem analysis and routing system supports time-series data analysis, extracting time-series-related features, including timestamp coordinates, through a three-layer vector database; adjusting the retrieval scope of the three-layer vector database based on the time-series features, and outputting time-series analysis results. This process parses the time column using a sorted index (such as a B-Tree) to organize the time-series data, with the B-Tree order being 5; adjusting the retrieval scope based on time-series features, and outputting time-series analysis results. The process applies an ARIMA model to predict trends, or uses sliding window aggregation (such as mean, sum) to generate a time-series report. The ARIMA parameters include p=1, d=1, q=1, and the window size is 7.

[0027] This method first constructs a multi-dimensional coordinate index system to accurately locate data at the cell level, improving retrieval accuracy and efficiency, avoiding the ambiguity of traditional vector retrieval, and reducing query time costs. It then constructs a three-layer vector database to perform collaborative retrieval at the cell, row, and column levels, improving the accuracy of contextual understanding. Furthermore, it performs complex query diagnosis based on problem routing and region expansion, enhancing the system's scalability and its reference value for practical applications.

[0028] The intelligent question analysis and routing system, intelligent retrieval and context building system, and answer generation and output system built on machine learning can fully learn data features and quickly and accurately extract features to predict and output corresponding answers when new questions are input. The automated question answering avoids the limitations of manual queries and improves accuracy. The intelligent problem analysis and routing system is based on the TAPAS (Table Parser) algorithm. The TAPAS algorithm replaces the attention mechanism of the existing Transformer model with table-specific encoding and uses a row / column aggregation layer to obtain an improved TAPAS algorithm. It receives natural language questions input by users. In practical applications, users only need to input the question once to respond quickly, avoiding the impact of multiple interactions on efficiency. This method has the following advantages and technical effects compared with the existing technology: 1. High query accuracy: It achieves precise cell-level positioning through coordinate indexing, avoiding the ambiguity of traditional vector retrieval; 2. Strong semantic understanding: Combining table structure with semantic information, it supports complex queries, such as aggregation, association, and region expansion; 3. Excellent retrieval efficiency: Three-layer vector library collaborative retrieval and intelligent routing improve response speed and accuracy; 4. Excellent scalability: Supports advanced scenarios such as multi-table joins and time-series data analysis, and is easy to integrate and extend; 5. Excellent user experience: Supports natural language questions, and the returned results are intuitive and include source coordinates for easy verification.

Claims

1. A method for intelligent question answering of tabular data based on multidimensional coordinate indexing, characterized in that: This includes constructing a multi-dimensional coordinate indexing system, a three-layer vector database, an intelligent question analysis and routing system, an intelligent retrieval and context building system, and an answer generation and output system; receiving natural language questions input by users, inputting the user-input natural language questions into the intelligent question analysis and routing system, determining the question type, and dynamically selecting the type of vector database to retrieve; Based on the question type, an intelligent retrieval and context building system is used to retrieve relevant cell, row, and column information in parallel within a multi-dimensional coordinate index system. The system intelligently expands adjacent area data based on the retrieved data coordinates to build a structured context. The system generates and outputs answers to structured contexts and user-input natural language questions. It uses a large language model to generate natural language answers and returns relevant data coordinates and retrieval statistics.

2. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: The multidimensional coordinate indexing system is built based on the coordinates of table cells.

3. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1 or 2, characterized in that: The multidimensional coordinate indexing system generates unique coordinates for each cell in the table, establishes a two-way mapping between cell coordinates and values, and supports wildcard queries.

4. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: The three-tiered vector database includes a cell-level vector database, a row-level vector database, and a column-level vector database.

5. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 4, characterized in that: The cell-level vector library stores the coordinates, column name, and value of each cell; the row-level vector library stores the concatenated text of the entire row of data; and the column-level vector library stores column statistics, including non-empty numbers, unique values, and example values.

6. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: The intelligent problem analysis and routing system determines the problem type of the natural language problem input by the user, including through keyword recognition. After recognition, the natural language problem is further determined by aggregation, entity, relation and coordinate query.

7. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: Based on the question type, the intelligent retrieval and context building system, after constructing a structured context, will re-analyze the question type and perform supplementary retrieval if the structured context is incomplete.

8. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: The answer generation and output system uses large language models, including DeepSeek or Ollam; it generates accurate and concise natural language answers; and returns relevant data coordinates and search statistics.

9. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: If a user's natural language question involves multiple tables, the intelligent question analysis and routing system expands the retrieval of related tables and updates the structured context through intelligent retrieval and context building systems.

10. The intelligent question-answering method for tabular data based on multidimensional coordinate indexing according to claim 1, characterized in that: The intelligent problem analysis and routing system supports time-series data analysis, extracts time-series related features, including timestamp coordinates, through a three-layer vector database, adjusts the retrieval range of the three-layer vector database based on the time-series features, and outputs time-series analysis results.

Images