Table question and answer method based on large model and retrieval enhancement

Abstract

The invention relates to a table question and answer method based on a large model and retrieval enhancement, and the method comprises the following steps: selecting CodeLLaMA-7B-Instruct-hf, and carrying out the fine adjustment of parameters through a LoRA technology, so as to obtain an LLMform Table; the method comprises the following steps: constructing a multi-modal knowledge base, converting an unstructured document in the multi-modal knowledge base into a plain text, and converting the text into a 768-dimensional knowledge vector by using Sension-BERT in a partitioning manner according to semantic granularity; a natural language question input by a user is converted into vectors, similarity between the vectors and knowledge vectors is calculated, the knowledge vectors corresponding to Top-K similarities with the similarity larger than a threshold value and arranged in a descending order are selected, and then knowledge fragments corresponding to the Top-K knowledge vectors are extracted; and making a structured prompt Prompt, taking the Prompt as the input of the LLMformTable, and outputting an SQL statement.

Description

Technical Field

[0001] This invention relates to the field of natural language processing, and in particular to a table-based question-answering method based on large models and retrieval enhancement. Background Technology

[0002] In today's increasingly data-driven decision-making environment, table-based question answering technology has become a crucial bridge connecting ordinary users with databases. This technology typically encompasses table retrieval, semantic parsing, question generation, and text generation: first, it accurately retrieves tables relevant to the user's question from a large pool of tables; then, semantic parsing transforms the natural language question into Structured Query Language (SQL); finally, combining question generation and text generation techniques, the SQL execution result is returned to the user in natural language. However, current table-based question answering technology has significant shortcomings: on the one hand, it is difficult to accurately retrieve the corresponding table, leading to a disconnect between the user's question and the target data; on the other hand, the accuracy of SQL generation is low, greatly limiting its application in real-world scenarios.

[0003] Traditionally, users need to master Structured Query Language (SQL) to extract information from databases, writing complex statements such as table joins and conditional filtering. However, this method has a high technical threshold, making it difficult for non-technical personnel (such as project managers and product managers) to complete the analysis independently. This results in low data utilization efficiency and an inability to quickly respond to R&D decision-making needs. Furthermore, it is prone to syntax errors or logical deviations when dealing with scenarios involving multiple table joins and complex aggregate functions, exhibiting extremely poor flexibility.

[0004] With the development of large language models (such as CodeLLaMA and the GPT series), natural language to SQL (NLP) technology has gradually matured. Text-to-SQL technology, as a core component, has evolved from rule-based methods to large model-based approaches. Early rule-based methods relied heavily on manually defining numerous grammar rules and dictionaries to map natural language to SQL, but their capabilities were limited when handling complex statements and diverse scenarios. Today, large model-based Text-to-SQL methods, with their powerful learning and generation capabilities, have become the mainstream development direction. Currently, data retrieval primarily relies on Text-to-SQL technology and TableQA systems. The mainstream TableQA solutions fall into three categories:

[0005] Semantic parsing frameworks (such as Seq2SQL and Tapas) map natural language to SQL through predefined grammar rules. In single-table query scenarios, the accuracy can reach 70%. However, they rely heavily on manually designed dictionaries and rules, making it difficult to adapt to the dynamically changing terminology system in the software development field. This results in poor generalization across business scenarios and a development cycle of more than 2 weeks for new requirements.

[0006] While general pre-trained models (such as CodeLLaMA and T5-TableQA) support end-to-end SQL generation, they have significant drawbacks: they ignore Chinese grammatical features (such as omitting quantifiers and flexible word order), confuse synonyms such as "work hours" and "work hour percentage," and achieve an accuracy of only 24%-35% in tests on the Chinese NL2SQL dataset (TableQA2023); they are weak in handling multi-table hierarchical relationships commonly found in software development (such as task table ←→ work hour table ←→ member table), and their accuracy drops sharply to 28% when JOIN operations involving more than 3 tables are involved, with nested aggregation queries having an error rate exceeding 65%.

[0007] The aforementioned existing technologies have significant drawbacks: a lack of understanding of Chinese software terminology, leading to confusion in the definition of professional indicators; insufficient understanding of domain-specific table structures; isolated storage of database schemas (DDL), business documents (such as performance indicator definitions), and historical SQL samples; failure to construct a domain knowledge graph; and the generation of SQL prone to "illusions" (such as fictitious fields and incorrect table joins). Performance degrades significantly when handling scenarios involving multi-table joins and complex aggregate functions, making it difficult to meet the cross-stage and multi-dimensional data analysis needs during the R&D process; and lagging knowledge updates, failing to dynamically adapt to changes in database structure or the addition of new business indicators, resulting in poor timeliness of query results.

[0008] While there has been some progress in specialized technologies for table-based question answering (such as TableQA), there are still shortcomings: there are misunderstandings about the tables and the corresponding questions; and there is insufficient support for multi-turn dialogues, making it impossible to optimize subsequent answers based on historical query context, which makes it difficult to meet the needs of complex analysis scenarios.

[0009] In summary, existing technologies are insufficient to meet the analytical needs of tabular data that require "low barriers to entry, high precision, and deep reasoning." Therefore, there is an urgent need for a tabular question-answering technology that is adapted to Chinese software development scenarios, integrates domain knowledge, supports multi-table queries, and enables natural language interaction, in order to overcome the technical barriers of traditional methods and improve efficiency and accuracy. Summary of the Invention

[0010] To address the aforementioned problems in existing technologies, the purpose of this invention is to provide a table-based question-and-answer technology that is adapted to Chinese software development scenarios, integrates domain knowledge, supports multi-table queries, and supports natural language interaction, thereby breaking through the technical barriers of traditional methods and improving efficiency and accuracy.

[0011] To achieve the above objectives, the present invention adopts the following technical solution: a table-based question-answering method based on a large model and retrieval enhancement, comprising the following steps:

[0012] S1: CodeLLaMA-7B-Instruct-hf was selected as the basic model. LoRA technology was used to fine-tune the parameters to obtain the Text-to-SQL model LLMforTable. Low-rank matrices were added to the Query and Value projection matrices of the Transformer layer of the basic model to obtain the modified model.

[0013] Extracted from the TableQA large-scale Chinese NL2SQL dataset, which contains structured samples consisting of several tables, several questions, and several SQL statements. Each structured sample consists of a natural language question and a corresponding SQL statement.

[0014] S2: Construct a multimodal knowledge base, preprocess the content of the multimodal knowledge base, convert unstructured documents into plain text, divide the text into blocks according to semantic granularity, use Sentence-BERT to convert the text into 768-dimensional knowledge vectors, and use the IVF-PQ algorithm to build an index.

[0015] The multimodal knowledge base includes database schemas, domain documents, and historical question-and-answer pairs.

[0016] S3: Convert the natural language question input by the user into a vector, calculate the similarity with the knowledge vectors stored in the multimodal knowledge base, select the top-K similarity vectors with similarity greater than the threshold and sorted in descending order, and then extract the knowledge fragments corresponding to the top-K knowledge vectors.

[0017] The retrieved knowledge fragments are combined with natural language questions to form a structured prompt that includes "role setting (Text-to-SQL tool), task description (generate SQL), instructions distinguished by #### delimiter, contextual information (search results), and format constraints (SQL statement)".

[0018] The context information includes database schemas retrieved from a multimodal knowledge base, domain documents, and historical question-and-answer pairs. The Prompt is used as input to LLMforTable, and the output is an SQL statement.

[0019] If the similarity scores are all below the threshold, it means that the search results are empty. In this case, the "fallback mechanism" is triggered, which inputs the natural language question from the user into LLMforTable and outputs an SQL statement.

[0020] The generated SQL statement is syntax-validated based on MySQL syntax rules. If the syntax is incorrect, it is filtered out and returned to LLMforTable to regenerate the SQL statement. After successful validation, the SQL statement is executed in the MySQL database to obtain the query results, such as tables and values. The query results, numbers, or lists are then converted into natural language answers (e.g., "There are 12 tasks that were not delivered on time in the third quarter of 2024, with task IDs XXX, XXX, etc.").

[0021] As an improvement, in S1, the process of fine-tuning the parameters to obtain LLMforTable adopts an incremental training strategy.

[0022] Structured samples that query only one table in an SQL statement are selected as basic samples. All basic samples constitute the basic dataset, which is then divided into a basic training set and a basic validation set. Other structured samples besides the basic samples are selected as advanced samples. All advanced samples constitute the advanced dataset, which is then divided into an advanced training set and an advanced validation set.

[0023] S1-1: Basic Training Phase: The samples in the basic training set are used as input to the revised model. After every T training steps, the revised model is validated using the basic validation set.

[0024] If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current corrected model is output as the pre-trained model.

[0025] If the loss decreases or the SQL syntax accuracy improves, update the parameters of the corrected model and return to continue training;

[0026] If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current corrected model is output as the pre-trained model.

[0027] S1-2: In the advanced training phase, samples from the advanced training set are used as input to the pre-trained model. After every T training steps, the current pre-trained model is validated using the advanced validation set.

[0028] If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current pre-trained model is output as LLMforTable.

[0029] If the loss decreases or the SQL syntax accuracy improves, update the parameters of the pre-trained model and return to continue training;

[0030] If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current pre-trained model is output as LLMforTable.

[0031] The generated SQL statement is syntax-validated based on MySQL syntax rules. Validation includes, but is not limited to, keyword spelling, field name matching with table structure, logical operator validity, and statement format completeness. If validation fails, the statement is deemed invalid and filtered, and LLMforTable is re-triggered to generate the SQL statement. If validation passes, the SQL statement is executed in the MySQL database to retrieve query results, which may include tables, numerical values, or lists. The query results are then converted into natural language answers. The conversion rules are as follows: for table-type results, key fields and corresponding values ​​are extracted and integrated; for numerical results, units and semantic descriptions are added; for list-type results, they are enumerated in logical order. The final output contains a detailed description of the query results (e.g., "There were 12 tasks not delivered on time in the third quarter of 2024, with task IDs XXX, XXX…").

[0032] Compared with the prior art, the present invention has at least the following advantages:

[0033] 1. By using CodeLLaMA-7B-Instruct-hf as the base model and combining it with LoRA parameter fine-tuning technology, a Text-to-SQL model LLMforTable specifically for Chinese table-based question answering was constructed by training the TableQA Chinese NL2SQL dataset. This technical solution solves the problem of low SQL generation accuracy caused by the lack of domain knowledge in traditional methods in Chinese software development scenarios. It makes the model more accurate in understanding Chinese terminology; compared with traditional rule-based or machine translation methods, this solution improves the SQL generation accuracy for Chinese terminology (such as "on-time delivery rate" and "work time deviation rate") by 35%. Experiments show that fine-tuning based on the TableQA Chinese NL2SQL dataset improves the model's accuracy by more than 27% in cross-domain multi-table query tasks, and the LoRA technology reduces the number of training parameters by 90% while maintaining model performance.

[0034] 2. An innovative two-stage training approach with increasing difficulty is adopted: Basic stage: Training is conducted using single-table query samples, with an early stopping mechanism (termination occurs when there is no change in loss / SQL accuracy after two consecutive validations) to prevent overfitting, outputting a pre-trained model. Advanced stage: Multi-table association samples are introduced for further training, dynamically adjusting model parameters. This strategy allows the model to maintain 98% accuracy in single-table queries while increasing multi-table query accuracy from 63% to 89%, and reducing training time by 35%. 3. By constructing a multimodal knowledge base containing database schemas, domain documents, and historical question-answer pairs, and using Sentence-BERT to generate 768-dimensional knowledge vectors and establish an IVF-PQ index, dynamic retrieval and injection of domain knowledge are achieved. When a user inputs a natural language question, Top-K relevant knowledge fragments (with configurable similarity thresholds) are retrieved and concatenated with the question to form a structured Prompt (containing elements such as role settings, task descriptions, and contextual information). This design solves the semantic ambiguity problem caused by the lack of domain adaptability in traditional methods. Tests show that in software project management scenarios, knowledge augmentation improves the accuracy of generating complex queries by 41%.

[0035] 4. After users directly input their questions in natural language, they don't need to worry about how to query. This invention automatically converts the SQL query results (tables, values, etc.) into expressions that conform to Chinese habits (e.g., "A total of 12 incomplete tasks were found, including 3 high-risk tasks: ID-101, ID-205..."). This allows non-technical users (such as product managers) to directly obtain database information by asking questions in Chinese, without needing to learn SQL syntax. This reduces the query threshold in software development scenarios by up to 70%, while ensuring professional-level accuracy. Attached Figure Description

[0036] Figure 1 This is an intelligent table-based question-and-answer framework.

[0037] Figure 2 A simplified flowchart of the method of this invention. Detailed Implementation

[0038] The present invention will now be described in further detail.

[0039] This invention addresses key issues in existing technologies for question-and-answer sessions using Chinese software development data tables, including low accuracy in natural language queries, difficulty in adapting to multi-table structures, and a lack of domain knowledge. It proposes an innovative solution. The core of this solution lies in employing LoRA fine-tuning technology (rank r = 64, scaling factor α = 128) to efficiently train the CodeLLaMA base model with optimized parameters. This approach effectively solves the aforementioned technical challenges: LoRA's low-rank characteristic significantly reduces the number of parameters and computational resources required for fine-tuning (training only 0.11B parameters reduces resource consumption by 98.3%), making domain-customized model training feasible. Simultaneously, through a carefully designed dataset (containing complex scenarios such as single tables, multi-table joins, and nested aggregations) and a phased, progressively challenging training strategy, the model learns and adapts to complex multi-table structures and query logic.

[0040] A table-based question-answering method based on large models and retrieval enhancement includes the following steps:

[0041] S1: CodeLLaMA-7B-Instruct-hf was selected as the base model. LoRA (low-rank adaptation) technique was used to fine-tune the parameters to obtain the Text-to-SQL model LLMforTable. A low-rank matrix was added to the Query and Value projection matrices of the Transformer layer of the base model to obtain the modified model (rank r = 64, scaling factor α = 128). This allows the model to be adapted to the domain task by training only 0.11B parameters, which reduces the computational resource consumption by 98.3% compared to full fine-tuning.

[0042] For the Query projection matrix, the low-rank decomposition matrix is ​​defined as:

[0043] in, The original Query projection matrix has a dimension of d. model *d model (d model =4096), that is, the original number of parameters is 4096*4096≈16.77 million.

[0044] Let A be a low-rank matrix with dimension d. model *r (r = 64, i.e., rank is 64), the number of parameters is 4096 * 64 = 262144.

[0045] Let B be a low-rank matrix with dimensions r*d. model The number of parameters is 64 * 4096 = 262144.

[0046] After low-rank decomposition, the number of new parameters is 262144 + 262144 = 524288 (approximately 524,000), which is only 3.125% of the number of parameters in the original matrix (524,000 / 16.77 million), significantly reducing the training cost.

[0047] Similarly, a low-rank decomposition matrix is ​​added to the Value projection matrix. Dimension d model ×r (r=64), The dimension is r×d model The number of parameters has also been significantly reduced.

[0048] We utilize the TableQA large-scale Chinese NL2SQL dataset (containing over 6000 tables, 64891 questions, and 20311 SQL statements), covering scenarios such as single-table queries, multi-table joins, and nested aggregate functions. We inject software development process management data (such as task schedules and performance indicator tables) to form a domain-customized training set, supplemented by enterprise-owned software process management data to form a hybrid training set. Data is labeled according to SQL complexity (simple queries / multi-table joins / nested aggregate functions).

[0049] Extracted from the TableQA large-scale Chinese NL2SQL dataset, which contains structured samples consisting of several tables, several questions, and several SQL statements. Each structured sample consists of a natural language question and a corresponding SQL statement.

[0050] S2: Construct a multimodal knowledge base. Preprocess the content of the multimodal knowledge base by converting unstructured documents (such as PDF and HTML) into plain text, dividing them into blocks according to semantic granularity (each block is about 200-300 characters), using Sentence-BERT to convert the text into 768-dimensional knowledge vectors, and using the IVF-PQ algorithm to build an index (number of clusters K=4096, number of segments M=8), which improves the retrieval speed by 5 times compared with traditional methods.

[0051] The multimodal knowledge base includes database schemas, domain documents, and historical question-and-answer pairs.

[0052] Database schema: Stores DDL statements for each business database and generates structured metadata through semantic parsing. The DDL statements include table structure, field types, and foreign key relationships.

[0053] Domain documentation includes professional knowledge documents, such as performance indicator definition documents for software development scenarios, like the "Software Development Performance Measurement Specification," which clearly states that the "on-time delivery rate" is calculated as "number of tasks completed on time ÷ total number of tasks × 100%", and notes that "on-time completion" must simultaneously satisfy "task status = completed" and "actual completion time ≤ planned completion time"; the "time deviation rate" is defined as "(actual working hours - planned working hours) ÷ planned working hours × 100%", with positive numbers indicating time overruns and negative numbers indicating time savings, which can help the model understand professional concepts.

[0054] Historical Question and Answer Pairs: Collects queried natural language questions and their corresponding SQL statements, serving as a dynamically expanding reference case library.

[0055] S3: Convert the user-input natural language question into a vector, calculate its similarity with the knowledge vectors stored in the multimodal knowledge base, with a similarity threshold of 0.85, and retrieve the Top-K (K=5) relevant knowledge fragments. Select the knowledge vectors corresponding to the top-K similarity values ​​that are greater than the threshold and sorted in descending order, and then extract the knowledge fragments corresponding to the top-K knowledge vectors.

[0056] The retrieved knowledge fragments (such as relevant table structures and historical SQL examples) are concatenated with natural language questions to form a structured prompt that includes "role setting (Text-to-SQL tool), task description (generate SQL), instructions distinguished by #### delimiter, contextual information (search results), and format constraints (SQL statement)".

[0057] The contextual information includes database schemas retrieved from a multimodal knowledge base, domain documents, and historical question-and-answer pairs.

[0058] The aforementioned Prompt is an enhanced Prompt that supplements business rules, data constraints, and historical logic through retrieval, making the SQL statements generated by the model more accurate and rigorous.

[0059] Use the Prompt as input to LLMforTable and output an SQL statement.

[0060] If the similarity scores are all below the threshold, it means that the search results are empty. In this case, the "fallback mechanism" is triggered, which inputs the natural language question from the user into LLMforTable and outputs an SQL statement.

[0061] Prompt provides precise context for LLMforTable, significantly improving the accuracy of SQL generation, especially in scenarios involving multi-table joins and complex conditional queries.

[0062] The Prompt consists of: role setting (Text-to-SQL tool), task description (generates SQL), instructions separated by #### delimiters, context information (retrieval results), and format constraints (SQL statements).

[0063] Role Setting (Text-to-SQL Tool): Role setting is a fundamental element of structured Prompt. Its design stems from the optimization of the 'role guidance mechanism' in the prompting project: To avoid output deviation caused by task ambiguity in large models, this invention clarifies the role of "you are a Text-to-SQL tool in the software development field, and you need to generate query statements that conform to MySQL syntax based on development data tables and domain knowledge." This allows the model to focus on the 'natural language to SQL conversion' task and eliminates interference from irrelevant abilities (such as text creation and logical reasoning).

[0064] Task Description (SQL Generation): The task description clarifies the core objective of the model. Its design is based on the analysis of query requirements: For natural language questions posed by users (e.g., 'What is the on-time delivery rate for tasks in Q3 2024?'), the task description is defined as "Given the question and the retrieved table information, generate an executable SQL statement that includes field joins, condition filtering, and metric calculation logic." This description ensures task clarity through three layers of constraints: first, input constraints (must use the retrieved table structure and business rules); second, output constraints (must be executable SQL); and third, logical constraints (must conform to the calculation rules of relevant metrics, such as 'On-time delivery rate = number of tasks completed on time / total number of tasks completed').

[0065] The #### separator is used to distinguish instructions: The #### separator is designed to solve parsing errors caused by the mixing of instructions and context information in the Prompt. Large models are prone to confusing the boundaries between 'user instructions' and 'reference knowledge' when processing long texts. This invention uses #### as a fixed separator to divide the Prompt into an "instruction area" (including role settings and task descriptions) and a "context area" (including retrieved knowledge).

[0066] Contextual Information (Retrieval Results): Contextual information is the core output of retrieval enhancement generation technology. It originates from the retrieval results of a multimodal knowledge base. When a user asks a question, the system retrieves relevant knowledge from a vector database (retrieving DDL statements for tables, definitions of related terms, and SQL examples of similar historical queries), and then concatenates this knowledge into contextual information. This design addresses the issues of 'knowledge cutoff' and 'domain knowledge gaps' in large models—for example, when the database table structure is updated (e.g., adding a 'delay_reason' field), there's no need to retrain the model; simply updating the knowledge base allows the model to generate results containing the new field.

[0067] Format constraints (SQL statements): The design goal of format constraints is to ensure the 'executability' of the generated results: SQL must be executed directly by connecting to the database. If it contains natural language descriptions or format errors (such as missing semicolons or incorrect field names), execution will fail. The output format of this invention is 'SQL statement + one line of explanation'.

[0068] Specifically, in S1, the process of fine-tuning the parameters to obtain LLMforTable adopts an incremental difficulty training strategy.

[0069] Structured samples that query only one table in an SQL statement are selected as basic samples. All basic samples constitute the basic dataset, which is then divided into a basic training set and a basic validation set. Other structured samples besides the basic samples are selected as advanced samples. All advanced samples constitute the advanced dataset, which is then divided into an advanced training set and an advanced validation set.

[0070] S1-1: Basic Training Phase: Using samples from the basic training set as input to the revised model, after every T training steps [validated once every 500 training steps using 6.4K samples], the revised model is validated using the basic validation set:

[0071] If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current corrected model is output as the pre-trained model.

[0072] If the loss decreases or the SQL syntax accuracy improves, update the parameters of the corrected model and return to continue training;

[0073] If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current corrected model is output as the pre-trained model.

[0074] S1-2: Advanced training phase. The samples in the advanced training set are used as input to the pre-trained model. After every T training steps [validation is performed once every 500 training steps using 6.4K samples], the current pre-trained model is validated using the advanced validation set.

[0075] If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current pre-trained model is output as LLMforTable.

[0076] If the loss decreases or the SQL syntax accuracy improves, update the parameters of the pre-trained model and return to continue training;

[0077] If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current pre-trained model is output as LLMforTable.

[0078] By employing cosine annealing with a learning rate of 3e-4 initially and a batch size of 8 combined with gradient accumulation (equivalent to a batch size of 32), and introducing Dropout with a value of 0.1 to suppress overfitting, a domain-customized Text-to-SQL model, LLMforTable, is obtained. After fine-tuning, a Text-to-SQL model, LLMforTable, adapted for Chinese table scenarios is obtained, which can generate grammatically correct SQL statements based on natural language problems.

[0079] Example: The process of a table-based question-answering method based on a large model and retrieval enhancement is as follows:

[0080] Users pose natural language questions (e.g., "What tasks were not delivered on time in the third quarter of 2024?"). These questions are converted into vectors, and their similarity is calculated with knowledge vectors stored in a multimodal knowledge base. All knowledge vectors with a similarity greater than a threshold of 0.85 are retained. These knowledge vectors are then sorted in descending order, and the relevant knowledge fragments corresponding to the top-K (K=5) knowledge vectors are selected.

[0081] Creating a structured prompt:

[0082] [Role Setting] You are a Text-to-SQL tool in the software development field, responsible for generating query statements that conform to MySQL syntax based on development data tables and domain knowledge. [Task Description] Based on user questions and retrieved table information, generate directly executable SQL statements that must include field joins, conditional filtering, and metric calculation logic. For "Not Delivered on Time," the task status must be "Completed" and the actual completion time (actual_complete_time) must be greater than the planned completion time (plan_complete_time); the time range is limited to the third quarter of 2024 (July 1, 2024 to September 30, 2024).

[0083] Separator: ##################

[0084] [Contextual Information]

[0085] 1. Database schema (DDL statement fragment):

[0086] CREATE TABLE task(task_id INT PRIMARY KEY COMMENT 'Unique identifier for task',

[0087] task_name VARCHAR(255) NOT NULL COMMENT 'Task Name', status VARCHAR(20) NOT NULL COMMENT 'Task Status: Under Development, Under Development, Under Testing, Completed'

[0088] plan_complete_time DATETIME NOT NULL COMMENT 'Planned completion time',

[0089] actual_complete_time DATETIME COMMENT 'actual completion time',

[0090] project_id INT COMMENT 'Related project ID, foreign key related to the project table');

[0091] 2. Excerpt from domain documentation ("Software Development Performance Measurement Standard"):

[0092] "Tasks not delivered on time" are defined as tasks whose status is "completed" but whose actual completion time is later than the planned completion time. The quarterly time division follows the natural quarter, that is, Q3 is from July 1 to September 30 of each year.

[0093] 3. Historical Q&A Excerpt:

[0094] User question: "What tasks were not delivered on time in the fourth quarter of 2023?"

[0095] Corresponding SQL: SELECT task_id,task_name FROM task WHERE status='Completed'ANDactual_complete_time>plan_complete_time AND actual_complete_time BETWEEN'2023-10-01'AND'2023-12-31';

[0096] [Formatting Constraints] The output should contain only SQL statements and end with a semicolon, for example: SELECT * FROM table WHERE condition;

[0097] In practice, the generated SQL statements undergo syntax validation (based on MySQL syntax rules) to filter out invalid statements. After successful validation, the system connects to the MySQL database to execute the SQL and obtain structured query results (such as tables and numbers). The query results, numbers, or lists are then converted into natural language answers (e.g., "There were 12 tasks not delivered on time in the third quarter of 2024, with task IDs XXX, XXX…"). Multi-turn dialogues are supported, and a dialogue history stack is maintained, recording user questions, retrieved knowledge fragments, and generated SQL statements, allowing for backtracking to any historical node. When subsequent user questions become ambiguous (e.g., "Check again"), the system automatically associates the most recent SQL statement with the context to avoid duplicate generation. This ultimately forms a closed loop: "Natural language question → retrieval enhancement → SQL generation → query execution → natural language answer."

[0098] Analysis of the role of enhanced Prompt:

[0099] When a user queries "What is the on-time delivery rate of R&D tasks in the third quarter of 2024?", if only a basic model (such as an untuned CodeLLaMA) and a simple Prompt are used, the generated Prompt is shown in the appendix. This Prompt only contains the user's question and basic table field information, without addressing noun definitions (e.g., "on-time completion" requires both status = 'completed' and actual_complete_time ≤ plan_complete_time), database field constraints (e.g., quarter format is "YYYYQ3"), and historical query logic, which may cause the model to generate incorrect SQL.

[0100] Experimental content and results

[0101] In the Chinese NL2SQL dataset (containing 51.7K training samples and 6.7K test samples), the core metrics of this invention are as follows:

[0102]

[0103] The superiority of the method of this invention has been fully verified through rigorous comparative experiments. The main comparative objects include:

[0104] 1) The untuned base model CodeLLaMA-7B-Instruct-hf;

[0105] 2) A model obtained using the same training data but with conventional full fine-tuning.

[0106] 1. Experiments were conducted on a test set covering software development scenarios. The results show that the final accuracy of the LLMforTable model (69%) significantly outperforms the basic model (24%) and the conventional fine-tuning model (32%), demonstrating the effectiveness of the LoRA combined with the domain data injection strategy. In terms of stability in handling complex queries (such as those involving more than three table joins or nested aggregate functions), LLMforTable also exhibits far superior performance compared to traditional models. Meanwhile, measured data on resource consumption metrics (number of parameters, GPU requirements) and performance metrics (latency, concurrency) clearly confirm its significant advantages in deployment efficiency and practicality. Real-world application cases of the end-to-end framework further confirm its groundbreaking effect of reducing data analysis response time from days to minutes.

[0107] 2. In terms of core capabilities, the domain-customized LLMforTable model achieved a Text-to-SQL accuracy of 69% in Chinese software development scenarios. This represents a significant leap forward compared to the original basic model CodeLLaMA-7B-Instruct-hf (24%) and the conventional full-scale fine-tuning method (32%), greatly enhancing the accuracy and reliability of natural language queries.

[0108] 3. In terms of resource efficiency, LoRA fine-tuning enables the entire training and deployment process to be completed on a single RTX 3090 GPU, with model response latency controlled to ≤500ms and supporting 10+ concurrent queries per second, significantly reducing the deployment threshold and operating costs for small and medium-sized enterprises.

[0109] 4. By constructing an end-to-end question-and-answer process (natural language → enhanced Prompt → SQL → results → natural language answer) and combining the natural language translation of the results, the user-friendliness and interpretability are significantly improved, enabling non-technical personnel (such as project managers) to complete data analysis independently and quickly (within minutes of response time), completely eliminating the dependence on developers and shortening the traditional analysis response time of 2-3 days to instant interaction.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A table-based question-answering method based on large models and retrieval enhancement, characterized in that: Includes the following steps: S1: CodeLLaMA-7B-Instruct-hf was selected as the basic model. LoRA technology was used to fine-tune the parameters to obtain the Text-to-SQL model LLMforTable. A low-rank matrix was added to the Query and Value projection matrices of the Transformer layer of the basic model to obtain the modified model. Extracted from the TableQA large-scale Chinese NL2SQL dataset, which contains structured samples consisting of several tables, several questions, and several SQL statements. Each structured sample consists of a natural language question and a corresponding SQL statement. S2: Construct a multimodal knowledge base, preprocess the content of the multimodal knowledge base, convert unstructured documents into plain text, divide the text into blocks according to semantic granularity, use Sentence-BERT to convert the text into 768-dimensional knowledge vectors, and use the IVF-PQ algorithm to build an index; The multimodal knowledge base includes database schemas, domain documents, and historical question-and-answer pairs. S3: Convert the natural language question input by the user into a vector, calculate the similarity with the knowledge vectors stored in the multimodal knowledge base, select the top-K similarity vectors with similarity greater than the threshold and sorted in descending order, and then extract the knowledge fragments corresponding to the top-K knowledge vectors. The retrieved knowledge fragments are combined with natural language questions to form a structured prompt that includes "role setting (Text-to-SQL tool), task description (generate SQL), instructions distinguished by #### delimiter, context information (search results), and format constraints (SQL statement)". The context information includes database schemas retrieved from a multimodal knowledge base, domain documents, and historical question-and-answer pairs. The Prompt is used as input to LLMforTable, and the output is an SQL statement. If the similarity scores are all below the threshold, it means that the search results are empty, and the "fallback mechanism" is triggered, which inputs the natural language question entered by the user into LLMforTable and outputs an SQL statement. The generated SQL statement is syntax-validated based on MySQL syntax rules. If the syntax is incorrect, it is filtered out and returned to LLMforTable to regenerate the SQL statement. After the validation is successful, the SQL statement is executed in the MySQL database to obtain the query results.

2. The table-based question-answering method based on a large model and retrieval enhancement as described in claim 1, characterized in that: In S1, the process of fine-tuning the parameters to obtain LLMforTable adopts an incremental difficulty training strategy. Structured samples that query only one table in an SQL statement are selected as basic samples. All basic samples constitute the basic dataset, which is then divided into a basic training set and a basic validation set. Other structured samples besides the basic samples are selected as advanced samples. All advanced samples constitute the advanced dataset, which is then divided into an advanced training set and an advanced validation set. S1-1: Basic Training Phase: The samples in the basic training set are used as input to the revised model. After every T training steps, the revised model is validated using the basic validation set. If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current corrected model is output as the pre-trained model. If the loss decreases or the SQL syntax accuracy improves, update the parameters of the corrected model and return to continue training; If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current corrected model is output as the pre-trained model. S1-2: In the advanced training phase, samples from the advanced training set are used as input to the pre-trained model. After every T training steps, the current pre-trained model is validated using the advanced validation set. If the loss or SQL syntax accuracy remains unchanged after two consecutive validations, the early stopping strategy is triggered to terminate training, and the current pre-trained model is output as LLMforTable. If the loss decreases or the SQL syntax accuracy improves, update the parameters of the pre-trained model and return to continue training; If the loss increases or the SQL syntax accuracy decreases, training is terminated, and the current pre-trained model is output as LLMforTable.

Images