Multi-round text-to-sql method, system, storage medium and electronic device based on edit chain

Abstract

Embodiments of the present application provide a multi-round text-to-SQL method, system, storage medium and electronic equipment based on edit chain. The method comprises: using the current round SQL query statement obtained in the training dialogue data set and the previous SQL query statement corresponding to the current round to construct the current abstract syntax tree and the previous abstract syntax tree; determining the difference sub-node between the current abstract syntax tree and the previous abstract syntax tree; using the difference sub-node to construct the edit chain transformed between the current abstract syntax tree and the previous abstract syntax tree; using the edit chain to learn the context of the large language model, and the large language model after learning is used for multi-round text-to-SQL. Embodiments of the present application do not need to rely on large-scale and high-quality data sets for model training fine-tuning, saving a large amount of computing power and time cost. The method does not need to convert the multi-round text-to-SQL task into a single-round text-to-SQL task, the whole process is automated, and personnel with expert knowledge are not needed for manual annotation, and better performance is achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice, and more particularly to a multi-round text-to-SQL method, system, storage medium, and electronic device based on edit chains. Background Technology

[0002] Text-to-SQL (Text to Structured Query Language) is a semantic parsing task that translates natural language questions into appropriate SQL queries based on a given database schema. This technique is crucial for building natural language interfaces for relational databases, alleviating the burden of annotation in natural language dialogue.

[0003] Existing technologies typically use the following methods to implement text-to-SQL:

[0004] 1. RASAT+PICARD; RASAT is a Transformer sequence-to-sequence architecture with added relation-aware self-attention mechanism, which can handle various relational structures while inheriting the pre-trained parameters of the T5 model; PICARD is a decoding method that rejects illegal tokens during autoregressive decoding.

[0005] 2. DIN-SQL: This is a context learning method that decomposes the text-to-SQL problem into several sub-problems based on a large model, and solves them by calling the API of the large model.

[0006] 3. ACT-SQL; it is also a context learning method that constructs a thought chain based on a large model using a pattern connection method and calls the large model API to solve the problem.

[0007] In the process of realizing this invention, the inventors discovered at least the following problems in the related technology:

[0008] 1. RASAT+PICARD is a traditional deep learning method. Due to the limitations of traditional deep learning, the model itself has a complex structure and is difficult to transfer to different systems; it relies on large-scale and high-quality datasets for model training and tuning; model training and tuning require a lot of computing power and time.

[0009] 2. DIN-SQL: The workflow is relatively complex, requiring multiple steps to call the large model API, resulting in long latency and high cost; the problem decomposition design is difficult to extend to multi-round text-to-SQL tasks.

[0010] 3. ACT-SQL: The schema-connected thought chain design is difficult to extend to multi-round text-to-SQL tasks. Summary of the Invention

[0011] In order to at least address the problem that the complex model structure in existing technologies makes it difficult to apply to multi-round text-to-SQL tasks.

[0012] In a first aspect, embodiments of the present invention provide a multi-round text-to-SQL method based on edit chains, including:

[0013] Using the current round SQL query statement obtained from the training dialogue dataset and the previous SQL query statement corresponding to the current round, construct the current abstract syntax tree and the previous abstract syntax tree;

[0014] Determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree;

[0015] The edit chain between the current abstract syntax tree and the previous abstract syntax tree is constructed using the difference child nodes;

[0016] The large language model is used to learn context using the edit chain, so that the learned large language model can be used for multiple rounds of text-to-SQL.

[0017] Secondly, embodiments of the present invention provide a multi-round text-to-SQL system based on edit chains, comprising:

[0018] The syntax tree determination module is used to construct the current abstract syntax tree and the prior abstract syntax tree using the current round SQL query statement obtained from the training dialogue dataset and the prior SQL query statement corresponding to the current round.

[0019] The difference child node determination module is used to determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree;

[0020] The edit chain determination module is used to construct an edit chain for transformation between the current abstract syntax tree and the prior abstract syntax tree using the difference child nodes;

[0021] The learning module is used to perform context learning on the large language model using the edit chain, so that the learned large language model can be used for multiple rounds of text-to-SQL.

[0022] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-round text-to-SQL method based on edit chains according to any embodiment of the present invention.

[0023] Fourthly, embodiments of the present invention provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the multi-round text-to-SQL method based on edit chains according to any embodiment of the present invention.

[0024] Fifthly, embodiments of the present invention provide a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the steps of the multi-round text-to-SQL method based on edit chains according to any embodiment of the present invention.

[0025] The beneficial effects of this invention are as follows: This method provides a novel large-scale model context learning approach for multi-round text-to-SQL tasks. Compared to existing deep learning methods, this method's edit-chain-based context learning does not require large-scale, high-quality datasets for model training and fine-tuning, saving significant computational and time costs. Furthermore, this method eliminates the need to transform multi-round text-to-SQL tasks into single-round text-to-SQL tasks, and the entire process is automated, eliminating the need for manual annotation by personnel with expert knowledge, thus achieving better performance. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a multi-round text-to-SQL method based on an edit chain, provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram comparing two ASTs of a multi-round text-to-SQL method based on edit chains provided in an embodiment of the present invention;

[0029] Figure 3 This is a performance diagram of a CoE-SQL based on a multi-round text-to-SQL method using an edit chain, provided by an embodiment of the present invention, and existing technologies on the SParC development set;

[0030] Figure 4 This is a performance diagram of CoE-SQL based on a multi-round text-to-SQL method using edit chains, provided by an embodiment of the present invention, and existing technologies on the CoSQL development set;

[0031] Figure 5 This is a performance diagram of CoE-SQL, a multi-round text-to-SQL method based on edit chains provided by an embodiment of the present invention, and existing technologies on the SParC and CoSQL test sets;

[0032] Figure 6 This is a performance diagram of a CoE-SQL method based on an edit chain and a baseline method on the SParC development set, provided by an embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram illustrating three prompt text styles on the SParC development set, based on a multi-round text-to-SQL method using an edit chain, according to an embodiment of the present invention, where CoE-SQL performance is conditionally affected.

[0034] Figure 8 This is a schematic diagram illustrating the CoE-SQL performance on the SParC development set based on a multi-round text-to-SQL method using an edit chain, according to an embodiment of the present invention, as a result of different Python code styles of version chain hint text. The "*" mark indicates that this specific component is used to complete the postprocessing.

[0035] Figure 9 This is a schematic diagram illustrating the CoE-SQL performance on the SParC development set, based on a multi-round text-to-SQL method using an edit chain, as provided in an embodiment of the present invention, affected by the "no changes" prompt text.

[0036] Figure 10 This is a schematic diagram illustrating the impact of a multi-round text-to-SQL method based on edit chains, provided by an embodiment of the present invention, on the CoE-SQL performance on the SParC development set.

[0037] Figure 11 This is a schematic diagram illustrating how the performance of CoE-SQL on the SParC development set, based on a multi-round text-to-SQL method using edit chains, is affected by the maximum length of the edit chain, according to an embodiment of the present invention.

[0038] Figure 12 This is a schematic diagram illustrating the impact of cell editing rule definitions on the CoE-SQL performance of the SParC development set based on a multi-round text-to-SQL method using edit chains, according to an embodiment of the present invention. Here, F represents the cell editing rule for the FROM clause, S represents the cell editing rule for the SELECT clause, and W represents the cell editing rule for the WHERE clause.

[0039] Figure 13This is a schematic diagram of the structure of a multi-round text-to-SQL system based on an edit chain, provided in an embodiment of the present invention;

[0040] Figure 14 This is a schematic diagram of an embodiment of an electronic device based on an edit chain and multi-round text-to-SQL, provided by an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] like Figure 1 The diagram shows a flowchart of a multi-round text-to-SQL method based on an edit chain according to an embodiment of the present invention, which includes the following steps:

[0043] S11: Construct the current abstract syntax tree and the previous abstract syntax tree using the current round SQL query statement obtained from the training dialogue dataset and the previous SQL query statement corresponding to the current round;

[0044] S12: Determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree;

[0045] S13: Construct the edit chain of transformations between the current abstract syntax tree and the prior abstract syntax tree using the difference child nodes;

[0046] S14: Use the edit chain to perform context learning on the large language model, so that the learned large language model can be used for multiple rounds of text-to-SQL.

[0047] In this implementation, instead of transforming the multi-turn text-to-SQL task into a single-turn text-to-SQL task, the problem is solved directly using LLM (Large Language Models) within the context. In a multi-turn dialogue context, because the two questions are related, the SQL statements corresponding to the two questions often only require a few simple rules to transform into each other. This method can be called CoE (Chain-of-Edition)-SQL.

[0048] For step S11, this method can utilize a small number of samples for context learning. The multi-round text-to-SQL task in the setup can be formulated as follows:

[0049]

[0050] Among them, R n This represents the response to the current question created by the LLM. I represents the instruction. D represents the training dialogue dataset. Q = [Q1, Q2, ..., Q...] n [] represents the entire context consisting of n questions in the training dialogue dataset. R = [R1, R2, ..., R] n-1 ] represents the LLM's answer to the preceding question Q in the training dialogue dataset. ε = [E1, E2, ..., E |ε| [] is a list of |E| examples used in minority sample learning.

[0051] In real-world scenarios, users are more likely to begin a conversation with a relatively simple question because they are unfamiliar with the system's detailed structure. As the number of sessions increases, user questions and corresponding SQL queries become more complex. Generating complex SQL queries from scratch using LLM becomes more difficult and redundant because the entire thought process and logical reasoning are often intricate. Instead, leveraging the previous query to generate the current SQL query is a better option.

[0052] This method provides the definition of cell editing rules, which can help SQL queries. To edit an SQL query into another SQL query, this method defines a set of cell editing rules. A total of 14 cell editing rules are defined based on different SQL queries, as shown below:

[0053] Cell editing rules - Explanation:

[0054] 1. EditSelectItem(oldItem,newItem)

[0055] Replace oldItem with newItem in the SELECT clause.

[0056] Add newItem into the SELECT clause if oldItem is "-".

[0057] Delete oldItem from the SELECT clause if newItem is "-".

[0058] 2. EditFromTable(oldTable,newTable)

[0059] Replace oldTable with newTable in the FROM clause.

[0060] Add newTable into the FROM clause if oldTable is“-”.

[0061] Delete oldTable from the FROM clause if newTable is“-”.

[0062] 3、EditNestedFromClause(SQL)

[0063] Edit the nested FROM clause with SQL.

[0064] Delete the nested FROM clause if SQL is“-”.

[0065] 4、EditJoinCondition(oldCondition,newCondition)

[0066] Replace oldCondition with newCondition in the ON clause.

[0067] Add newCondition into the ON clause if oldCondition is“-”.

[0068] Delete oldCondition from the ON clause if newCondition is“-”.

[0069] 5、EditJoinLogicalOperator(and / or)

[0070] Edit the logical operator in the ON clause.

[0071] 6、EditWhereCondition(oldCondition,newCondition)

[0072] Replace oldCondition with newCondition in the WHERE clause.

[0073] Add newCondition into the WHERE clause if oldCondition is“-”.

[0074] Delete oldCondition from the WHERE clause if newCondition is“-”.

[0075] 7、EditWhereLogicalOperator(and / or)

[0076] Edit the logical operator in the WHERE clause.

[0077] 8、EditGroupByColumn(oldColumn,newColumn)

[0078] Replace oldColumn with newColumn in the GROUP BY clause.

[0079] Add newColumn into the GROUP BY clause if oldColumn is“-”.

[0080] Delete oldColumn from the GROUP BY clause if newColumn is“-”.

[0081] 9、EditHavingCondition(oldCondition,newCondition)

[0082] Replace oldCondition with newCondition in the HAVING clause.

[0083] Add newCondition into the HAVING clause if oldCondition is“-”.

[0084] Delete oldCondition from the HAVING clause if newCondition is“-”.

[0085] 10、EditHavingLogicalOperator(and / or)

[0086] Edit the logical operator in the HAVING clause.

[0087] 11、EditOrderByItem(oldItem,newItem)

[0088] Replace oldItem with newItemin the ORDER BY clause.

[0089] Add newIteminto the ORDER BY clause if oldItem is“-”.

[0090] Delete oldItem from the ORDER BY clause if newItem is“-”.

[0091] 12、EditOrder(asc / desc)

[0092] Edit the order in the ORDER BY clause.

[0093] 13、EditLimit(oldLimit,newLimit)

[0094] Replace oldLimit with newLimit in the LIMIT clause.

[0095] Add newLimit into the LIMIT clause if oldLimit is“-”.

[0096] Delete oldLimit from the LIMIT clause if newLimit is“-”.

[0097] 14、EditIUE(intersect / union / except,left / right,SQL)

[0098] Append SQL to the left / right side of the previous SQL with intersect / union / except keyword.

[0099] Delete the left / right side of the previous SQL with intersect / union / except keyword if SQL is "-".

[0100] Take the following dialogue example as an illustration,

[0101] Turn 1

[0102] Q:What are the phone names available in Alberta?

[0103] SQL:SELECT T1.Name FROM phone AS T1 JOIN phone_market AS T2 JOIN market AS T3 ON T1.Phone_ID=T2.Phone_ID AND T2.Market_ID=T3.Market_ID WHERE T3.District="Alberta"

[0104] Edit:no edition

[0105] Turn 2

[0106] Q:Show the district as well.

[0107] SQL:SELECT T1.Name,T3.District FROM phone AS T1 JOIN phone_market AS T2 JOIN market AS T3 ON T1.Phone_ID=T2.Phone_ID AND T2.Market_ID=T3.Market_ID WHERE T3.District="Alberta"

[0108] Edit:add select column T3.District

[0109] Turn 3

[0110] Q:Remove the restriction of those in Alberta only,show all instead.

[0111] SQL:SELECT T1.Name,T3.District FROM phone AS T1 JOIN phone_market AST2 JOIN market AS T3 ON T1.Phone_ID=T2.Phone_ID AND T2.Market_ID=T3.Market_ID

[0112] Edit:delete where clause T3.District="Alberta"

[0113] You can use the cell editing rule EditSelectItem(-, market.District) to edit SQL 1 into SQL 2. You can also use the cell editing rule EditWhereCondition(market.District="Alberta",-) to edit SQL 2 into SQL 3.

[0114] Suppose the dataset example consists of n questions [Q1, Q2, ..., Q... n It consists of n corresponding SQL queries [S1, S2, ..., Sn]. The i-th query is considered as the current count. It is worth noting that a reasonable way to extract the edit chain is to compare two ASTs (abstract syntax trees). Figure 2 An example comparison between two ASTs is shown. The FROM clause component is omitted from the figure.

[0115] For step S12, Figure 2 Taking the two ASTs in the example diagram as examples, the syntax rules are as follows:

[0116] 1. root=Root(sqlUnit,intersect,union,except)

[0117] 2. sqlUnit=SQL(valUnit*,from,where,groupBy,orderBy,limit)

[0118] 3. valUnit=ValUnit(agg,distinct,calcOp,col1,col2)

[0119] 4. where=Conds(cond*,logicOp)

[0120] 5. cond=Cond(valUnit,compOp,val1,val2)

[0121] 6. ...

[0122] The difference child nodes of the syntax rules are found using a depth-first traversal, comparing each node pair in the two ASTs. As shown in the figure, the difference child nodes found are Rule5, valUnit→Rule3, compOp→">", vall→3, and coll→Stu.GPA.

[0123] For step S13, the edit chain is formed by the current SQL query S i And the previous SQL query S j The differences between (j < i) are determined. For determining the edit chain, this method uses a few-shot learning approach to activate the LLM's ability to utilize predefined unit editing rules. Therefore, this method requires selecting samples from the training dataset and then extracting the edit chain from each session. This method randomly selects k. d Database schema, and then randomly select k for each database schema. e Example dataset. Therefore, the total k d ×k e The samples are placed in the cue text for few-shot learning. In the following process, this method needs to extract the edit chains from these examples.

[0124] Edit chains can be constructed based on unequal parts. By identifying the syntax rules of nodes in the unequal parts, it can be determined that the edit chain contains exactly one unit edit rule, i.e., EditWhereCondition(-, Stu.GPA>3). The following algorithm outlines the detailed process:

[0125]

[0126]

[0127] The algorithm described above can be used to determine the current abstract syntax tree and the edit chain between previous abstract syntax trees.

[0128] In one implementation, the current round of the SQL query statement is the i-th round, and the previous round of the SQL query statement is the (i-1)-th round.

[0129] As another implementation, the current round of the SQL query statement is the i-th round, and the previous round of the SQL query statement is the j-th round, where j < i.

[0130] In this implementation, within the context of the SQL query statement in the dialogue, the current question Q... i It may not be inherited from the previous question Q. i-1 Instead, it is inherited from the previous question Q. j (j <i-1)。

[0131] For example, in a hospital setting: The user says, "I want to register," and the smart device replies, "Which department do you want to register with?" The user then asks, "My hand is numb, which department should I register with?" The smart device responds, "There are many factors that can cause hand numbness; we recommend neurology, orthopedics, or endocrinology." The user asks, "What are the differences between these departments?" The smart device then replies, "If your numbness is caused by condition A, you can register with neurology; if it's caused by…", this is the approach used for the current question Q. i Is it inherited from the previous question Q? i-1 .

[0132] In another scenario, a navigation scenario, the user asks, "What are some good restaurants nearby?" The smart device replies, "Restaurant A, Restaurant B, Restaurant C." The user says, "Let's go to Restaurant A." The smart device replies, "Okay, I'll navigate to Restaurant A." The user says, "Never mind, let's go to Restaurant B instead." The smart device replies, "Okay, I'll navigate to Restaurant B." In this case, the current question Q... i It may not be inherited from the previous question Q. i-1 Instead, it is inherited from the previous question Q. j (j <i-1)。

[0133] Furthermore, the current problem may raise a new topic unrelated to the original problem. To address these two issues, this method enumerates all i-1 SQL queries and, for each SQL pair (S... i S j An edit chain CoE was constructed. i,j , where j = 1, 2, ..., i-1.

[0134] As one implementation, the method further includes: removing edit chains whose length is greater than a preset threshold.

[0135] In this embodiment, the method eliminates those with a length greater than l c The edit chain (consisting of more than l) c An edit chain consisting of individual edit rules, where l c This is a threshold. If the edit chain contains too many unit edit rules, the two corresponding issues are considered unrelated. In this case, writing the current SQL query directly is more efficient than editing from the previous query. For the edit chains on the left, this method selects the edit chain with the fewest tokens as the final edit chain.

[0136] For step S14, the edit chain is input into the large language model for learning. Specifically, the context learning of the large language model using the edit chain includes: determining the edit chain as model prompt words for the large language model according to editing rules, Python code, and natural language; using the large language model to understand and learn the model prompt words, so that the large language model imitates the context output format of the model prompt words; specifically, the edit chain is directly used as model prompt words for the large language model according to editing rules; the SQL query statements stored in the large language model are determined as Python dicts, and the edit chain is converted into Python code as model prompt words to update the Python dicts; the edit chain is converted into model prompt words in natural language.

[0137] In this implementation, using edit rule styles, this method directly writes the unit edit rules into the prompt text. In the Python code style, the SQL query is treated as a Python dict, and the unit edit rules are converted into corresponding Python code to update the Python dict. In the natural language style, the unit edit rules are converted into plain text that more closely resembles the corpus used in the LLM pre-training step. Specifically, this method attempts to place the edit chain in the prompt text using three different styles. The example edit chain consists of two unit edit rules: EditSelectItem(*, COUNT(*)) and EditWhereCondition(-, Stu.GPA>3).

[0138] Modify the above edit chain according to the edit rules to use the following model prompt words:

[0139] FROM clause:

[0140] No change is needed.

[0141] SELECT clause:

[0142] -EditSelectItem(*,COUNT(*))

[0143] WHERE clause:

[0144] -EditWhereCondition(-,Stu.GPA>3)

[0145] GROUP BY clause:

[0146] No change is needed.

[0147] ORDER BY clause:

[0148] No change is needed.

[0149] LIMIT clause:

[0150] No change is needed.

[0151] INTERSECT / UNION / EXCEPT:

[0152] No change is needed.

[0153] Modify the above edit chain according to the Python code to give the model prompt words as follows:

[0154] sql['select'].remove('*')

[0155] sql['select'].append('COUNT(*)')

[0156] sql['where']['conditions'].append('Stu.GPA>3')

[0157] The above edit chain is modified into model prompts in natural language as follows:

[0158] FROM clause:

[0159] No change is needed.

[0160] SELECT clause:

[0161] -change*to COUNT(*)

[0162] WHERE clause:

[0163] -add WHERE condition Stu.GPA>3

[0164] GROUP BY clause:

[0165] No change is needed.

[0166] ORDER BY clause:

[0167] No change is needed.

[0168] LIMIT clause:

[0169] No change is needed.

[0170] INTERSECT / UNION / EXCEPT:

[0171] No change is needed.

[0172] The model prompts for the large language model are obtained through the above method. The large language model is then used to learn and understand these prompts. In other words, only the inference API of the large language model is called to perform contextual learning on the prompts. This contextual learning is not the residual training of prediction and baseline results commonly used in existing technologies, but rather uses the model prompt engineering to enable the large language model's inference API to understand and mimic the output format of the context. After contextual learning, when processing text-to-SQL functions in multi-turn dialogues, the large language model can infer the current turn's SQL statement based on the SQL statements mentioned above.

[0173] As can be seen from this implementation, this method provides a novel large-scale model context learning approach for multi-round text-to-SQL tasks. Compared to existing deep learning methods, this edit-chain-based context learning method does not require large-scale and high-quality datasets for model training and fine-tuning, saving significant computational and time costs. This method does not require converting multi-round text-to-SQL tasks into single-round text-to-SQL tasks, and the entire process is automated, eliminating the need for manual annotation by personnel with expert knowledge, thus achieving better performance.

[0174] Experiments illustrate this method, using the GPT-3.5-turb-16k model to evaluate its CoE-SQL. This method expects the LLM to have a long context window. This is why GPT-3.5-turb-16k with a 16385-terminal context window was chosen instead of GPT-3.5mTurbo with a 4096-terminal context window.

[0175] In the LLM API, setting the `temperature` parameter to 0 indicates that a greedy decoding strategy is being used. Text-to-SQL tasks require the model to generate SQL queries that conform to strict grammar rules. If set too high, LLM may generate invalid or irrelevant SQL queries. Regarding examples for few-shot learning, consider the number of database patterns, k. d Set it to 4, and set the number of examples k for each database schema. e Set it to 4.

[0176] The performance of our proposed CoE-SQL method was evaluated on the SParC and CoSQL datasets. SParC consists of 4298 coherent question sequences, including over 12000 individual questions and associated SQL queries. CoSQL contains over 10k annotated SQL queries. Each dialogue in CoSQL is designed to simulate a real-world scenario where a casual user is exploring a database and an expert is retrieving answers using SQL. An evaluation script is also provided, categorizing the SQL queries into four difficulty levels (easy, medium, hard, and very hard).

[0177] The performance of this method is evaluated using three commonly used evaluation metrics for text-to-SQL tasks: Exact Match Accuracy (EM), Execution Accuracy (EX), and Test Suite Accuracy (TS). EM requires that each component of the predicted SQL is identical to the corresponding component of the golden SQL, regardless of the values ​​in the query. EX evaluates the correctness of the execution results of the predicted SQL, which is generally more accurate than EM.

[0178] Since this method evaluates the performance of LLM in multi-turn text-to-SQL tasks, it is necessary to consider question matching accuracy (QM) and interaction matching accuracy (IM) separately. QM is 1 if the predicted SQL query for a single question is correct, and IM is 1 if all predicted SQL queries in the context are correct.

[0179] In the experiment, a natural language style was selected for the edit chain. When testing on SParC, the maximum length of the edit chain was set to l. c It was set to 4, and then to 3 when testing on CoSQL. Figure 3 and Figure 4 The performance of our proposed CoE-SQL method and other existing techniques on the development sets of SParC and CoSQL is shown. It is worth noting that the EM evaluation metric is not particularly relevant when comparing context learning methods with fine-tuning models. Fine-tuning models learn dataset features from the training set. These models are more likely to generate SQL queries with the same structure as the SQL query, thus achieving higher EM scores. Conversely, LLM tends to learn SQL queries based on its raw knowledge during the pre-training phase. Only a few samples from the training dataset cannot provide sufficient information about the dataset's features. Therefore, LLM is more likely to generate SQL queries with accurate semantics and logic, and correct execution results. Generally, our approach aims to focus primarily on the EX and TS evaluation metrics in the following discussion. Most fine-tuning models only provide EM scores. Our approach is compared with existing techniques GAZP and RASAT, as these two models provide their EX scores, with RASAT being the state-of-the-art (SOTA) score.

[0180] Compared with the fine-tuning model, the proposed CoE-SQL method achieved an EX(QM) score of 70.3% and an EX(IM) score of 50.5% on the SParC development set, which significantly outperformed the GAZP+BERT model and was comparable to the RASAT+PICARD model.

[0181] Experimental results demonstrate that LLMs possess strong capabilities for handling complex multi-turn text-to-SQL tasks. Using the GPT-3.5-turbo-16k LLM, the CoE-SQL method performs almost as well as the previous best fine-tuned model (EX score). With the application of a larger LLM (e.g., GPT-4), the CoE-SQL method can achieve even better performance.

[0182] Furthermore, the CoE-SQL method achieved the highest EM, EX, and TS scores among existing context learning methods. The ACTSQL method transforms multi-turn datasets into single-turn datasets by rewriting and completing context-dependent questions. Comparing the baseline methods and the ACT-SQL method, it can be concluded that paraphrasing multi-turn datasets using LLM is not a good choice; its performance is even worse than the baseline methods. Based on edit operations, CoE-SQL performs significantly better than the baseline methods that only use the original database schema and question as LLM input. This indicates that editing SQL queries is an effective method for multi-turn text-to-SQL tasks.

[0183] This method was also tested on the SParC and CoSQL test sets. The experimental results are as follows: Figure 5 As shown. Under the EX metric, the CoE-SQL method of this approach maintains good performance, even surpassing the SOTA fine-tuning model on the test set.

[0184] This method demonstrates an example from the SParC development set:

[0185] Turn 1

[0186] Question:Find all employees who are under age 30.

[0187] Baseline:SELECT*FROM employee WHERE age<30√

[0188] CoE-SQL:SELECT*FROM employee WHERE age<30√

[0189] Turn 2

[0190] Question:Which cities did they come from?

[0191] Baseline:SELECT DISTINCT city FROM employee WHERE age<30√

[0192] CoE-SQL:SELECT City FROM employee WHERE age<30√

[0193] Turn 3

[0194] Question:Show the cities from which more than one employee originated.

[0195] Baseline:SELECT city FROM employee GROUP BY city HAVING count(*)>1X

[0196] CoE-SQL:SELECT City FROM employee WHERE age<30GROUP BY City HAVINGCOUNT(*)>1√

[0197] The first question mentions the condition "employees under 30 years old". The baseline method successfully preserved this condition in the second query but forgot it in the last query. However, due to the edit chain mechanism, the CoE-SQL method preserved this condition from beginning to end.

[0198] In addition to the baseline method, this method also attempts a high-level method based on CoT. For example... Figure 6 The results demonstrate that the simplified CoT method is beneficial for LLM compared to the most trivial baselines. Equally important, the CoE-based method is better suited for multi-turn text-to-SQL tasks.

[0199] like Figure 7The performance of our CoE-SQL method on the SParC development set is shown, and the experimental results demonstrate that the natural language style is the most suitable language style. This is because LLMs like GPT models are mostly trained on natural language corpora. The edit chain style with edit rules performs relatively poorly because the unit edit rules are self-designed and unlikely to appear in the pre-training corpus. Although the LLM must have seen a lot of Python code during pre-training, the Python code style still scores low. This is attributed to the fact that the Python code generated by the LLM is used to update the Pythondict representing the SQL query.

[0200] When using Python code styles, the suggestion text consists primarily of three parts: the Python code representing the editing rules, the Python dict representing the SQL components, and the current SQL query. Therefore, this method completes further ablation studies on LLM generation. Results are as follows... Figure 8 As shown. First, this method changes the order of the Python dict and SQL query in the example. During post-processing, the LLM-generated code is run to update the dict, and the SQL is obtained by parsing the updated dict. During post-processing of the dict, the SQL is directly obtained by parsing the LLM-generated dict. Experiments show that adding the dict after the SQL is generally a better choice. The Python code generated by LLM is not very reliable. It is best to use either Pythondict or the SQL itself directly. Figure 9 The experimental results demonstrate the necessity of the cue text. By adding this specific sentence, the CoE cue text in the example can be regular and canonical because all components and clauses can be referenced in the context. LLMs are generally better at receiving and processing regularized and canonical contexts. Figure 10 This demonstrates that the difference analysis between the current and previous problems is effective. Analyzing the differences between the current and previous problems helps LLMs to consider more possible editing rules.

[0201] If the edit chain is too long, LLM will directly generate an SQL query instead of using an edit-based approach. This method can be modified by changing the maximum length of the edit chain. c This controls the coverage of the edit chain on the training dataset. Figure 11 Showing l c The impact on performance on the SParC dev set. Based on the experimental results, l will be used in the main experiment. c Set to 4. If l c If it's too small, an LLM is more likely to directly generate SQL queries rather than using an edit-based approach. If l cIf the value is too large, LLM will always edit the SQL query, even though the CoE-SQL approach may not be suitable for the current test case.

[0202] This method also attempts another approach to control coverage in the training dataset. Relatively complex cell editing rules are removed, retaining only commonly used rules. Therefore, if the SQL query contains complex clauses, LLM will not force the SQL query to be edited. Figure 12 The experimental results are shown under different definitions of cell editing rules. In the first row, only cell editing rules involving the SELECT and WHERE clauses are retained. In the second row, cell editing rules related to the FROM clause are added. In the third row, no restrictions are set. The experimental results show that this approach is not very effective. The performance of LLM does not change significantly with the changes in cell editing rules. This explains why this method uses the maximum length of the edit chain to limit the edit-based approach.

[0203] In summary, this paper proposes a CoE-SQL method for multi-turn text-to-SQL tasks. The definition and style of cell editing rules are explained. A method for extracting edit chains by comparing two ASTs of two different SQL queries is also provided. Furthermore, the proposed CoE framework follows human thought processes. Experimental results show that among existing context learning methods, this method achieves state-of-the-art performance on the SParC and CoSQL development sets, and is comparable to state-of-the-art fine-tuned models.

[0204] like Figure 13 The diagram shows a schematic of a multi-round text-to-SQL system based on an edit chain according to an embodiment of the present invention. This system can execute the multi-round text-to-SQL method based on an edit chain as described in any of the above embodiments and is configured in the terminal.

[0205] This embodiment provides a multi-round text-to-SQL system 10 based on edit chains, which includes: a syntax tree determination module 11, a difference child node determination module 12, an edit chain determination module 13, and a learning module 14.

[0206] The syntax tree determination module 11 is used to construct a current abstract syntax tree and a prior abstract syntax tree using the current round SQL query statement obtained from the training dialogue dataset and the prior SQL query statement corresponding to the current round; the difference child node determination module 12 is used to determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree; the edit chain determination module 13 is used to construct an edit chain that transforms between the current abstract syntax tree and the prior abstract syntax tree using the difference child nodes; and the learning module 14 is used to perform context learning on the large language model using the edit chain, so that the learned large language model can be used for multi-round text-to-SQL.

[0207] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the multi-round text-to-SQL method based on edit chains in any of the above method embodiments;

[0208] In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows:

[0209] Using the current round SQL query statement obtained from the training dialogue dataset and the previous SQL query statement corresponding to the current round, construct the current abstract syntax tree and the previous abstract syntax tree;

[0210] Determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree;

[0211] The edit chain between the current abstract syntax tree and the previous abstract syntax tree is constructed using the difference child nodes;

[0212] The large language model is used to learn context using the edit chain, so that the learned large language model can be used for multiple rounds of text-to-SQL.

[0213] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the multi-round text-to-SQL method based on edit chains in any of the above method embodiments is executed.

[0214] Figure 14 This is a schematic diagram of the hardware structure of an electronic device based on a multi-round text-to-SQL method using edit chains, as provided in another embodiment of this application. Figure 14 As shown, the device includes:

[0215] One or more processors 1410 and memory 1420, Figure 14 Taking a processor 1410 as an example, the device based on the multi-round text-to-SQL method of edit chains may also include an input device 1430 and an output device 1440.

[0216] The processor 1410, memory 1420, input device 1430, and output device 1440 can be connected via a bus or other means. Figure 14 Taking the example of a connection between China and Israel via a bus.

[0217] The memory 1420, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-round text-to-SQL method based on edit chains in the embodiments of this application. The processor 1410 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1420, thereby implementing the multi-round text-to-SQL method based on edit chains in the above-described method embodiments.

[0218] The memory 1420 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 1420 may optionally include memory remotely located relative to the processor 1410, and these remote memories may be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0219] Input device 1430 can receive input numerical or character information. Output device 1440 may include display devices such as a display screen.

[0220] The one or more modules are stored in the memory 1420, and when executed by the one or more processors 1410, they execute the multi-round text-to-SQL method based on the edit chain in any of the above method embodiments.

[0221] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0222] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0223] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the multi-round text-to-SQL method based on edit chains according to any embodiment of this invention.

[0224] The electronic devices described in this application exist in various forms, including but not limited to:

[0225] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0226] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0227] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0228] (4) Other electronic devices with data processing functions.

[0229] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0230] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0231] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-round text-to-SQL method based on edit chains, comprising: Using the current round SQL query statement obtained from the training dialogue dataset and the previous SQL query statement corresponding to the current round, construct the current abstract syntax tree and the previous abstract syntax tree; Determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree; The edit chain between the current abstract syntax tree and the previous abstract syntax tree is constructed using the difference child nodes; The large language model is used to learn context using the edit chain, so that the learned large language model can be used for multiple rounds of text-to-SQL.

2. The method of claim 1, wherein, The context learning of the large language model using the edit chain includes: The edit chain is determined as the model prompt words of the large language model according to the edit rules, Python code, and natural language form; The large language model is used to learn and understand the model prompt words, so that the large language model can imitate the contextual output format of the model prompt words.

3. The method of claim 2, wherein, The process of determining the edit chain as model prompts for the large language model according to edit rules, Python code, and natural language includes: The edit chain is directly used as the model prompt word of the large language model according to the editing rules; The SQL query statements stored in the large language model are identified as Python dicts, and the edit chain is converted into Python code as model prompt words to update the Python dicts; The edit chain is converted into model prompts in natural language.

4. The method of claim 1, wherein, The current round of the SQL query statement is the i-th round, and the previous round of the SQL query statement is the (i-1)-th round.

5. The method of claim 1, wherein, The current round of the SQL query statement is round i, and the previous round of the SQL query statement is round j, where j < i.

6. The method of claim 1, wherein, After constructing the current abstract syntax tree and the edit chain between the previous abstract syntax tree using the difference child nodes, the method further includes: removing edit chains with a length greater than a preset threshold.

7. A multi-round text-to-SQL system based on edit chains, comprising: The syntax tree determination module is used to construct the current abstract syntax tree and the prior abstract syntax tree using the current round SQL query statement obtained from the training dialogue dataset and the prior SQL query statement corresponding to the current round. The difference child node determination module is used to determine the difference child nodes between the current abstract syntax tree and the prior abstract syntax tree; The edit chain determination module is used to construct an edit chain for transformation between the current abstract syntax tree and the prior abstract syntax tree using the difference child nodes; The learning module is used to perform context learning on the large language model using the edit chain, so that the learned large language model can be used for multiple rounds of text-to-SQL.

8. A storage medium having stored thereon a computer program product, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

9. A computer program product having instructions embedded on a storage medium, the instructions implementing the steps of the method according to any one of claims 1-6.

10. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-6.

Images