A Text2SQL Dataset Generation Method Based on Dynamic Difficulty Adjustment

Abstract

This invention discloses a method for generating Text2SQL datasets based on dynamic difficulty adjustment. The method proposes a complexity metric to evaluate the complexity of SQL statements and text. A large language model is used to generate a Text2SQL dataset based on a prompt word generation template, incorporating dynamic constraints into the template. The complexity of the data generated by the large language model is evaluated using the complexity metric, and compared with a preset difficulty target distribution. The dynamic constraints are then modified to guide the model to generate data that conforms to the expected target distribution. Simultaneously, validation and correction templates are set to ensure the execution and semantic correctness of the generated data, improving the efficiency and quality of dataset construction, ensuring dataset diversity and relevance, and providing the model with more comprehensive and practical training data. This helps improve the generalization ability and performance of the Text2SQL model in different scenarios.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing technology, and relates to the generation of Text2SQL datasets, specifically to a method for generating Text2SQL datasets based on dynamic difficulty adjustment. Background Technology

[0002] Databases, as core tools for data storage and management, are widely used in numerous fields such as scientific research, finance, healthcare, and e-commerce. The Text2SQL task aims to convert natural language questions into SQL queries, enabling users to easily retrieve the information they need from databases, playing a crucial role in data analysis and decision support.

[0003] With the rapid development of artificial intelligence and machine learning technologies, the demand for high-quality, large-scale Text2SQL datasets with a reasonable difficulty distribution is becoming increasingly urgent. However, the construction of traditional Text2SQL datasets faces significant bottlenecks: manual annotation is time-consuming and labor-intensive, and simply automatically generated datasets often suffer from difficulty imbalances and cannot be dynamically adjusted according to the needs of actual application scenarios. This results in poor controllability of the generated datasets in terms of diversity and difficulty. Specifically, the following issues are particularly prominent:

[0004] (1) Insufficient diversity: SQL queries generated by fixed-template methods are usually concentrated on certain specific patterns, lacking sufficient variation and diversity. For example, if a template can only generate simple queries involving a single table, it is difficult for the generated dataset to include complex queries involving multiple table joins. This uniformity not only limits the richness of the dataset but may also lead to poor model performance when faced with diverse queries. In reality, database queries in the real world often involve complex relationships between multiple tables, and fixed-template methods struggle to simulate these complex scenarios, thus reducing the effectiveness of the dataset.

[0005] (2) Insufficient difficulty control: Existing technologies struggle to flexibly adjust the complexity of SQL queries when generating datasets. For example, in some application scenarios, it may be necessary to generate more complex queries to test the model's ability to handle complex logic; while in other scenarios, it may be necessary to generate simpler queries for preliminary evaluation or debugging. However, the fixed template method cannot dynamically adjust the complexity of queries according to requirements, resulting in datasets that are either too simple to fully test the model's performance or too complex, exceeding the current stage of requirements.

[0006] (3) Lack of dynamic optimization mechanism: Existing technologies typically use one-time templates or rule settings when generating datasets, lacking a subsequent dynamic adjustment mechanism. This means that even if problems are found in the generated dataset, they cannot be corrected without redesigning the template. For example, if the proportion of subqueries in the generated dataset is found to be too high, or certain types of queries are missing, the only solution is to manually modify the template or rules. This static approach not only increases workload but may also lead to a cumbersome and inefficient generation process.

[0007] Against this backdrop, the implementation of an automatic Text2SQL dataset construction system based on dynamic difficulty adjustment is of great significance. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a Text2SQL dataset generation method based on dynamic difficulty adjustment. It introduces a dynamic constraint component into the prompt word generation template, evaluates the complexity of the data generated by the large language model, compares it with a preset difficulty target distribution, modifies the dynamic constraints, and guides the model to generate data that conforms to the expected target distribution to meet training requirements.

[0009] The method for generating Text2SQL datasets based on dynamic difficulty adjustment includes the following steps:

[0010] Step 1: Initialize parameters

[0011] The complexity metrics of SQL statements and text are mapped to three difficulty ranges: low, medium, and high, and a target distribution is defined. The complexity metrics of SQL statements include structural complexity, semantic complexity, and syntactic complexity. The complexity metrics of text include text perplexity and text complexity.

[0012] Step 2: Data Generation

[0013] Define a default prompt word template. Use a large language model to generate SQL statements and text based on the default prompt word template.

[0014] The default prompt template includes a role description, schema, problem description, SQL generation requirements, and an output format example. The SQL generation requirements include basic rules and dynamic constraints. The dynamic constraints are used to adjust the complexity of the generated SQL statement.

[0015] Step 3: SQL statement verification

[0016] For the SQL statement generated in step 2, verify its execution correctness and semantic correctness.

[0017] Execution correctness verification: Create real data using the schema in the default prompt word template and execute the generated SQL statement. If it cannot be executed, proceed to step 4. Otherwise, perform semantic correctness verification.

[0018] Semantic correctness verification: Compare the execution result of the generated SQL statement with the problem description to determine if the execution result is the answer required by the problem. If so, return to step 2 to generate the next piece of data, until the number of generated data reaches the preset batch size, then proceed to step 5. Otherwise, proceed to step 4.

[0019] Step 4: Data Correction

[0020] Define a correction template. Using the large language model, correct the generated data that failed validation based on the correction template, generate corrected data, and return to step 3 for validation. If validation passes, replace the original data with the corrected data; if validation still fails after three corrections, retain the original data and mark it as an error sample, then return to step 2.

[0021] The corrected template replaces the SQL generation requirements in the default prompt word template with erroneous SQL statements and error feedback from performing correctness verification.

[0022] Step 5: Complexity Index Evaluation

[0023] Evaluate the complexity metrics of SQL statements and text in the generated data:

[0024] s5.1, Structural Complexity

[0025] The Structural Complexity Index (SCI) is used to measure the structural complexity of SQL statements.

[0026] SCI=α×Nsubquery+β×Njoin+γ×Dnest

[0027] Where Nsubquery represents the number of subqueries in the SQL statement. Njoin represents the number of join operations in the SQL statement. Dnest represents the maximum nesting depth in the SQL statement. α, β, and γ are the corresponding weighting coefficients.

[0028] s5.2, Semantic Complexity

[0029] The Semantic Complexity Index (SEI) is used to measure the semantic complexity of SQL statements.

[0030] SEI=δ×Crange+∈×Nagg+ζ×Ccondition

[0031] Here, Crange represents the number of tables and the sum of the number of fields in the tables in the query data, reflecting the complexity of the query data range. Nagg represents the number of aggregate functions in the SQL statement. Ccondition represents the number of conditional expressions, reflecting the complexity of the conditional expressions. δ, ∈, and ζ are the corresponding weight coefficients.

[0032] s5.3, Syntax Complexity

[0033] The Syntax Complexity Index (SGI) is used to measure the syntactic complexity of SQL statements.

[0034] SGI = η × Lsql + θ × Dsyntax

[0035] Where Lsql represents the length of the SQL statement. Dsyntax represents the number of different syntactic structures in the SQL statement, reflecting the diversity of syntactic structures. η and θ represent the corresponding weighting coefficients.

[0036] s5.4, Text perplexity

[0037] The generated Text T is segmented into words, where T = (t1, t2, ..., t3). n ), where n represents the number of tokens in text T. Predict the probability of the next token based on previous tokens, and take the logarithm:

[0038]

[0039] Wherein, P(t) i |t ＜i ) indicates that the first i-1 words are t1, t2, ... t i-1 When the i-th word is t i The probability of.

[0040] Then, the average log probability AvgLogProb is calculated, and by taking the exponentiation, it is mapped to the scale of the original probability to obtain the text perplexity P:

[0041]

[0042] P = exp(-AvgLogProb)

[0043] s5.5, text complexity

[0044] Perform syntactic dependency parsing on the text T, construct a syntactic dependency tree, obtain the maximum depth MaxDepth(T), average dependency distance AvgDepDist(T), and average branching factor AvgBranchingFactor(T) of the syntactic dependency tree, and calculate the text complexity:

[0045] Complexity=μ·MaxDepth(T)+ν·AvgDepDist(T)+ω·AvgBranchingFactor(T)

[0046] Among them, μ, ν, and ω are preset weight parameters that can be adjusted according to the data distribution or desired effect of the specific task.

[0047] Step 6: Dynamic Constraint Feedback

[0048] Based on the evaluation results of step 5, a feedback template is generated. A large language model is then used to generate dynamic constraints based on the feedback template, and these generated dynamic constraints replace the dynamic constraint portion of the default prompt word template described in step 2.

[0049] Return to step 2, use the large language model to generate SQL statements and text based on the modified prompt word template, until the amount of generated data reaches the preset number, then proceed to step 7.

[0050] The feedback template includes a role description, task requirements, task example, task background, and output format example.

[0051] The task requirements define the focus of the analysis and the conflict resolution methods. The task example provides a step-by-step example of completing the task requirements. The task background includes the evaluation results of the indicators and the defined target distribution.

[0052] Step 7: Text Difficulty Control

[0053] Based on the evaluation results of step 5, the direction for modifying the text difficulty is determined, and a difficulty control template is generated. The large language model is then used to modify the text in the generated data according to the difficulty control template, and its semantic correctness is verified, ensuring that the complexity index distribution of the generated data is consistent with the target distribution defined in step 1.

[0054] The difficulty control template includes character settings, task requirements, and task examples. The task requirements define dynamic adjustment instructions, adjustment constraints, and collaboration strategies. The collaboration strategies specify task priority and conflict resolution methods, the adjustment constraints define the text difficulty modification requirements, and the dynamic adjustment instructions are invoked based on the direction of the text difficulty modification.

[0055] The present invention has the following beneficial effects:

[0056] This method designs a series of quantitative metrics to measure the difficulty of the generated text and SQL statements, and dynamically adjusts the difficulty of the generated dataset based on the measurement results to conform to a preset difficulty distribution. This not only improves the efficiency and quality of dataset construction but also ensures the diversity and relevance of the dataset, providing the model with more comprehensive and practical training data. This helps improve the model's generalization ability and performance in different scenarios, promoting the development and application of Text2SQL technology. Attached Figure Description

[0057] Figure 1 Flowchart for generating a Text2SQL dataset based on dynamic difficulty adjustment;

[0058] Figure 2 This is a schematic diagram of a grammatical dependency tree. Detailed Implementation

[0059] The present invention will be further explained below with reference to the accompanying drawings;

[0060] like Figure 1 As shown, the method for generating Text2SQL datasets based on dynamic difficulty adjustment includes the following steps:

[0061] Step 1: Initialize parameters

[0062] The complexity metrics of SQL statements and text text are mapped to three difficulty ranges: low, medium, and high, and a preset target distribution is defined, as shown in Table 1:

[0063] index Low difficulty range Medium difficulty range High difficulty range Target distribution Structural complexity [0,2) [2,3) [3,+∞) 20％:50％:30％ semantic complexity [0,0.5) [0.5,1.5) [1.5,+∞) 25％:50％:25％ Syntax complexity [0,2) [2,3) [3,+∞) 30％:40％:30％ Text perplexity [0,5.5) [5.5,9.5) [9.5,+∞) 30％:50％:20％ Text complexity [0,4) [4,5) [5,+∞) 40％:50％:10％

[0064] Step 2: Data Generation

[0065] Define a default prompt word template. Use a large language model to generate SQL statements and text based on the default prompt word template.

[0066] The default prompt template includes a role description, schema, problem description, SQL generation requirements, and an output format example. Below is an example of a default prompt template:

[0067] ①Character Description

[0068] You are a meticulous SQL engineer who needs to generate SQL query questions and corresponding SQL statements that meet the requirements based on the given schema and user needs.

[0069] ②Schema

[0070] CREATE TABLE`stadium`(

[0071] Stadium_ID INT PRIMARY KEY,

[0072] Location TEXT,

[0073] Name TEXT,

[0074] Capacity INT,

[0075] Highest INT,

[0076] Lowest INT,

[0077] Average INT );

[0079] Sample rows from`stadium`:

[0080] 1,Raith Rovers,Stark's Park,10104,4812,1294,2106

[0081] 2,Ayr United,Somerset Park,11998,2363,1057,1477

[0082] 3,East Fife,Bayview Stadium,2000,1980,533,864

[0083] CREATE TABLE`singer`(

[0084] Singer_ID INT PRIMARY KEY,

[0085] Name TEXT,

[0086] Country TEXT,

[0087] Song_Name TEXT,

[0088] Song_release_year TEXT,

[0089] Age INT,

[0090] Is_male bool );

[0092] Sample rows from`singer`:

[0093] (1,Joe Sharp,Netherlands,You,1992,52,F),

[0094] (2,Timbaland,United States,Dangerous,2008,32,T),

[0095] (3,Justin Brown,France,Hey Oh,2013,29,T)

[0096] ③ Problem Description

[0097] How many female singers from France released songs before 2000?

[0098] ④ SQL generation requirements

[0099] ##Basic Rules

[0100] 1. Use standard SQL-92 syntax

[0101] 2. The result must be executable in MySQL 8.0.

[0102] ##Dynamic Constraints

[0103] Must comply with:

[0104] {must_items}

[0105] Recommended to follow:

[0106] {suggest_items}

[0107] Strictly prohibited:

[0108] {forbid_items}

[0109] ⑥ Output format example

[0110] Please strictly adhere to the following JSON format for output:

[0111] {

[0112] Question: "How many singers are there in total?"

[0113] "SQL":"SELECT count(*)FROM singer",

[0114] }

[0115] In the default prompt word template, the schema describes the data table structure in the database to be generated and provides some sample data.

[0116] The dynamic constraint section is used to adjust the complexity of the generated SQL statements. When the data is initially generated, the dynamic constraints are empty. When the amount of generated data reaches the preset batch number, the complexity of the generated data will be evaluated, and targeted dynamic generation constraints will be generated based on the difficulty distribution of the current data to fit the parameters set in the initialization.

[0117] The following is a data example generated by the large language model qwen-omni-turbo based on the default prompt word template during code debugging:

[0118] {

[0119] "schema":"CREATE TABLE`singer`(\n Singer_ID INT PRIMARY KEY,\n NameTEXT,\n Country TEXT,\n Song_Name TEXT,\n Song_release_year TEXT,\n Age INT,\nIs_male bool\n);\nSample rows from`singer`:\n1,Joe Sharp,Netherlands,You,1992,52,F\n2,Timbaland,United States,Dangerous,2008,32,T\n3,Justin Brown,France,Hey Oh,2013,29,T",

[0120] "question":"How many singers are there in total?"

[0121] "sql":"SELECT count(*)FROM singer",

[0122] "count":"1",

[0123] "turn":"1"

[0124] }

[0125] Here, `schema` represents the data table structure, `question` represents the generated text, and `sql` represents the generated SQL statement. `count` and `turn` are used to count the generated data, representing the number of generated data items and the corresponding generation batch, respectively. In the actual data generation, the model will output the generated text and SQL statement according to the output format example defined in the default prompt word template.

[0126] Step 3: SQL statement verification

[0127] For the SQL statement generated in step 2, verify its execution correctness and semantic correctness.

[0128] Execution correctness verification: Create real data using the schema in the default prompt word template and execute the generated SQL statement. If it cannot be executed, proceed to step 4. Otherwise, perform semantic correctness verification.

[0129] Semantic correctness verification: Compare the execution result of the generated SQL statement with the problem description to determine if the execution result is the answer required by the problem. If so, return to step 2 to generate the next piece of data, until the number of generated data reaches the preset batch size, then proceed to step 5. Otherwise, proceed to step 4.

[0130] The following is a data example output by the large language model qwen-omni-turbo based on the SQL statement validation results during code debugging:

[0131]

[0132]

[0133] Among them, `sqlValid` is used to identify the validation result of the correctness of the current SQL statement execution, `sqlMatch` is used to identify the validation result of semantic correctness, and `error` is used to output the reason for the error. In the actual data generation, the model will output the corrected SQL statement according to the output format example defined in the correction template.

[0134] Step 4: Data Correction

[0135] Define a correction template. Using the large language model, correct the generated data that failed validation based on the correction template, generate corrected data, and return to step 3 for validation. If validation passes, replace the original data with the corrected data; if validation still fails after three corrections, retain the original data and mark it as an error sample, then return to step 2.

[0136] The corrected template replaces the SQL generation requirements in the default prompt word template with erroneous SQL statements and error feedback from performing correctness verification.

[0137] Here is a sample correction template:

[0138] ①Character Description

[0139] You are a meticulous SQL engineer who needs to modify the currently erroneous SQL statement based on the given schema, issues, and error feedback.

[0140]

[0141] Sample rows from `Employees`:

[0142] (1,'Alice','Engineering',75000.00,'2020-06-15'),

[0143] (2,'Bob','HR',55000.00,'2018-03-21'),

[0144] (3,'Charlie','Engineering',82000.00,'2019-11-10'),

[0145] (4,'David','Marketing',62000.00,'2021-09-05');

[0146] ③ Problem Description

[0147] Retrieve the names and salaries of all employees in the Engineering department, and sort them by salary from highest to lowest.

[0148] ④ Incorrect SQL statement

[0149] SELECT Name,Salary FROM Employee WHERE Department='Engineering'ORDERBY Salary DESC;

[0150] ⑤ Error Feedback

[0151] ERROR:relation "Employee" does not exist

[0152] LINE 1:SELECT Name,Salary FROM Employee WHERE Department='Engineering'...

[0153] HINT:Did you mean "Employees"?

[0154] ⑥ Output format example

[0155] Please strictly adhere to the following JSON format for output:

[0156] {

[0157] "SQL":"SELECT Name,Salary FROM Employees WHERE Department='Engineering'ORDER BY Salary DESC;"

[0158] }

[0159] Step 5: Complexity Index Evaluation

[0160] After generating a batch of data, the complexity metrics of the SQL statements and text in the generated data are evaluated:

[0161] s5.1, Structural Complexity

[0162] The Structural Complexity Index (SCI) is used to measure the structural complexity of SQL statements.

[0163] SCI=α×Nsubquery+β×Njoin+γ×Dnest

[0164] Here, Nsubquery represents the number of subqueries in the SQL statement. The more subqueries, the more nested logic and data dependencies need to be handled. Njoin represents the number of join operations in the SQL statement. The more tables joined, the more relationships and join conditions need to be handled. Dnest represents the maximum nesting depth in the SQL statement. The greater the depth, the more levels and nested logic need to be handled. α, β, and γ represent the weights of the number of subqueries, join operations, and maximum nesting depth, respectively.

[0165] s5.2, Semantic Complexity

[0166] The Semantic Complexity Index (SEI) is used to measure the semantic complexity of SQL statements.

[0167] SEI=δ×Crange+∈×Nagg+ζ×Ccondition

[0168] Here, Crange represents the number of tables and the sum of the number of fields in the tables in the query data, reflecting the complexity of the query data range. Nagg represents the number of aggregate functions in the SQL statement. Ccondition represents the number of conditional expressions, reflecting the complexity of the conditional expressions. δ, ∈, and ζ are the corresponding weight coefficients.

[0169] s5.3, Syntax Complexity

[0170] The Syntax Complexity Index (SGI) is used to measure the syntactic complexity of SQL statements.

[0171] SGI = η × Lsql + θ × Dsyntax

[0172] Where Lsql represents the length of the SQL statement. Dsyntax represents the number of different syntactic structures in the SQL statement, reflecting the diversity of syntactic structures. η and θ represent the corresponding weighting coefficients.

[0173] s5.4, Text perplexity

[0174] Different expressions of the same meaning vary in terms of comprehension difficulty and clarity. To measure this clarity, text perplexity is introduced to assess the large language model's ability to predict the meaning of individual words in a text. A high prediction probability for a word indicates that the word is reasonable given the context, meaning the text's semantics are clear. Conversely, a low prediction probability suggests that the word is unreasonable given the context, indicating potential ambiguity or unclear expression. Lower text perplexity indicates more accurate predictions from the model, meaning the text structure and word usage better align with the language model's expectations, typically indicating clearer expression. Conversely, higher perplexity may indicate ambiguity, grammatical errors, or incoherent expression.

[0175] The generated Text T is segmented into words, where T = (t1, t2, ..., t3). n ), where n represents the number of tokens in text T. Predict the probability of the next token based on previous tokens, and take the logarithm:

[0176]

[0177] Wherein, P(t) i |t ＜i ) indicates that the first i-1 words are t1, t2, ... t i-1 When the i-th word is t i The probability of.

[0178] Then, the average log probability AvgLogProb is calculated, and by taking the exponentiation, it is mapped to the scale of the original probability to obtain the text perplexity P:

[0179]

[0180] P = exp(-AvgLogProb)

[0181] s5.5, text complexity

[0182] Syntactic dependency parsing is a method that reveals sentence structure by analyzing the dependency relationships between words. Dependency relationships refer to the grammatical and semantic connections between words, such as subject-verb, verb-object, and attributive-head relationships. Syntactic dependency parsing can parse a sentence into a grammatical dependency tree, where each node represents a word and the edges represent the dependency relationships between words.

[0183] Perform syntactic dependency parsing on the text T to construct, such as Figure 2 Given the syntactic dependency tree, obtain the maximum depth MaxDepth(T), average dependency distance AvgDepDist(T), and average branching factor AvgBranchingFactor(T) of the syntactic dependency tree, and calculate the text complexity:

[0184] Complexity=μ·MaxDepth(T)+ν·AvgDepDist(T)+ω·AvgBranchingFactor(T)

[0185] Among them, μ, ν, and ω are preset weight parameters that can be adjusted according to the data distribution or experimental results of a specific task.

[0186] Step 6: Dynamic Constraint Feedback

[0187] Based on the evaluation results of step 5, a feedback template is generated. A large language model is then used to generate dynamic constraints based on the feedback template, and these generated dynamic constraints replace the dynamic constraint portion of the default prompt word template described in step 2.

[0188] Return to step 2, use the large language model to generate SQL statements and text based on the modified prompt word template, until the amount of generated data reaches the preset number, then proceed to step 7.

[0189] The feedback template includes a role description, task requirements, a task example, task background, and an output format example. The task requirements define the analytical focus and conflict resolution methods for the task. The task example provides a step-by-step guide to completing the task requirements. The task background includes the evaluation results of the metrics and the defined target distribution, used to indicate the generation of targeted dynamic constraints.

[0190] The following is an example of a default prompt word template:

[0191] ①Character Description

[0192] You are an intelligent Text2SQL generation controller, dynamically adjusting the generation strategy based on complexity metrics. You possess the dual skills of an SQL syntax expert and a data analyst.

[0193] ②Task Requirements

[0194] 1. Analysis Focus: Prioritize addressing the metrics that deviate most from the target. Identify the segments with the largest absolute differences between the current performance and the target at each level of SEI, SCI, and SGI. The larger the absolute value, the higher the priority. Then, process them in order of priority.

[0195] 2. Conflict Resolution: If there are conflicting suggestions, handle them according to priority.

[0196] 3. Feasibility check: Ensure the suggestion will not lead to syntax errors or execution failure.

[0197] ③Task Example

[0198] 1. Gap Identification: This involves identifying the areas with the largest absolute gaps between the current level and the target level at each of the SEI, SCI, and SGI metrics.

[0199] Example: "sci":{"Current Distribution":{"Low":40%,"Medium":55%,"High":5%},"Target Distribution":{"Low":10%,"Medium":40%,"High":50%},

[0200] "sei":{"Current Distribution":{"Low":25%,"Medium":50%,"High":25%}",Target Distribution":{"Low":10%,"Medium":40%,"High":50%},

[0201] "sgi":{"Current Distribution":{"Low":60%,"Medium":20%,"High":20%},"Target Distribution":{"Low":10%,"Medium":40%,"High":50%}

[0202] SCI high-difficulty level has a 20% shortage, SEI high-difficulty level has a 15% shortage, and SGI high-difficulty level exceeds the standard by 50%.

[0203] 2. Weighted Priority: If multiple indicators deviate from the target, the indicator with the largest deviation will be adjusted first.

[0204] Example: SGI high difficulty level exceeds the standard by 50% > SCI high difficulty level has a 20% gap > SEI high difficulty level has a 15% gap.

[0205] 3. Attribution Analysis: Inferring causes based on indicator formulas.

[0206] Example: A low SEI might be due to a small number of tables involved.

[0207] 4. Recommendation Generation: Propose specific measures based on the attribution results.

[0208] Correct examples: "The query must involve at least 3 tables", "The query must involve four or more fields".

[0209] Incorrect example: "Increases complexity" (too vague)

[0210] 5. Instruction Conversion: Converts suggestions into constraints that can include prompt words.

[0211] ④Task Background

[0212] {userinput}

[0213] ⑤ Output format example

[0214] Please strictly adhere to the following JSON format for output:

[0215] {

[0216] "gap_analysis":["List no more than 3 key gaps, sorted by priority"],

[0217] "action_plan":["Specific adjustment actions (each must be associated with the influencing indicators)"],

[0218] "prompt_modifications":{

[0219] "must":["Add mandatory constraints (maximum 3)"],

[0220] "suggestion":["Suggested guidelines (maximum 2)"],

[0221] "forbidden items (maximum 2 items)"

[0222] }

[0223] }

[0224] Here, `userinput` is a placeholder used to fill in the metric evaluation results from step 5. The following is an example of a structured input design for the task context, which structurally transmits information such as metric definitions, current state, and historical adjustment records to the large model, ensuring it can accurately understand the task context:

[0225] {

[0226] "Task Type":"Text2SQL Dataset Complexity Control",

[0227] "Indicator Definition":{

[0228] "SCI":{

[0229] Formula: SCI = 0.4 * number of subqueries + 0.3 * number of join operations + 0.3 * nesting depth

[0230] "threshold":{"low":[0,5),"medium":[5,15),"high":[15,100]},

[0231] Explanation: "The number of subqueries has the greatest impact on SCI (weight 0.4)."

[0232] },

[0233] "SEI":{

[0234] Formula: "SEI = 0.5 * number of tables involved + 0.3 * number of aggregate functions + 0.2 * complexity of conditional expressions",

[0235] "threshold":{"low":[0,10),"medium":[10,25),"high":[25,100]},

[0236] Explanation: "Focus on the number of tables and the complexity of WHERE conditions."

[0237] },

[0238] "SGI":{

[0239] Formula: "SGI = 0.6 * SQL length + 0.4 * Syntax structure diversity",

[0240] "threshold":{"low":[0,20),"medium":[20,50),"high":[50,100]},

[0241] Explanation: "Long statements and complex syntax (such as window functions) can significantly improve SGI performance."

[0242] }

[0243] },

[0244] Current status:{

[0245] "SCI":{"Average":14.2,"Current Distribution":{"Low":15%,"Medium":60%,"High":25%},"Target Distribution":{"Low":20%,"Medium":50%,"High":30%}},

[0246] "SEI":{"Average":8.7,"Distribution":{"Low":30%,"Medium":55%,"High":15%},"Target Distribution":{"Low":20%,"Medium":50%,"High":30%}},

[0247] "SGI":{"Average":35.2,"Distribution":{"Low":10%,"Medium":40%,"High":50%},"Target Distribution":{"Low":20%,"Medium":50%,"High":30%}},

[0248] },

[0249] "Historical Adjustment Records":[

[0250] {"Round":1,"Action":"Increase the number of JOINs","Result":"SCI increased by 5%, but SGI exceeded the limit"}

[0251] {"Round":2,"Action":"Restrict Group By","Result":"SGI decreases by 10%"}

[0252] ],

[0253] Task Instruction: "Please generate a strategy suggestion for the next batch of data."

[0254] }

[0255] The following is an example of a dynamic constraint generated by the large language model qwen-omni-turbo based on a feedback template:

[0256] {

[0257] "gap_analysis":[

[0258] "The number of high-difficulty SEI samples is severely insufficient (15% vs 30%), mainly due to the average number of tables involved being only 2.1."

[0259] "SGI has an overabundance of high-difficulty samples (50% vs 30%) due to the average SQL length reaching 120 characters."

[0260] "SCI score is very high and close to the target (25% vs 30%), requiring slight improvement."

[0261] ],

[0262] "action_plan":[

[0263] "Improving SEI: Forcing queries across 3 tables (affecting SEI+SCI)"

[0264] "Reduce SGI: Limit SQL length to <100 characters (affects SGI)",

[0265] "Fine-tuning SCI: Using one subquery is recommended but not mandatory (it affects SCI)"

[0266] ],

[0267] "prompt_modifications":{

[0268] "must":[

[0269] "At least 3 tables must be joined using a JOIN statement".

[0270] The WHERE clause must contain at least two AND / OR conditions.

[0271] ],

[0272] "suggest":[

[0273] "Can contain one subquery to increase nesting depth"

[0274] ],

[0275] "forbid":[

[0276] "SQL statements cannot exceed 100 characters",

[0277] "Disable the use of window functions" ]

[0279] }

[0280] }

[0281] The `must`, `suggest`, and `forbid` fields in `prompt_modifications` are used to populate the placeholders `must_items`, `suggest_items`, and `forbid_items` corresponding to the dynamic constraints of the default prompt word template.

[0282] Step 7: Text Difficulty Control

[0283] When the number of generated data reaches the preset number, based on the evaluation results of the text perplexity and text complexity indices in step 5, the direction for modifying the text difficulty is determined, and a difficulty control template is generated. The large language model is then used to modify the generated text according to the difficulty control template, and its semantic correctness is verified, ensuring that the complexity index distribution of the generated data is consistent with the target distribution defined in step 1.

[0284] The difficulty control template includes character settings, task requirements, and task examples. The task requirements define dynamic adjustment instructions, adjustment constraints, and collaboration strategies. The collaboration strategies specify task priority and conflict resolution methods, the adjustment constraints define the text difficulty modification requirements, and the dynamic adjustment instructions match predefined rules to invoke commands based on the direction of the text difficulty modification. Below is an example of a difficulty control template:

[0285] ①Character Description

[0286] Multidimensional text complexity control expert

[0287] ②Task Requirements

[0288] Strictly adhere to the adjustment constraint reference collaboration strategy to perform precise rewriting while maintaining the semantics of the text:

[0289] Original sentence: {text}

[0290] Adjust direction: {direction}

[0291] ##Dynamic Adjustment Commands

[0292] Execute according to priority:

[0293] 1. [Difficulty of understanding] {perplexity_instruction}

[0294] 2. [Structural Complexity] {max_depth_instruction}

[0295] 3. [Distance Modifier]{dep_distance_instruction}

[0296] 4. [Logical Density]{branching_instruction}

[0297] ## Adjusting Constraints

[0298] - Maintain the integrity of text semantic elements (retain all entities, attributes, and conditions), but you can change how semantic elements are expressed.

[0299] - Be sure to strictly adhere to the constraints of the original sentence and do not introduce any additional information.

[0300] - You cannot change the number itself, but you can change how it is expressed, such as changing half to fifty percent.

[0301] ## Collaborative Strategies

[0302] 1. Priority order: perplexity > depth > distance > branching factor

[0303] 2. Conflict resolution:

[0304] - When structural simplification conflicts with increasing perplexity, the perplexity objective should be prioritized.

[0305] - Use compensatory adjustments (such as shortening the distance to compensate for increased depth).

[0306] ③Task Example

[0307] Original sentence: "List the stores in Hangzhou with monthly sales exceeding the average."

[0308] Adjustment direction: Complicate

[0309] Dynamic adjustment instructions:

[0310] [Difficulty of understanding] Deliberately create local syntactic anomalies (such as omitting prepositions)

[0311] [Structural complexity] Increase 2 levels of nesting (insert conditional clauses)}

[0312] [Modifier distance] Extend the word distance by 1.5 (postpone geographical qualification)

[0313] [Logical density] Add 1 coordinate condition

[0314] The following explains the dynamic adjustment instructions given in the task example:

[0315] For [Difficulty of understanding], it can be achieved by changing the text perplexity. For example, make the sentence pattern more convoluted, use "Find those stores in Hangzhou, and these stores'..." instead of "List the stores in Hangzhou...", or introduce local syntactic anomalies by omitting the preposition "de" to create slight confusion.

[0316] For [Structural complexity], increase the maximum depth in the syntactic dependency tree. For example, introduce "and these stores'..." as a conditional clause to increase nesting.

[0317] For [Modifier distance], postpone the geographical qualification to make the modifying element more backward to extend the word distance.

[0318] For [Logical density], insert a coordinate structure to increase the logical level.

[0319] After executing the dynamic adjustment instructions, "List the stores in Hangzhou with monthly sales exceeding the average" can be made more complex and rewritten as "Find those stores in Hangzhou, and these stores' monthly sales are higher than the average monthly sales of all stores."

Claims

1. A method for generating Text2SQL datasets based on dynamic difficulty adjustment, characterized by: Specifically, the following steps are included: Step 1: Map the complexity metrics of SQL statements and text to different difficulty ranges, and define the target difficulty range distribution for the generated dataset; the complexity metrics of the SQL statements include structural complexity, semantic complexity, and syntactic complexity; the complexity metrics of the text include text perplexity and text complexity. Step 2: Define a default prompt template; use a large language model to generate SQL statements and text based on the default prompt template; Dynamic constraints are introduced into the SQL generation requirements section of the default prompt template to adjust the complexity of the generated SQL statements; Step 3: For the SQL statement generated in Step 2, verify the correctness of execution and the correctness of semantics in turn; if there is an incorrect execution, proceed to Step 4; if there is an incorrect semantics, proceed to Step 4; otherwise, return to Step 2 to generate the next piece of data, until the number of generated data reaches the preset batch number, then proceed to Step 5. Step 4: Define a correction template; use the large language model to correct the generated data that failed the verification according to the correction template, generate corrected data and return to Step 3 for verification; if the verification passes, replace the original data with the corrected data; if the verification still fails after several corrections, retain the original data and mark it as an error sample, and return to Step 2. Step 5: Evaluate the complexity metrics of the SQL statements and text in the generated data. Step 6: Based on the evaluation results of the indicators in Step 5, generate a feedback template; use the large language model to generate dynamic constraints based on the feedback template, and use the generated dynamic constraints to replace the dynamic constraint part in the default prompt word template described in Step 2; return to Step 2, use the large language model to generate SQL statements and text based on the modified prompt word template, until the number of generated data reaches the preset number, then proceed to Step 7; Step 7: Based on the evaluation results of Step 5, determine the direction of the text difficulty modification and generate a difficulty control template; use the large language model to modify the text in the generated data according to the difficulty control template, and verify its semantic correctness, so that the complexity index distribution of the generated data is consistent with the target distribution defined in Step 1.

2. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The Structural Complexity Index (SCI) is used to measure the structural complexity of SQL statements. SCI=α×Nsubquery+β×Njoin+γ×Dnest Where Nsubquery represents the number of subqueries in the SQL statement; Njoin represents the number of join operations in the SQL statement; Dnest represents the maximum nesting depth in the SQL statement; α, β, and γ represent the weights of the number of subqueries, the number of join operations, and the maximum nesting depth, respectively.

3. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The Semantic Complexity Index (SEI) is used to measure the semantic complexity of SQL statements. Where Crange represents the number of tables and the sum of the number of fields in the tables in the query data, reflecting the complexity of the query data range; Nagg represents the number of aggregate functions in the SQL statement; Ccondition represents the number of conditional expressions, reflecting the complexity of the conditional expressions; δ, ζ and ζ are the corresponding weighting coefficients.

4. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The Syntax Complexity Index (SGI) is used to measure the syntactic complexity of SQL statements. SGI = η × Lsql + θ × Dsyntax Where Lsql represents the length of the SQL statement; Dsyntax represents the number of different syntax structures of the SQL statement, reflecting the diversity of syntax structures; η and θ represent the corresponding weight coefficients.

5. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The generated Text text T is segmented into words. , where n represents the number of tokens in text T; predict the probability of the next token based on previous tokens, and take the logarithm: Wherein, P(t) i |t ＜i ) indicates that the first i-1 words are t1, t2, ... t i-1 When the i-th word is t i The probability of; Then, the average log probability AvgLogProb is calculated, and by taking the exponentiation, it is mapped to the scale of the original probability to obtain the text perplexity P: 。 6. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: Perform syntactic dependency parsing on the text T, construct a syntactic dependency tree, obtain the maximum depth MaxDepth(T), average dependency distance AvgDepDist(T), and average branching factor AvgBranchingFactor(T) of the syntactic dependency tree, and calculate the text complexity. Complexity=μ×MaxDepth(T)+ν×AvgDepDist(T)+ω×AvgBranchingFactor(T) Where μ, ν, and ω are preset weight parameters.

7. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The default prompt template includes a role description, schema, problem description, SQL generation requirements, and an output format example; the SQL generation requirements include basic rules and dynamic constraints. The corrected template replaces the SQL generation requirements in the default prompt word template with erroneous SQL statements and error feedback from performing correctness verification.

8. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The method for verifying the correctness of execution is as follows: create real data using the schema in the default prompt word template and execute the generated SQL statement; the method for verifying the semantic correctness is as follows: compare the execution result of the generated SQL statement with the problem description to determine whether the execution result is the answer required for the problem.

9. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The feedback template includes a role description, task requirements, task example, task background, and output format example; The task requirements define the focus of the analysis and the conflict resolution methods; the task example provides a step-by-step example of completing the task requirements; the task background includes the evaluation results of the indicators and the defined target distribution.

10. The Text2SQL dataset generation method based on dynamic difficulty adjustment as described in claim 1, characterized in that: The difficulty control template includes character settings, task requirements, and task examples; the task requirements define dynamic adjustment instructions, adjustment constraints, and collaboration strategies. The collaboration strategy specifies the task priority order and conflict resolution method, the adjustment constraint specifies the text difficulty modification requirements, and the dynamic adjustment instruction calls the instruction according to the text difficulty modification direction.

Images