Prompt word optimization method and device suitable for nl2sql

Abstract

The invention relates to the field of artificial intelligence, and particularly provides a cue word optimization method and device suitable for nl2sql, and the method comprises the following steps: S1, constructing an instance library and test data; s2, based on the large model, discovering error concepts and constructing a concept enhancement library; and S3, improving user prompt words by utilizing concept enhancement and a dynamic example optimization strategy. Compared with the prior art, the method has the advantages that generation errors caused by cognition errors of the model can be avoided, the labor cost of cue word design is reduced, the model generation guiding capability of the cue words is effectively improved, and the accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and specifically provides a prompt word optimization method and device suitable for nl2sql. Background Art

[0002] With the development of big model technology, intelligent question and answer based on artificial intelligence is increasingly widely used in the industry, such as conversational BI, intelligent customer service and other applications are booming in various industries. These applications allow people to obtain the information they need through daily conversations, providing more convenient services. Behind this technology is the prompt word engineering, which is the language for people to interact with big models. A good prompt word can maximize the performance of big models and provide high-quality answer generation. At present, there have been many studies on prompt word engineering. By dividing roles, instructions, thinking processes, examples, etc., and guiding the model to generate answers step by step, the model question and answer effect can be significantly improved. However, in the specific prompt word construction process, it still relies too much on human participation. Due to differences in training data, different big models have different understandings of common sense concepts, which makes the constructed prompt words unable to effectively guide model generation, and the trial and error cost is high.

[0003] The main reasons why the current prompt words are not enough to improve the large model are as follows:

[0004] Cognitive defects of large models: The knowledge of large models comes from training samples, which often cannot fully cover the scope of users' problems, especially some models with smaller parameters. Their performance is weak in specific fields and colloquial problems, resulting in a certain gap between the user's prompt word description and the cognition of the large model. Currently, there is a lack of an effective knowledge enhancement method to assist in discovering the description defects in the user's prompt words and automatically enhance the supplementary instructions for the defects (especially for small models, whose knowledge reserves are often scarce and insufficient compared to large models), thereby enhancing the stability and accuracy of the model.

[0005] User questions are complex and diverse: The instruction descriptions in user prompts often only provide a broad guide, while user questions are often highly targeted. Similar questions under the same topic often have different focuses, making it difficult for prompts to work effectively. Users who maintain multiple prompts for different questions also face the problem of matching questions and prompt templates, which increases the cost of use. Currently, there is a lack of an effective matching method to select corresponding prompts for user questions. Summary of the invention

[0006] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a highly practical method for optimizing prompt words applicable to nl2sql.

[0007] A further technical task of the present invention is to provide a prompt word optimization device suitable for nl2sql that is reasonably designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A prompt word optimization method applicable to nl2sql has the following steps:

[0010] S1, build instance library and test data;

[0011] S2. Based on the big model, discover wrong concepts and build a concept enhancement library;

[0012] S3. Use concept enhancement and dynamic example optimization strategies to improve user prompt words.

[0013] Furthermore, in step S1, the prompt words are divided into two parts: instructions and examples;

[0014] The examples refer to the question-answer examples provided in the prompt words, which are used to guide the large model to generate results;

[0015] The instruction refers to the specific task description that the large model needs to perform, namely: Instruction.

[0016] Furthermore, in step S1, the specific construction steps are as follows:

[0017] S1.1. Obtain the database table information required for the nl2sql task;

[0018] S1.2. Manually construct corresponding prompt word examples based on database information. A prompt word example includes at least two items: the user question and the corresponding question-and-answer result;

[0019] S1.3. Based on the prompt word examples, query intent and query index are annotated, where query intent refers to the specific task described in the question, and query index refers to the specific field that the user wants to obtain information about;

[0020] S1.4, construct a sample library;

[0021] S1.5. Construct test data.

[0022] Furthermore, in step S1.4, each example in the example library includes a question, a generated result, an intention and an indicator. The intention and indicator annotation information is used to screen candidate examples in step S1.3. The question and the generated result of the candidate example will be added to the user prompt word for prompt word optimization.

[0023] Furthermore, in step S1.5, each type of query intent and query index includes multiple examples, some of which are used only as test data and do not participate in constructing the user's final prompt word. They are only used to evaluate the prompt word optimization performance. Thus, a prompt word example library and test data are obtained.

[0024] Furthermore, in step S2, the specific construction steps are as follows:

[0025] S2.1. Use the user questions in the test set to input the big model and get the generated results.

[0026] S2.2. Compare the correct SQL results and record the SQL generation errors;

[0027] S2.3. Use the wrong results and correct results to construct concept correction prompts, use a large model with a larger number of parameters to find the wrong concepts that need to be corrected in the original model, and record the concepts and their correct descriptions;

[0028] S2.4. Repeat step S2.3 on the test set to obtain all concepts that need to be updated and their correct descriptions in the incorrect questions and answers, and record them as the concept enhancement library.

[0029] Furthermore, in step S3, the specific construction steps are as follows:

[0030] S3.1. Input the user's question and the user's original prompt words;

[0031] S3.2, query whether the user's question contains the concept words in the concept enhancement library in step S2, and if so, splice the concept words into the knowledge background of the prompt words to enhance the user prompt words;

[0032] S3.3. Use the big model to extract the user's query indicators, classify the user's questions, and determine the user's query intent;

[0033] S3.4, based on the extracted query indicators and query intent, filter the sample library in step S1 to obtain all examples of indicators under the corresponding intent;

[0034] S3.5. Vectorize the user question and calculate the similarity with the vectorized matrix of the candidate examples to obtain the topN examples, which are recorded as dynamic examples. They are concatenated with the prompt words in step S3.5 to obtain the final optimized prompt words.

[0035] A prompt word optimization device applicable to nl2sql, comprising: at least one memory and at least one processor;

[0036] The at least one memory is used to store a machine-readable program;

[0037] The at least one processor is used to call the machine-readable program to execute a prompt word optimization method applicable to nl2sql.

[0038] Compared with the prior art, the prompt word optimization method and device applicable to nl2sql of the present invention have the following outstanding beneficial effects:

[0039] The present invention can avoid generation errors caused by cognitive errors of the model itself, reduce the labor cost of prompt word design, effectively improve the generation guidance ability of the prompt words on the model, and improve the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Attached Figure 1 The present invention is a flowchart of a prompt word optimization method applicable to nl2sql. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] A best embodiment is given below:

[0044] like Figure 1 As shown, a prompt word optimization method applicable to nl2sql in this embodiment has the following steps:

[0045] S1, build instance library and test data;

[0046] The prompt words are divided into two parts: instructions and examples;

[0047] Examples refer to the question-answer examples provided in the prompt words, which are used to guide the large model to generate results;

[0048] An instruction refers to the specific task description that the large model needs to perform, namely: Instruction.

[0049] The specific construction steps are as follows:

[0050] S1.1. Obtain the database table information required for the nl2sql task;

[0051] S1.2. Manually construct corresponding prompt word examples for database information. A prompt word example includes at least two items: user question and corresponding question-and-answer result (SQL statement);

[0052] S1.3. Based on the prompt word examples, query intent and query index are annotated. Query intent refers to the specific task used to describe the question, including but not limited to: value screening, dimension grouping, dimension sorting, etc.; query index refers to the specific field that the user wants to obtain information about;

[0053] S1.4, construct a sample library;

[0054] Each example in the example library includes: question, generation result, intention and indicator. The annotation information such as intention and indicator is used to screen candidate examples in step 3. The question and generation result of the candidate instance will be added to the user prompt word for prompt word optimization.

[0055] The sample library looks like this:

[0056]

[0057] S1.5. Construct test data;

[0058] Each type of query intent and query indicator includes multiple examples, some of which are only used as test data and do not participate in constructing the user's final prompt word. They are only used to evaluate the prompt word optimization performance. Thus, the prompt word example library and test data are obtained.

[0059] The sample library classifies questions from the perspective of query intent and query indicators. Questions in the same category often have consistent dimensional screening ranges. For example, questions about GDP screen data within the time range, and their descriptions often conform to the storage period of GDP data. Under the same intent and indicator, semantic matching of examples based on user questions will avoid example confusion with examples with similar descriptions under different indicators.

[0060] S2. Based on the big model, discover wrong concepts and build a concept enhancement library;

[0061] The specific construction steps are as follows:

[0062] S2.1. Use the user questions in the test set to input the big model and get the generated results.

[0063] S2.2. Compare the correct SQL results and record the SQL generation errors;

[0064] S2.3. Use the wrong results and correct results to construct concept correction prompts, use a large model with a larger number of parameters to find the wrong concepts that need to be corrected in the original model, and record the concepts and their correct descriptions;

[0065] For example, in the Baichuan (13b) model, there is a misunderstanding of the concept of the year before last in the user question: "What was the total GDP value of Qingdao City the year before last (this year is 2024)", that is, it is believed that the year before last is the year before 2024, while the correct definition of the year before last is the two years before 2024. If the cognitive errors of the model itself are not supplemented and corrected, such problems will continue to exist.

[0066] The following is an example of an error correction prompt word template:

[0067] For the question: "What was the total GDP of Qingdao two years ago (this year is 2024)", the standard answer is: select gdp_value from GDP where year=2022. For the wrong answer: select gdp_value from GDP where year=2023. Try to identify the reason for the error in which concept is misunderstood and give the correct definition.

[0068] Running this prompt with Qianwen(72b) yields the following results:

[0069] The reason for the wrong answer is that the time concept of "the year before last" is misunderstood. When referring to "the year before last", especially when the question clearly states that "this year is 2024", the year before last should refer to the two years before 2024, that is, 2022. However, the query condition in the wrong answer is set as year = 2023, which actually refers to last year, not the year before last.

[0070] Based on the analysis results of Qianwen (72b), the updated concepts extracted are:

[0071] The year before last: the two years before 2024.

[0072] S2.4. Repeat step S2.3 on the test set to obtain all concepts that need to be updated and their correct descriptions in the incorrect questions and answers, and record them as the concept enhancement library.

[0073] S3, using concept enhancement and dynamic example optimization strategies to improve user prompt words;

[0074] The specific construction steps are as follows:

[0075] S3.1. Input the user's question and the user's original prompt words;

[0076] S3.2, query whether the user's question contains the concept words in the concept enhancement library in step S2, and if so, splice the concept words into the knowledge background of the prompt words to enhance the user prompt words;

[0077] S3.3. Use the big model to extract the user's query indicators, classify the user's questions, and determine the user's query intent;

[0078] Optionally, if there is sufficient data, a separate classification model can be trained based on samples to replace the large model;

[0079] Do an information extraction task, extract the user's query indicators from the user question (a database query problem), and analyze the user's query intention, classify the intention into value screening, dimension grouping, and dimension sorting. Strictly refer to the examples and do not generate them arbitrarily.

[0080] S3.4, based on the extracted query indicators and query intent, filter the sample library in step S1 to obtain all examples of indicators under the corresponding intent;

[0081] Different indicators may have similar examples, but these examples may not be common at the semantic level. For example, the indicator GDP is usually queried at a monthly granularity, and there is no query at a daily granularity. The indicator visit volume is usually queried at a daily or hourly granularity. If the question about GDP is mistakenly matched to a similar example of visit volume, the user's vague time description may be guided to the wrong granularity, causing an error. Step S3.4 can filter out interference examples to the maximum extent at the minimum cost, eliminating the interference of similar examples under different indicators, and thus obtaining candidate examples.

[0082] S3.5. Vectorize the user question and calculate the similarity with the vectorized matrix of the candidate examples to obtain the topN examples, which are recorded as dynamic examples. They are concatenated with the prompt words in step S3.5 to obtain the final optimized prompt words.

[0083] Based on the above method, a prompt word optimization device applicable to nl2sql in this embodiment includes: at least one memory and at least one processor;

[0084] The at least one memory is used to store a machine-readable program;

[0085] The at least one processor is used to call the machine-readable program to execute a prompt word optimization method applicable to nl2sql.

[0086] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A prompt word optimization method applicable to nl2sql, characterized in that: The steps are as follows: S1, build instance library and test data; S2. Based on the big model, discover wrong concepts and build a concept enhancement library; S3. Use concept enhancement and dynamic example optimization strategies to improve user prompt words.

2. A prompt word optimization method applicable to nl2sql according to claim 1, characterized in that: In step S1, the prompt words are divided into two parts: instructions and examples; The examples refer to the question-answer examples provided in the prompt words, which are used to guide the large model to generate results; The instruction refers to the specific task description that the large model needs to perform, namely: Instruction.

3. A prompt word optimization method applicable to nl2sql according to claim 2, characterized in that: In step S1, the specific construction steps are as follows: S1.

1. Obtain the database table information required for the nl2sql task; S1.

2. Manually construct corresponding prompt word examples based on database information. A prompt word example includes at least two items: the user question and the corresponding question-and-answer result; S1.

3. Based on the prompt word examples, query intent and query index are annotated, where query intent refers to the specific task described in the question, and query index refers to the specific field that the user wants to obtain information about; S1.4, construct a sample library; S1.

5. Construct test data.

4. The prompt word optimization method applicable to nl2sql according to claim 3, characterized in that: In step S1.4, each example in the example library includes a question, a generated result, an intention, and an indicator. The intention and indicator annotation information is used to screen candidate examples in step S1.

3. The question and generated result of the candidate example will be added to the user prompt word for prompt word optimization.

5. The prompt word optimization method applicable to nl2sql according to claim 4, characterized in that: In step S1.5, each type of query intent and query index includes multiple examples, some of which are used only as test data and do not participate in constructing the user's final prompt word. They are only used to evaluate the prompt word optimization performance. At this point, a prompt word example library and test data are obtained.

6. The prompt word optimization method applicable to nl2sql according to claim 5, characterized in that: In step S2, the specific construction steps are as follows: S2.

1. Use the user questions in the test set to input the big model and get the generated results. S2.

2. Compare the correct SQL results and record the SQL generation errors; S2.

3. Use the wrong results and correct results to construct concept correction prompts, use a large model with a larger number of parameters to find the wrong concepts that need to be corrected in the original model, and record the concepts and their correct descriptions; S2.

4. Repeat step S2.3 on the test set to obtain all concepts that need to be updated and their correct descriptions in the incorrect questions and answers, and record them as the concept enhancement library.

7. A prompt word optimization method applicable to nl2sql according to claim 6, characterized in that: In step S3, the specific construction steps are as follows: S3.

1. Input the user's question and the user's original prompt words; S3.2, query whether the user's question contains the concept words in the concept enhancement library in step S2, and if so, splice the concept words into the knowledge background of the prompt words to enhance the user prompt words; S3.

3. Use the big model to extract the user's query indicators, classify the user's questions, and determine the user's query intent; S3.4, based on the extracted query indicators and query intent, filter the sample library in step S1 to obtain all examples of indicators under the corresponding intent; S3.

5. Vectorize the user question and calculate the similarity with the vectorized matrix of the candidate examples to obtain the topN examples, which are recorded as dynamic examples. They are concatenated with the prompt words in step S3.5 to obtain the final optimized prompt words.

8. A prompt word optimization device suitable for nl2sql, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 7.

Images