Large model cue word automatic optimization method

Abstract

The invention provides a large model cue word automatic optimization method which comprises the following steps: segmenting a structure cue word into a plurality of first fields according to a structure mark of the structure cue word, obtaining category themes contained in the first fields, and combining text descriptions corresponding to single category themes to generate second fields; calling a training set of a business scene corresponding to the structure cue word, aggregating the same category theme in the training set to generate a third field, and combining the second field and the third field of the contract category theme to generate a category field; and independently obtaining a first reasoning result of each category field, comparing the first reasoning results with the labeling information of the training set, outputting distinguishing information, fusing the distinguishing information of all category themes to generate optimization feedback, optimizing the structure cue word according to the optimization feedback, and outputting the optimal cue word. According to the method, the cue word can be optimized in a fine-grained manner on the premise that the description of the user is not changed.

Description

Technical Field

[0001] The present invention relates to the technical field of large language prompt optimization, and particularly to an automatic optimization method for large model prompts. Background Art

[0002] With the rapid development of large models, inferring large models through prompts has become the most common paradigm for large model applications. In industrial applications, in order to make large models output more professional answers when facing specific business scenarios, it is often necessary to add more and richer scenario content to the prompts to guide large model inference.

[0003] In the prior art, markdown format can be used to define prompts that contain clearly defined and semantically distinct parts such as domain knowledge, rule constraints, and operation steps, and such prompts are called structured prompts.

[0004] When manually adjusting the prompts reaches the bottleneck of inference effect, prompt optimization tools such as Dspy and TextGrad are used to automatically optimize the prompts. However, Dspy, TextGrad, etc. often regard the prompts as a whole and adjust them through repeated iteration and trial and error, and there are the following problems:

[0005] 1. The overall optimization of complex structured prompts has a relatively coarse granularity, making it difficult to capture the optimization points, and the optimization process cannot be refined and controlled.

[0006] 2. It will disrupt the structure of the structured prompts, resulting in a decrease in the readability of the prompts. Summary of the Invention

[0007] In view of this, the problem to be solved by the present invention is to provide an automatic optimization method for large model prompts, which can optimize the prompts with fine granularity without changing the user's own description.

[0008] To solve the above technical problems, the technical solution adopted by the present invention is:

[0009] An automatic optimization method for large model prompts, including splitting the prompts: splitting the structured prompts into several first fields according to the structure markers of the structured prompts, obtaining the category topics included in the first fields, and combining the text descriptions corresponding to individual category topics to generate second fields;

[0010] Generating category fields, calling the sample set corresponding to the business scenario of the structured prompts, aggregating the same category topics in the sample set to generate third fields, and combining the second fields and third fields of the same category topics to generate category fields;

[0011] Optimization prompt: Individually obtain the first inference result of each category field, compare the first inference result with the annotation information in the training set and output the difference information, fuse the difference information of all category themes to generate an optimization feedback, and optimize the structure prompt according to the optimization feedback and output the optimal prompt.

[0012] Further, the structure prompt includes content such as a task description for recording the business scenario, rule constraints for recording the business theme and event theme, and domain knowledge for recording the common knowledge within the business scenario.

[0013] The category theme contains several event themes, and the annotation information records the category themes and event themes included in the business scenario.

[0014] Further, the sample set is composed of several sample prompts. Eighty percent of the sample prompts in the sample set are used as the training set, and twenty percent of the sample prompts are used as the validation set.

[0015] Further, the process of obtaining the optimal prompt is as follows: The large language model fills the event theme in the structure prompt according to the optimization feedback and outputs several candidate prompts.

[0016] Individually obtain the second inference result of each candidate prompt through the large language model, obtain the third inference result of the sample prompts in the validation set through the large language model, calculate the theme similarity between the third inference result and each second inference result, and the candidate prompt with the highest theme similarity is the optimal prompt.

[0017] The advantages and positive effects of the present invention are:

[0018] (1) By using structural markers and category theme variable granularity to segment the structure prompt, and then gradually optimizing it automatically in segments, without changing the original structure of the structure prompt, the theme and text granularity included in a single optimization are reduced, and the optimization effect is improved.

[0019] (2) By setting a sample set that includes all themes of the business scenario, the theme data included in the sample set can be more fully utilized during the optimization process, accurately feedback the theme defects of the prompt, and improve the accuracy of optimization.

[0020] (3) By using a large language model with stronger capabilities to optimize the structure prompt, giving full play to the text understanding ability and creative ability of the large model, abandoning the heavy work of manually repeatedly debugging the prompt in the past, the automatic optimization of more fine-grained and accurate prompts can be realized, and the intelligent degree of the large model application is improved. Brief Description of the Drawings

[0021] The accompanying drawings are used to provide a further understanding of the present invention and form a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the accompanying drawings:

[0022] Figure 1 is the overall flowchart of an automatic optimization method for large model prompt words of the present invention. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] The present invention provides an automatic optimization method for large model prompt words. As Figure 1 shown, when a user uses AI software, they need to manually and temporarily construct structural prompt words. In the prior art, markdown format is generally used to construct structural prompt words, and the structural prompt words include content such as task descriptions, rule constraints, and domain knowledge. The task description is used to record the business scenario, that is, the actual problem to be solved. The domain knowledge is used to record the well-known information that needs to be understood in the field to which the business scenario belongs. The rule constraint is used to record the category themes and event themes included in the business scenario.

[0026] The large language model determines the business scenario based on the task description and calls a sample set composed of a number of sample prompt words based on the business scenario. One embodiment of the present application is: 80% of the sample prompt words in the sample set are used as the training set, and 20% of the sample prompt words are used as the validation set. The training set is used to optimize the prompt words, and the validation set is used to verify the optimization effect of the prompt words in order to determine the optimal prompt words. A business scenario includes a number of category themes, and a category theme contains a number of event themes. The sample set contains all the category themes and event themes under this business scenario. The sample set includes annotation information, and the annotation information is used to record all the category themes and event themes included in the business scenario.

[0027] One embodiment of the present application is as follows: The business scenario is the extraction of harmful information on the network; the corresponding category themes include categories such as extreme ideological content, privacy-infringing content, and online fraud content; the event themes corresponding to online fraud include events such as online scams, false advertisements, and fictional transactions.

[0028] The annotation information can be obtained through a large language model that is more powerful than AI software capabilities and then supplemented manually. The specific process includes: inputting the distribution of sample prompts in the sample set into the large language model, receiving all inference outputs, extracting the category themes and corresponding event themes included in the inference outputs, and jointly generating annotation information by aggregating all event themes and category themes. The more comprehensive the themes filled in the annotation information, the better the optimization effect of subsequent prompts.

[0029] Split the prompt: Split it into several first fields according to the structure markers of the structure prompt, obtain the category themes included in the first fields, and combine the text descriptions corresponding to individual category themes to generate the second field.

[0030] One embodiment of the structure prompt is as follows:

[0031] ## Task description

[0032] You are a xxx, please according to xxx.

[0033] ## Field regulations

[0034] 1. xxxxxx.

[0035] 2. xxxxxx.

[0036] ## Judgment criteria

[0037] 1. xxxxxx.

[0038] 2. xxxxxx.

[0039] The structure prompt contains fixed format markers such as "##xxxx, ##xxxx, 1 2…, ##xxxx, 1 2…", and divides the structure prompt into several first fields with the format markers as the splitting points.

[0040] The structure prompt is temporarily input by the user. When the user fills in the category themes and event themes, it is easy to mix multiple category themes together. When optimizing the prompt later, it is easy to mislocate the category themes included in the first fields, affecting the subsequent optimization effect. Method for extracting category themes: Identify the category themes included in the first fields based on the theme recognition model and separately output the text descriptions related to the category themes, and combine the text descriptions to generate the second field. When generating the second field, the original text description of the user is not changed.

[0041] One embodiment of generating the second field is as follows:

[0042] ## Judgment Criteria

[0043] 1. Internet fraud

[0044] Including information such as online scams, false advertisements, fictitious transactions, personal privacy, and disclosure of personal information.

[0045] The theme recognition model recognizes that the above first field includes two category themes, namely "Internet fraud" and "information leakage". Internet fraud includes: event themes such as online scams, false advertisements, and fictitious transactions. Information leakage includes: event themes such as personal privacy and disclosure of personal information.

[0046] Based on the category theme and the original description, split the first field to generate two second fields, which can reduce the text granularity of the field to be optimized. That is: 1. Internet fraud: including information such as online scams, false advertisements, and fictitious transactions. 2. Information leakage: including information such as personal privacy and disclosure of personal information.

[0047] Generate a category field, call the training set corresponding to the structural prompt word for the business scenario, aggregate the same-category themes in the training set to generate a third field, and combine the second field and the third field of the same-category theme to generate a category field.

[0048] The generation process of the third field includes: aggregating the text descriptions of the relevant event themes of the same-category themes in the training set, removing duplicate and similar text descriptions, and then fusing them into the third field. Use the same method to obtain the third fields of all category themes included in the business scenario. Fuse the third field and the second field of the same-category theme to generate a category field. An embodiment of the present application is: when there is no second field for the category theme, directly define the third field as the category field.

[0049] Optimization prompt word: separately obtain the first inference result of each category field, compare the first inference result with the annotation information in the training set and output the difference information, fuse the difference information of all category themes to generate an optimization feedback, and optimize the structural prompt word based on the optimization feedback and output the optimal prompt word.

[0050] Supplement all category fields into the structural prompt word to generate the current prompt word, freeze the category fields that are not optimized in the current prompt word, input it into the large language model and output the first inference result, use the large language model to compare the inference information and the annotation information, and output the difference feedback of this category theme. Use the same method to traverse other category fields, and aggregate all the difference feedbacks to jointly form an optimization feedback.

[0051] The large language model optimizes the current prompt based on the optimization feedback and outputs several candidate prompts. The optimization feedback separately records the event topics missing in each category field. When the large language model optimizes, it supplements keywords in the category field based on the missing event topics to generate candidate prompts.

[0052] The candidate prompts are respectively input into the large language model and the corresponding second inference results are output. Each second inference result corresponds to a candidate prompt. The validation set is input into the second large language model and the corresponding third inference result is output. The third inference result includes the inference results of all sample prompts in the validation set. By comparing the topic similarity between the third inference result and each second inference result through the large language model, the candidate prompt with the highest similarity is output as the optimal result.

[0053] The further optimization of the prompt includes: iteratively optimizing the optimal prompt multiple times. When the number of optimizations meets the set threshold, or when the candidate prompts have not been changed in several recent optimizations, the iteration ends and the optimal candidate prompt is output, which can improve the accuracy and comprehensiveness of the optimization in the structured prompt.

[0054] The above has described the embodiments of the present invention in detail, but the content described is only the preferred embodiments of the present invention and cannot be considered as used to limit the scope of implementation of the present invention. All equal changes and improvements made according to the scope of the present invention should still fall within the scope covered by this patent.

Claims

1. A large model prompt word automatic optimization method, characterized in that: include, Segmentation of prompt words: Segmenting the structural prompt words into a plurality of first fields according to their structural tags, obtaining the category topics contained in the first fields, and combining the text descriptions corresponding to the individual category topics to generate the second fields; Generate a category field, call a sample set of business scenarios corresponding to the structure prompt words, aggregate the same category topics in the sample set to generate a third field, and combine the second field and the third field of the same category topics to generate a category field; Optimize prompt words: Get the first inference result of each category field separately, compare the first inference result with the annotation information of the training set and output the difference information, integrate the difference information of all category topics to generate optimization feedback, optimize the structural prompt words based on the optimization feedback and output the optimal prompt words.

2. The automatic optimization method for large model prompt words according to claim 1, characterized in that: The structural prompt words include task descriptions for recording business scenarios, rule constraints for recording business topics and event topics, and domain knowledge for recording publicly known information in business scenarios. The category topic includes several event topics, and the annotation information records the category topics and event topics included in the business scenario.

3. The automatic optimization method for large model prompt words according to claim 1 is characterized in that: The sample set is composed of a number of sample prompt words, 80 percent of the sample prompt words in the sample set are used as a training set, and 20 percent of the sample prompt words are used as a verification set.

4. The automatic optimization method for large model prompt words according to claim 3 is characterized in that: The process of obtaining the optimal prompt word is as follows: the large language model fills the event theme in the structural prompt word according to the optimization feedback, and outputs a number of prompt words to be selected; The second inference result of each candidate prompt word is obtained separately through the large language model, and the third inference result of the sample prompt word in the verification set is obtained through the large language model. The topic similarity between the third inference result and each second inference result is calculated, and the candidate prompt word with the highest topic similarity is the optimal prompt word.

Images