Cue word generation model, optimization method thereof and cue word generation method

Abstract

The invention provides a cue word generation model, an optimization method thereof and a cue word generation method, belongs to the field of artificial intelligence, and can solve the problem that an existing cue word generation model is poor in cue word optimization effect. The optimization method comprises the steps of training an initial cue word generation model based on a first training data set to obtain a first cue word generation model; the first training data set comprises a plurality of first samples, and the first samples comprise original cue words and initial optimization cue words; training the first cue word generation model based on the second training data set to obtain a second cue word generation model; the second training data set comprises a plurality of second samples, and the second samples comprise the first type of original cue words and corresponding preference cue words and non-preference cue words; the preference cue words and the non-preference cue words are obtained based on a first cue word generation model; the first type of original cue words represent cue words used in combination with the first type of task information, and the first type of task information refers to task information with standard answers.

Description

Technical Field

[0001] The present disclosure belongs to the field of artificial intelligence technology, and particularly relates to an optimization method for a prompt word generation model and a prompt word generation model. Background Art

[0002] Large language models (LLMs) have had a significant impact on various industries, including healthcare, finance, education, and law. The rise of LLMs, particularly ChatGPT, GPT-4, and BERT, has triggered a new wave of AI innovation. Currently, popular LLMs are Transformer-based models with billions to hundreds of billions of parameters. During pre-training, LLMs learn a wide range of human knowledge from a variety of publicly available data. For specific domains, the pre-trained LLMs are retrained and fine-tuned using task data from that domain. Through this process, LLMs acquire domain-specific knowledge, enabling them to accomplish tasks such as text translation, summarization, conference summarization, and code generation.

[0003] When using LLM, users need to provide user prompts to LLM, which then outputs the desired content based on the prompts. For example, prompts could be "Help me translate the following sentence into English," "Help me summarize the following meeting content," or "Help me generate Python code that prints hello world." Ideally, LLM will output the desired and correct content based on the prompts. However, due to issues such as model design, inaccurate prompts, and limited model training data, LLM inevitably experiences "hallucinations" in its output. For example, for subjective or unclear questions like "analyzing the sentiment associated with user comments," LLM struggles to provide accurate answers.

[0004] With the model design and performance unchanged, providing LLM with appropriate and precise user prompts can effectively reduce the probability of LLM outputting "hallucinations". Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. On the one hand, a method for optimizing a prompt word generation model is provided, wherein the optimization method includes: collecting a first training data set, and training an initial prompt word generation model based on the first training data set to obtain a first prompt word generation model; the first training data set includes multiple first samples, and the first samples include original prompt words and initial optimized prompt words corresponding to the original prompt words; collecting a second training data set, and training the first prompt word generation model based on the second training data set to obtain a second prompt word generation model; the second training data set includes multiple second samples, and the second samples include a first category of original prompt words, preferred prompt words corresponding to the first category of original prompt words, and non-preferred prompt words; the preferred prompt words and non-preferred prompt words corresponding to the first category of original prompt words are obtained based on the first prompt word generation model; the first category of original prompt words represent prompt words used in combination with first category task information, and the first category of task information refers to task information with standard answers.

[0006] In some examples, the optimization method also includes: collecting a third training data set, and training the second prompt word generation model based on the third training data set to obtain a third prompt word generation model; the third training data set includes multiple third samples, and the third samples include the second category of original prompt words, preference prompt words and non-preferential prompt words corresponding to the second category of original prompt words; the preference prompt words and non-preferential prompt words corresponding to the second category of original prompt words are obtained based on the second prompt word generation model or the first prompt word generation model; the second category of original prompt words represent prompt words used in combination with the second category of task information, and the second category of task information refers to task information with diverse answers.

[0007] In some examples, the first prompt word generation model is capable of generating first optimized prompt words based on the first category of original prompt words; collecting the second training data set includes collecting a plurality of second samples; collecting the second samples includes: inputting the first category of original prompt words into the first prompt word generation model multiple times to obtain a plurality of first optimized prompt words corresponding to the first category of original prompt words; marking each of the first optimized prompt words, wherein some of the first optimized prompt words are marked positively and the remaining first optimized prompt words are marked negatively; and selecting, from the plurality of first optimized prompt words, a first optimized prompt word marked positively as a preferred prompt word and a first optimized prompt word marked negatively as a non-preferred prompt word.

[0008] In some examples, marking the first optimization prompt word includes: using the first optimization prompt word and the first type of task information as inputs to a universal large language model, and obtaining, by the universal large language model, an output answer corresponding to the first type of task information; determining whether the output answer is consistent with a standard answer to the first type of task information; and if so, marking the first optimization prompt word as positive; otherwise, marking the first optimization prompt word as negative.

[0009] In some examples, the first prompt word generation model is capable of generating a second optimized prompt word based on the second category of original prompt words; collecting the third training data set includes: collecting a plurality of the third samples; collecting the third samples includes: inputting the second category of original prompt words into the first prompt word generation model multiple times to obtain a plurality of second optimized prompt words corresponding to the second category of original prompt words; scoring each of the second optimized prompt words to obtain an average score of the plurality of scores; and selecting, from the plurality of second optimized prompt words, a second optimized prompt word with a score greater than the average score as a preferred prompt word for the second category of original prompt words, and selecting a second optimized prompt word with a score not greater than the average score as a non-preferred prompt word for the second category of original prompt words.

[0010] In some examples, the second prompt word generation model is capable of generating a second optimized prompt word based on the second category of original prompt words; collecting the third training data set includes: collecting a plurality of the third samples; collecting the third samples includes: inputting the second category of original prompt words into the second prompt word generation model multiple times to obtain a plurality of second optimized prompt words corresponding to the second category of original prompt words; scoring each of the second optimized prompt words to obtain an average score of the plurality of scores; and selecting, from the plurality of second optimized prompt words, a second optimized prompt word with a score greater than the average score as a preferred prompt word for the second category of original prompt words, and selecting a second optimized prompt word with a score not greater than the average score as a non-preferred prompt word for the second category of original prompt words.

[0011] In some examples, scoring the second optimization prompt word includes: using the second optimization prompt word and the second type of task information as inputs to a universal large language model, and obtaining, by the universal large language model, an output answer corresponding to the second type of task information; manually evaluating the output answer corresponding to the second type of task information, and scoring the second optimization prompt word based on the evaluation result.

[0012] In some examples, the initial prompt word generation model is trained based on the first training data set using a supervised learning algorithm.

[0013] In some examples, the training of the first prompt word generation model based on the second training data set and the training of the second prompt word generation model based on the third training data set are both performed using a direct preference optimization algorithm.

[0014] In a second aspect, the present disclosure provides a prompt word generation model, which is obtained by optimizing the initial prompt word generation model using the optimization method described in any example of the first aspect.

[0015] In a third aspect, the present disclosure provides a method for generating a prompt word, which includes: inputting an original prompt word into a prompt word generation model, and obtaining an optimized prompt word corresponding to the original prompt word by at least the prompt word generation model; the prompt word generation model is as described in the second aspect.

[0016] In some examples, obtaining the optimized prompt word corresponding to the original prompt word includes multiple iterations; i is an iteration number index, which is an integer not less than 1; when i=1, for the i-th iteration, including: using the original prompt word as the i-th prompt word, obtaining the score of the i-th prompt word; judging whether the score of the output answer corresponding to the i-th prompt word reaches the target score; if so, using the i-th prompt word as the optimized prompt word and ending the iteration; if not, setting i=i+1 and proceeding to the next iteration; when i≥2, for the i-th iteration, including: judging whether i is not greater than the preset number of iterations, and if so, , then execute step S1; if not, execute step S2; step S1 includes: inputting the i-1th prompt word and the score of the i-1th prompt word into the prompt word generation model, and generating the i-th prompt word by the prompt word generation model; obtaining the score of the i-th prompt word; judging whether the score of the output answer corresponding to the i-th prompt word reaches the target score; if so, taking the i-th prompt word as the optimized prompt word; if not, setting i=i+1 and returning to the step of judging whether i is not greater than the preset number of iterations; step S2 includes: taking the i-1th prompt word as the optimized prompt word and ending the iterative process.

[0017] In some examples, when the original prompt word belongs to the first category of original prompt words, the first category of original prompt words represents prompt words used in combination with the first category of specific task information; the first category of specific task information refers to task information with a standard answer; obtaining the score of the i-th prompt word includes: inputting the i-th prompt word and n first category specific task information into a specific large language model multiple times, and obtaining the output answer of each first category specific task information respectively; n is an integer not less than 1; for each first category specific task information, comparing its corresponding output answer with the standard answer, counting the number of first category specific information whose output answer meets the standard answer, and obtaining the score of the i-th prompt word based on the number.

[0018] In some examples, when the original prompt word belongs to the second category of original prompt words, the second category of original prompt words represents prompt words used in combination with the second category of specific task information; the second category of specific task information refers to task information with diverse answers; obtaining the score of the i-th prompt word includes: inputting the i-th prompt word and the second category of specific task information into a specific large language model to obtain the output answer of the second category of specific task information; using manual evaluation of the output answer of the second category of specific task information, and obtaining the score of the i-th prompt word based on the evaluation result.

[0019] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the optimization method as described in any example of the first aspect.

[0020] In a fifth aspect, the present disclosure further provides a computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the optimization method as described in any example of the first aspect.

[0021] In a sixth aspect, the present disclosure also provides an electronic device comprising: at least one memory; and a processor coupled to the memory, the processor being configured to execute the optimization method as described in any example of the first aspect based on instructions stored in the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of the optimization method provided in Example 1.

[0023] Figure 2 Flowchart for collecting the second sample.

[0024] Figure 3 Flowchart for collecting the third sample.

[0025] Figure 4 This is a flow chart of the optimization method provided in Example 2.

[0026] Figure 5 It is an iterative process of obtaining optimized prompt words based on the original prompt words.

[0027] Figure 6 Schematic diagram of the process of obtaining the i-th original prompt word score. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0029] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0030] As used herein, "parallel" and "perpendicular" include the conditions described and conditions similar to the conditions described, and the range of the similar conditions is within an acceptable deviation range, wherein the acceptable deviation range is determined by a person of ordinary skill in the art taking into account the measurement in question and the errors associated with the measurement of the specific quantity (i.e., the limitations of the measurement system). For example, "parallel" includes absolute parallelism and approximate parallelism, wherein the acceptable deviation range for approximate parallelism can be, for example, a deviation within 5°; "perpendicular" includes absolute perpendicularity and approximate perpendicularity, wherein the acceptable deviation range for approximate perpendicularity can also be, for example, a deviation within 5°.

[0031] It will be understood that when a layer or element is referred to as being on another layer or substrate, it can be directly on the other layer or substrate, or intervening layers may be present therebetween.

[0032] As used herein, "electrically connected" includes components connected together via an element having some electrical function. There are no particular limitations on the "element having some electrical function" as long as it enables the transfer of electrical signals between the connected components. Examples of "element having some electrical function" include not only electrodes and wiring, but also switching elements such as transistors, resistors, inductors, capacitors, and other components with various functions.

[0033] Exemplary embodiments are described herein with reference to cross-sectional and / or plan views that are idealized exemplary drawings. In the drawings, the thicknesses of layers and regions are exaggerated for clarity. Therefore, variations in shape relative to the drawings due to, for example, manufacturing techniques and / or tolerances are contemplated. Therefore, the exemplary embodiments should not be construed as limited to the shapes of the regions shown herein, but rather include deviations in shape due to, for example, manufacturing. Therefore, the regions shown in the drawings are schematic in nature, and their shapes are not intended to illustrate the actual shape of regions of the device and are not intended to limit the scope of the exemplary embodiments.

[0034] LLM output "hallucination" occurs when the generated content, while seemingly plausible, is inaccurate or inconsistent with the facts. This phenomenon typically manifests as non-adherence to the original text (faithfulness), fabrication of facts (factualness), logical errors, or overconfidence. Common causes of this phenomenon include limitations in model training data, inadequate model design, or the model's inability to correctly interpret user prompts. To minimize LLM output "hallucination" and improve the accuracy of LLM output while maintaining consistent performance, it is necessary to provide the LLM with appropriate user prompts.

[0035] In order to improve the quality of LLM output content, the related art provides artificial prompt word templates (PromptTemplate), such as the preset prompt word templates provided by Baidu Qianfan Big Model Platform. In short, the artificial prompt word template is a structured text format used to guide LLM to generate a specific type of output, which can help users more accurately guide LLM to generate output that meets task requirements. However, the limitations of artificial prompt word templates are quite obvious. On the one hand, it can easily restrict the user's thinking and make the user fall into a certain fixed thinking mode, making it impossible to obtain innovative and effective content from LLM; on the other hand, it is time-consuming and has low generation efficiency. In addition, the related art also provides Automatic Prompt Engineering (APE). APE is a technology that uses algorithms to automatically generate and optimize prompt words, which can improve the performance of LLM on specific tasks; however, APE also has some shortcomings, such as it consumes a lot of computing resources and time costs, the evaluation indicators are single and therefore not applicable to all tasks, the generalization ability is limited, and it lacks interpretability.

[0036] The following lists two solutions for constructing prompt words or optimizing prompt words provided in related technologies, and explains the technical problems therein one by one.

[0037] The first method is to automatically construct prompt words based on LLM reinforcement learning. This method generally includes the following steps:

[0038] Step S1a: setting basic information related to the reinforcement learning task, wherein the basic information is used to provide background information related to the reinforcement learning task for the LLM and integrate it into prior knowledge.

[0039] Step S1b: updating the state module and memory module in the reinforcement learning in real time, so as to update the state of each decision step in real time during the decision-making process, and optimize the decision performance based on the previous decision trajectory.

[0040] Step S1c: Combine the prompt words generated by reinforcement learning into complete prompt words, ask questions to the LLM based on the complete prompt words and obtain answers.

[0041] Step S1d: parse the obtained answer to determine the legality of the answer format and the correctness of the answer.

[0042] The above method uses a reinforcement learning algorithm, which has the problem of high complexity because it involves interaction with the environment, acquisition of reward signals, and optimization of strategies. In addition, reinforcement learning requires storing a large amount of state, action, and reward information, so it has a large demand for memory. Furthermore, due to the need for interaction and exploration with the environment, reinforcement learning training time is long, and therefore there is also the problem of poor stability.

[0043] The second method is an automatic iterative optimization method of prompt words based on context learning. This method includes the following steps:

[0044] Step S2a: construct initial iteration data based on the task, and obtain initial processing results corresponding to the initial iteration data based on the localized private LLM.

[0045] Step S2b: Construct a meta-prompt word, use the initial iteration data as the target iteration data, use the initial processing result as the target processing result, and execute the following steps:

[0046] Step S2b1: Fill the meta prompt words based on the target iteration data and the target processing results, and input the filled meta prompt words into the public LLM to obtain optimized prompt words.

[0047] Step S2b2: Evaluate the optimized prompt words based on the private LLM to obtain the current processing result.

[0048] Step S2b3: In response to determining that the current processing result does not meet the preset conditions, the optimized prompt word is used as the target iteration data and the current processing result is used as the target processing result to perform the next round of iteration process; until the current processing result meets the preset conditions, the iteration process ends.

[0049] As can be seen, the second method uses a large public model to continuously iterate and optimize prompt words, evaluates and tests the optimized prompt words on a local private LLM, and then returns a supervised tuning signal to the public model, thus achieving automatic prompt word optimization. However, this second method does not generate a prompt word generation model or involve training the prompt word generation model. Therefore, it lacks the ability to independently generate optimized prompt words, limiting its application scenarios.

[0050] Before introducing the technical solutions provided by this disclosure, the following terms are explained:

[0051] (1) Prompts refer to instructions or guiding texts input by users to the LLM. They are used to clarify the task objectives, constraints, and generation direction of the LLM, and directly affect the quality, relevance, and controllability of the generated content. For example, prompts can be {Summarize the following text}, {Explain the following nouns}, {Judge whether the emotion of the following sentence is positive, negative, or neutral}, {Write a seven-character quatrain based on the following keywords}, etc. It should be noted that original prompts refer to unoptimized prompts, and optimized prompts refer to prompts that have been optimized manually or generated by the prompt generation model.

[0052] (2) Task information, also known as source material, input data, input material, or original information, together with prompt words, constitutes the input content of LLM. Continuing with the example of prompt words above, {prompt words} + {task information} can be the following examples: {Summarize the following text} + {text content XXX}, {Explain the following nouns} + {Noun 1: Absorption coefficient of the material, ...}, {Judge whether the emotion of the following sentence is positive, negative, or neutral} + {Sentence 1: XXX}, {Write a seven-character quatrain based on the following keywords} {Keyword 1, Keyword 2, ...}.

[0053] (3) First-category prompt words and first-category task information, wherein the first-category task information refers to task information with a standard answer, and the first-category prompt words refer to prompt words used in conjunction with the first-category task information. For example, {first-category prompt words} + {first-category task information} can be {determine whether the emotion of the following sentence is positive, negative, or neutral} + {Sentence 1: XXX} in the above example, wherein the emotion of Sentence 1 is certain, i.e., it has a standard answer; or it can be {select the correct answer based on the question} + {Question 1, Option A, Option B, Option C}, wherein the answer to Question 1 is certain, i.e., Question 1 has a standard answer. It should be noted that the first-category original prompt words are the first-category prompt words that have not been optimized.

[0054] (4) Second-category prompt words and second-category task information, wherein the second-category task information refers to task information that does not have a standard answer or has open-ended or diverse answers, and the second-category prompt words refer to prompt words used in conjunction with the second-category task information. For example, {second-category prompt words} + {second-category task information} can be {summarize the following text} + {text content XXX} in the above example, wherein the summary of text content XXX can be diverse, or it can be {write a seven-character quatrain based on the following keywords} {keyword 1, keyword 2, ...}, wherein the seven-character quatrain can also be open-ended.

[0055] (5) General Large Language Model (General LLM) and Specific Large Language Model (Specific LLM). General LLM and Specific LLM are two types of artificial intelligence models with different positioning. The core differences between the two are reflected in the training objectives, knowledge coverage and application scenarios. Specifically, the knowledge of General LLM covers multiple fields, such as science, literature, news, social networking, etc., but lacks depth in professional fields. It is mainly used for educational assistance or cross-domain knowledge retrieval. Common General LLMs include GPT-4, Llama3, and PalM. Specific LLMs are based on General LLMs and use specialized corpora in specific fields to enhance training. They can accurately parse professional terms in the field and output accurate answers for specific tasks in the field. Common specific LLMs include BioBERT (biomedical field), LegalGPT (legal field), and FinBERT (financial field).

[0056] (6) The first type of specific task information and the second type of specific task information, wherein the first type of specific task information is similar to the first type of task information described above, and refers to task information with a standard answer. The only difference from the first type of task information is that the first type of specific task information refers to task information input into a specific LLM; similarly, the second type of specific task information is similar to the second type of task information described above, and refers to task information without a standard answer. The only difference from the second type of task information is that the second type of specific task information refers to task information input into a specific LLM. For example, when the specific LLM is an LLM in the field of law, the first type of prompt words and the first type of specific task information can be {Select the correct answer according to the question} + {Question: The content of Article X of the XX Law is Option A, Option B, Option C}; the second type of prompt words and the second type of specific task information can be {Please analyze the following case} + {Case text}.

[0057] Those skilled in the art will understand that for relatively simple chat scenarios or question-and-answer scenarios with publicly known or objective answers, even if the input is relatively simple prompt words, the LLM model has a high probability of outputting the correct answer. For example, when the input of the LLM is {Explain the following noun} + {Noun 1: Absorption coefficient of the material, ...}, the prompt words are relatively simple, but the LLM still has a high probability of outputting the correct answer; however, for question-and-answer scenarios with strong subjectivity, the output of the LLM model is heavily dependent on the quality of the prompt words. For example, when the input of the LLM is respectively the following two situations, there is a significant difference in the output quality of the LLM. The first is {Judge the emotion of the following sentence} + {sentence}, and the second is {Please judge the emotion of the following sentence, which is a comment made by the user on the news event; assuming you are a public opinion analyst, please read the report of the news event and analyze the emotion of the user's comment based on the content of the report} + {sentence} (the prompt words are more specific). For the former, the output answer of the LLM may be wrong or one-sided, but for the latter, the output answer of the LLM will be more comprehensive and accurate. In other words, for highly subjective question-and-answer scenarios, such as those involving sentiment analysis, the quality of the prompt words directly impacts the accuracy of the LLM output. The technical solution provided by this disclosure is applicable not only to optimizing prompt words for question-and-answer scenarios with known answers, but also to those involving highly subjective questions and answers. The following describes the technical solution of this disclosure in detail.

[0058] In order to solve at least one of the above technical problems, the present disclosure provides a method for optimizing a prompt word generation model. To enable those skilled in the art to fully understand the technical solution of the present disclosure, the specific implementation of the optimization method provided by the present disclosure is described in detail below through multiple examples.

[0059] Example 1:

[0060] Figure 1 This is a flow chart of the first prompt word generation model optimization method provided in Example 1. Figure 1 As shown, the optimization method provided by the present disclosure includes at least the following steps S101-S102:

[0061] Step S101: collect a first training data set, and train an initial prompt word generation model V0 based on the first training data set to obtain a first prompt word generation model V1.

[0062] The first training data set includes a plurality of first samples, and each first sample includes an original prompt word and an initial optimized prompt word corresponding to the original prompt word.

[0063] Specifically, the initial prompt word generation model V0 can be selected from existing prompt word generation models. The initial prompt word generation model V0 is expected to be a Chat model that already possesses a large amount of knowledge and can provide corresponding responses based on user instructions. The initial prompt word generation model V0 is trained based on the first training data set to obtain the first prompt word generation model V1. This can be achieved using a supervised learning algorithm, such as one or more of a neural network, linear regression, SVM, and decision tree. The first sample includes {original prompt words, initial optimized prompt words}; in specific implementations, the first training data set can be manually collected, or by sequentially inputting multiple original prompt words into an existing prompt word generation model to obtain the initial optimized prompt words corresponding to each original prompt word, thereby obtaining pairs of original prompt words and initial optimized prompt words.

[0064] The resulting first prompt word generation model V1 is capable of generating optimized prompt words based on the original prompt word. Specifically, when the first prompt word generation model V1 is fed with the following input: {Please optimize the following prompt word} + {Original prompt word XXX}, the first prompt word generation model V1 can output: {Original prompt word XXX can be optimized to: YYY}. For example, when the first prompt word generation model V1 is fed with the following input: {Please optimize the following prompt word} + {Please summarize the following text}, the first prompt word generation model V1 can respond with: {The prompt word can be optimized to: Write an abstract for the following article, with the following specific requirements: 1. Use plain and easy-to-understand language; 2. The abstract should include a summary and a list of relevant key points; 3. Bold key parts of the abstract to improve readability}.

[0065] In this embodiment, step S101 uses a supervised learning algorithm to train the initial prompt word generation model V0 using the first training data set. By adjusting the initial prompt word generation model V0 using labeled data, the model weights are adjusted according to the difference with the labels. This can greatly improve the prompt word generation model's ability to optimize the original prompt words, improve the model's output accuracy, and provide a good task instruction foundation for users when using the general LLM.

[0066] Step S102: Collect a second training data set, and train the first prompt word generation model V1 based on the second training data set to obtain a second prompt word generation model V2.

[0067] The second training data set includes multiple second samples. A second sample includes the first category original prompt words, the preference prompt words corresponding to the first category original prompt words, and the non-preference prompt words. That is, the second sample is {first category original prompt words, preference prompt words, non-preference prompt words}. The first category original prompt words represent prompt words used in conjunction with the first category task information. The definitions and examples of the first category original prompt words and the first category task information can be referred to the description above and will not be repeated here. The preference prompt words and non-preference prompt words corresponding to the first category original prompt words can be obtained based on the first prompt word generation model V1. The specific acquisition process is as follows.

[0068] The first prompt word generation model V1 obtained in step S101 can generate the first optimized prompt word based on the first type of original prompt word. When step S102 is specifically implemented, collecting the second training data set includes collecting multiple second samples; Figure 2 Schematic diagram of the steps for collecting the second sample, as shown in Figure 2 As shown, collecting the second sample includes:

[0069] Step S1021: Input the first category original prompt word P1 multiple times into the first prompt word generation model V1 to obtain multiple first optimized prompt words P11, P12, ..., P1n corresponding to the first category original prompt word P1.

[0070] Step S1022: Each first optimization prompt word P11, P12, ..., P1n is labeled separately, with some first optimization prompt words labeled positive and others labeled negative. The labeling process is as follows: the first optimization prompt word and the first type of task information are used as input to a universal large language model (i.e., universal LLM), which obtains the output answer corresponding to the first type of task information; a determination is made as to whether the output answer is consistent with the standard answer for the first type of task information; if so, the first optimization prompt word is labeled positive; if not, the first optimization prompt word is labeled negative. Subsequently, from the multiple first optimization prompt words P11, P12, ..., P1n, a first optimization prompt word labeled positive is selected as a preferred prompt word, and a first optimization prompt word labeled negative is selected as a non-preferred prompt word. For example, the output answer A11 corresponding to the first optimized prompt word P11 is consistent with the standard answer, and the output answer A12 corresponding to the first optimized prompt word P12 does not meet the standard answer. At this time, the first optimized prompt word P11 can be used as a preferred prompt word, and the first optimized prompt word P12 can be used as a non-preferred prompt word.

[0071] For ease of understanding, we'll use the example of the first category of original prompts and the first category of task information, {Select the correct answer based on the question} + {Question 1, Options A, B, C}, as a specific example. {Select the correct answer based on the question} is a specific example of the first category of original prompts P1, and {Question 1, Options A, B, C} is a specific example of the first category of task information.

[0072] First, {Please optimize the following prompts} + {Choose the correct answer based on the question} are repeatedly input into the first prompt generation model V1 to generate multiple first optimized prompts. For example, the first optimized prompt P11 is {Please first understand the core requirements of the question, systematically analyze the relevance of each option, and ultimately select the correct answer that best matches the question based on logical reasoning}; the first optimized prompt P12 is {Please list the supporting evidence and contradictions for each option based on the question, and then select the answer that is logically consistent and correct based on reasoning}. Here, we only use the two first optimized prompts P11 and P12 as an example.

[0073] Next, {first optimized prompt word P11: ...} + {Question 1, Option A, Option B, Option C} are input into the universal LLM to obtain the output answer A11, assuming that the content of A11 is Option A. {first optimized prompt word P12: ...} + {Question 1, Option A, Option B, Option C} are input into the universal LLM to obtain the output answer A12, assuming that the content of A12 is Option B. The output answers A11 and A12 are compared with the standard answer (for example, the standard answer is Option A). It is found that the output answer A11 meets the standard answer, while the output answer A12 does not meet the standard answer. In this case, the first optimized prompt word P11 is marked as positive and used as the preferred prompt word for the first category of original prompt words P1. The first optimized prompt word P12 is marked as negative and used as the non-preferred prompt word for the first category of original prompt words P1.

[0074] Specifically, in step S102, the first prompt word generation model V1 is trained based on the second training data set, which can be completed using a direct preference optimization algorithm, such as DPO, IPO, SPPO, etc. Taking the DPO algorithm as an example, the DPO algorithm is a preference learning algorithm that bypasses explicit reward modeling and directly optimizes the policy model through an implicit reward function. Its core innovation lies in merging the reward learning and policy optimization stages in reinforcement learning into a single training objective, which can avoid the multi-stage training process of traditional RLHF (reinforcement learning based on human feedback). The loss function of the DPO algorithm is:

[0075]

[0076] Among them, (x,y w ,y l) is the second sample collected above, x represents the first type of original prompt word P1, y w Indicates the preference prompt word corresponding to the first type of original prompt word P1, y l represents the non-preferred prompt word corresponding to the first type of original prompt word P1; π ref Refers to the reference model, which corresponds to the first prompt word generation model V1 in step S102, that is, the model after supervised learning fine-tuning; σ represents the Sigmoid function; β is the temperature parameter; Represents the policy model π θ (The second prompt word generation model V2 in step S102) and the reference model π ref Preference cue word y w The log-odds ratio of Indicates preference cue word y w and the non-preferred cue word y l The difference in the log-odds ratio.

[0077] In the above steps, the first prompt word generation model V1, which has undergone supervised fine-tuning, is trained using the direct preference optimization algorithm based on the second training dataset. The resulting second prompt word generation model V2 can further improve the optimization quality of the prompt word generation model for the original prompt words compared to the first prompt word generation model V1, thereby improving the output quality of the general large language model. In addition, since the DPO algorithm can significantly reduce training time compared to traditional reinforcement learning algorithms (for example, compared to RLHF, DPO training time only requires one-third of RLHF training time), while also occupying less memory and having a lower harmful response rate, the optimization method provided in this embodiment can not only effectively improve the optimization effect of the prompt word generation model, but also minimize memory usage, save training time, and improve generation security.

[0078] Continue to refer to Figure 1 In some examples, the optimization method in embodiment 1 further includes step S103:

[0079] Step S103: Collect a third training data set, and train the second prompt word generation model V2 based on the third training data set to obtain a third prompt word generation model V3.

[0080] The third training data set includes multiple third samples, and the third samples include the second-category original prompt words P2, the preference prompt words corresponding to the second-category original prompt words P2, and the non-preferential prompt words. In general, the preference prompt words and non-preferential prompt words corresponding to the second-category original prompt words P2 can be obtained based on the second prompt word generation model V2, or based on the first prompt word generation model V1. Here, the second-category original prompt words refer to prompt words used in conjunction with the second-category task information, and the second-category task information refers to task information with diverse answers. The definitions and examples of the second-category original prompt words P2 and the second-category task information can be found above and will not be repeated here.

[0081] The first prompt word generation model V1 obtained in step S101 can generate a second optimized prompt word based on the second type of original prompt word P2, and the second prompt word generation model V2 obtained in step S102 can also generate a second optimized prompt word based on the second type of prompt word P2. In the specific implementation of step S103, collecting the third training data set includes collecting multiple third samples; Figure 3 Schematic diagram of the steps for collecting the third sample using the second prompt word generation model V2 or the first prompt word generation model V1, as shown in FIG. Figure 3 As shown, collecting the third sample includes:

[0082] Step S1031: Input the second category original prompt word P2 multiple times into the second prompt word generation model V2 or the first prompt word generation model V1 to obtain multiple second optimized prompt words P21, P22, ..., P2n corresponding to the second category original prompt word P2.

[0083] Step S1032: Score each second optimized prompt word P21, P22, ..., P2n to obtain an average score of the multiple scores. Scoring each second optimized prompt word P21, P22, ..., P2n includes: using the second optimized prompt word and the second type of task information as input to a universal large language model, and using the universal large language model to obtain an output answer corresponding to the second type of task information; manually evaluating the output answer corresponding to the second type of task information, and scoring the third optimized prompt word based on the evaluation result. From the multiple third optimized prompt words P21, P22, ..., P2n, a third optimized prompt word with a score greater than the average score is selected as the preferred prompt word for the second type of original prompt word P2, and a third optimized prompt word with a score not greater than the average score is selected as the non-preferred prompt word for the second type of original prompt word P2.

[0084] For ease of explanation, step S103 will be specifically described here using the example of the second type of original prompt and the second type of task information above: {Write a seven-character quatrain based on the following keywords} + {Keyword 1, Keyword 2, ...}. Here, {Write a seven-character quatrain based on the following keywords} is the second type of original prompt, and {Keyword 1, Keyword 2, ...} is the second type of task information.

[0085] First, {Please optimize the following prompt} + {Write a seven-character quatrain based on the following keywords} are repeatedly input into the first prompt generation model V1 or the second prompt generation model V2 to obtain multiple second optimized prompts. For example, the second optimized prompt P21 is {Please create a seven-character quatrain in the XX style, with a consistent rhyme scheme and containing the following keywords}; the second optimized prompt P12 is {Please use the XX theme and create a seven-character quatrain containing the following keywords in a progressive emotional structure, with the first sentence describing the scene, the second sentence describing the action details, the three sentences shifting time and space, and the last sentence expressing emotions implicitly; the rhyme scheme should be flat, and the last sentence should avoid closed-mouth sounds}. Here, we only use the two second optimized prompts P21 and P22 as an example.

[0086] Next, {second optimized prompt word P21: ...} + {keyword 1, keyword 2, ...} are input into the universal LLM to obtain output answer A21; {second optimized prompt word P22: ...} + {keyword 1, keyword 2, ...} are input into the universal LLM to obtain output answer A22. Output answers A21 and A22 are manually evaluated, and the second optimized prompt words P21 and P22 are scored based on the evaluation results. In this case, the second optimized prompt word with the higher score is used as the preferred prompt word for the second category of original prompt words P2, while the second optimized prompt word with the lower score is used as the non-preferred prompt word for the second category of original prompt words P2.

[0087] Specifically, in step S103, the second prompt word generation model V2 is trained based on the third training data set, and can also be completed using a direct preference optimization algorithm, such as DPO, IPO, SPPO, etc.

[0088] Since the training data collected in step S102 is only for the second type of task information, its application scope and effect are still insufficient. In step S103, by collecting third training data and using the direct preference optimization method to further optimize the second prompt word generation model V2, the effect of the prompt word generation model in generating optimized prompt words can be further improved, thereby expanding the application scenarios and scope of the prompt word generation model.

[0089] It is worth noting that the specific examples mentioned above are illustrative and are intended to facilitate understanding of the technical solutions of the present disclosure by those skilled in the art. In actual application, the prompt word generation model optimized using the optimization method provided by the present disclosure is more effective in question-and-answer scenarios with strong subjectivity. A highly subjective question-and-answer scenario refers to, for example, the need for the LLM to analyze the emotions expressed in all comments on a certain media account over the past ten years. Since different users have different attitudes towards events and different language habits, the answer to "what emotion a certain comment expresses" is not objective and cannot be obtained solely by querying the LLM's massive knowledge base. In this case, if the prompt word is too simplified, it is difficult for the LLM to output an accurate answer. Moreover, since the data that needs to be analyzed is usually huge, in order to enable the LLM to accurately analyze the "emotion of each comment" as much as possible, it is necessary to improve the prompt word as much as possible to make it universally applicable to "comments" of different lengths, different wordings, and different language habits.

[0090] Example 2:

[0091] Figure 4 This is a schematic diagram of the optimization method provided in Example 2. Figure 4 As shown, the optimization method includes steps S201 to S203, wherein step S201 is the same as step S101 in embodiment 1. To avoid repetition, please refer to the above step S101.

[0092] Step S202: Collect a fourth training data set, and train the first prompt word generation model V1 based on the fourth training data set to obtain a fourth prompt word generation model V4.

[0093] The fourth training dataset includes multiple fourth samples, including the second-category original prompt words P2, the preferred prompt words corresponding to the second-category original prompt words P2, and the non-preferred prompt words. The preferred prompt words and non-preferred prompt words corresponding to the second-category original prompt words P2 are obtained based on the first prompt word generation model V1. The second-category original prompt words here refer to prompt words used in conjunction with the second-category task information. The definitions and examples of the second-category original prompt words P2 and the second-category task information can be found above and are not further elaborated here.

[0094] It can be understood that the fourth sample here is similar to the third sample in Example 1, the only difference being that the preference prompt words and non-preference prompt words in the fourth sample are obtained based on the first prompt word generation model V1. Therefore, the collection process of the fourth sample can refer to Figure 3The relevant descriptions of step S103 in Example 1 are not repeated here to avoid repetition. Similar to Example 1, in step S202, the first prompt word generation model V1 is trained based on the fourth training data set, which can be completed using a direct preference optimization algorithm, such as DPO, IPO, SPPO, etc.

[0095] Step S203: Collect a fifth training data set, and train the fourth prompt word generation model V4 based on the fifth training data set to obtain a fifth prompt word generation model V5.

[0096] The fifth training dataset includes multiple fifth samples, including the first-category original prompt word P1, the preferred prompt word corresponding to the first-category original prompt word P1, and the non-preferred prompt word. The preferred prompt word and the non-preferred prompt word corresponding to the first-category original prompt word P1 are obtained based on the first prompt word generation model V1 or the fourth prompt word generation model V4. The first-category original prompt word here refers to the prompt word used in conjunction with the first-category task information. The definitions and examples of the first-category original prompt word P1 and the first-category task information can be found above and are not further elaborated here.

[0097] It can be understood that the fifth sample here is similar to the second sample in Example 1, with the only difference being that the preference prompt words and non-preference prompt words in the fifth sample can be obtained based on either the first prompt word generation model V1 or the fourth prompt word generation model V4. Therefore, the collection process of the fifth sample can refer to Figure 3 The relevant descriptions of step S102 in Example 1 are not repeated here to avoid repetition. Similar to Example 1, in step S203, the fourth prompt word generation model V4 is trained based on the fifth training dataset, which can be completed using a direct preference optimization algorithm, such as DPO, IPO, SPPO, etc.

[0098] In short, the optimization process of Example 2 is similar to that of Example 1, differing only in that Example 2 reverses the order of Steps S102 and S103 in Example 1, and accordingly, the source of the training dataset in each step is also changed. However, it should be noted that while the fifth prompt word generation model V5 obtained in Example 2 significantly improves the prompt word optimization effect compared to the initial prompt word generation model, it has been verified that the third prompt word generation model V3 ultimately obtained in Example 1 achieves a better optimization effect on prompt words than Example 2. In practical applications, those skilled in the art can select the optimal one based on the specific scenario.

[0099] Based on the same inventive concept, in a second aspect, the present disclosure provides a prompt word generation model, which is obtained by optimizing the initial prompt word generation model V0 using the optimization method provided in the first aspect. For example, the prompt word generation model can be the second prompt word generation model V2 or the third prompt word generation model V3 in Example 1, or the fourth prompt word generation model V4 or the fifth prompt word generation model V5 in Example 2.

[0100] As an optional example, the prompt word generation model provided by the present disclosure can be a second prompt word generation model V2 obtained through steps S101 and S102 of the optimization method provided in the first aspect. In conjunction with the description in Example 1, the second prompt word generation model V2 is obtained by optimizing the initial prompt word generation model through two training steps, wherein the first training step includes: training the initial prompt word generation model based on a first training dataset to obtain a first prompt word generation model V1, and the training step is completed using a supervised learning algorithm; the second training step includes: training the first prompt word generation model V1 based on a second training dataset to obtain a second prompt word generation model V2, and the training step is completed using a direct preference optimization algorithm. The second prompt word generation model V2 obtained after two training steps has model parameters optimized by learning from a large amount of data, compared to the initial prompt word generation model. Therefore, the optimization quality of the original prompt words can be improved, especially the optimization quality of the "first type of original prompt words", thereby providing high-quality prompt words for the general large language model and improving the accuracy of the answers output by the general large language model, especially the accuracy of the answers output by the general large language model for the "first type of task information". Taking a specific application scenario as an example, the "first-category original prompt words" are input into the second prompt word generation model V2 provided in the present disclosure, which can optimize the "first-category original prompt words" to obtain high-quality first-category prompt words; at this time, if the high-quality first-category prompt words and the corresponding first-category task information are input into the general large language model, the general large language model has a high probability of outputting the standard answer to the first-category task information.

[0101] As another optional example, the prompt word generation model provided by the present disclosure can be a third prompt word generation model V3 obtained through steps S101, S102, and S103 of the optimization method provided in the first aspect. In combination with the description in Example 1, the third prompt word generation model V3 is obtained by optimizing the initial prompt word generation model through three training steps, wherein the first training process includes: training the initial prompt word generation model based on a first training data set to obtain a first prompt word generation model V1, and the training process is completed using a supervised learning algorithm; the second training process includes: training the first prompt word generation model V1 based on a second training data set to obtain a second prompt word generation model V2, and the training process is completed using a direct preference optimization algorithm; the third training process includes: training the second prompt word generation model V2 based on a third training data set to obtain a third prompt word generation model V3, and the training process is also completed using a direct preference optimization algorithm. Because the third training process further adds a large amount of training data to further optimize the parameters of the second prompt word generation model V2, the third prompt word generation model V3 not only possesses the capabilities of the second prompt word generation model V2, but can also further improve the optimization quality of the "second-category original prompt words," thereby providing high-quality second-category prompt words for the general large language model, thereby improving the quality of the answers output by the general large language model for the "second-category task information." Taking a specific application scenario as an example, inputting the "second-category original prompt words" into the third prompt word generation model V3 provided by the present disclosure can optimize the "second-category original prompt words" to obtain high-quality second-category prompt words. At this time, if the high-quality second-category prompt words and the corresponding second-category task information are input into the general large language model, since the second-category task information refers to tasks without standard answers, the general large language model will output high-quality answers to this second-category task information. Those skilled in the art will understand that this is because the third prompt word generation model V3 after the third training can optimize the "second type of original prompt words" into "higher-scoring" preference prompt words, and use the "higher-scoring" preference prompt words as prompt words for the general large language model, which of course outputs "higher-scoring" high-quality answers.

[0102] As another optional example, the prompt word generation model provided by the present disclosure can be the fourth prompt word generation model V4 obtained through steps S201 and S202 of the optimization method provided in the first aspect. In conjunction with the description in Example 2, the fourth prompt word generation model V4 is obtained by optimizing the initial prompt word generation model through two training steps. The first training step includes: training the initial prompt word generation model based on a first training dataset to obtain a first prompt word generation model V1, and this training step is performed using a supervised learning algorithm; the second training step includes: training the first prompt word generation model V1 based on a fourth training dataset to obtain a fourth prompt word generation model V4, and this training step is performed using a direct preference optimization algorithm. Compared to the initial prompt word generation model, the fourth prompt word generation model V4 obtained through two training steps has its model parameters optimized through extensive data learning. This improves the optimization quality of the original prompt words, particularly the optimization quality of the "second type of original prompt words," thereby providing high-quality prompt words for the general large language model and improving the quality of the answers output by the general large language model. Taking a specific application scenario as an example, the "second-category original prompt words" are input into the fourth prompt word generation model V4 provided in the present disclosure, which can optimize the "second-category original prompt words" to obtain high-quality second-category prompt words; at this time, if the high-quality second-category prompt words and the corresponding second-category task information are input into the general large language model, the general large language model should output a high-quality answer to the second-category task information.

[0103] It should be noted that although both the third prompt word generation model V3 and the fourth prompt word generation model V4 can optimize the "second category prompt words", since the training samples of the fourth prompt word generation model V4 do not include the "first category original prompt words" and their preferred prompt words and non-preferred prompt words, the fourth prompt word generation model V4 is lacking in the optimization effect on the "first category original prompt words". This is an important difference between the fourth prompt word generation model V4 and the third prompt word generation model V3 in the previous example.

[0104] As another optional example, the prompt word generation model provided by the present disclosure can be the fifth prompt word generation model V5 obtained through steps S201, S202, and S203 of the optimization method provided by the first aspect. In conjunction with the description in Example 2, the fifth prompt word generation model V5 is obtained by optimizing the initial prompt word generation model through three training steps, wherein the first training step includes: training the initial prompt word generation model based on a first training data set to obtain a first prompt word generation model V1, and the training step is completed using a supervised learning algorithm; the second training step includes: training the first prompt word generation model V1 based on a fourth training data set to obtain a fourth prompt word generation model V4, and the training step is completed using a direct preference optimization algorithm; and the third training step includes: training the fourth prompt word generation model V4 based on a fifth training data set to obtain a fifth prompt word generation model V5, and the training step is completed using a direct preference optimization algorithm. Because the third training process further adds a large amount of training data to further optimize the parameters of the fourth prompt word generation model V4, the fifth prompt word generation model V5 not only has the capabilities of the fourth prompt word generation model V4, but can also further improve the optimization quality of the "first-category original prompt words", thereby providing high-quality first-category prompt words for the general large language model, thereby improving the accuracy of the answers output by the general large language model for the "first-category task information". Taking a specific application scenario as an example, the "first-category original prompt words" are input into the fifth prompt word generation model V5 provided by the present disclosure, which can optimize the "first-category original prompt words" to obtain high-quality first-category prompt words. At this time, if the high-quality first-category prompt words and the corresponding first-category task information are input into the general large language model, the general large language model will output the standard answer for the first-category task information. Those skilled in the art will understand that this is because the fifth prompt word generation model V5, after the third training, can optimize the "first-category original prompt words" into their corresponding preferred prompt words, use the preferred prompt words as prompt words for the general large language model, and of course output the standard answer.

[0105] It should be noted that although both the third prompt word generation model V3 and the fifth prompt word generation model V5 are derived from the initial prompt word generation model after three training cycles, the difference between the two lies in the order of some training steps. Specifically, the former places the training step requiring manual feedback / scoring in the third step, while the latter places it in the second step. However, the inventors discovered that while both the third prompt word generation model V3 and the fifth prompt word generation model V5 can optimize the first and second category prompt words, the third prompt word generation model V3 achieves superior optimization results. This may be because the fifth prompt word generation model V5 prematurely executes the training step requiring manual feedback / scoring, introducing a large amount of error correction noise from manual feedback when the model's initial capabilities are insufficient, or causing the initial model to overfit based on manual scoring, resulting in a model that caters to subjective human preferences. Therefore, the preferred prompt word generation model provided in the present disclosure is the third prompt word generation model V3.

[0106] Based on the same inventive concept, in a third aspect, the present disclosure provides a method for generating a prompt word. The method comprises: inputting an original prompt word into the prompt word generation model provided in the second aspect, and obtaining an optimized prompt word corresponding to the original prompt word from at least the prompt word generation model.

[0107] In order to further optimize the application effect of the prompt word generation model provided by the present disclosure in a specific LLM, the present disclosure uses the OPRO method to further improve the optimization ability of the prompt word generation model in a specific LLM. The process of obtaining the optimized prompt word using the OPRO method includes multiple iterations, Figure 5 A schematic diagram of the iterative process.

[0108] like Figure 5 As shown, when i=1, for the i-th iteration, including:

[0109] Use the original prompt word P0 as the prompt word for the i-th time and obtain the score of the prompt word for the i-th time;

[0110] Determine whether the score of the output answer corresponding to the i-th prompt word reaches the target score; if so, use the i-th prompt word as the optimized prompt word and end the iteration; if not, set i = i + 1 and proceed to the next iteration.

[0111] When i ≥ 2, for the i-th iteration, including:

[0112] Determine whether i is not greater than a preset number of iterations. If so, execute step S1; if not, execute step S2.

[0113] Step S1 includes: inputting the i-1th prompt word and its score into the prompt word generation model, which then generates the i-th prompt word. Obtaining the score for the i-th prompt word. Determining whether the score of the output answer corresponding to the i-th prompt word meets the target score; if so, using the i-th prompt word as the optimized prompt word and terminating the iteration; if not, setting i = i + 1 and returning to the step of determining whether i is less than the preset number of iterations.

[0114] Step S2 includes: taking the prompt word of the i-1th time as the optimized prompt word, and ending the iterative process.

[0115] Specifically, the score of the i-th prompt word in the iterative process can be obtained in the following way.

[0116] When the original prompt word P0 belongs to the first category of original prompt words, the first category of original prompt words here refers to prompt words used in combination with the first category of specific task information, and the first category of specific task information refers to task information with standard answers; the definitions of the first category of original prompt words and the first category of specific task information can be found above and will not be detailed here. Figure 6 This is the process of obtaining the score of the i-th prompt word when the original prompt word is the first type of original prompt word, such as Figure 6 As shown in , the scoring process for obtaining the i-th prompt word includes:

[0117] The i-th prompt word and n first-category specific task information M1, M2, ..., Mn are input into a specific large language model multiple times, obtaining output answers AM1, AM2, ..., AMn for each first-category specific task information M1, M2, ..., Mn. For each first-category specific task information, the corresponding output answer AM1, AM2, ..., AMn is compared with the respective standard answer. The number of first-category specific task information whose output answers match the standard answer is counted, and a score for the i-th prompt word is obtained based on this number. For example, assuming there are n first-category specific task information, the number of correct output answers is m, where n is an integer not less than 1 and m is an integer not greater than n. In this case, the score can be recorded as m / n.

[0118] When the original prompt word P0 belongs to the second category of original prompt words, where the second category of original prompt words refers to prompt words used in conjunction with the second category of specific task information, which refers to task information with diverse answers; the definitions and examples of the second category of original prompt words and the second category of specific task information can be found above. When the original prompt word belongs to the second category of original prompt words, the score of the prompt word for the i-th time is obtained, including:

[0119] The i-th prompt word and the second-category specific task information are input into a specific large language model to obtain an output answer for the second-category specific task information. The output answer for the second-category specific task information is manually evaluated, and a score for the i-th prompt word is obtained based on the evaluation result. It will be understood by those skilled in the art that the higher the quality of the output answer, the higher the score for the i-th prompt word.

[0120] Those skilled in the art will understand that when the prompt word belongs to the first category of prompt words, the scoring of the i-th prompt word can be implemented by a program or manually; but when the prompt word belongs to the second category of prompt words, the scoring of the i-th prompt word should be implemented by manual evaluation.

[0121] In the related art, the OPRO method, when optimizing prompt words, typically only uses samples with standard answers. Specifically, it performs multiple iterations using only the first category of prompt words, the first category of specific task information, and a specific LLM to obtain optimized prompt words. This approach has limited applicability, and its optimization effect on the second category of original prompt words still needs improvement. However, the present disclosure expands the dataset used by the OPRO method to include not only data with standard answers but also data without standard answers. This further improves the prompt word generation model's optimization effect on prompt words for specific LLMs.

[0122] Based on the same inventive concept, in a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the optimization method as described in any embodiment of the first aspect.

[0123] Based on the same inventive concept, in a fifth aspect, the present disclosure provides a computer program product, comprising instructions, which, when executed by a processor, enable the processor to execute the optimization method as described in any embodiment of the first aspect.

[0124] Based on the same inventive concept, in the sixth aspect, the present disclosure also provides an electronic device, comprising: at least one memory; and a processor coupled to the memory, the processor being configured to execute the optimization method as described in any embodiment of the first aspect based on instructions stored in the memory.

[0125] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for optimizing a prompt word generation model, wherein: The optimization method comprises: Collecting a first training data set, and training an initial prompt word generation model based on the first training data set to obtain a first prompt word generation model; the first training data set includes a plurality of first samples, each of which includes an original prompt word and an initial optimized prompt word corresponding to the original prompt word; Collecting a second training data set, and training the first prompt word generation model based on the second training data set to obtain a second prompt word generation model; The second training data set includes a plurality of second samples, and the second samples include a first category of original prompt words, preferred prompt words corresponding to the first category of original prompt words, and non-preferred prompt words.

2. The optimization method according to claim 1, wherein: The preference prompt words and non-preference prompt words corresponding to the first category of original prompt words are obtained based on the first prompt word generation model; the first category of original prompt words represent prompt words used in combination with the first category of task information, and the first category of task information refers to task information with standard answers.

3. The optimization method according to claim 1, wherein: Also includes: Collecting a third training data set, and training the second prompt word generation model based on the third training data set to obtain a third prompt word generation model; The third training data set includes multiple third samples, and the third samples include the second category of original prompt words, the preferred prompt words and non-preferred prompt words corresponding to the second category of original prompt words; the preferred prompt words and non-preferred prompt words corresponding to the second category of original prompt words are obtained based on the second prompt word generation model or the first prompt word generation model; the second category of original prompt words represent prompt words used in combination with the second category of task information, and the second category of task information refers to task information with diverse answers.

4. The optimization method according to claim 2, wherein: The first prompt word generation model is capable of generating a first optimized prompt word according to the first type of original prompt words; The collecting of the second training data set includes collecting a plurality of the second samples; collecting the second sample, comprising: Inputting the first category of original prompt words multiple times into the first prompt word generation model to obtain multiple first optimized prompt words corresponding to the first category of original prompt words; Each of the first optimization prompt words is marked, wherein some of the first optimization prompt words are marked as positive and the remaining first optimization prompt words are marked as negative; from the plurality of first optimization prompt words, one of the first optimization prompt words marked as positive is selected as a preferred prompt word, and one of the first optimization prompt words marked as negative is selected as a non-preferred prompt word.

5. The optimization method according to claim 4, wherein: Marking the first optimization prompt word includes: Using the first optimization prompt word and the first type of task information as input to a universal large language model, and obtaining an output answer corresponding to the first type of task information by the universal large language model; Determine whether the output answer is consistent with the standard answer of the first type of task information; if so, mark the first optimization prompt word as positive; if not, mark the first optimization prompt word as negative.

6. The optimization method according to claim 3, wherein: The first prompt word generation model is capable of generating second optimized prompt words according to the second type of original prompt words; The collecting of the third training data set includes: collecting a plurality of the third samples; Collecting the third sample includes: Inputting the second category of original prompt words into the first prompt word generation model multiple times to obtain multiple second optimized prompt words corresponding to the second category of original prompt words; Scoring each of the second optimized prompt words to obtain an average score of the multiple scores; selecting, from the multiple second optimized prompt words, a second optimized prompt word with a score greater than the average score as a preferred prompt word for the second category of original prompt words, and selecting a second optimized prompt word with a score not greater than the average score as a non-preferred prompt word for the second category of original prompt words.

7. The optimization method according to claim 3, wherein: The second prompt word generation model is capable of generating second optimized prompt words according to the second type of original prompt words; The collecting of the third training data set includes: collecting a plurality of the third samples; Collecting the third sample includes: Inputting the second category of original prompt words into the second prompt word generation model multiple times to obtain multiple second optimized prompt words corresponding to the second category of original prompt words; Scoring each of the second optimized prompt words to obtain an average score of the multiple scores; selecting, from the multiple second optimized prompt words, a second optimized prompt word with a score greater than the average score as a preferred prompt word for the second category of original prompt words, and selecting a second optimized prompt word with a score not greater than the average score as a non-preferred prompt word for the second category of original prompt words.

8. The optimization method according to claim 6 or 7, wherein: Scoring the second optimization prompt word includes: Using the second optimized prompt word and the second type of task information as input to a universal large language model, and obtaining an output answer corresponding to the second type of task information from the universal large language model; The output answer corresponding to the second type of task information is manually evaluated, and the second optimized prompt word is scored according to the evaluation result.

9. The optimization method according to claim 1, wherein: The initial prompt word generation model is trained based on the first training data set and is completed using a supervised learning algorithm.

10. The optimization method according to claim 3, wherein: The training of the first prompt word generation model based on the second training data set and the training of the second prompt word generation model based on the third training data set are both completed by using a direct preference optimization algorithm.

11. A prompt word generation model, which is obtained by optimizing an initial prompt word generation model using the optimization method according to any one of claims 1 to 10.

12. A method for generating a prompt word, wherein: include: Inputting the original prompt word into the prompt word generation model, and obtaining at least the optimized prompt word corresponding to the original prompt word by the prompt word generation model; The prompt word generation model is as described in claim 11.

13. The generation method according to claim 12, wherein: Obtaining the optimized prompt word corresponding to the original prompt word includes multiple iterative processes; i is the iteration number index, which is an integer not less than 1; When i=1, for the i-th iteration, including: The original prompt word is used as the prompt word for the i-th time, and the score of the prompt word for the i-th time is obtained; whether the score of the output answer corresponding to the prompt word for the i-th time reaches the target score is determined; if so, the prompt word for the i-th time is used as the optimized prompt word, and the iteration ends; if not, i is set to i + 1 and the next iteration is performed; When i ≥ 2, for the i-th iteration, including: Determine whether i is not greater than a preset number of iterations. If so, execute step S1; if not, execute step S2; Step S1 includes: inputting the i-1th prompt word and the score of the i-1th prompt word into a prompt word generation model, and generating the i-th prompt word by the prompt word generation model; Obtain the score of the i-th prompt word; determine whether the score of the output answer corresponding to the i-th prompt word reaches the target score; if so, use the i-th prompt word as the optimized prompt word; if not, set i = i + 1 and return to the step of determining whether i is not greater than the preset number of iterations; Step S2 includes: taking the prompt word of the i-1th time as the optimized prompt word, and ending the iterative process.

14. The generation method according to claim 13, wherein: When the original prompt words belong to the first category of original prompt words, the first category of original prompt words represents prompt words used in combination with the first category of specific task information; The first type of specific task information refers to task information with standard answers; The step of obtaining the score of the prompt word for the i-th time includes: Input the i-th prompt word and n first-category specific task information into the specific large language model multiple times, and obtain the output answer of each first-category specific task information respectively; n is an integer not less than 1; For each of the first-category specific task information, the corresponding output answer is compared with the standard answer, and the number of the first-category specific information whose output answer meets the standard answer is counted, and the score of the i-th prompt word is obtained based on the number.

15. The generation method according to claim 13, wherein: When the original prompt words belong to the second category of original prompt words, the second category of original prompt words represent prompt words used in combination with the second category of specific task information; The second type of specific task information refers to task information with diverse answers; The step of obtaining the score of the prompt word for the i-th time includes: Input the i-th prompt word and the second type of specific task information into the specific large language model to obtain the output answer of the second type of specific task information; The output answers of the second type of specific task information are manually evaluated, and the score of the i-th prompt word is obtained according to the evaluation result.

16. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the optimization method according to any one of claims 1 to 10 is implemented.

17. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform the optimization method according to any one of claims 1 to 10.

18. An electronic device comprising: at least one memory; and A processor coupled to the memory, wherein the processor is configured to execute the optimization method according to any one of claims 1 to 10 based on instructions stored in the memory.

Images