Large model prompt optimization method and device based on generalization from weak to strong, and medium

Abstract

The invention discloses a large model prompt optimization method and device based on generalization from weak to strong, and a medium, and the method comprises the steps: driving a large model to generate a group of candidate prompts for each data pair of an original data set; an output is generated with each candidate cue and a similarity to the target text is calculated. The highest K1% is a first prompt, the lowest K2% is a second prompt, and the rest is a third prompt; taking the input text, the first prompt and the third prompt as a first data set; taking the input text, the first prompt, the second prompt and the third prompt as a second data set; based on the first data set, taking the input text and the third prompt as input, taking the first prompt as target output, and optimizing a prompt corrector through a first loss function; based on the second data set, taking the input text and the third prompt as input, and finely adjusting the optimized prompt corrector through a second loss function; for a to-be-processed task, an initial prompt is generated by using the large model, and an optimized prompt can be obtained by inputting the initial prompt into the corrector.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large model prompt optimization, and in particular to a large model prompt optimization method based on weak-to-strong generalization, device and medium. BACKGROUND

[0002] In recent years, the remarkable progress of large language models (LLMs) has demonstrated their superior capabilities in various tasks. Despite these models' human-like understanding and response abilities, their performance is largely influenced by the quality of prompts. To ensure that LLMs can generate accurate responses, many studies have focused on automatically generating higher-quality prompts. However, these methods mostly concentrate on generating a single fixed prompt for a specific task, which remains unchanged when facing different inputs, often making it difficult to dynamically guide LLMs to meet diverse user needs.

[0003] To address this issue, some researchers have shifted their focus to input-dependent prompt optimization methods, which generate personalized prompts for each specific input. A common approach to achieving input-dependent prompt optimization is to train a model to generate corresponding prompts from inputs. During training, the model is typically optimized using exploratory methods such as reinforcement learning, guided by real-time feedback from LLMs that compare the differences between model outputs and true responses. However, these exploratory methods often require the model to start from scratch and search for the optimal solution in a vast prompt space, making the optimization process exceptionally complex and ultimately generating suboptimal prompts. Although existing studies have attempted to reduce the search space by designing fixed prompt templates, i.e., limiting the model to insert words within the templates, these templates often need to be manually adjusted to function effectively in different tasks. While these methods are effective in specific scenarios, they lack flexibility and automatic adaptability, making it difficult to address the diverse needs of real-world applications. As real-world scenarios become increasingly dynamic and LLM technology rapidly evolves, finding an efficient and adaptable input-dependent prompt optimization method is more important than ever. SUMMARY

[0004] To address the shortcomings of existing technologies, the present application provides a large model prompt optimization method based on weak-to-strong generalization, device and medium.

[0005] In a first aspect, the embodiments of the present application provide a large model prompt optimization method based on weak-to-strong generalization, which comprises:

[0006] For each (input text, target text) data pair in the original dataset for text generation tasks, a large model is driven to generate a set of candidate prompts that can assist it in generating target texts under the given input texts and prompt examples;

[0007] input the input text and each candidate prompt into the large model to obtain an output text corresponding to each candidate prompt; calculate the similarity between each output text and the target text, take the K1% candidate prompts with the highest similarity as the first prompts, take the K2% candidate prompts with the lowest similarity as the second prompts, and take the remaining candidate prompts as the third prompts;

[0008] take the input text, the first prompt and the third prompt as a first data set, and take the input text, the first prompt, the second prompt and the third prompt as a second data set;

[0009] construct a prompt corrector, input the input text and the third prompt into the prompt corrector as input and take the first prompt as target output, and optimize the prompt corrector through a first loss function;

[0010] input the input text and the third prompt into the optimized prompt corrector through a second loss function based on the second data set, and the second loss function is used to make the output of the prompt corrector close to the first prompt and far away from the second prompt;

[0011] in response to the input text to be processed and the prompt example, the large model outputs the second prompt; input the input text to be processed and the second prompt into the fine-tuned prompt corrector to obtain an optimized prompt.

[0012] In a second aspect, an electronic device is provided, including a memory and a processor, the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the large model prompt optimization method based on weak-to-strong generalization described above.

[0013] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the program is executed by a processor to realize the large model prompt optimization method based on weak-to-strong generalization described above.

[0014] In a fourth aspect, a computer program product is provided, including computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the large model prompt optimization method based on weak-to-strong generalization described above.

[0015] Compared with the prior art, the present application has the beneficial effects that:

[0016] The present invention provides a large-model prompt optimization method based on weak-to-strong generalization, constructs a weak-to-strong prompt training data set, takes a weak prompt (i.e., the third prompt) as input, trains a weak-to-strong prompt corrector, and outputs a strong prompt (i.e., the first prompt) with better structure and stronger semantics, thereby providing a non-zero starting point for prompt optimization and enabling it to learn the migration mapping relationship from weak prompts to strong prompts; adopts a direct preference optimization method to enable the corrector to learn in the direction of strong prompts (i.e., the first prompt) and away from poor prompts (i.e., the second prompt), identify and reduce the erroneous components in weak prompts, and gradually improve the generalization ability of the corrector from weak to strong, so that personalized prompts can be generated for each input text, which is more in line with actual needs and further alleviates the adaptability problem faced by prompt optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A flowchart of a large model prompt optimization method based on weak-to-strong generalization provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0022] like Figure 1 As shown, an embodiment of the present invention provides a large model prompt optimization method based on generalization from weak to strong, the method comprising the following steps:

[0023] Step S1: For each (input text, target text) data pair in the original dataset of the text generation task, drive the large model to generate a set of candidate prompts that can assist it in generating the target text under the conditions of given input text and prompt examples.

[0024] Further, in order to reduce the search space of the prompt and obtain high-quality prompt data within a limited number of interactions with the large model, the present example precisely demonstrates the guidance prompt generation, which specifically includes:

[0025] Step S101, obtaining an original data set of a text generation task x (i) is the i-th input text, y (i)* is the i-th target text, and N is the number of samples.

[0026] Step S102, randomly extracting a subset from the original data set wherein The size M of the subset can be flexibly set according to actual computing resources to balance the search cost and the representativeness of the example.

[0027] Step S103, generating a candidate prompt set corresponding to each data pair (x (j) , y (j)* ) in the subset by the large language model The expression is as follows:

[0028]

[0029] In the formula, I is an instruction for guiding the LLM to generate prompt data, for example, "The above shows an input text and its corresponding target output of a text generation task. Based on this example, write a prompt to make the model accurately generate the target text under the condition of only receiving the input text."

[0030] The candidate prompt set is evaluated to select the optimal prompt p (j)* corresponding to each data pair (x (j) , y (j)* ); the above operation is performed for each step of the subset , thereby constructing a context example consisting of the input text x (j) , the target text y (j)* , and the optimal prompt p (j)* .

[0031] Step S104, the large language model responds to the input text x (i) , the target text y (i)* , and the context example ε M to generate a set of candidate prompts for each data pair in the original data set using the large language model The expression is as follows:

[0032]

[0033] It should be noted that a large number of prompt data sets customized for each training data can be obtained by the step S1, which serves as the basis for realizing weak to strong generalization. This method searches a large number of samples to obtain accurate and representative context examples to assist prompt generation, thereby significantly reducing search overhead while significantly improving prompt quality. At the same time, in order to ensure the diversity of generated prompts, the temperature coefficient of the large model is generally set to 0.95 or higher. The core of the precise example guidance method proposed in step S1 is "small search + high-quality example guidance", that is, only a small number of samples need to be searched to generate efficient prompts for large-scale data through these high-quality examples, which greatly improves the efficiency and effectiveness of prompt generation. At the same time, it also provides high-quality supervision signals for subsequent weak-to-strong prompt correction training, supporting the overall performance improvement of prompt optimization.

[0034] Step S2, input the input text and each candidate prompt into the large model to obtain the output text corresponding to each candidate prompt; calculate the similarity between each output text and the target text, and take the candidate prompt with the highest similarity K1% as the first prompt, the candidate prompt with the lowest similarity K2% as the second prompt, and the remaining candidate prompts as the third prompt. Take the input text, the first prompt, and the third prompt as the first data set; take the input text, the first prompt, the second prompt, and the third prompt as the second data set.

[0035] Further, since the optimal prompt corresponding to each data pair is unknown, the present example measures the effectiveness of the prompt by comparing the difference between the output generated by the LLM under a given prompt and the corresponding target text (i.e. the real reference answer) based on an indirect prompt quality evaluation strategy. Specifically, the following steps are included:

[0036] Step S201, concatenate the input text x (i) with any candidate prompt p in the candidate prompt set (i) and input it into the large language model to obtain the output text y (i) .

[0037] Step S202, calculate the similarity between each output text y (i) and the target text y (i)* , and take the similarity as the quality evaluation score of the candidate prompt p (i) ; the expression is as follows:

[0038] r(p (i) )=metric(y (i)y (i)* )

[0039] where metric(·) can be any text similarity evaluation index, such as BLEU, ROUGE or other similarity indexes, etc.

[0040] Step S203, sort all candidate prompts from low to high according to the quality evaluation score, take the top K1% candidate prompts as the first prompt i.e. strong prompt; take the lowest K2% candidate prompts as the second prompt i.e. poor prompt; and take the remaining candidate prompts as the third prompt i.e. weak prompt.

[0041] Based on this classification result, two training data sets are constructed: the first data set contains weak prompts and their corresponding input texts and strong prompts, denoted as The second data set contains poor prompts, weak prompts and their corresponding input texts and strong prompts, denoted as These two data sets will be used to train the subsequent prompt corrector, so that it can learn the ability to map from low-quality prompts to high-quality prompts, thereby realizing the automatic optimization and enhancement of the prompts.

[0042] Step S4, constructing a prompt corrector; based on the first data set, taking the input text and the third prompt as input, and taking the first prompt as target output, the prompt corrector is optimized by the first loss function.

[0043] Specifically, based on the first data set, the weak-to-strong prompt corrector is trained with weak prompts as input, and outputs strong prompts with better structure and stronger semantics, thereby providing a non-zero starting point for prompt optimization and learning the transfer mapping relationship from weak prompts to strong prompts; including:

[0044] The prompt corrector is constructed, and the prompt corrector is selected from T5 model, GPT-4, GPT-3.5, Qwen, Llama and other generative language models, and in this example, T5 model is selected.

[0045] Based on the first data set taking the input text x (i) , the third prompt as input, and the target output is the first prompt The cross-entropy loss function is used as the first loss function, and the prompt corrector is optimized by the first loss function to learn the transfer mode from weak prompts to strong prompts, and is used to correct weak prompts to generate strong prompts; wherein the expression of the first loss function is as follows:

[0046]

[0047] wherein, The prompt correcter is represented.

[0048] It should be noted that learning to generate high-quality strong prompts directly from input text usually faces an extremely large prompt space search problem. Such a "zero-to-one" prompt optimization process not only has large computational overhead, but also has an unclear optimization path, resulting in significantly increased overall optimization difficulty. To alleviate this challenge, the present application proposes to use weak prompts as the starting point for prompt optimization, and converts the optimization process from input to high-quality prompts into a gradual prompt correction method for weak prompts. This converts the originally complex global optimization problem into a relatively simplified local adjustment task, thereby reducing the dimensionality of the search space and the complexity of the optimization process to some extent, and improving the optimization efficiency and stability.

[0049] Step S5, based on the second data set, taking the input text and the third prompt as input, fine-tuning the optimized prompt corrector through the second loss function; the second loss function is used to make the output of the prompt corrector close to the first prompt and far away from the second prompt.

[0050] Further, to avoid retaining the error information in the weak prompt in the generated prompt, in step S4, the corrector is proposed to perform prompt preference optimization, and the difference prompt is set to contain the most error information, while the strong prompt is set as a positive sample that can most accurately guide the large model, and the corrector is used for direct preference learning.

[0051] The second data set is composed of pairs of prompt samples, including strong prompts (positive samples) and corresponding difference prompts (negative samples). Each pair of prompt samples is associated with the same original input text x (i) , weak prompt . Then, the prompt preference loss function is designed using the ranking loss idea in human feedback preference learning (RLHF), so that the corrector can learn to distinguish and prefer to generate high-quality prompts.

[0052] Specifically, it includes:

[0053] Based on the second data set , taking the input text x (i) , the third prompt as input, minimizing the second loss function, making the output of the prompt corrector close to the first prompt and far away from the second prompt , thereby fine-tuning the prompt corrector; wherein the expression of the second loss function is as follows:

[0054]

[0055] where σ is a sigmoid function, β is a hyperparameter, π ref is the reference model. The hyperparameter β controls the training aggressiveness / conservativeness, the smaller β is, the more high-reward routes are selected, and the model changes more (biased towards strong hints); the larger β is, the more the model retains the original behavior.

[0056] where the second loss function encourages the corrector to assign higher preference scores to positive samples hints. Next, the gradient descent algorithm is used to optimize the corrector to gradually enhance its preference for strong hints and reduce the generation of weak hints containing misleading information or invalid guidance during the training process. After direct preference optimization, the corrector will have stronger discrimination and generation capabilities, and when faced with weak hints containing ambiguity, errors or noise, it can effectively correct and enhance them, thereby further improving the response quality and robustness of the downstream large model.

[0057] Step S6, in response to the input text to be processed and the hint examples, the large model outputs a second hint; the input text to be processed and the second hint are input into the fine-tuned hint corrector to obtain an optimized hint.

[0058] Embodiment

[0059] This embodiment verifies the effect on three text generation tasks: KG-to-Text, Summarization and Data-to-Text. The KG-to-Text task uses the WebNLG dataset, the Summarization uses the CNN / Daily Mail dataset, and the Clean E2E NLG uses the dataset. The experimental results are shown in Tables 1 and 2:

[0060] Table 1

[0061]

[0062] Table 2

[0063]

[0064]

[0065] Tables 1 and 2 summarize the comparison of the effects of the existing methods on various different large models using the method. The experimental results show that the present application achieves better performance than other methods based on three large models of LLaMA-3, Qwen-2 and GLM-4 in the three tasks of KG-to-Text, Summarization and Data-to-Text, and is significantly better than StandardPrompting, APE, InstructZero and DSP series methods. Especially in the KG-to-Text task, the present application is the highest in all evaluation indicators, showing strong versatility and stability; in the Summarization and Data-to-Text tasks, it also leads in key indicators such as BLEU, ROUGE-L and METEOR. The method proposed in the present application exhibits the strongest consistency and optimal performance in the multi-task and multi-model scenarios, fully verifying the effectiveness of the prompt correction and preference optimization mechanism in improving the prompt quality, and thereby enhancing the quality of the large model generated text.

[0066] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the large model prompt optimization method based on generalization from weak to strong as described above. As shown in Figure 2 The hardware structure diagram of the device based on the large model prompt optimization method based on generalization from weak to strong provided by the embodiment of the present application in any data processing capable device, in addition to the processor, memory and network interface shown in the embodiment, the device based on the large model prompt optimization method based on generalization from weak to strong provided by the embodiment of the present application in any data processing capable device can also include other hardware according to the actual function of the data processing capable device, which will not be described again. Figure 2 The hardware structure diagram of the device based on the large model prompt optimization method based on generalization from weak to strong provided by the embodiment of the present application in any data processing capable device, in addition to the processor, memory and network interface shown in the embodiment, the device based on the large model prompt optimization method based on generalization from weak to strong provided by the embodiment of the present application in any data processing capable device can also include other hardware according to the actual function of the data processing capable device, which will not be described again.

[0067] Correspondingly, the application further provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement the large model prompting optimization method based on generalization from weak to strong as described above. The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit of any data processing capable device and the external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0068] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application embrace any and all variations of the application that fall within the scope of the general inventive concept as defined by the appended claims and their equivalents. The specification and examples are illustrative only.

[0069] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application.

Claims

1. A large model hint optimization method based on generalization from weak to strong, characterized by: The method comprises: For each (input text, target text) data pair in the original dataset of the text generation task, the large model is driven to generate a set of candidate prompts that can help it generate the target text given the input text and prompt examples; Input the input text and each candidate prompt into the large model to obtain the output text corresponding to each candidate prompt; calculate the similarity between each output text and the target text, and use the candidate prompt with the highest similarity (K1%) as the first prompt, the candidate prompt with the lowest similarity (K2%) as the second prompt, and the remaining candidate prompts as the third prompt; The input text, the first prompt, and the third prompt are taken as the first data set; the input text, the first prompt, the second prompt, and the third prompt are taken as the second data set; Constructing a prompt corrector; based on the first dataset, taking the input text and the third prompt as input, taking the first prompt as the target output, and optimizing the prompt corrector through the first loss function; Based on the second data set, taking the input text and the third prompt as input, fine-tuning the optimized prompt corrector using a second loss function; the second loss function is used to make the output of the prompt corrector closer to the first prompt and away from the second prompt; In response to the input text to be processed and the prompt example, the large model outputs a second prompt; the input text to be processed and the second prompt are input into the fine-tuned prompt corrector to obtain an optimized prompt.

2. A large model prompt optimization method based on generalization from weak to strong according to claim 1, characterized in that: The process of generating candidate hints includes: Get the original dataset for the text generation task x (i) For the i-th input text, y (i)* is the i-th target text, N is the number of samples; From the original dataset Randomly select a subset Generate subsets through large language models Each data pair (x (j) ,y (j)* ) corresponding candidate prompt set Candidate hint set Evaluate and select each data pair (x (j) ,y (j)* ) corresponds to the optimal prompt p (j)* ;Write the input text x (j) , target text y (j)* , optimal prompt p (j)* Composition context example ε M ; The large language model responds to the input text x (i) , target text y (i)* , context example ε M , generate a set of (i) and context example ε M Generate target text y under the condition (i)* Candidate hint set 3. The large model prompt optimization method based on generalization from weak to strong according to claim 1, characterized in that: The process of obtaining the first prompt, the second prompt, and the third prompt includes: Enter the text x (i) and its candidate hint set Any candidate hint p in (i) Concatenate and input into the large language model to obtain the output text y (i) ; Calculate each output text y (i) With the target text y (i)* The similarity is used as the candidate hint p (i) Quality assessment score; All candidate prompts are sorted from low to high according to the quality assessment score, and the top K1% candidate prompts are taken as the first prompts, the bottom K2% candidate prompts are taken as the second prompts, and the remaining candidate prompts are taken as the third prompts.

4. The large model prompt optimization method based on generalization from weak to strong according to claim 3, characterized in that: Calculate each output text y (i) With the target text y (i)* The similarity uses BLEU or ROUGE similarity indicators.

5. The large model prompt optimization method based on generalization from weak to strong according to claim 1, characterized in that: The prompt corrector is selected from T5 model, GPT-4, GPT-3.5, Qwen, and Llama.

6. The large model prompt optimization method based on generalization from weak to strong according to claim 1, characterized in that: Based on the first dataset, taking the input text and the third prompt as input and the first prompt as the target output, the process of optimizing the prompt corrector through the first loss function includes: Based on the first dataset To enter text x (i) , Third Tip is the input, and the target output is the first prompt The cross entropy loss function is used as the first loss function, and the prompt corrector is optimized by the first loss function; wherein, the expression of the first loss function is as follows: Where, Indicates a prompt corrector.

7. The large model prompt optimization method based on generalization from weak to strong according to claim 1, characterized in that: Based on the second dataset, taking the input text and the third prompt as input, the process of fine-tuning the optimized prompt corrector using the second loss function includes: Based on the second dataset To enter text x (i) , Third Tip As input, minimize the second loss function to make the output of the prompt corrector close to the first prompt Stay away from the second tip The prompt corrector is fine-tuned in this way; the expression of the second loss function is as follows: Where σ is the sigmoid function, β is the hyperparameter, and π ref For reference model.

8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the large model prompt optimization method based on generalization from weak to strong as described in any one of claims 1 to 7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the large model prompt optimization method based on generalization from weak to strong is implemented as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the large model prompt optimization method based on weak to strong generalization described in any one of claims 1 to 7 is implemented.

Images