An automatic thought chain prompt generation method based on black box optimization and vulnerability quantification

Abstract

The present invention discloses a method for generating automatic thought chain prompts based on black-box optimization and vulnerability quantification, comprising the following steps: obtaining an unlabeled training data set; screening a subset of high-difficulty questions through a difficulty judgment module to generate a labeled data set; utilizing a large language model to generate multiple reasoning chains for each question in the labeled data set, retaining reasoning chains consistent with the correct answer to form an example library; utilizing a variance reduction strategy gradient estimator to optimize the reasoning chain selection strategy to generate common style prompts; and performing style diversification processing on the common style prompts through a novel format prompt module to generate anti-fragility automatic thought chain prompts.

Description

Technical Field

[0001] The present invention relates to the technical field of black box optimization, and in particular to a method for generating automatic thought chain prompts based on black box optimization and vulnerability quantification. Background Art

[0002] In recent years, with the rapid development of large language models, a variety of innovative prompt engineering technologies have emerged. Among them, zero-shot prompting technology guides the model to complete a task by simply providing a task description without relying on any training data. In contrast, few-shot prompting technology goes a step further by incorporating a small number of carefully designed input-output examples into the prompts, significantly improving the model's understanding of the task intent and boundaries. These two breakthrough technologies have greatly lowered the application threshold for large language models, enabling them to quickly adapt to various new task scenarios without the need for time-consuming fine-tuning training. To solve complex reasoning tasks, two advanced prompt engineering technologies are currently being used: thought chain prompting and automatic thought chain prompting. Thought chain prompting technology significantly improves the model's performance in logical reasoning tasks by guiding large language models to demonstrate the complete reasoning process. However, this method requires manual and meticulous design of step-by-step reasoning examples, which is limited by low efficiency and insufficient scalability. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method for automatically generating thought chain prompts based on black-box optimization and fragility quantification. The method reduces the training cost and time of automatically generating prompts for large language models by screening training samples, generates detailed thought chain prompts for reasoning problems in different fields, and sets different prompt styles to reduce the fragility of prompts, thereby helping large language models understand prompts and questions and give more reasonable and appropriate answers.

[0004] Technical solution: The present invention provides a method for automatically generating thought chain prompts based on black box optimization and vulnerability quantification, comprising the following steps:

[0005] (1) Obtain an unlabeled training dataset; filter a subset of high-difficulty questions through the difficulty judgment module to generate a labeled dataset;

[0006] (2) Use a large language model to generate multiple reasoning chains for each question in the annotated dataset, and retain the reasoning chains that are consistent with the correct answer to form an example library;

[0007] (3) Adopting the variance reduction strategy gradient estimator to optimize the inference chain selection strategy and generate common style cues;

[0008] (4) Through the novel format prompt module, the common style prompts are diversified and the automatic thought chain prompts with anti-fragility are generated.

[0009] Furthermore, the difficulty judgment module operates by generating a predicted probability distribution of k answers for the large language model. Calculate entropy:

[0010] ;

[0011] in, Representing a parameterized model In the given problem Generate a specific answer when The predicted probability of Represents the parameters of the language model; adjustable hyperparameters and Used to control the probability scaling ratio; select the first n questions with the largest entropy values ​​as high-difficulty questions.

[0012] Furthermore, when the number of high-difficulty questions exceeds G, M questions are randomly selected; the construction of the labeled data set is achieved through automatic labeling by a large language model; where G is the set threshold for the number of high-difficulty questions.

[0013] Furthermore, the generation and pruning of reasoning chains include: generating l reasoning chains for each question; comparing the generated answers with the labeled answers, retaining only the correctly deduced reasoning chains, and pairing the questions with the correct reasoning chains to form an initial example set.

[0014] Furthermore, the optimization process of the variance reduction policy gradient estimator includes: defining the latent variable Obey categorical distribution ;in, The probability vector of N candidate examples is updated by the gradient estimation formula :

[0015] ;

[0016] Where T represents the complete sample set, is the i-th example, is the i-th example The probability of being selected; A collection of independently sampled examples ; is the gradient estimate, used to update ; is the probability vector gradient;

[0017] Represents the current sample set The difference between the loss of and the average loss of all example sets;

[0018] Update the example distribution using projected stochastic gradient descent:

[0019] ;

[0020] in, is the learning rate, I is the size of the example set, Calculate for projection.

[0021] Furthermore, during the optimization process: loss function Based on the cross entropy calculation between the black box model output and the true label; the number of context examples is dynamically adjusted to 3-6.

[0022] Furthermore, the operation of the novel format hint module includes: designing a unique style template for each few-sample example. The template types include: non-semantic markup format rewriting, visual structure adjustment, hierarchical optimization reconstruction, and nesting common style hints into the style template to generate the final hint.

[0023] An electronic device described in the present invention includes a memory and a processor, wherein the memory stores a computer program, and the processor implements any one of the methods described above when executing the program.

[0024] The computer-readable storage medium of the present invention stores a computer program, which implements the steps of any one of the methods when executed by a processor.

[0025] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: (1) Using difficulty as an indicator to select sample problems that help improve the reasoning ability of the large language model, reducing the training cost and time of the model. (2) Using the variance reduction strategy gradient estimator and projection calculation to allow the large language model to autonomously generate correct thought chain prompts, avoiding the appearance of toxic and harmful prompt content. (3) Building a new style prompt template, the large language model uses the novel prompts generated by the template to effectively reduce the vulnerability of the prompt itself. (4) The novel prompts improve the question-answering effect of the large language model in the field of reasoning problems, and the accuracy and stability of the answers are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 is a diagram of the reasoning process of the present invention for difficult problem samples;

[0028] Figure 3 This is a performance graph of the random prompts and novel prompts of the present invention on 6 reasoning datasets;

[0029] Figure 4 This is a comparison chart of the effects of novel prompts, automatic thought chain prompts, and self-improvement under different pool sizes of the present invention;

[0030] Figure 5 3 is a comparison chart of the performance differences between the common style prompt and the novel prompt of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0032] like Figure 1 As shown, an embodiment of the present invention provides a method for automatically generating thought chain prompts based on black box optimization and vulnerability quantification, comprising the following steps:

[0033] Given an unlabeled training dataset (where each q is a question without an inference step), construct a new dataset , the dataset contains The n labeled samples screened out will be described. How to screen the most challenging n questions based on the difficulty of the questions and label them. After obtaining the dataset E, a large language model is used to generate l reasoning chains for each sample, and the reasoning chains consistent with the correct answer are retained. New prompts are formed by pairing questions with reasoning chains. These prompts can be used as efficient demonstration samples. Using the "novel format" strategy, these automatic thinking chain prompts are integrated into a variety of formats, and the style sensitivity differences are quantified by calculating the performance fluctuation (that is, the difference in accuracy between the best and worst prompt formats). Among them,

[0034] (1) The specific method for determining difficulty is as follows: When selecting a subset of questions from a large dataset, the difficulty score of the large language model is used as a metric to evaluate the difficulty of each question. Based on the thought chain prompt framework, the large language model is required to generate k answers for each question, and then the importance index is calculated by the entropy value of the predicted answers. The specific calculation formula is as follows:

[0035] ;

[0036] in, Representing a parameterized model In the given problem Generate a specific answer when The predicted probability of Represents the parameters of the language model; adjustable hyperparameters and This is used to control probability scaling; the top n questions with the highest entropy values ​​are selected as high-difficulty questions. The logarithmic contribution of prediction results is adjusted by increasing the weight of top predictions or weakening the influence of low-frequency predictions. This design balances high-confidence and low-confidence predictions, enabling more accurate difficulty grading for problem selection across different complexity reasoning tasks.

[0037] (2) The selection and annotation are as follows: After evaluating the difficulty of each question, a ranking is generated based on the difficulty score, and the top n most difficult questions are selected for annotation. If there are a large number of difficult questions (more than G, G = 500), m questions are randomly selected from the subset. Subsequently, these selected questions are annotated with the correct answers by the large language model to form a new dataset. E contains the final selected sample cases. This dataset will be used by the large language model to generate thought chain prompts.

[0038] (3) Generate and prune automatically generated thought chain prompts, as follows: Based on the sample set E, use the large language model to generate l reasoning chains for each question q. For cost and efficiency considerations, l=3 is set in the experiment. By comparing the generated answers with the standard answers, incorrect reasoning chains are screened out, and only the chains that can derive the correct answers are retained, and finally n high-quality reasoning chains are obtained. These reasoning chains and the sample set E together constitute the example library. Figure 2 As shown, the reasoning process of the large language model on difficult questions and the accuracy of the answers obtained.

[0039] (4) Reasoning chain optimization; specifically, after building a high-quality example set, the next step is to apply it to a large language model, but the following limitations must be considered: (41) The input context length of the LLM is limited, and all examples cannot be used at once; (42) The order in which examples are presented may affect model performance; (43) Different downstream tasks may require different examples to achieve the best results. A reasonable solution is to automatically select 3-6 examples based on the specific context and task. This process can be regarded as optimizing a supervised model with latent variables:

[0040] For each thought chain index i, initialize a class distribution Hidden variables , the random variable According to probability distribution Sampling is performed on N candidate example indices. Where the probability vector Belongs to set C, represents the probability of selecting each candidate example, and The constraints defined by C must be satisfied. .because Distributionally independent, the joint probability of all input examples is , where T represents the complete sample example set, is the i-th example, Represents the i-th example is the first one selected from N candidate examples indivual, is the i-th example t iThe probability of being selected. The corresponding loss function is defined as , G represents the generative model, S is the current problem, Represents the composition hint, and y is the true label of the question. Since the black box model cannot obtain gradient information, it is not possible to directly use the gradient Backpropagation update prompt ( is the probability vector The gradient of ), using the variance reduction strategy gradient estimator, optimizes the loss function through forward propagation:

[0041] ;

[0042] in, To represent the loss function The expectation about the example set T. Then The gradient estimation formula is:

[0043] ;

[0044] A collection of examples independently sampled from P(T) , is the gradient estimate, used to update .

[0045] Represents the current sample set The difference between the loss of and the average loss of all example sets is then used to update the example distribution using projected stochastic gradient descent:

[0046] ;

[0047] in, is the learning rate, I is the size of the example set, For projection calculation. For black box models that cannot obtain gradients (such as GPT-3.5), the user's question is used as input and updated through iterative ,The model automatically selects the optimal example set, which consists of reasoning chains and questions, achieving the optimization effect of reasoning chains. The example set can be used as a general style prompt.

[0048] (5) Prompt style changes. The thought chain prompts generated by the large language model in the present invention still have prompt fragility caused by style differences. This is similar to the phenomenon observed in the field of computer vision - subtle changes in image style (such as color or background changes) may affect the prediction accuracy of the model. To this end, the present invention proposes a "novel format" prompt strategy. The characteristic of the strategy is that a different style format is used for each small sample example in the existing thought chain prompt, where the style template contains the initial sample example and the prompt words that rewrite the content of the sample example, for example: only changing the format of non-semantic tags to rewrite the example, reorganizing the visual structure of the example without changing the semantic expression, and optimizing the hierarchy of the example (such as title classification, indentation adjustment) while maintaining the integrity of the information. To further enhance model understanding, the default model nests the prompt content of the ordinary style prompt in the style template to generate novel prompts.

[0049] The automated thought chaining prompts designed using this "novel format" can be applied to reasoning tasks on large language models and compared with existing common prompting techniques. Examples include automated thought chaining prompts, self-consistency, self-improvement, thought growth, and self-checking. This invention will compare the novel prompts with existing prompting techniques. The effectiveness of the prompts will be tested on six different reasoning tasks. The generated answers will then be compared with the actual answers to obtain the accuracy of the large language model's responses.

[0050] To ensure the richness and effectiveness of the experiments, we conducted experiments on six reasoning tasks, including four arithmetic reasoning datasets: GSM8K, ASDiv, SVAMP, and AQuA. We also used a commonsense reasoning dataset: StrategyQA, and a symbolic reasoning dataset: Letter. Table 1 provides detailed statistics for these datasets. The evaluation metric for each task is the accuracy of the model's answer.

[0051] In experiments, this method was compared with four existing prompting techniques: automatic thought chaining, self-consistency, self-improvement, and thought growth. The experiments primarily used GPT-3.5-Turbo, with API calls made through OpenAI's services. The Llama2-70b-chat model was also used to ensure the method's adaptability to the model.

[0052] The experimental results are shown in Table 1. The present invention is significantly superior to other prompting technologies. In the six benchmark datasets, large language models using novel prompts consistently demonstrate excellent performance. On the gpt-3.5-turbo model, the average value of novel prompts is 3.6% and 5.3% higher than self-consistency and automatic thought chain prompts, respectively. Similarly, on the lama2-70b-chat model, the average value of novel prompts is 0.8% and 1.9% higher than self-improvement and thinking growth, respectively. The results of arithmetic reasoning, common sense reasoning, and symbolic reasoning will be discussed separately. It is worth noting that novel prompts improve the answer accuracy on both gpt-3.5 and Llama2-70b, and the improvement on Llama2-70b is even greater, which shows that the method of the present invention is applicable to large language models with different parameter levels.

[0053] Table 1 Performance comparison of novel hints and other hint methods on 6 reasoning tasks

[0054] ;

[0055] Across four arithmetic reasoning tasks, novel prompts demonstrated significant improvement over automatic thought chaining prompts, with an average gain of 5.325%. Compared to a strong baseline approach using self-improvement, answer accuracy still improved by 1.875%. In the Llama2-70b-chat model test, novel prompts also achieved the best average performance. Compared to self-consistency, they achieved an average improvement of 2.625% across the four tasks. Specifically, novel prompts achieved a 6.6% improvement over automatic thought chaining prompts on the ASDIV reasoning task. Because ASDIV relies more on fine-grained semantic representations and displays standard reasoning paths, novel prompts leverage language models to generate appropriate reasoning chains during multi-round reasoning path generation, making them useful as sample prompts.

[0056] In common sense and symbolic reasoning tasks, novel prompts achieved significant improvements over self-consistency and automatic thought chaining prompts. On the gpt-3.5-turbo-1106 model, novel prompts improved STRATEGY by 2.0% compared to thought growth on the common sense reasoning task, and improved LETTER by 1.0% compared to self-improvement on the symbolic reasoning task. When tested using Llama2-70b-chat, novel prompts improved STRATEGY by 2.7% and LETTER by 6.3% compared to automatic thought chaining prompts. These improvements also outperformed other strong baseline methods on both types of reasoning tasks, demonstrating the effectiveness of our method across diverse reasoning tasks.

[0057] Next, we conduct a series of ablation experiments to examine the impact of each component of our proposed method design. First, we explore the contribution of our proposed difficult question selection strategy. Next, we investigate the impact of the answer pool size. Finally, we analyze how the "novel format" prompt strategy enhances the comprehension capabilities of large language models.

[0058] In Table 2, we compare the language model accuracy of random prompts, self-consistency, self-improvement, and novel prompts on three reasoning tasks. We observe that novel prompts significantly outperform random prompts, while random prompts are only slightly better than self-consistency and below self-improvement. Figure 3 As shown in Figure 2, random prompts yielded lower answer accuracy than novel prompts across all six reasoning tasks. This suggests that the performance improvement stems not from an advantage in the annotation process, but rather from the strategy of selecting challenging questions. This also highlights the need for further exploration in automatic prompting engineering in terms of difficulty, diversity, and style.

[0059] Table 2 Accuracy of random prompts, self-consistency, self-improvement, and novel prompts in three reasoning tasks

[0060] ;

[0061] In the process of judging the difficulty of sample questions, this invention generates k answers for each input question to build a prediction pool. It is assumed that the size of k will affect the difficulty judgment and performance of the large language model on downstream tasks. To verify this, a series of experiments were conducted to test different pool sizes. The experiments were conducted jointly on six reasoning tasks. Figure 4 The results plot the accuracy of predicted answers for different numbers of answers (k = 1, 5, 10, and 20). It is clear from the results that when the pool size is only 1, novel prompts perform worse than automatic thought chaining prompts. This suggests that with small pool sizes, high-quality reasoning steps cannot be selected. However, when the pool size is increased to 10 or more, the performance of selection prompts surpasses that of self-improvement methods. This trend indicates that as the pool size increases, more complex and diverse reasoning steps can be selected, leading to continuous improvement in the performance of large language models. The results also converge at k = 20, and further increases in the pool size yield diminishing returns. Intuitively, smaller values ​​of k introduce noise into the reasoning selection process, while larger values ​​of k make the selection of difficult questions more stable. Due to computational limitations, the maximum pool size was limited to 20 in the experiments. This suggests that while larger pool sizes contribute to more robust difficulty judgments, there is a practical upper limit above which performance plateaus.

[0062] During the experiment, the self-improvement method was used as the baseline, in which the large language model would use the automatic thinking chain prompts after the reasoning chain optimization. At this time, the prompts have not yet adopted the "novel format". This prompt is called a normal style prompt. Compare the performance of the large language model when applying normal style prompts and novel prompts. The main focus is on whether the novel prompts can narrow the performance differences of the large language model when the prompt style changes. Difference is a measure of the fragility of the prompt induced by the style. The performance difference mainly refers to the difference between the highest accuracy prompt and the lowest accuracy prompt. In Table 3, the results of the best and worst prompts among the normal style prompts and novel prompts are reported. It is observed that the novel prompts not only reduce the difference, but also improve the minimum and maximum accuracy. As Figure 5 As shown in Figure 5, novel cues outperform common style cues in most datasets, while common style cues outperform in AQUA and LETTER.

[0063] Table 3 Results of the best and worst prompts for common style prompts and novel prompts

[0064] ;

[0065] In the experiment, we use the training data (denoted as ) and test data (denoted as The number of difficult problems selected for different reasoning tasks is as follows: 6 difficult problems each for GSM8K, ASDiv, SVAMP, and StrategyQA, and 4 difficult problems each for AQuA and Last Letter. During the reasoning phase, the temperature parameter was set to 0.68, 10 inference runs were performed for each problem, and the maximum number of input-output tokens was set to 256. The default version of GPT-3.5-turbo used was gpt-3.5-turbo-1106. To ensure the generalizability of the method, Llama2-70b-chat was also tested.

[0066] In the present invention, manually labeled question samples are first used to ensure the reliability of the results of the difficulty judgment stage. These question samples usually come from the source data set itself. Even in the absence of manually labeled question samples, the experiment can continue. During the experiment, the size of the candidate question pool is limited to 500 - if the original training data set exceeds 500, 500 questions are randomly selected; if it is less than 500, all data are used. After testing and verification of data pools of different sizes, 500 questions can achieve a better balance in performance. Generally speaking, a larger data pool will bring higher performance gains. When the number of answer generation k=15, the model performance tends to converge. The main experimental report is based on the entropy-based difficulty judgment method.

[0067] In the sample set E, 200 question samples are selected as the training set and the validation set respectively to balance performance and cost. The cross-entropy loss between the answers generated by the API and the true answers is calculated, and the AdamW optimizer is used to perform multiple rounds of iterative optimization on the latent variables. The learning rate is set to , with a batch size of 5. After the inference chain optimization is completed, the common style hints have been stored in the default file. It is also necessary to reduce the performance fluctuations (i.e., discreteness) caused by changes in the prompt format style. The numerical range of discreteness is between 0.0 and 1.0. The closer to 0.0, the more robust the model is and the less sensitive it is to style changes; conversely, the closer to 1.0, the more sensitive the model is to prompt style changes. We use the designed "novel format" hint template and require the large language model to rewrite each common style hint as the final novel hint. By default, when asking the large language model an inference question, the large language model will query for novel hints that are similar to the question content, and then use the novel hints to combine with the question and pass it to the large language model. The answer generated by it is then compared with the true answer to calculate the accuracy of the large language model's answer.

Claims

1. A method for automatically generating thought chain prompts based on black box optimization and vulnerability quantification, characterized in that: The following steps are involved: (1) Obtain unlabeled training dataset; Use the difficulty judgment module to filter out a subset of difficult questions and generate a labeled dataset; The operation of the difficulty judgment module includes: generating the predicted probability distribution of k answers for the large language model Calculate entropy: ； in, Representing a parameterized model In the given problem Generate a specific answer when The predicted probability of Represents the parameters of the language model; adjustable hyperparameters and Used to control the probability scaling ratio; select the top n questions with the largest entropy values ​​as the high-difficulty questions; (2) Use a large language model to generate multiple reasoning chains for each question in the annotated dataset, and retain the reasoning chains that are consistent with the correct answer to form an example library; (3) Using the variance reduction policy gradient estimator to optimize the inference chain selection strategy and generate common style hints; the optimization process of the variance reduction policy gradient estimator includes: defining the latent variable Obey categorical distribution ;in, The probability vector of N candidate examples is updated by the gradient estimation formula : ； Where T represents the complete sample set, is the i-th example, is the i-th example The probability of being selected; A collection of independently sampled examples ; is the gradient estimate, used to update ; is the probability vector gradient; Represents the current sample set The difference between the loss of and the average loss of all example sets; Update the example distribution using projected stochastic gradient descent: ； in, is the learning rate, I is the size of the example set, Calculate for projection; (4) Through the novel format prompt module, the common style prompts are diversified and the automatic thought chain prompts with anti-fragility are generated.

2. The method for generating automatic thought chain prompts based on black box optimization and vulnerability quantification according to claim 1 is characterized in that: When the number of high-difficulty questions exceeds G, m questions are randomly selected; the construction of the labeled dataset is achieved through automatic labeling using a large language model; where G is the set threshold for the number of high-difficulty questions.

3. The method for generating automatic thought chain prompts based on black box optimization and vulnerability quantification according to claim 1 is characterized in that: The generation and pruning of reasoning chains include: generating l reasoning chains for each question; comparing the generated answers with the labeled answers, retaining only the correct reasoning chains, and pairing the questions with the correct reasoning chains to form an initial example set.

4. The method for generating automatic thought chain prompts based on black box optimization and vulnerability quantification according to claim 1 is characterized in that: During optimization: loss function Based on the cross entropy calculation between the black box model output and the true label; the number of context examples is dynamically adjusted to 3-6.

5. The method for generating automatic thought chain prompts based on black box optimization and vulnerability quantification according to claim 1 is characterized in that: The operation of the novel format hint module includes: designing a unique style template for each few-sample example. The template types include: non-semantic markup format rewriting, visual structure adjustment, hierarchical optimization reconstruction, and nesting common style hints into the style template to generate the final hint.

6. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the program.

7. A computer-readable storage medium, characterized in that A computer program is stored, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images