A prompt word optimization method and related apparatus
By using an automated prompt word optimization method, the prompt word rules of a large language model are optimized using error sample data. This solves the problem of low efficiency in manual adjustment and achieves efficient and reliable prompt word optimization, which is suitable for complex classification tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CTRIP COMP TECH SHANGHAI
- Filing Date
- 2026-06-02
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, classification tasks based on large language models require manual rewriting and adjustment of prompt words, which is inefficient and cannot meet the accuracy requirements of complex classification needs.
By automatically acquiring initial prompt words and the original dataset, calling the business big model to perform category determination, constructing an error sample set and generating learning samples, and optimizing the main body of the prompt word rules until the performance is better than the initial prompt words, the automatic optimization of prompt words is achieved.
It significantly reduces labor and time costs, improves the efficiency of prompt word optimization, focuses on error patterns, and ensures the reliability and accuracy of the optimization process.
Smart Images
Figure CN122489758A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for optimizing prompt words. Background Technology
[0002] The widespread use of online service platforms has spurred a surge in demand for automated categorization. For example, in review appeal tasks, it's necessary to determine whether a user review is factual to support a decision to retain or reject it. In these categorization scenarios, accurate classification results are fundamental to triggering differentiated follow-up processing. However, the objects to be categorized (such as user reviews, conversational texts, or other materials reflecting interaction events) are often diverse in expression and semantically complex, making traditional keyword-based or simple rule-based categorization methods insufficient to meet accuracy requirements.
[0003] With the rise of Large Language Models (LLMs), leveraging their powerful semantic understanding capabilities to perform the aforementioned classification tasks has gradually become the mainstream approach. However, the output quality of large models is highly dependent on the quality of prompts, which means that to meet complex classification requirements, experts often need to repeatedly write and adjust prompts based on their experience, resulting in low efficiency. Summary of the Invention
[0004] In view of the above problems, this application provides a cue word optimization method and related apparatus to optimize the cue word performance for complex classification tasks.
[0005] The specific plan is as follows:
[0006] The first aspect of this application provides a method for optimizing prompt words, including:
[0007] Obtain the initial prompt words applied to the target classification task, and the original dataset corresponding to the target classification service; for any sample data in the original dataset, the data includes: user input content related to the target classification service, and the corresponding real category label;
[0008] Invoke the business big model to instruct the business big model to classify the input content in the original dataset based on the initial prompt words;
[0009] Compare the category determination result of the business model with the corresponding real category label;
[0010] An original evaluation set is constructed based on the comparison results, and the original evaluation set includes at least the set of error samples.
[0011] Construct learning samples and optimize the rule body in the initial prompt words based on the learning samples; the learning samples are constructed based at least on the error sample data in the error sample set;
[0012] If the performance of the optimized prompt word is better than that of the initial prompt word, the optimized prompt word shall be used as the final optimal prompt word; wherein the optimized prompt word is determined based on the optimized rule body.
[0013] A second aspect of this application provides a prompt word optimization device, comprising:
[0014] An initialization unit is used to obtain initial prompt words for the target classification task and the original dataset corresponding to the target classification service; for any sample data in the original dataset, the data includes: user input content related to the target classification service and the corresponding real category label;
[0015] An initial processing unit is used to call a large business model to instruct the large business model to perform category determination on the input content in the original dataset based on the initial prompt words, compare the category determination result of the large business model with the corresponding true category label, and construct an original evaluation set based on the comparison result. The original evaluation set includes at least an error sample set.
[0016] An optimization unit is configured to construct learning samples and optimize the rule body in the initial prompt words based on the learning samples; the learning samples are constructed based at least on the error sample data in the error sample set;
[0017] The final processing unit is used to select the optimized prompt word as the final optimal prompt word if the performance of the optimized prompt word is better than that of the initial prompt word; wherein the optimized prompt word is determined based on the optimized rule body.
[0018] A third aspect of this application provides a prompt word optimization device, comprising at least one processor and a memory connected to the processor, wherein:
[0019] The memory is used to store computer programs;
[0020] The processor is used to execute the computer program to implement the prompt word optimization method described in the first aspect above.
[0021] The fourth aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the prompt word optimization method described in the first aspect.
[0022] By employing the aforementioned technical solution, the prompt word optimization scheme provided in this application automatically acquires initial prompt words and the original dataset, calls the business big data model for initial category determination, automatically compares the determination results with the true labels, constructs an error sample set, and generates learning samples based on this to optimize the main body of the prompt word rules. When the optimized prompt word performs better, the optimal prompt word is determined, ultimately achieving automatic prompt word optimization. The entire process eliminates the need for repeated manual debugging of prompt words, significantly reducing labor and time costs. This application uses sample data with incorrect category determinations in the business big data model (i.e., error sample data) as learning samples when optimizing the main body of the rules. This allows the prompt word optimization process to focus on sample data or error patterns that are difficult to classify correctly for the current prompt word, guiding the main body of the prompt word rules towards error correction, thus improving the efficiency of prompt word optimization. This application applies a best-choice adoption mechanism in the prompt word optimization process, ensuring the reliability of the optimization process. Attached Figure Description
[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0024] Figure 1 A schematic diagram of an implementation system architecture for the prompt word optimization method provided in this application embodiment;
[0025] Figure 2 A flowchart illustrating a prompt word optimization method provided in an embodiment of this application;
[0026] Figure 3 This is a schematic diagram of the structure of a prompt word optimization device provided in an embodiment of this application;
[0027] Figure 4 This is a schematic diagram of the structure of a prompt word optimization device provided in an embodiment of this application. Detailed Implementation
[0028] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0029] This application provides a method and related apparatus for optimizing prompt words, which can be applied to solving the prompt word optimization task for complex classification tasks.
[0030] First, let's explain the prompt word optimization scenarios described in this application.
[0031] In business scenarios where large models are used to implement complex classification tasks such as automatic review and appeal judgment, it is necessary to optimize the prompt words when calling the large model to improve the accuracy of category determination. The prompt word optimization solution provided in this application can be applied to the business preparation stage to pre-configure high-quality prompt words before business implementation, or it can be applied during the business process to realize online prompt word optimization tasks, such as automatically optimizing the actual prompt words entered by the user.
[0032] To distinguish between the two, this application refers to the large model used to perform the classification task as the "business large model." The objects that the business large model needs to process in a single classification task, i.e., the objects to be classified, can include: business input content and auxiliary input content. Taking a review appeal scenario as an example, the business data content can be the review content of the first user (e.g., a customer), and the auxiliary input content can be the appeal content of the second user (e.g., a merchant).
[0033] The complete prompt word described in this application is obtained by concatenating three parts: the rule body, the input format, and the output format.
[0034] The output format specifies that the large business model must output category determination results (also known as classification results) in a structured JSON format. For example, this JSON structure may include the following parameters: category name, the highest priority category (first_category), confidence score (ranging from 0.00 to 1.00), reasoning, and supporting input content. Taking a review appeal scenario as an example, supporting input content may include supporting review snippets and supporting appeal snippets. It should be noted that this application optimizes only the main body of the rules when optimizing prompt words, ensuring that the large business model always outputs category determination results in a unified JSON structure.
[0035] Based on the above, this application provides a prompt word optimization method that can be applied to, for example... Figure 1 The system architecture shown may include a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).
[0036] Either terminal 100 or server 200 can be used independently to execute the prompt word optimization method provided in this application embodiment. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the prompt word optimization method provided in this application embodiment. Terminal 100 in this application embodiment can be a mobile phone, computer, etc., and this application embodiment does not impose any limitations on this.
[0037] This application provides a prompt word optimization method, which is illustrated by applying the method to a computer device. Specifically, the computer device may be... Figure 1 The system consists of terminal 100 or a combination of terminal 100 and server 200. (Refer to...) Figure 2 The specific steps for optimizing prompt words include:
[0038] Step S101: Obtain the initial prompt words applied to the target classification task, and the original dataset corresponding to the target classification business.
[0039] The target classification task can be any of the complex classification tasks such as review and appeal judgment, customer service ticket classification, or artificial intelligence (AI) incident judgment. The initial prompt words can be prompt words actually entered by the user in the business model; specifically, they can be prompt words used in historical business processes or in the current business process. Alternatively, they can be pre-configured prompt word templates.
[0040] For any sample data in the original dataset, the data includes: user input content related to the target classification business, and the corresponding real category label.
[0041] The aforementioned user input content can be actual content obtained from an online service platform or simulated content generated according to preset rules. The true category label corresponding to the user input content is the category determination result expected to be output by the large business model. For example, in the review appeal determination task, the true category labels contained in the original dataset may include: category A1 corresponding to reviews that do not match reality, and category A2 corresponding to reviews that match reality; in the customer service ticket classification task, the true category labels contained in the original dataset may include: category B1 corresponding to refund applications, category B2 corresponding to complaints and suggestions, category B3 corresponding to inquiries, and category B4 corresponding to technical issues, etc.; in the AI event liability determination task, the true category labels contained in the original dataset may include: category C1 corresponding to platform liability, category C2 corresponding to merchant liability, category C3 corresponding to customer liability, and category C4 corresponding to third-party liability, etc.
[0042] Step S102: Invoke the business big model to instruct the business big model to classify the input content in the original dataset based on the initial prompt words.
[0043] Specifically, when calling the business big model to classify an input, the input placeholder in the initial prompt or the historical input can be replaced with the current input to construct a complete LLM request. This request is then used to call the business big model and obtain the model response, i.e., the classification result.
[0044] Because the prompts described in this application contain a fixed output format, the output text of the large business model can be automatically parsed. In one possible implementation, a predictive parsing function (parse_prediction) can be configured based on regular expressions to achieve the output parsing task. Specifically, this function uses regular expressions to extract JSON blocks from the text returned by the large model, and then parses its categories[0].name field to obtain the category name, which serves as the category determination result for the large business model. In particular, if JSON parsing fails, it can degenerate into using keyword matching to extract the category determination result.
[0045] Step S103: Compare the category determination result of the business big model with the corresponding real category label, and construct the original evaluation set based on the comparison result.
[0046] In one possible implementation, a class determination function (check_binary_classification) can be pre-configured to compare the class determination results of the large business model with the true class labels in order to automatically determine the correctness of the classification of each sample data.
[0047] The original evaluation set includes at least an error case dataset, which consists of sample data from errors identified by the large business model. Optionally, it can be obtained by reading a structured tabular data CSV file using the error case loading function (load_error_cases). This function employs a multi-encoding automatic detection mechanism; specifically, it automatically attempts to parse multiple character encodings until successful parsing, and maps each row of data to a dictionary object containing input content and category labels, which are then compiled into a list and returned.
[0048] Step S104: Construct learning samples and optimize the rule body in the initial prompt words based on the learning samples.
[0049] The learning samples are constructed based at least on the error sample data in the error sample set. That is, the error sample data in the error sample set can be used to provide learning material for the optimization process, allowing for targeted improvement of prompt words based on the error sample data. Optionally, for any error sample data in the error sample set, the data may also include historical category determination results, used to record the category determination results of the business big model for the corresponding input content.
[0050] For example, before optimizing the rule body, a pre-configured extraction function (extract_rule_body) can be called to extract the rule body from the complete prompt words for optimization and modification.
[0051] In another possible implementation, the input and output format content in the prompt words can be explicitly prohibited from being modified in the optimization instructions, so as to optimize only the main body of the rule, thereby ensuring the invariance and stability of the input and output format throughout the optimization process.
[0052] Step S105: If the performance of the optimized prompt word is better than that of the initial prompt word, the optimized prompt word shall be taken as the final optimal prompt word.
[0053] The optimized prompt word is determined based on the optimized rule body. For example, a pre-configured assembly function (assemble_full_prompt) can be called to concatenate the optimized rule body with the original input and output formats to obtain the complete optimized prompt word.
[0054] The aforementioned superior performance can be expressed as higher accuracy and / or lower error rate. Accuracy and error rate can be obtained by testing the judgment effect of prompt words on the corresponding dataset.
[0055] This application automatically acquires initial prompt words and the original dataset, calls the business model for initial category determination, automatically compares the determination results with the true labels, constructs an error sample set, and generates learning samples to optimize the main body of the prompt word rules. When the optimized prompt word performs better, the optimal prompt word is determined, ultimately achieving automatic prompt word optimization. The entire process eliminates the need for repeated manual debugging of prompt words, significantly reducing labor and time costs. This application uses sample data of incorrect category determinations from the business model (i.e., error sample data) as learning samples for optimizing the main body of the rules. This allows the prompt word optimization process to focus on sample data or error patterns that are difficult to classify correctly, guiding the main body of the prompt word rules towards error correction, thus improving the efficiency of prompt word optimization. This application employs a merit-based adoption mechanism during the prompt word optimization process, ensuring the reliability of the optimization process. Furthermore, this application can be widely applied to classification business scenarios such as hotel review appeals, possessing good versatility.
[0056] In one or more embodiments provided in this application, constructing learning samples and optimizing the rule body in the initial prompt words based on the learning samples may include:
[0057] Perform M iterations of the optimization process.
[0058] Where M is the preset maximum number of iterations. Specifically, each iteration optimization process may include the following steps:
[0059] Step S201: Determine the current optimal prompt word as the prompt word to be optimized.
[0060] The optimal prompt word at the beginning is the initial prompt word.
[0061] Step S202: Determine the learning samples for this iteration from the erroneous sample data generated before this iteration optimization.
[0062] The error sample data generated before the first iteration of optimization is the sample data in the error sample set. For any iteration of optimization, the learning samples used can be sample data from the error sample set corresponding to the initial prompt word, or error sample data generated in previous iterations, such as error sample data generated when evaluating the performance of previous candidate prompt words.
[0063] For example, the batch size can be preset. When constructing or determining the learning samples to be used for each iteration of optimization, batch_size of error samples can be randomly selected from the corresponding error sample dataset. By randomly selecting, the prompt word optimization process can learn different error cases.
[0064] Step S203: Invoke the prompt word optimization model to instruct the prompt word optimization model to optimize the rule body in the prompt word to be optimized based on the current learning sample.
[0065] The aforementioned prompt word optimization model can be the same as or different from the business model described above; this application does not impose any limitations. In this embodiment, error sample data is provided as exemplary feedback material to the prompt word optimization model, enabling the model to rewrite the main rules for prompt words accordingly. Optionally, before inputting the current learning samples into the prompt word optimization model, the learning samples can be formatted to standardize the input.
[0066] Step S204: Optimize the output content of the large model based on the prompt words to determine the candidate prompt words for this time.
[0067] Specifically, the optimized rule body output by the model can be concatenated with the original input and output format content to obtain the complete prompt word, which can be used as the candidate prompt word for this time.
[0068] Step S205: Determine whether the performance of the current candidate suggestion word is better than the suggestion word to be optimized. If so, proceed to step S206.
[0069] Step S206: Update the current optimal suggestion word to the candidate suggestion word for this time.
[0070] If the performance of the candidate suggestion word is not better than that of the suggestion word to be optimized (i.e., the current best suggestion word), it can be indicated that the optimization purpose has not been achieved in this iteration. In this case, the current best suggestion word will not be updated.
[0071] This application achieves the prompt word optimization task through a finite number of iterations, which improves the optimization effect compared to a single-step optimization scheme. Since the learning samples used in each iteration are previously generated error sample data, subsequent iterations can learn from initial errors and emerging errors generated during the iteration process, providing a foundation for improving the comprehensiveness of prompt word optimization. Furthermore, because each iteration's candidate prompt words undergo performance evaluation, the current optimal prompt word is updated only when the performance of a candidate prompt word is better. This avoids prompt word performance degradation caused by fluctuations in the quality generated by the large model or bias in the learning samples, ensuring the reliability of the optimization process.
[0072] In one or more embodiments provided in this application, when the M-step optimization process is divided into K optimization cycles, determining the current learning sample from the error sample data generated before the current iteration optimization may include:
[0073] Step S301: If this iteration optimization is the first iteration optimization in the current optimization cycle, then sample the sample data in the original evaluation set to generate the evaluation dataset for the current optimization cycle, and extract sample data from the error sample subset of the current optimization cycle as the learning samples for this iteration.
[0074] The evaluation dataset includes at least a subset of error samples. Optionally, the evaluation batch size (eval_batch_size) can be preset to control the amount of data used to evaluate the performance of candidate prompts in each optimization cycle, or the learning range for each optimization cycle. Typically, the evaluation batch size is smaller than the original dataset (or original evaluation set). In this case, the original evaluation set can be stratified according to the true label category to ensure that the evaluation dataset includes sample data from all categories. Specifically, the data in the combination is grouped according to the true category label carried by the sample data, and the proportion of sample data in each category is calculated. While ensuring that at least one sample is extracted from each category, the corresponding number or proportion of sample data is extracted sequentially from the sample data of each category in ascending order of the number of data in each category. Specifically, if insufficient sample data is extracted due to rounding operations, random sampling can be performed from the remaining sample data until the preset evaluation batch size is reached.
[0075] In one possible implementation, the original evaluation set may also include a set of correct samples, also known as a regression dataset, which consists of sample data correctly determined by the large business model. Optionally, it can be obtained by reading a CSV file using the dataset loading function (load_regression_dataset). In this case, stratified sampling can be performed on the error sample set and the correct sample set separately to construct the evaluation dataset, which includes subsets of error samples and subsets of correct samples.
[0076] In addition, in some other embodiments, a regression validation dataset can be used to detect whether the optimized prompt words introduce regression errors, that is, whether the sample data that is correctly classified is misclassified after the prompt words are optimized.
[0077] The evaluation dataset configured based on the aforementioned steps has the same data structure as the original dataset, but the set size is smaller, which reduces the computational overhead of evaluation in the iterative optimization process. In addition, through the stratified sampling mechanism, the distribution ratio of each category can be kept consistent with the original dataset, thereby preserving the distribution characteristics of the original evaluation set.
[0078] In another possible implementation, if this iteration is the first iteration within the current optimization cycle, the sample data in the original evaluation set can be sampled to generate the evaluation dataset for the current optimization cycle. The business big model is then invoked to instruct it to classify the input content in the evaluation dataset for the current optimization cycle based on the current optimal prompt word. The classification result of the business big model is compared with the corresponding true category label to determine the erroneous sample data corresponding to the current optimal prompt word. Sample data is then extracted from the erroneous sample data corresponding to the current optimal prompt word as the learning samples for this iteration.
[0079] Step S302: If this iteration is not the first iteration in the current optimization cycle, then extract sample data from the error sample data generated in the previous iteration and use it as the learning sample for this iteration.
[0080] Among them, the erroneous sample data generated in any iteration optimization includes: sample data that was incorrectly identified by the business big model during the performance evaluation of the candidate prompt words in that iteration optimization.
[0081] Based on this, for any iteration of optimization, the learning samples are all sample data in the evaluation dataset of the corresponding optimization period, but they may be sample data that was misjudged by the initial prompt word, or they may be sample data that was misjudged by the previous candidate prompt word.
[0082] Based on the above, compared to a fixed evaluation set, this embodiment, by periodically rotating the evaluation dataset, can avoid overfitting to a specific set to a certain extent, and can also ensure fairness when comparing the performance of different candidate suggestion words to a certain degree. Within an optimization cycle, the first iteration uses newly sampled error sample data from the error sample set to optimize the rule body of the suggestion words, and subsequent iterations use error sample data generated from previous iterations to optimize the rule body of the suggestion words, enabling the current optimization cycle to optimize suggestion words for representative errors and marginal errors under the current data distribution. Based on the spiral iterative optimization mechanism in this embodiment, while ensuring the optimization depth of each cycle, diverse optimization tasks are also achieved.
[0083] In one or more embodiments provided in this application, when the M-times iteration optimization process is divided into K optimization cycles, the process may further include the following after the iteration optimization process of the current optimization cycle is completed and before the iteration optimization process of the next optimization cycle is executed:
[0084] Step S401: Extract at least one prompt word from the best prompt words of the previous optimization cycle and all previous optimization cycles.
[0085] Optionally, after the iterative optimization process of each optimization cycle is completed, the best suggestion word of the current subset cycle can be recorded to construct a set of historical best suggestion words. Optionally, the feature values corresponding to each historical best suggestion word can also be recorded to characterize the performance of the suggestion words.
[0086] Step S402: Take the extracted prompt words and the current optimal prompt words as objects to be merged, and perform merging processing on the rule subjects in the objects to be merged.
[0087] Optionally, the aforementioned merging process can be a direct concatenation. In another possible implementation, the aforementioned merging process may include: filling the prompt words (or rule bodies) and corresponding feature values from the objects to be merged into a pre-configured merging instruction template, constructing a merging request, and invoking the prompt words to optimize the large model to instruct the large model to merge the rule bodies according to the merging request. For example, the aforementioned merging instruction template is used to guide the large model to merge based on at least one of the following rules, which include extracting unique rules, comparing similar rules, resolving conflicts, and maintaining priority order.
[0088] Step S403: Determine the merge prompt words based on the merged rule body.
[0089] Step S404: Determine whether the preset conditions for merging prompt words are met. If so, proceed to step S405.
[0090] Step S405: Update the current optimal prompt word to the merged prompt word.
[0091] The conditions for merging suggestion words may include: the performance of the merged suggestion word has not deteriorated or the deterioration is less than a preset limit relative to the current best suggestion word. If the conditions for merging suggestion words are not met, it indicates that the current suggestion word merging operation has introduced a large merging error; in this case, the current best suggestion word is not updated. Optionally, the evaluation dataset used when evaluating the merged suggestion word may be the same dataset used when evaluating the current best suggestion word to ensure fairness in performance comparison.
[0092] This embodiment improves the initial performance of prompts during subsequent iterations by merging the best prompts from multiple optimization cycles, providing a foundation for enhancing the overall performance of the optimized prompts. This allows the application to achieve better performance improvements with fewer iterations. Furthermore, by evaluating the performance of the merged prompts and updating the current best prompt only when the merged prompts perform better, the risk of merging can be avoided, ensuring the robustness and reliability of the optimization process.
[0093] In one or more embodiments provided in this application, extracting at least one prompt word from the optimal prompt words of the previous optimization cycle and all previous optimization cycles may include:
[0094] Extract the optimal prompt word from the optimal prompt words of the previous optimization cycle and all previous optimization cycles. Extract the optimal prompt word from at least one optimization cycle that is closest to the current optimization cycle.
[0095] For example, one or two historically optimal suggestion words can be obtained.
[0096] Based on the above, this embodiment does not simply accumulate all historical best suggestion words, but extracts the best suggestion words from the nearest historical period as the objects to be merged with the current best suggestion word. Therefore, as the iteration process progresses, recent optimization results have higher weights in the optimized suggestion words. Although the weights of older optimization results are gradually reduced through chain propagation, older optimization results can still be retained through previous merging operations. Thus, even if the rules from the early stages of the iteration are outdated or conflict with new data distributions, they will not cause significant interference to the current suggestion words. In summary, this embodiment integrates past experience and eliminates interference from older periods, further improving the efficiency and quality of suggestion word optimization.
[0097] In one or more embodiments provided in this application, for any prompt word involved in the prompt word optimization process, the performance evaluation process of the current prompt word may include:
[0098] Step S501: Determine the evaluation dataset corresponding to the current prompt word.
[0099] The evaluation dataset is either the original evaluation set or a subset thereof, and the original evaluation set further includes a set of correct samples. For example, the evaluation dataset corresponding to the initial prompt word can be the original evaluation set; the evaluation dataset corresponding to other prompt words can be a subset of the original evaluation set, such as the evaluation dataset corresponding to the optimization period.
[0100] Step S502: Invoke the business big model to instruct the business big model to classify the input content in the evaluation dataset based on the current prompt word.
[0101] Step S503: Compare the category determination result of the business model with the corresponding real category label to determine the correct sample data and incorrect sample data corresponding to the current prompt word.
[0102] Step S504: Calculate the feature value corresponding to the current prompt word based on the quantitative characteristics of the correct sample data and incorrect sample data corresponding to the current prompt word.
[0103] The feature values are used to characterize the performance of the prompt words.
[0104] In one possible implementation, data features may include precision, recall, and F1 score for each class. Precision can be expressed as Precision = TP / (TP + FP), which is the proportion of samples that are correctly identified as belonging to that class; Recall = TP / (TP + FN), which is the proportion of samples that are correctly identified as belonging to that class; F1 score = 2 × Precision × Recall / (Precision + Recall), which is the harmonic mean of precision and recall.
[0105] Based on this, the feature value corresponding to the current prompt word can include the overall F1 score, which is the weighted average of the macro average F1 score of the correct sample data and the macro average F1 score of the incorrect sample data, with the corresponding weights being the proportion of the sample data. It can be expressed as: Overall F1 score = (Macro average F1 score of the correct sample dataset × Number of correct samples + Macro average F1 score of the incorrect sample dataset × Number of incorrect samples) / (Number of correct samples + Number of incorrect samples). The macro average F1 score of any set can be expressed as: Macro_F1 = 2 × Macro_Precision × Macro_Recall / (Macro_Precision + Macro_Recall), where Macro_Precision represents the macro average precision of the set, i.e., the arithmetic mean of the precision of each class, and Macro_Recall represents the macro average recall of the set, i.e., the arithmetic mean of the recall of each class.
[0106] This embodiment quantifies the performance of prompt words using objective feature values. The entire evaluation process requires no human intervention, improving the objectivity of performance evaluation and avoiding human error. Furthermore, because the evaluation dataset can be flexibly configured, using the full dataset can improve evaluation accuracy, while using component datasets, especially those corresponding to the training samples used, can improve evaluation efficiency and relevance, and help reduce evaluation overhead.
[0107] In one or more embodiments provided in this application, the process of performing M iterations of optimization may further include:
[0108] Step S601: If the preset prompt word simplification conditions are met, the rule body in the current optimal prompt word is simplified, and the simplified prompt word is determined based on the simplified rule body.
[0109] The simplification condition for the prompt words can be that the current iteration count has reached a preset number. For example, a simplification period of `simplify_every` can be preset, and the simplification condition is determined to be met when the current iteration count is an integer multiple of `simplify_every`. Optionally, this step can be performed before the current optimization or after obtaining the optimal prompt word corresponding to the current iteration optimization.
[0110] Optionally, the aforementioned simplification process may include: extracting the rule body from the current optimal prompt word and recording its original length; configuring a simplification request based on the extracted rule body and a pre-configured simplification instruction template; and invoking the prompt word optimization model to instruct the prompt word optimization model to simplify the rule body according to the simplification request. For example, the simplification instruction template may include at least one of the following rules: merging duplicate content, simplifying expression, retaining core logic, optimizing structure, and controlling the length to a preset proportion (e.g., 60%~80%).
[0111] Step S602: Determine whether the preset simplified prompt word application conditions are met. If so, proceed to step S603.
[0112] Step S603: Update the current optimal prompt word to the simplified prompt word.
[0113] The simplified prompt application conditions may include: the performance of the simplified prompt has not deteriorated or the deterioration is less than a preset limit compared to the current optimal prompt. If the simplified prompt application conditions are not met, it indicates that the current prompt simplification operation has introduced a large simplification error; in this case, the current optimal prompt will not be updated.
[0114] Optionally, the evaluation dataset used when evaluating the performance of simplified prompt words can be the same dataset used when evaluating the current best prompt word, in order to ensure the fairness of the prompt word performance comparison.
[0115] This embodiment can effectively control the length of prompt words by automatically simplifying the current optimal prompt word, thereby reducing the risk of context collapse. In addition, by simplifying the application judgment steps based on the performance of the simplified prompt word, the risk of simplification can be avoided, thereby improving the safety and reliability of the simplified operation.
[0116] In summary, this application provides a scheme for automatically optimizing prompt words based on spiral iterative optimization. This scheme is applicable to complex classification scenarios and can, to a certain extent, avoid overfitting and optimization regression, thus ensuring the efficiency and effectiveness of prompt word optimization.
[0117] The prompt word optimization device provided in the embodiments of this application is described below. The prompt word optimization device described below can be referred to in correspondence with the prompt word optimization method described above.
[0118] See Figure 3 , Figure 3 This is a schematic diagram of a prompt word optimization device disclosed in an embodiment of this application.
[0119] like Figure 3 As shown, the device may include:
[0120] Initialization unit 11 is used to obtain initial prompt words applied to the target classification task and the original dataset corresponding to the target classification service; for any sample data in the original dataset, the data includes: user input content related to the target classification service and the corresponding real category label;
[0121] The initial processing unit 12 is used to call the business big model to instruct the business big model to perform category determination on the input content in the original dataset according to the initial prompt words, compare the category determination result of the business big model with the corresponding real category label, and construct an original evaluation set based on the comparison result. The original evaluation set includes at least an error sample set.
[0122] Optimization unit 13 is used to construct learning samples and optimize the rule body in the initial prompt words based on the learning samples; the learning samples are constructed based at least on the error sample data in the error sample set;
[0123] The final processing unit 14 is used to select the optimized prompt word as the final optimal prompt word if the performance of the optimized prompt word is better than that of the initial prompt word; wherein the optimized prompt word is determined based on the optimized rule body.
[0124] Each unit in the prompt word optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in the processor of a computer device in hardware form or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each unit.
[0125] The prompt word optimization device provided in this application embodiment can be applied to prompt word optimization devices, such as terminals with data processing capabilities: mobile phones, computers, servers, etc. Optionally, Figure 4 The hardware structure block diagram of the prompt word optimization device is shown, referencing... Figure 4 The hardware structure of the prompt word optimization device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;
[0126] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0127] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0128] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0129] The memory stores computer programs, and the processor executes the computer programs to enable the prompt word optimization device to implement any of the prompt word optimization methods described above.
[0130] This application also provides a storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the prompt word optimization methods provided in this application.
[0131] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement any of the prompt word optimization methods provided in this application.
[0132] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0133] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.
[0134] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method of prompt word optimization, the method comprising: The method includes: Obtain the initial prompt words applied to the target classification task, and the original dataset corresponding to the target classification service; for any sample data in the original dataset, the data includes: user input content related to the target classification service, and the corresponding real category label; Invoke the business big model to instruct the business big model to classify the input content in the original dataset based on the initial prompt words; Compare the category determination result of the business model with the corresponding real category label; An original evaluation set is constructed based on the comparison results, and the original evaluation set includes at least the set of error samples. Construct learning samples and optimize the rule body in the initial prompt words based on the learning samples; the learning samples are constructed based at least on the error sample data in the error sample set; If the performance of the optimized prompt word is better than that of the initial prompt word, the optimized prompt word shall be used as the final optimal prompt word; wherein the optimized prompt word is determined based on the optimized rule body.
2. The prompt word optimization method of claim 1, wherein, The process of constructing learning samples and optimizing the rule body in the initial prompt words based on the learning samples includes: The optimization process is executed M times, where M is a preset maximum number of iterations; each iteration includes: The current optimal suggestion word is determined as the suggestion word to be optimized; wherein, the initial optimal suggestion word is the initial suggestion word; The learning samples for this iteration are determined from the error sample data generated before the current iteration optimization; wherein, the error sample data generated before the first iteration optimization is the sample data in the error sample set; Invoke the prompt word optimization model to instruct it to optimize the main rules in the prompt words to be optimized based on the current learning samples; Based on the aforementioned prompts, optimize the output of the large model to determine the candidate prompts for this iteration; If the performance of the current candidate suggestion word is better than the suggestion word to be optimized, then the current best suggestion word is updated to the current candidate suggestion word; otherwise, the current best suggestion word is not updated.
3. The prompt word optimization method of claim 2, wherein, When the M iterations of optimization are divided into K optimization cycles, the learning samples for this iteration are determined from the erroneous sample data generated before this iteration, including: If this iteration is the first iteration in the current optimization cycle, the sample data in the original evaluation set is sampled to generate the evaluation dataset for the current optimization cycle. The evaluation dataset includes at least a subset of error samples. Sample data is extracted from the subset of error samples in the current optimization cycle as the learning samples for this iteration. If this iteration is not the first iteration within the current optimization cycle, then sample data is extracted from the erroneous sample data generated in the previous iteration and used as the learning sample for this iteration. Among them, the erroneous sample data generated in any iteration optimization includes: sample data that was incorrectly identified by the business big model during the performance evaluation of the candidate prompt words in that iteration optimization.
4. The prompt word optimization method of claim 3, wherein, When the M-step optimization process is divided into K optimization cycles, the process further includes the following steps after the current optimization cycle's iterative optimization process is completed and before the next optimization cycle's iterative optimization process is executed: Extract at least one prompt word from the best prompt words of the previous optimization cycle and all previous optimization cycles; The extracted prompt words and the current optimal prompt words are used as objects to be merged, and the rule subjects in the objects to be merged are merged. The merge prompt words are determined based on the merged rule body; If the preset conditions for merging suggestion words are met, the current optimal suggestion word will be updated to the merged suggestion word; otherwise, the current optimal suggestion word will not be updated. The conditions for merging suggestion words include: the performance of the merged suggestion word has not deteriorated or the degree of deterioration is less than a preset limit relative to the current optimal suggestion word.
5. The prompt word optimization method of claim 4, wherein, Extract at least one prompt word from the optimal prompt words of the previous optimization cycle and all previous optimization cycles, including: Extract the optimal prompt word from the optimal prompt words of the previous optimization cycle and all previous optimization cycles. Extract the optimal prompt word from at least one optimization cycle that is closest to the current optimization cycle.
6. The prompt word optimization method of any one of claims 1-5, wherein, For any prompt word involved in the prompt word optimization process, the performance evaluation process for the current prompt word includes: Determine the evaluation dataset corresponding to the current prompt word. The evaluation dataset is the original evaluation set or a subset of the original evaluation set. The original evaluation set also includes a set of correct samples. Invoke the business big model to instruct it to classify the input content in the evaluation dataset based on the current prompt word; The category determination result of the business model is compared with the corresponding real category label to determine the correct sample data and incorrect sample data corresponding to the current prompt word; Based on the quantity characteristics of correct and incorrect sample data corresponding to the current prompt word, the feature value corresponding to the current prompt word is calculated, and the feature value is used to characterize the performance of the prompt word.
7. The prompt word optimization method of any one of claims 2-5, wherein, The process of executing M iterations of optimization also includes: If the preset prompt word simplification conditions are met, the rule body in the current optimal prompt word is simplified, and the simplified prompt word is determined based on the simplified rule body; If the preset simplified prompt word application conditions are met, the current optimal prompt word will be updated to the simplified prompt word; otherwise, the current optimal prompt word will not be updated. The simplified prompt word application conditions include: the performance of the simplified prompt word has not deteriorated or the degree of deterioration is less than a preset limit relative to the current optimal prompt word.
8. A prompt word optimization apparatus characterized by comprising: include: An initialization unit is used to obtain initial prompt words for the target classification task and the original dataset corresponding to the target classification service; For any sample data in the original dataset, the data includes: user input content related to the target classification business, and the corresponding real category label; An initial processing unit is used to call a large business model to instruct the large business model to perform category determination on the input content in the original dataset based on the initial prompt words, compare the category determination result of the large business model with the corresponding true category label, and construct an original evaluation set based on the comparison result. The original evaluation set includes at least an error sample set. An optimization unit is configured to construct learning samples and optimize the rule body in the initial prompt words based on the learning samples; the learning samples are constructed based at least on the error sample data in the error sample set; The final processing unit is used to select the optimized prompt word as the final optimal prompt word if the performance of the optimized prompt word is better than that of the initial prompt word; wherein the optimized prompt word is determined based on the optimized rule body.
9. A prompt word optimization device characterized by comprising: It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the prompt word optimization device can implement the prompt word optimization method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, enable the electronic device to implement the prompt word optimization method as described in any one of claims 1 to 7.