An automatic optimization method, system, and device for prompt words based on large language model policy priors

Abstract

This invention discloses an automatic prompt word optimization method, system, and device based on a large language model's policy prior, belonging to the field of artificial intelligence and natural language processing technology. The method includes: constructing a dual-model collaborative architecture with a large language model as the policy prior network and a target small model as the evaluation environment; using the large language model to analyze error cases of the target model on the validation set, generating candidate rule modification actions and their confidence scores; incorporating the confidence scores as prior probabilities into a Monte Carlo tree search process based on the PUCT formula to dynamically balance semantic intuition and experimental feedback; employing a dynamic funnel evaluation and safety rollback mechanism to implement zero-tolerance pruning for modifications that lead to performance degradation; and combining incremental optimization and history freezing strategies to sequentially optimize prompt words in multi-label classification scenarios. This invention can efficiently and automatically generate high-quality, conflict-free prompt word rule sets, significantly improving the performance of the small model on specific tasks.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method, system and device for automatic optimization of prompt words based on large language model strategy priors. Background Technology

[0002] With the maturation of Large Language Model (LLM) technology, it has demonstrated powerful capabilities in various natural language processing tasks. However, in resource-constrained edge computing or private deployment scenarios, models with smaller parameter counts (such as 7B or 14B levels) have become the practical choice due to their lower deployment costs. To improve the accuracy of these small models in specific domain tasks (such as fine-grained text classification), cue word engineering has become a key optimization technique.

[0003] Existing prompt word optimization methods mainly face the following challenges: 1) Manual design relies on expert experience, which is costly and difficult to cover marginal cases; 2) In multi-rule scenarios, modifying rules can easily lead to "catastrophic forgetting", that is, correcting one type of error while causing other correct samples to be misjudged; 3) The search space for prompt word combinations is huge, and traditional optimization algorithms are inefficient; 4) Automated methods (such as APE) lack deep semantic understanding and the search process is blind.

[0004] Therefore, there is a need for a new type of automated system that can integrate the semantic reasoning capabilities of large models with efficient search algorithms to generate high-performance and robust prompt words at low cost and high efficiency. Summary of the Invention

[0005] To address these issues, this application provides an automatic suggestion word optimization method, system, and device based on a large language model strategy prior, aiming to overcome the shortcomings of existing suggestion word optimization technologies, such as high manual costs, susceptibility to "catastrophic forgetting" between rules, low efficiency due to a huge search space, and blind searching without semantic guidance.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides an automatic optimization method for prompt words based on policy priors from a large language model, including: S1: Initialize a Monte Carlo search tree by taking the initial prompt word rule list as the state. The nodes in the tree record the number of visits to the corresponding state, the historical average reward value, and the prior probability. S2: Perform an iterative Monte Carlo tree search. In a single iteration, execute the following sub-steps: S2.1: Starting from the root node, recursively select child nodes based on the prior probabilities and historical average reward values ​​recorded in each child node until a leaf node to be expanded is reached. S2.2: Use a preset second language model to perform classification reasoning on the current rule list corresponding to the leaf node to be expanded, and obtain classification error information; input the leaf node to be expanded and the classification error information into the first language model, and have the first language model analyze the classification error information to generate at least one candidate modification action for optimizing the current rule list, and output the confidence level corresponding to each candidate modification action; S2.3: Create a new child node for each candidate modification action, and convert the confidence level corresponding to the candidate modification action obtained from the first language model into the prior probability recorded by the newly created child node; S2.4: For each newly created child node, a conflict detection evaluation is performed using a dynamically constructed confusion set; if the evaluation result meets the preset performance requirements, the verification set used for final performance evaluation is called for precise evaluation to obtain a reward value; otherwise, the node is marked as invalid. S2.5: Propagate the reward value backward along the path from the node to the root node, and update the number of visits and the historical average reward value of each node on the path. S3: Repeat step S2 until the preset number of iterations is reached or the preset convergence condition is met, and output the list of prompt word rules corresponding to the node with the highest historical average reward value. The first language model is used to generate rule modification actions and their confidence levels, while the second language model is the object to be optimized, used to perform inference and provide classification error information and evaluation feedback.

[0007] Furthermore, in step S2.1, the recursive selection of child nodes based on the prior probabilities and historical average reward values ​​recorded by each child node is achieved by calculating the score of each candidate child node and selecting the one with the highest score using the following formula: Score=Q(s,a)+c*P(s,a)*sqrt(N(s)) / (1+N(s,a))(1) Where Q(s, a) is the historical average reward value of the child node reached by taking action a in state s, P(s, a) is the prior probability recorded by the child node, N(s) is the number of visits to state s, N(s, a) is the number of times action a is selected in state s, and c is the balance coefficient; wherein, the reached child node is the new rule list state obtained after applying action a to the rule list of state s, and this new rule list state is a child node of the current node.

[0008] Furthermore, step S2.2 also includes a step of controlling the generation strategy for the first language model switching action, specifically including: The first strategy is adopted by default, which constrains the first language model to prioritize generating actions that modify existing rules. If, starting from the same parent node, a preset number of attempts are made to modify the existing rules, and none of these attempts result in the corresponding child node receiving a higher reward value, then the second strategy is switched to allow the first language model to generate actions that add new rules. When the total number of rules in the rule list reaches the preset limit, switch back to the first strategy and prohibit the generation of new rules.

[0009] 4. The method according to claim 1, characterized in that, in step S2.4, the dynamically constructed confusion set is constructed based on the following samples: All positive samples of the target category to be optimized; In historical evaluations, the sample from at least one other category that has the highest frequency of mutual misclassification with the target category; In the current or previous preset rounds of Monte Carlo tree search iterations, the misclassified samples generated by the second language model.

[0010] Furthermore, it also includes the historical rule freezing step: According to a predetermined order, one of the multiple categories to be optimized is determined as the target category in turn; When optimizing the current target category, the set of prompt word rules from other categories that have been optimized beforehand will be used as a fixed system background context.

[0011] Furthermore, it also includes caching steps: The inference results of the second language model on a specific list of rules and a specific evaluation dataset are cached. The cached keys are generated based on the hash value of the rule list and the identifier of the evaluation dataset.

[0012] Furthermore, the node selection process of the Monte Carlo tree search is configured to support multi-threaded parallel execution: During the node selection process, when a thread selects a node for evaluation, the statistical parameters used for the selection calculation of that node are temporarily adjusted to reduce the attractiveness of that node to other threads, thereby achieving exploration differentiation among threads.

[0013] Furthermore, in step S2.2, the first language model is output in a preset structured format; The structured format includes at least the following fields: action type, target rule index, new rule content, confidence level, and reason for modification.

[0014] Secondly, this application provides an automatic prompt word optimization system based on large language model policy priors, applied to any of the above-described automatic prompt word optimization methods based on large language model policy priors, including: The strategy generation module is used to call the first language model and generate rule modification actions and corresponding confidence scores based on classification error analysis. The environmental assessment module is used to run the second language model, perform inference on the assessment dataset based on the input prompt word rules, and output classification error information and performance feedback; The tree search control module is connected to the strategy generation module and the environment assessment module respectively, and is used to manage the Monte Carlo search tree and perform search iterations; The tree search control module is configured as follows: Initialize a Monte Carlo search tree, in which each node records the number of visits to its corresponding rule list state, the historical average reward value, and the prior probability; In each round of search iteration: Based on the prior probability and historical average reward value recorded by each child node, recursively select child nodes starting from the root node until a leaf node to be expanded is reached. The strategy generation module is invoked to generate at least one candidate rule modification action and its corresponding confidence level based on the current rule list corresponding to the leaf node and the classification error information output by the environment assessment module. Create a new child node for each candidate modification action, and convert the corresponding confidence level into the prior probability recorded by the newly created child node; The environment assessment module is invoked to perform a graded assessment on each newly created child node: first, a conflict detection assessment is performed using a dynamically constructed confusion set; if the assessment result meets the preset performance requirements, a precise assessment is performed using a verification set for final performance evaluation to obtain a reward value; otherwise, the node is marked as invalid. The reward value is propagated backward along the path from the node to the root node, updating the number of visits to each node on the path and the historical average reward value. The search iteration is repeated until a preset number of iterations is reached or a preset convergence condition is met, outputting a list of prompt word rules corresponding to the node with the highest historical average reward value. Thirdly, this application provides an automatic prompt word optimization device based on a large language model policy prior, the device comprising: Memory, on which executable programs are stored; A processor for executing the executable program in the memory to implement the steps of any of the methods described above.

[0015] The application employs the above technical solution and has at least the following beneficial effects: 1. Significantly improves search efficiency and optimization quality, solving the combinatorial explosion problem. This method creatively uses the inference confidence of a first-language model (such as DeepSeek) as the prior probability (P-value) in Monte Carlo Tree Search (MCTS). This transforms the search process from traditional "blind random exploration" to "semantic intelligence-guided heuristic search." The system can prioritize exploring the most promising modification directions for large models based on semantic analysis, greatly reducing ineffective searches in the huge discrete rule combination space, thus converging quickly to a high-performance prompt word rule set with fewer iterations and lower computational cost.

[0016] 2. Effectively prevents performance degradation and "catastrophic forgetting," ensuring the safety and stability of the optimization process. By introducing a "dynamic confusion set for conflict detection and evaluation" and a "performance requirement" judgment mechanism, this invention establishes a safety barrier during the search expansion phase. Specifically, any newly generated rule modification is first quickly tested on a set containing easily confused negative samples (especially optimized samples from other categories). If the new rule leads to an increase in the misclassification rate of historically correct cases, it will be immediately marked as invalid and pruned. This "prior safety filtering" mechanism fundamentally eliminates the "robbing Peter to pay Paul" problem of sacrificing the accuracy of other categories to improve the recall rate of one category, ensuring that the overall performance of the model does not monotonically decrease during the optimization process.

[0017] 3. This method achieves automated, low-cost, high-quality prompt word engineering, reducing reliance on expert experience. It constructs a complete automated optimization loop through a dual-model collaborative approach: a first language model analyzes errors and generates actions, while a second language model provides feedback. The first language model replaces domain experts in performing complex error root cause analysis and rule drafting; the second language model, as a realistic testing environment, provides objective performance evaluation. This significantly reduces the reliance on human experience in high-quality prompt word development and the resulting high human and time costs, making rapid and efficient performance tuning of small models for specific tasks possible.

[0018] 4. Particularly suitable for complex multi-label incremental optimization scenarios, possessing excellent scalability and scenario adaptability. The mechanism of using the "current rule list" and "classification error information" as inputs in the method naturally supports incremental optimization. When facing a complex scenario with 20 classification labels, the system can strictly follow the "history freeze strategy": when optimizing subsequent labels, the rules of already optimized labels serve as a fixed context, and the current optimization can only adjust the rules of the target label. This method, by having the first language model analyze error information containing confusion with historical labels, can automatically generate boundary delimitation rules and negative exemption rules, thereby accurately handling semantic overlap and conflict between labels, achieving harmonious coexistence and synergistic effects of multiple label rule sets. This solves the problem of rule coupling and conflict that traditional methods struggle to handle in multi-label scenarios.

[0019] 5. It provides an interpretable and controllable optimization path, enhancing the credibility and reusability of the process. The node statistics (number of visits N, average reward Q, prior probability P) during the Monte Carlo tree search process fully record the "experiences" and "lessons learned" of each exploration. The final output includes not only a list of optimal rules but also a decision graph of the entire optimization process. This allows researchers to trace the logic behind rule generation, understand why some rules were adopted while others were discarded, increasing the transparency and interpretability of the optimization process. Simultaneously, the accumulated search tree experience can provide a starting point for transfer learning in optimizing similar tasks, improving the reusability of the method.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating an automatic optimization method for prompt words based on a large language model policy prior, according to an exemplary embodiment. Figure 2 This is a schematic diagram illustrating the composition of an automatic prompt word optimization system based on a large language model policy prior, according to an exemplary embodiment. Figure 3 This is a schematic diagram illustrating the composition of an automatic prompt word optimization device based on a large language model strategy prior, according to an exemplary embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be described in detail below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In practice, before starting the automated optimization process, the following data and environment initialization work needs to be completed to provide a stable and reliable operating foundation for the subsequent core algorithms.

[0025] First, for specific business scenarios (e.g., a 20-classification task based on 5,000 passenger arrival voice-to-text records), a high-quality, fully labeled sample library is constructed. The data in this library is cleaned and preprocessed, including removing colloquial expressions, correcting automatic speech recognition (ASR) errors, and desensitizing privacy information, forming a benchmark dataset for model evaluation.

[0026] Secondly, to adapt to the incremental optimization logic at the core of this invention, a corresponding verification environment must be planned and constructed in advance. In this embodiment, the label optimization order is set as: Label_1, Label_2, …, Label_20. The core rule is that when the system optimizes the Nth label, the optimized rules of the previous N-1 labels will serve as a frozen, unchangeable background context. This means that the optimization process must be carried out under the dual constraints of preventing new rules from mistakenly affecting already frozen labels and preventing the current label sample from being mistakenly covered by frozen rules. As the optimization order progresses, the semantic definition space of subsequent labels will be limited by more and more frozen rules, increasing the optimization difficulty.

[0027] Therefore, the system needs to dynamically maintain a dynamic confusion set. This set is not static, but focuses on the most prominent classification boundary issues. For example, when optimizing the "harassment" (Label_5) label, this set will primarily include samples that the model is most likely to confuse with the frozen labels such as "sexual harassment" (Label_1) and "outcasting" (Label_2). Its core function is to quickly detect whether new rules encroach on the semantic space of other labels in subsequent searches.

[0028] Finally, a benchmark evaluation environment was established. The target inference model (i.e., the second language model, such as Qwen-14B) was stably deployed and a reliable inference interface was provided. The model was run on the full validation set using a minimal set of initial prompts, and baseline values ​​of performance metrics such as precision, recall, and F1 score were recorded as an objective starting point for comparing the effects of all subsequent optimization iterations.

[0029] Example 1 Please see Figure 1 As shown, Figure 1 This is a flowchart illustrating an automatic prompt word optimization method based on a large language model policy prior, according to an exemplary embodiment. The method includes: S1: Initialize a Monte Carlo search tree by taking the initial prompt word rule list as the state. The nodes in the tree record the number of visits to the corresponding state, the historical average reward value, and the prior probability. S2: Perform an iterative Monte Carlo tree search. In a single iteration, execute the following sub-steps: S2.1: Starting from the root node, recursively select child nodes based on the prior probabilities and historical average reward values ​​recorded in each child node until a leaf node to be expanded is reached. S2.2: Use a preset second language model to perform classification reasoning on the current rule list corresponding to the leaf node to be expanded, and obtain classification error information; input the leaf node to be expanded and the classification error information into the first language model, and have the first language model analyze the classification error information to generate at least one candidate modification action for optimizing the current rule list, and output the confidence level corresponding to each candidate modification action; S2.3: Create a new child node for each candidate modification action, and convert the confidence level corresponding to the candidate modification action obtained from the first language model into the prior probability recorded by the newly created child node; S2.4: For each newly created child node, a conflict detection evaluation is performed using a dynamically constructed confusion set; if the evaluation result meets the preset performance requirements, the verification set used for final performance evaluation is called for precise evaluation to obtain a reward value; otherwise, the node is marked as invalid. S2.5: Propagate the reward value backward along the path from the node to the root node, and update the number of visits and the historical average reward value of each node on the path. S3: Repeat step S2 until the preset number of iterations is reached or the preset convergence condition is met, and output the list of prompt word rules corresponding to the node with the highest historical average reward value. The first language model is used to generate rule modification actions and their confidence levels, while the second language model is the object to be optimized, used to perform inference and provide classification error information and evaluation feedback.

[0030] It should be noted that selecting the node with the highest Q value means that the final output is the set of rules that has been verified as the best and most reliable in terms of overall performance throughout the entire search history.

[0031] The method constructs a PUCT-MCTS (Prior-Based Monte Carlo Tree Search) framework that works in collaboration between a first language model (policy network) and a second language model (evaluation environment).

[0032] S1: Initialize the search tree. The system defines the initial list of prompt rules (e.g., an empty list []) as the starting state of the search. Using this state as the root node, a Monte Carlo search tree is initialized. Each node corresponds precisely to a complete list of rules and records three main statistics: the number of visits N (initially 0), the historical average reward Q (initially 0), and the prior probability P (initially set to the default value or 0). The node also contains a safety flag (e.g., Is_Dead) to record whether it has been pruned due to performance degradation.

[0033] S2: A single Monte Carlo tree search iteration. This iteration is the main loop of the algorithm and includes the following sub-steps: S2.1: Selection. Starting from the root node, the system recursively selects child nodes until a leaf node is reached. Selection is not random, but rather based on intelligent decision-making according to the prior probability P recorded for each child node and its historical average reward value Q. The specific selection strategy is detailed in Example 2.

[0034] S2.2: Expansion and Policy Generation. Upon reaching a leaf node, the system invokes the first language model (e.g., DeepSeek) as the policy prior. Input information includes: 1) the current rule list corresponding to the leaf node; 2) specific classification errors recently generated by the second language model on the validation set (e.g., false positives, false negatives and their context). The first language model analyzes the errors and outputs at least one candidate modification action (of type MODIFY or ADD) and the confidence level for each action.

[0035] S2.3: Node Creation and Prior Injection. A new child node is created for each candidate action, its state being the list of rules with the modified version applied. A key step is to normalize the confidence score output by the first language model and directly assign it to the prior probability P recorded by this new node. This completes the injection of the large model's "intuition" into a search guidance signal.

[0036] S2.4: Hierarchical Evaluation. Each new node is evaluated using a dynamic funnel model to improve efficiency and security. Conflict Detection Evaluation (Fast Filtering): First, fast inference is performed using a dynamically constructed confusion set (construction method see Example 4). This step is specifically designed to quickly detect "catastrophic forgetting," i.e., whether a new rule degrades existing classification performance. If the performance metric drops beyond a threshold (e.g., -1%), the node is immediately marked as Is_Dead (pruned).

[0037] Precise evaluation: Only for nodes that pass the fast filtering, inference is performed using the validation set (such as the full validation set) used for final performance evaluation to calculate the precise reward value, such as the F1 score.

[0038] S2.5: Backpropagation. The reward value (and penalty value for invalid nodes) is propagated backward along the search path to the root node. The number of visits N and the average reward value Q for each node on the path are updated (calculated as: Q_new = (Q_old * (N_old - 1) + Reward) / N_new). This process reinforces successful paths.

[0039] S3: Iteration and Output. Repeat step S2 hundreds to thousands of times until the preset number of iterations is reached or the convergence condition is met (e.g., the optimal reward value does not improve for several consecutive times). Finally, output the list of rules corresponding to the node with the highest historical average reward value Q in the entire tree, which is the result of automatic optimization.

[0040] Example 2: This embodiment refines step S2.1 and discloses its specific implementation method, namely the PUCT formula.

[0041] The "recursive selection of child nodes based on the prior probability and historical average reward value recorded by each child node" is achieved by calculating the PUCT score of each candidate child node. For a child node that arrives after taking action a from the parent node state s, its score is calculated as follows: Score(s,a)=Q(s,a)+c*P(s,a)*sqrt(N(s)) / (1+N(s,a))(1) in: Q(s, a): The historical average reward value (utilization item) of this child node.

[0042] P(s, a): The prior probability (prior term) recorded by this child node is directly derived from the value injected in step S2.3 of Example 2.

[0043] N(s): The total number of visits to the parent node s.

[0044] N(s, a): The number of times action a is selected under the parent node s.

[0045] c: Adjustable balance coefficient used to control the intensity of exploration.

[0046] In each selection step, the system calculates the PUCT score of all child nodes and selects the one with the highest score. This formula dynamically balances historical experience (Q), large model prior (P), and encouragement for underexplored nodes (few visits N(s,a)), and is the core of achieving efficient and intelligent search.

[0047] Example 3 This embodiment details the steps for dynamically constraining candidate modification action types, which is crucial for controlling the refinement of the rule set.

[0048] In step S2.2 of Example 1, the first language model can generate two types of actions: Modify (MODIFY) or Add (ADD). In specific implementation, the following intelligent constraints are implemented: Default priority modification: The system defaults to constraining the first language model to generate MODIFY actions first, encouraging the resolution of problems by refining existing rules and preventing the rule set from expanding too early.

[0049] Allowing additions after failure: If, under the same parent node, the reward value of child nodes created by the MODIFY action does not exceed that of the parent node after a preset number of consecutive attempts (e.g., 4), the system determines that the modification may have reached a bottleneck. In this case, the system removes the constraint, allowing the first language model to generate the ADD action during the next expansion to introduce new dimensions.

[0050] Total rule limit control: The system presets a maximum number of rules for a single-label rule list (e.g., 20 rules). Once the total number of rules reaches the limit, the constraint is restored, prohibiting the first language model from generating any ADD actions, and subsequent optimizations are limited to MODIFY. This mechanism fundamentally ensures the conciseness of the output rule set.

[0051] Example 4 This embodiment provides a detailed explanation of the "dynamically constructed obfuscation set" used in step S2.4.

[0052] This confusion set is the core dataset for rapid collision detection evaluation. It is constructed based on the following four types of samples and is dynamically updated as the optimization process progresses: Core set: All positive samples of the target category to be optimized. Used to ensure that new rules do not impair the identification of this category.

[0053] Easily confused negative sample set: Samples from other categories that are most easily confused with the current target category (especially samples with frozen labels). This is crucial for detecting rule boundaries and preventing false positives.

[0054] Recent Error Set: Misclassified samples (especially False Positives) recently generated by the second language model. Used to test whether new rules can correct the latest errors.

[0055] Random negative sample set: A small number of randomly selected samples from other classes, used to provide a generalization reference and prevent overfitting.

[0056] Before each evaluation, the system dynamically assembles this confusion set based on the latest error analysis report to ensure that it always focuses on the most pressing classification contradictions.

[0057] Example 5: This embodiment describes the historical rule freezing steps, which are applicable to serial optimization in multi-label classification scenarios.

[0058] The system optimizes labels one by one strictly according to a predetermined order (e.g., Label_1 to Label_20). Its core rule is: Serial optimization: At any given time, only one target label is optimized.

[0059] History Freeze: When optimizing the Nth tag, the rules for the previous N-1 tags have been optimized and frozen, serving as an unchangeable system prompt word prefix (fixed system background context).

[0060] Constrained Action Space: In this mode, in step S2.2 of Implementation 1, the first language model is prohibited from generating any modification actions against the frozen rules, and its action space is limited to serving the current target label.

[0061] When optimizing the "harassment" label, the system automatically generated an exclusive rule with the message "Note: This is different from 'sexual harassment'..." after discovering a conflict with the already frozen "sexual harassment" label. This allowed the system to complete its own system construction without disrupting the existing system.

[0062] Example 6: The caching steps described in this embodiment are designed to greatly improve optimization efficiency.

[0063] In step S2.4 of Example 1, the second language model needs to be frequently called for evaluation. To avoid redundant calculations, the system establishes an evaluation result cache: Cache key: generated by combining hash (current rule list content) and hash (the evaluation dataset ID used) to ensure uniqueness.

[0064] Cache values: Store the corresponding complete evaluation results (such as confusion matrix, F1 score).

[0065] Function: When the same list of rules needs to be evaluated on the same dataset again, the cached results are read directly, skipping model inference and significantly speeding up the iteration.

[0066] Simultaneously, the system maintains a state replacement table. Since different modification orders may lead to the same final rule state, the replacement table maps these identical states to the same MCTS node, allowing them to share the number of visits N and the average reward value Q, thereby improving the efficiency and accuracy of search statistics.

[0067] Example 7: This embodiment describes in detail the specific methods for configuring multi-threaded parallel execution to accelerate the search.

[0068] Step S2 in Example 1 supports multi-threaded parallel execution. To prevent multiple threads from evaluating the same node simultaneously, the system introduces a virtual loss mechanism: When a thread selects a node and begins its evaluation process in step S2.1, the statistical parameters used for the selection calculation of that node are immediately and temporarily adjusted.

[0069] The specific operation is to temporarily increase the number of visits to the node N and temporarily decrease its historical average reward value Q.

[0070] This temporary modification temporarily reduces the attractiveness (score) of the node when other threads calculate it according to the PUCT formula of Example 2, thereby guiding other threads to explore different nodes.

[0071] Once the thread completes its evaluation and updates the actual parameters, the temporary modifications are revoked. This mechanism effectively achieves differentiated exploration and load balancing among threads.

[0072] Example 8: This example illustrates the prompt word format requirements set to ensure effective interaction with large models.

[0073] In step S2.2 of Example 1, when the first language model is invoked, its system prompts are carefully designed to force the model to output in a preset, strictly structured format (such as JSON), which must include the following fields: action: Action type ("MODIFY" / "ADD").

[0074] target_index: If MODIFY, it specifies the index of the rule to be modified.

[0075] content: The complete text of the new rules.

[0076] confidence: confidence score.

[0077] Reasoning: The reasons for modification based on input error analysis.

[0078] This design transforms the natural language output of large models into structured data that can be directly parsed and processed by the program, which is key to ensuring the stable operation of automated processes.

[0079] Example 9: Please see Figure 2 This embodiment describes a hardware system for implementing any of the methods described above. The system includes: Strategy generation module 90: Encapsulates the interaction with the first language model and is responsible for generating structured modification suggestions and confidence levels.

[0080] Environment Assessment Module 91: Deploy and manage second language models, provide an efficient inference interface, and return performance feedback and error analysis.

[0081] Tree Search Control Module 92: The core of the system, used to execute the complete PUCT-MCTS process, including tree initialization, node selection (calling the formula of Example 2), extended scheduling, hierarchical evaluation scheduling (calling the confusion set of Example 4), backtracking update, and integrating all management strategies of Examples 3, 5, 6, 7, and 11.

[0082] Rule and Status Management Module: Maintains the current rule list and frozen rule sets, and performs logic such as checking the maximum number of rules.

[0083] The modules mentioned above work together through a software interface to automate the entire optimized pipeline.

[0084] Example 10: Please see Figure 3 This embodiment provides an electronic device for implementing the above-described method. The device includes at least one processor 11 (CPU / GPU), a memory 10 (such as RAM or a hard disk), and a computer program stored on the memory 10. When the processor executes the program, it controls the device to implement the method steps described in detail in any one or more of embodiments 1 to 9, 11. The device may be a server, workstation, or cloud computing instance.

[0085] Example 11 This embodiment simulates a specific business scenario to demonstrate in detail the complete execution process and technical effects of the method of the present invention in practical applications, so as to further illustrate and verify the foregoing embodiments.

[0086] I. Application Scenarios and Initialization This simulation targets a 20-category task of handling inbound passenger complaints for ride-hailing services, specifically optimizing the fifth label, "harassment." Following the incremental optimization strategy of this invention, the rules for the preceding labels, Label_1 (sexual harassment) and Label_2 (external submission), were optimized in advance and are frozen in this round of optimization as an unchangeable system background context. The dataset used for evaluation is a validation set containing 115 diverse passenger complaint texts, covering various situations such as direct allegations of harassment, descriptions of phone / text message bombardment, requests for contact information, upskirt photography, and conflation with fare disputes.

[0087] II. Simulation of Automated Optimization Process Based on PUCT-MCTS The system starts with an empty list of rules and iterates through a Monte Carlo tree search. The following is a simulation of several key iterations, demonstrating the operation of the core algorithm: First iteration (establishing core definitions): Error Analysis: Initially, the model missed all "harassment" samples. DeepSeek analysis revealed that high-frequency erroneous samples all contained keywords such as "making a phone call," "sending a text message," and "bombardment."

[0088] Policy generation and expansion: Based on this, the first language model (DeepSeek) generates an add-on action, suggesting the addition of the rule: "The driver contacts the passenger multiple times and continuously through channels such as phone, SMS, and WeChat...". Its confidence level is 0.98.

[0089] Evaluation and Backtracking: After evaluation on a dynamic confusion set, the recall rate was significantly improved. The system creates a new node, the confidence level is converted into a prior probability P, and the reward value Q is updated.

[0090] Second and third iterations (covering semantic variants): Similarly, by analyzing new erroneous samples (such as "adding WeChat", "asking about privacy", and "secretly taking photos"), the system drives DeepSeek to generate and verify new rules regarding "demanding privacy" and "behavioral violations", gradually improving the recall rate.

[0091] Fourth iteration (handling historical frozen tag conflicts – embodying incremental optimization): Error Analysis: The system detected that the new rules began to misclassify some samples that should have belonged to the frozen "sexual harassment" label (such as those involving sexual innuendo) as "harassment".

[0092] Policy Generation and Constraints: Based on historical freeze policies, the system prohibits DeepSeek from modifying the rules for the "Sexual Harassment" tag. Therefore, DeepSeek generated an exclusive rule (negative constraint) for the current "Harassment" tag: "Note: Unlike 'Sexual Harassment,' this tag focuses on non-sexual harassment behaviors..."

[0093] Fifth iteration: Error Analysis: It was found that the rules led to some legitimate collection actions "due to fee issues" being mistakenly judged as harassment (FalsePositive).

[0094] Policy Generation and Evaluation: DeepSeek generated a negative exemption rule: "Note: Repeatedly contacting passengers solely for fee-related issues does not constitute harassment..." This rule was only formally adopted after undergoing a rapid security assessment on a dynamic obfuscation set to confirm that it did not compromise existing key metrics.

[0095] III. Simulation Results and Technical Effects After multiple rounds of iteration, the system automatically outputs a set of optimized prompts for the "harassment" label. A set of rules (20 in total). This set of rules has the following characteristics, which verify the beneficial effects of the present invention: Hierarchical and comprehensive: The rules automatically cover core dimensions such as "communication harassment", "privacy harassment" and "behavioral harassment", and include boundary definitions and negative exemption clauses.

[0096] Boundary clarity: Successfully handled semantic overlap with frozen tags such as "sexual harassment" and "outward projection", demonstrating the effectiveness of the incremental optimization strategy.

[0097] Security: Throughout the process, rapid conflict detection through dynamic confusion sets ensures that no rules that would degrade the classification performance of frozen labels are introduced.

[0098] Conclusion: This simulation example, based on real business data, fully demonstrates how, starting from the initial state and under given constraints, this invention drives the collaborative work of two models through the PUCT-MCTS framework to automatically, safely, and efficiently generate a high-quality, conflict-free prompt word rule set. This fully demonstrates the practicality, effectiveness, and innovation of the method of this invention in solving the problem of automated prompt word optimization in complex scenarios.

[0099] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0100] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "multiple" or "more" means at least two.

[0101] It should be understood that when an element is referred to as "fixed to" or "set on" another element, it may be directly on the other element or may have an intervening element present at the same time; when an element is referred to as "connected to" another element, it may be directly connected to the other element or may have an intervening element present at the same time. In addition, the term "connected" as used herein may include wireless connections; the word "and / or" as used includes any unit and all combinations of one or more of the associated listed items.

[0102] Any process or method description in the flowchart or otherwise herein can be understood as: representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0103] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0106] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0107] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0108] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An automatic optimization method for prompt words based on policy priors of a large language model, characterized in that, include: S1: Initialize a Monte Carlo search tree by taking the initial prompt word rule list as the state. The nodes in the tree record the number of visits to the corresponding state, the historical average reward value, and the prior probability. S2: Perform an iterative Monte Carlo tree search. In a single iteration, execute the following sub-steps: S2.1: Starting from the root node, recursively select child nodes based on the prior probabilities and historical average reward values ​​recorded in each child node until a leaf node to be expanded is reached. S2.2: Use a preset second language model to perform classification reasoning on the current rule list corresponding to the leaf node to be expanded, and obtain classification error information; input the leaf node to be expanded and the classification error information into the first language model, and have the first language model analyze the classification error information to generate at least one candidate modification action for optimizing the current rule list, and output the confidence level corresponding to each candidate modification action; S2.3: Create a new child node for each candidate modification action, and convert the confidence level corresponding to the candidate modification action obtained from the first language model into the prior probability recorded by the newly created child node; S2.4: For each newly created child node, a conflict detection evaluation is performed using a dynamically constructed confusion set; if the evaluation result meets the preset performance requirements, the verification set used for final performance evaluation is called for precise evaluation to obtain a reward value; otherwise, the node is marked as invalid. S2.5: Propagate the reward value backward along the path from the node to the root node, and update the number of visits and the historical average reward value of each node on the path. S3: Repeat step S2 until the preset number of iterations is reached or the preset convergence condition is met, and output the list of prompt word rules corresponding to the node with the highest historical average reward value. The first language model is used to generate rule modification actions and their confidence levels, while the second language model is the object to be optimized, used to perform inference and provide classification error information and evaluation feedback.

2. The method according to claim 1, characterized in that, In step S2.1, the recursive selection of child nodes based on the prior probabilities and historical average reward values ​​recorded by each child node is achieved by calculating the score of each candidate child node and selecting the one with the highest score using the following formula: Score=Q(s,a)+c*P(s,a)*sqrt(N(s)) / (1+N(s,a))(1) Where Q(s, a) is the historical average reward value of the child node reached by taking action a in state s, P(s, a) is the prior probability recorded by the child node, N(s) is the number of visits to state s, N(s, a) is the number of times action a is selected in state s, and c is the balance coefficient; wherein, the reached child node is the new rule list state obtained after applying action a to the rule list of state s, and this new rule list state is a child node of the current node.

3. The method according to claim 1, characterized in that, Step S2.2 further includes a step of controlling the generation strategy for the switching action of the first language model, specifically including: The first strategy is adopted by default, which constrains the first language model to prioritize generating actions that modify existing rules. If, starting from the same parent node, a preset number of attempts are made to modify the existing rules, and none of these attempts result in the corresponding child node receiving a higher reward value, then the second strategy is switched to allow the first language model to generate actions that add new rules. When the total number of rules in the rule list reaches the preset limit, switch back to the first strategy and prohibit the generation of new rules.

4. The method according to claim 1, characterized in that, In step S2.4, the dynamically constructed confusion set is built based on the following samples: All positive samples of the target category to be optimized; In historical evaluations, the sample from at least one other category that has the highest frequency of mutual misclassification with the target category; In the current or previous preset rounds of Monte Carlo tree search iterations, the misclassified samples generated by the second language model.

5. The method according to claim 1, characterized in that, It also includes the historical rule freezing step: According to a predetermined order, one of the multiple categories to be optimized is determined as the target category in turn; When optimizing the current target category, the set of prompt word rules from other categories that have been optimized beforehand will be used as a fixed system background context.

6. The method according to claim 1, characterized in that, It also includes caching steps: The inference results of the second language model on a specific list of rules and a specific evaluation dataset are cached. The cached keys are generated based on the hash value of the rule list and the identifier of the evaluation dataset.

7. The method according to claim 1, characterized in that, The node selection process of the Monte Carlo tree search is configured to support multi-threaded parallel execution: During the node selection process, when a thread selects a node for evaluation, the statistical parameters used for the selection calculation of that node are temporarily adjusted to reduce the attractiveness of that node to other threads, thereby achieving exploration differentiation among threads.

8. The method according to claim 1, characterized in that, In step S2.2, the first language model is output in a preset structured format; The structured format includes at least the following fields: action type, target rule index, new rule content, confidence level, and reason for modification.

9. An automatic prompt word optimization system based on large language model policy prior, applied to the automatic prompt word optimization method based on large language model policy prior as described in any one of claims 1-8, characterized in that, include: The strategy generation module is used to call the first language model and generate rule modification actions and corresponding confidence scores based on classification error analysis. The environmental assessment module is used to run the second language model, perform inference on the assessment dataset based on the input prompt word rules, and output classification error information and performance feedback; The tree search control module is connected to the strategy generation module and the environment assessment module respectively, and is used to manage the Monte Carlo search tree and perform search iterations; The tree search control module is configured as follows: Initialize a Monte Carlo search tree, in which each node records the number of visits to its corresponding rule list state, the historical average reward value, and the prior probability; In each round of search iteration: Based on the prior probability and historical average reward value recorded by each child node, recursively select child nodes starting from the root node until a leaf node to be expanded is reached. The strategy generation module is invoked to generate at least one candidate rule modification action and its corresponding confidence level based on the current rule list corresponding to the leaf node and the classification error information output by the environment assessment module. Create a new child node for each candidate modification action, and convert the corresponding confidence level into the prior probability recorded by the newly created child node; The environment assessment module is invoked to perform a graded assessment on each newly created child node: first, a conflict detection assessment is performed using a dynamically constructed confusion set; if the assessment result meets the preset performance requirements, a precise assessment is performed using a verification set for final performance evaluation to obtain a reward value; otherwise, the node is marked as invalid. The reward value is propagated backward along the path from the node to the root node, updating the number of visits to each node on the path and the historical average reward value. Repeat the search iteration until the preset number of iterations is reached or the preset convergence condition is met, and output the list of prompt word rules corresponding to the node with the highest historical average reward value.

10. An automatic prompt word optimization device based on large language model strategy prior, characterized in that, The device includes: Memory, on which executable programs are stored; A processor for executing the executable program in the memory to implement the steps of the method according to any one of claims 1-8.

Images