Reward rule set updating method and device and reinforcement learning-based model training method and device

Abstract

The embodiment of the invention discloses a reward rule set updating method and device and a model training method and device based on reinforcement learning. The updating method comprises the following steps: firstly, based on a training sample library, sampling a batch of abnormal samples of which reward scores meet a preset abnormal condition; wherein each training sample comprises a sample question, an answer output by the first large language model for the sample question, and a reward score; the reward score is obtained by utilizing a second large language model to evaluate the reasoning step in the sample answer according to a current reward rule set; then, processing the batch of abnormal samples by using a third language model to obtain an optimization suggestion for the current reward rule set; and then, based on the optimization suggestion, updating the current reward rule set. The updated current reward rule set is used for reinforcement learning of the first large language model.

Description

Technical Field

[0001] This specification relates to the field of large model technology, and in particular to a method and apparatus for updating a set of reward rules, a method and apparatus for training a model based on reinforcement learning, a computer-readable storage medium, and a computing device. Background Technology

[0002] Large Language Models (LLMs) acquire powerful language generation and understanding capabilities through pre-training on massive amounts of text data. Leveraging this capability, LLMs are widely used in many fields such as intelligent customer service, content creation, code assistance, education, and information retrieval, significantly improving the intelligence level of human-computer interaction.

[0003] However, the behavioral patterns learned by pre-trained LLMs on open-domain data are highly uncertain, and their raw outputs are difficult to directly meet the stability, security, and controllability requirements necessary for integration into actual production systems. Specifically, LLMs may produce content that contradicts known facts, contains logical contradictions, or conflicts with system security protocols. These technical flaws severely hinder the reliable deployment and effective application of LLMs. Therefore, "alignment," as a key technical processing stage, has the core task of correcting these technical flaws, ensuring that the model's output behavior is consistent with established factual benchmarks, logical rules, and security boundaries, thereby making it technically feasible for integration and deployment in actual software systems and hardware platforms.

[0004] Therefore, an improved LLM alignment scheme is needed to meet the higher requirements in practical applications. Summary of the Invention

[0005] This specification describes an embodiment of an update of a reward rule set, a method and apparatus for training a reinforcement learning-based model, which can solve the above-mentioned technical problems.

[0006] According to the first aspect, a method for updating a reward rule set is provided. The method includes: sampling a batch of anomalous samples whose reward scores meet preset anomalous conditions based on a training sample library; wherein each training sample includes a sample question, an answer output by a first language model for the sample question, and a reward score; the reward score is obtained by evaluating the reasoning steps in the sample answer using a second language model based on the current reward rule set. The batch of anomalous samples is then processed using a third language model to obtain optimization suggestions for the current reward rule set. Based on the optimization suggestions, the current reward rule set is updated.

[0007] In one embodiment, the preset abnormal conditions include: the reward score is lower than the lower limit threshold, and / or the reward score is greater than the upper limit threshold.

[0008] In one embodiment, the preset abnormal condition includes: the probability of the second language model generating a reward score is less than a probability threshold.

[0009] In one embodiment, the preset abnormal condition includes: the variance among multiple reward scores corresponding to multiple reasoning steps in the answer is greater than a variance threshold.

[0010] In one embodiment, the preset abnormal condition includes: the answer is identified by the fourth language model as obtaining an excessively high reward score by exploiting a rule loophole.

[0011] In one embodiment, sampling a batch of abnormal samples whose reward scores meet preset abnormality conditions based on the training sample library includes: sampling the batch of abnormal samples based on the abnormal sample sub-library in the training sample library.

[0012] In one embodiment, the batch of abnormal samples is processed using a third language model to obtain optimization suggestions for the current reward rule set. This includes inputting the batch of abnormal samples, the current reward rule set, and a task description for the rule optimization task into the third language model to obtain the optimization suggestions.

[0013] In one embodiment, the optimization suggestion includes one or more of the following: adding new rules, modifying existing rules, and deleting existing rules.

[0014] According to the second aspect, a reinforcement learning-based model training method is provided. The method includes: querying a current set of reward rules, which is obtained by updating the set using the method described in claim 1; using a second large language model, evaluating the reasoning steps in historical answers based on the current set of reward rules to obtain corresponding reward scores; the historical answers are obtained by processing historical questions using a first large language model; and training the first large language model based on the historical questions, historical answers, and reward scores.

[0015] In one embodiment, the training period of the first large language model is shorter than the update period of the current reward rule set.

[0016] In one embodiment, the second largest language model is used to evaluate the reasoning steps in historical answers based on the reward rule set to obtain the corresponding reward score. This includes: filling the current reward rule set, historical questions, and historical answers into a preset prompt word template to obtain complete prompt words; and inputting the prompt words into the second largest language model to obtain the reward score.

[0017] In one embodiment, training the first large language model based on the historical questions, historical answers, and reward scores includes: using a policy gradient algorithm to update the parameters of the first large language model based on training samples composed of the historical questions, historical answers, and reward scores.

[0018] According to a third aspect, an apparatus for updating a set of reward rules is provided. The apparatus includes: an abnormal sample sampling unit configured to sample a batch of abnormal samples whose reward scores meet preset abnormal conditions based on a training sample library; wherein each training sample includes a sample question, an answer output by a first language model for the sample question, and a reward score; the reward score is obtained by evaluating the reasoning steps in the sample answer using a second language model according to the current set of reward rules; an optimization and prediction unit configured to process the batch of abnormal samples using a third language model to obtain optimization suggestions for the current set of reward rules; and a reward rule updating unit configured to update the current set of reward rules based on the optimization suggestions.

[0019] According to the fourth aspect, a model training device based on reinforcement learning is provided. The device includes: a reward rule query unit configured to query the current reward rule set, which is obtained by updating using the device provided in the third aspect; a reward score prediction unit configured to use a second large language model to evaluate the reasoning steps in historical answers based on the current reward rule set, and obtain the corresponding reward score; the historical answers are obtained by processing historical questions using a first large language model; and a policy model update unit configured to train the first large language model based on the historical questions, historical answers, and reward scores.

[0020] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first or second aspect.

[0021] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first or second aspect.

[0022] In summary, by employing the methods and apparatus disclosed in the embodiments of this specification, it is possible to achieve "adversarial" co-evolution between the strategy model and the reward rules, thereby significantly improving the alignment robustness of the model, and further improving the quality of responses when providing services to users using the strategy model, effectively enhancing the user experience. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A data flow diagram illustrating the implementation framework of the improved scheme disclosed in the embodiments of this specification;

[0025] Figure 2 This is a schematic diagram of the process steps for updating the reward rule set disclosed in the embodiments of this specification;

[0026] Figure 3 This is a schematic diagram illustrating the process steps of the reinforcement learning-based model training method disclosed in the embodiments of this specification.

[0027] Figure 4 This is a schematic diagram of the dual-loop collaborative architecture of the strategy model training system disclosed in the embodiments of this specification;

[0028] Figure 5 This is a functional structure diagram of the device for updating the reward rule set disclosed in the embodiments of this specification;

[0029] Figure 6 This is a schematic diagram of the functional structure of the reinforcement learning-based model training device disclosed in the embodiments of this specification. Detailed Implementation

[0030] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0031] As mentioned earlier, alignment is a crucial stage following pre-training, designed to ensure that the behavior patterns of the LLM meet the stability, controllability, and reliability requirements necessary for integration into a real-world production environment.

[0032] Reinforcement Learning (RL) provides a robust theoretical framework for model alignment. In this framework, a language model is viewed as an "agent," and the sequences of text it generates are considered "actions." The agent's goal is to learn a policy through interaction with the environment to maximize the accumulated "reward" signal. However, defining a precise reward function to capture the nuances of human preferences is a significant challenge for complex language tasks. This is because human preferences are often subjective, context-dependent, and difficult to describe with simple, predefined mathematical formulas.

[0033] This challenge gave rise to the core idea of ​​modern LLM alignment techniques: instead of having human engineers specify a fixed reward function, machine learning techniques are used to learn a reward function from data reflecting human preferences. This paradigm shift "from specifying rewards to learning rewards" is specifically embodied in Reinforcement Learning from Human Feedback (RLHF) techniques. However, it is precisely this new technological framework itself that introduces a series of inherent limitations, and overcoming these limitations forms the direct basis for the technical problem that this solution aims to address.

[0034] To aid understanding, the following will first introduce the standard process of human feedback-based reinforcement learning (RLHF), which serves as the current technological baseline, and further analyze its technological limitations, as well as the technological evolution path of the Process Reward Model (PRM) that has emerged from it, which aims to achieve fine-grained supervision.

[0035] I. Standard Alignment Process: Reinforcement Learning Based on Human Feedback (RLHF)

[0036] RLHF is currently the mainstream LLM alignment technique in the industry. It effectively injects human preference signals into model training through a standardized, multi-stage process. A typical RLHF process includes the following three core steps:

[0037] 1. Supervised Fine-Tuning (SFT): First, a high-quality "hint-response" pair dataset compiled by human experts is collected. Then, the pre-trained LLM is fine-tuned using this dataset. The purpose of this stage is to allow the model to initially adapt to specific task formats and dialogue styles, providing a good starting point for its subsequent reinforcement learning stages.

[0038] 2. Reward Model (RM) Training: This is the core step of RLHF. First, the model from the SFT stage generates multiple different responses to a set of cues (or cue words). Then, human annotators are invited to rank these responses to reflect their preferences (e.g., response A is better than response B). Using these paired preference data, an independent reward model is trained. This model takes a cue-response pair as input and outputs a scalar score representing the degree of human preference for that response.

[0039] 3. Reinforcement Learning Optimization: Finally, the trained reward model is used as the reward function in the reinforcement learning environment. The SFT model is further fine-tuned using RL algorithms such as Proximal Policy Optimization (PPO). In this stage, the policy model (i.e., the LLM as the agent) generates answers, the reward model scores these answers, and the PPO algorithm updates the parameters of the policy model based on these scores, making it more inclined to generate answers that yield higher rewards.

[0040] The above introduces the standard RLHF.

[0041] II. Advances in Granularity Supervision: The Emergence of Process Reward Models (PRM)

[0042] With the development of alignment techniques, researchers have found that simply evaluating the final output is far from sufficient for many complex tasks. This type of reward model, which focuses solely on the final result, is called an Outcome Reward Model (ORM). In multi-step tasks such as mathematical reasoning and code generation, a model might "luckily" arrive at the correct final answer through an incorrect reasoning process. If ORMs award high scores to such "convergent results," they are essentially encouraging the model to form flawed reasoning logic.

[0043] To address this issue, Process Reward Models (PRMs) were developed. Unlike ORMs, PRMs aim to evaluate and reward each intermediate step in the reasoning chain. PRMs provide denser, more granular feedback signals to the model's thinking process, ensuring that each step of reasoning is correct and beneficial. A "valid process" is typically defined as a logically coherent sequence of reasoning that ultimately leads to the correct answer.

[0044] The evolution from ORMs to PRMs marks a significant deepening of understanding in the field of alignment: the goal of alignment has shifted from "What correct answer should the model output?" to "How should the model think correctly?". This supervision of process, rather than just the result, is a crucial step towards more reliable and interpretable AI systems, and directly forms the technical foundation for the improvements and optimizations this solution aims to achieve. The core of this invention is built upon the concept of PRM and is dedicated to addressing the fundamental shortcomings of existing PRM implementations.

[0045] III. The fundamental limitations and technical origins of traditional PRM implementations

[0046] While Process Reward Models (PRMs) represent a significant leap in supervision granularity, moving from outcome-based to process-based approaches, current mainstream PRM implementations generally adhere to a "static reward paradigm." This paradigm means that after a one-time training, the parameters of the reward model are frozen and remain fixed during subsequent reinforcement learning optimization phases. This "train once, use forever" technical architecture suffers from the following three interrelated fundamental technical flaws:

[0047] 1. The inherent contradiction between static evaluation system and dynamic optimization objectives

[0048] The static PRM architecture creates an inherently non-adaptive alignment system. Once the model is trained, its criteria for judging "high-quality reasoning" are fixed. However, policy models continuously evolve and generate new behavioral patterns during reinforcement learning, and their optimization objective—a reasoning process that better aligns with human preferences—is itself a dynamic and complex system. This causes the training of the policy model to become an optimization of a fixed target, while the real target is a moving one. This fundamental misalignment between the evaluation system and the optimization objective is the core technical root cause of the degradation or failure of alignment performance.

[0049] 2. "Reward Hacking" Behavior Caused by Insufficient Generalization of Reward Models

[0050] Because static PRMs are merely poorly trained agents of human preferences learned on limited datasets, their generalization ability has an inherent bottleneck. When the policy model discovers reasoning patterns not present in the PRM training data during training, static PRMs struggle to provide accurate evaluations. This directly leads to the "reward hacking" phenomenon: instead of learning correct reasoning logic, the policy model seeks out and exploits loopholes in the static PRM's scoring rules to maximize rewards. For example, a PRM for mathematical reasoning might be trained to favor certain forms of derivation steps; the policy model would then mechanically insert these formalized steps instead of performing substantive reasoning, resulting in a deterioration in actual output quality.

[0051] 3. Inherent defects and high costs of the supervision signal generation mechanism

[0052] Current implementations of Proof-of-Reference (PRM) heavily rely on costly sources of supervised signals. On one hand, relying on manual annotation to score each inference step faces technical bottlenecks such as enormous data acquisition costs and poor scalability. On the other hand, automated methods adopted to reduce costs (such as self-game based on Monte Carlo estimation) often generate reward signals containing significant noise and low signal-to-noise ratios, severely impacting the training effectiveness and final performance of the PRM. This difficult trade-off between annotation costs and signal quality constitutes another key technical obstacle preventing the large-scale deployment of static PRMs in practical applications.

[0053] Based on the above observations and analysis, this specification discloses an improved scheme that enables "adversarial" co-evolution between the strategy model and reward rules, thereby significantly improving the model's alignment robustness. Specifically, the reward rule set can be dynamically updated according to the evolutionary behavior of the strategy model to continuously repair blind spots and loopholes in the evaluation system. This suppresses the tendency of the strategy model to obtain high scores by fitting specific rule loopholes rather than learning general abilities, ultimately guiding it to converge to a behavioral pattern that is more in line with human intentions and has stronger generalization capabilities.

[0054] Figure 1 The implementation framework of the improvement scheme is shown, in which the data flow is divided into two interrelated loops: the inner loop (or RL training loop) and the outer loop (rule evolution update).

[0055] In any model training iteration within the inner loop, the dynamic PRM module is read first. Figure 1 The system uses green to indicate the currently active (i.e., latest) reward rule set in the main components. Then, it uses the reward scoring model in the dynamic PRM module in combination with the latest reward rule set to determine the reward score of the reasoning steps output by the policy model for the problem, thereby constructing training samples to train the policy model.

[0056] During any rule update process in the outer loop, a batch of abnormal samples (or bad samples) with abnormal reward scores are sampled from the training sample library. The reward design model (RDM) in the dynamic PRM module is used to process this batch of abnormal samples, thereby outputting optimization suggestions for the reward rule set and updating the reward rule set.

[0057] Next, the process steps involved in the aforementioned inner and outer loops will be described in turn. It should be noted that the executing entity for each process step can be any device, platform, server, or equipment cluster with computing and processing capabilities.

[0058] See Figure 2 The diagram illustrates that any RL learning iteration involving the inner loop comprises the following process steps:

[0059] Step S210: Query the current reward rule set (or PRM rule set, rule set, etc.).

[0060] In one implementation, the current set of reward rules retrieved is an initial, basic set of rules. This set of rules can be predefined by human experts based on the task domain (such as intelligent customer service, information retrieval, etc.) and includes some general, basic evaluation criteria (e.g., "reasoning steps should be relevant to the question", "avoid factual errors", etc.).

[0061] In another implementation, the current set of reward rules retrieved is the set of reward rules after an update (the number of updates is unlimited). It should be noted that the specific rule update process will be described in detail later.

[0062] On the other hand, in one embodiment, currently active PRM rule sets can be read from a PRM Rules Cache Server. It should be understood that this cache server can be a highly available, low-latency distributed caching system where PRM rule sets can be stored using key-value pairs, where the key is the rule's sequence number and the value is the rule's content.

[0063] On the other hand, for the current RL learning session, the current reward rule set can be queried only at the beginning, and then reused directly when multiple historical answer-historical question pairs are identified. Alternatively, the reward rule set can be queried according to a preset query period (e.g., 1 minute), and the latest reward rule set can be used after each query.

[0064] From the above, we can obtain the current set of reward rules, which is the latest set of reward rules.

[0065] Step S220: Using a reward scoring model, the reasoning steps in historical answers are evaluated based on the current set of reward rules to obtain the corresponding reward score; the historical answers are obtained by processing historical questions using a strategy model.

[0066] It should be noted that in the text, "strategy model" may be replaced with "first major language model," and "reward scoring model" with "second major language model." Furthermore, the use of "first" in "first major language model," as well as "second," "third," and similar terms elsewhere in the text, are merely for distinguishing similar items and do not serve any ranking or other limiting function. Additionally, this step corresponds to... Figure 1 Steps a1 and a2 are shown in the figure.

[0067] The modalities of the aforementioned historical questions and answers are generally text-based. Alternatively, they can include voice, images, or video, in which case relevant components need to be added and combined with the policy model. For example, assuming voice input is supported, a speech recognition module can be added to convert the user's speech into text before inputting it into the policy model. Similarly, assuming image input or generation is supported, an image encoder or image decoder can be added accordingly.

[0068] The content of historical questions and answers is adapted to the specific business domain or scenario in which the strategy model is applied. For example, in the field of intelligent customer service, historical questions may include user-inputted inquiries, and historical answers may include responses from electronic customer service (generated by the strategy model). As another example, in the field of information retrieval, historical questions may include user-inputted search information, and historical answers may include search feedback generated by the strategy model. Yet another example, in the field of smart healthcare, historical questions may include medical texts (such as patient complaints and medical history records), medical imaging data, and various physiological or examination indicators, while historical answers may include auxiliary diagnostic analyses, differential diagnostic suggestions, or treatment plan recommendations generated by the strategy model.

[0069] There are multiple sources for historical questions and answers. One possible source is collection during online service provision (such as intelligent customer service or search service) using a strategy model. Accordingly, in this step, pre-stored historical question-and-answer pairs can be directly read. It should be noted that all user-related data mentioned in this text is legally collected and used after user authorization. Another possible source is real-time generation of corresponding answers. Specifically, in this step, a strategy model is invoked to process a large number of questions, thereby obtaining answers for each question.

[0070] It is understandable that historical questions generally include prompts (for example, these can be added to the user's original input by the system when calling the policy model to form a complete prompt), which are used to guide the policy model to generate reasoning steps in the answer.

[0071] For the policy model, in one possible case, it is an initialized, untrained large language model, such as a basic pre-trained large language model, or a large language model trained using SFT (for a specific business task). In another possible case, it can be a large language model trained several times using RL. It should be understood that "several" in this text refers to one or more, without specifying a particular numerical value.

[0072] The reward scoring model is used to score the reasoning steps in historical answers. Generally, a score is given for each of the several reasoning steps contained in the historical answer. It can be understood that compared with policy models used to handle complex tasks, reward scoring models can be models with a smaller number of parameters. Furthermore, it can be a pre-trained large oracle model as a basis, or it can be a large language model that has undergone SFT (for reward evaluation tasks).

[0073] In this step, the current set of reward rules, historical questions, and historical answers can be filled into a preset prompt template to obtain a complete reward prompt. This reward prompt is then input into the reward scoring model to obtain the corresponding reward score. For example, the prompt template can be:

[0074] You are a judge who evaluates the quality of an AI model's reasoning process. Your task is to provide fine-grained scores for an AI assistant's answers to user questions and its reasoning steps, based on given reward rules.

[0075] #Please strictly adhere to the following rules for evaluation:

[0076] {Insert reward rules here}

[0077] #Items to be evaluated:

[0078] User question: {Insert historical question here}

[0079] AI assistant's answer (including reasoning steps): {Insert historical answer here}

[0080] #Output Requirements:

[0081] Please strictly adhere to the following JSON format when outputting your evaluation results; do not output any other content:

[0082] {Insert JSON format here}

[0083] The above provides a template example of reward prompts.

[0084] Furthermore, for example, assuming that the historical answers include reasoning steps A, B, and C, the output of the reward scoring model may accordingly include reward score A, reward score B, and reward score C.

[0085] From the above, we can obtain the reward score corresponding to any pair of historical questions and historical answers.

[0086] Step S230: Train the strategy model based on the historical questions, historical answers, and reward scores.

[0087] Specifically, the parameters of the policy model can be updated based on training samples consisting of the historical questions, historical answers, and reward scores. Furthermore, this step corresponds to... Figure 1 Step a3 is shown in the figure.

[0088] This step implements the inner-layer optimization (policy learning) in the proposed improved co-evolutionary game model. Its goal is: given the current set of reward rules... In the defined evaluation environment, find a better strategy. This makes the response trajectory generated by the strategy... The trajectory formed by multiple reasoning steps in the answer yields the maximum expected cumulative reward. The mathematical expression of this optimization problem is as follows:

[0089] (1)

[0090] in, It is a strategy model Generated reasoning trajectory (historical answers), actions Reference The first in Each reasoning step, environmental state Referential generation action The context of time ( (For historical issues) It is the reward scoring model in the current reward rule set Under what circumstances, regarding the environmental state Next reasoning steps Reward rating; This represents the discount factor, with a value range of [0,1]. The specific value can be set by the staff. This indicates the expected cumulative reward.

[0091] Furthermore, policy gradient algorithms and other methods can be used to update the policy model. The parameters.

[0092] For example, algorithms such as Proximal Policy Optimization (PPO) or Grade-Reward Policy Optimization (GRPO) can be used to maximize Calculate the parameters of the policy model for the target. The gradient estimate is obtained, and the gradient direction points to the parameter space that can increase the probability of high-reward responses. This gradient estimate is then used to adjust the parameters of the policy model. Iterative updates will be performed.

[0093] This allows for parameter updates to the strategy model, driving continuous optimization of its behavioral patterns.

[0094] It should be noted that the reward scoring model can be updated during the RL learning process, or it may not. In fact, the reward scoring model only needs basic semantic understanding and logical judgment capabilities to function effectively. This is because the core engine driving the evolution of evaluation criteria is not the model's own parameters, but rather the dynamically updated set of reward rules. These rules, in the form of natural language, are injected into the reward scoring model through prompt word templates, thereby dynamically shaping its scoring behavior. This architecture, which externalizes "evaluation knowledge" into the rule set rather than fixing it into the model parameters, enables this solution to achieve efficient and low-cost co-evolution.

[0095] The above describes the workflow steps for any RL learning iteration within the RL training loop (inner loop). Next, we will introduce the workflow steps involved in rule evolution and update (outer loop).

[0096] See Figure 3 The diagram illustrates that any rule update in the outer loop involves the following steps:

[0097] Step S310: Based on the training sample library, sample a batch of abnormal samples whose reward scores meet the preset abnormal conditions; wherein each training sample includes a sample question, the answer output by the policy model for the sample question, and a reward score; the reward score is obtained by using a reward scoring model to evaluate the reasoning steps in the sample answer according to the current reward rule set.

[0098] It should be understood that the training samples in the training sample library can be constructed during the internal loop. The specific construction process can be found in the relevant descriptions in the preceding embodiments, and will not be repeated here. Furthermore, this step corresponds to... Figure 1 Step b1 is shown in the diagram.

[0099] The following provides an exemplary description of the aforementioned preset abnormal conditions. Those skilled in the art can select or combine the following abnormal condition examples as needed:

[0100] Anomaly Condition 1: The reward score in the training sample is lower than the lower score threshold, and / or the reward score is greater than the upper score threshold. It should be noted that this condition can capture samples with abnormal quality.

[0101] Scores below the lower threshold indicate extremely poor quality of responses generated by the strategy model, with its reasoning process severely violating the basic requirements of the current reward rule set (e.g., factual errors, logical inconsistencies, or complete deviation from the question). These samples are negative examples that need to be corrected.

[0102] A score exceeding the upper threshold indicates that the answer received an unusually high reward. This could be due to the strategy model accidentally generating an excellent answer, but it's more likely a potential sign of reward hacking—the strategy model may have found and exploited a vulnerability in the current reward rule set, obtaining an unreasonably high score through an unexpected method (such as keyword stuffing or using specific sentence templates). Such samples provide crucial clues for analyzing rule vulnerabilities and optimizing evaluation criteria.

[0103] Abnormal condition 2: The probability of generating reward scores in the training samples is less than the probability threshold.

[0104] It should be noted that this condition is used to capture anomalies in evaluation uncertainty. Generation probability refers to the predicted probability of a token when the reward scoring model generates a specific reward score for a given inference step. A low generation probability means the model is "hesitant" or "uncertain" when giving this score, resulting in a low-reliability evaluation. This typically occurs when the policy model generates novel, complex, or fuzzy inference patterns that are not adequately covered in typical scenarios of the current rule set or in the training data of the reward scoring model.

[0105] The purpose of sampling such samples is to proactively identify blind spots in the evaluation system and provide data support for further refining rule descriptions or enhancing the judgment confidence of reward scoring models.

[0106] Abnormal condition 3: For the multiple reasoning steps included in the answers in the training samples, the variance among the multiple reward scores corresponding to them is greater than the variance threshold.

[0107] It should be noted that this condition is used to detect anomalies in the consistency of the reasoning process. A high-quality reasoning chain should have relatively stable or reasonable reward scores for each step. Excessive variance indicates unstable reasoning quality. For example, an answer might contain a logically sound step (high score) followed by a logically flawed or erroneous step (low score). Such a mixed reasoning process, while potentially leading to a correct answer, raises questions about its reliability.

[0108] Sampling such samples helps to refine the correction of local errors in the reasoning process and promotes the evolution of rule sets towards encouraging "high-quality reasoning throughout the process," rather than just pursuing the correctness of the final result.

[0109] Anomaly 4: The answers in the training samples are identified by the audit model (or the fourth language model) as having obtained excessively high reward scores by exploiting rule loopholes.

[0110] It should be noted that this condition is used to proactively detect and verify potential reward-based hacking activities. The fourth language model plays the role of an independent auditor in this solution, and its design can follow these principles:

[0111] Independence: This model remains independent of the dynamically updated reward rule set and reward scoring model. It does not enforce specific scoring rules, but instead makes judgments based on a stable, universal, high-quality reasoning principle, thus providing a neutral perspective unaffected by potential flaws in the current rule set.

[0112] Fixed parameters: To ensure the consistency and objectivity of audit standards, the parameters of this model remain fixed throughout the audit process, avoiding judgment biases introduced by their own variations. This ensures that outliers discovered in different periods are identified under the same benchmark, making the evolution of the rule set more stable and traceable.

[0113] The core effectiveness of this auditing model lies in its specialized prompts, which are designed to identify opportunistic patterns in the reasoning process. An example of the auditing model's prompts is as follows:

[0114] You are a senior reasoning auditor. Your task is to analyze given AI responses to determine whether they are obtained through shortcuts or loopholes rather than sound reasoning. Focus on the following:

[0115] 1. Substance: Do the reasoning steps actually lead to a solution to the problem? Or are they merely repeating the problem, piling up irrelevant information, or using empty template sentences?

[0116] 2. Logical soundness: Are there circular arguments, fallacies, or logical leaps in the reasoning chain?

[0117] 3. Honesty: Does the answer involve pretending to know what you don't, fabricating facts or steps to appease the question?

[0118] #Items to be audited:

[0119] Question: {Insert sample question here}

[0120] Answer: {Insert sample answer here}

[0121] #Output Requirements:

[0122] Please output only one of the following two conclusions:

[0123] Conclusion: Opportunistic behavior exists.

[0124] Conclusion: The reasoning process is solid.

[0125] When an answer from the strategy model is deemed "opportunistic" by the audit model, it will be marked as an anomalous sample even if it scores highly under the current reward model. This implements a crucial cross-validation mechanism that effectively identifies behaviors that have successfully "deceived" the current reward rule set, providing the most direct feedback source for subsequent targeted patching of rule vulnerabilities.

[0126] Abnormal condition 5: The answers in the training samples triggered the preset heuristic filtering rules.

[0127] It should be noted that this condition is used to quickly and efficiently capture samples that have obvious defects in form or structure. These defects are usually not effectively identified by semantic-based reward models, or their definitions are too specific to be included in a general set of reward rules.

[0128] The heuristic filtering rules are a set of predefined, formalized criteria based on pattern matching or simple logical judgments. They serve as an effective supplement to semantic evaluation systems, focusing on checking the surface quality and standardization of responses. These rules can be implemented directly in the program logic without invoking large language models, thus enabling batch and rapid filtering with extremely low computational cost.

[0129] Rules Examples: 1. Formatting Correctness: For example, in code generation tasks, check if the brackets in the generated code snippets match, if the indentation is severely inconsistent, or if there are obvious syntax error markers. 2. Structural Integrity: For example, check if the reasoning path is unexpectedly truncated in the middle, or if the final answer is missing. 3. Symbol Abuse: For example, check if the answer contains an abnormally large number of repeated punctuation marks (such as "!!!", "..."), which may be a speculative attempt to emphasize to cover up empty content. 4. Keyword Violation: For example, check if the answer contains words from a pre-set list of banned or sensitive words.

[0130] When a response in the training samples triggers any of the heuristic filtering rules, it is marked as an anomalous sample. These samples represent systematic deviations in the basic generation quality or prescriptiveness of the policy model. Sampling them helps to quickly correct these low-level errors and improve the overall robustness and usability of the model output.

[0131] Based on the aforementioned pre-defined anomaly conditions, anomalous samples can be identified from the training sample library (forming an anomalous sample sub-library). Furthermore, a batch of anomalous samples can be sampled from the identified anomalous samples. This sampling operation is performed primarily for two reasons: First, to control analysis and computational costs. The anomalous sample sub-library may be enormous, and directly analyzing its entire set would result in extremely high computational and token consumption. By sampling a representative set of manageable size, the efficiency and economy of the rule update process can be significantly improved. Second, to enhance the quality and focus of subsequent analysis stages. Large language models exhibit greater analytical depth and reliability when processing inputs with high information density and moderate context length. A refined sample set helps the model eliminate noise interference and more accurately identify common problems and root causes.

[0132] The sampling methods are quite flexible. For example, random sampling can be used to ensure that each anomalous sample has an equal probability of being selected, thus unbiasedly representing the overall composition of the sub-database. Alternatively, a stratified sampling strategy can be adopted, that is, based on key attributes such as anomaly type, reward score range, and problem domain, the anomalous sample sub-database is first divided into several levels, and then sampling is performed from each level, thereby ensuring that the sampled sample can comprehensively cover the characteristics of different types of anomalies. In this way, through the above statistical sampling methods, it can be ensured that the final batch of samples can effectively infer the general characteristics of the entire anomalous sample sub-database from a statistical perspective.

[0133] Based on the above, a batch of abnormal samples can be collected.

[0134] Step S320: Process the batch of anomalous samples (multiple anomalous samples) using a reward design model (or the third major language model) to obtain optimization suggestions for the current reward rule set. It should be understood that this step corresponds to... Figure 1 Steps b2 and b3 are shown in the diagram.

[0135] It should be noted that the reward design model can include one or more independent, powerful large language models whose dedicated task is reward design. It automatically synthesizes or revises PRM rules for evaluation by analyzing "bad samples" in a database. This model elevates the concept of "LLM-as-a-Judge" from passive scoring to proactively and dynamically developing and refining evaluation criteria.

[0136] Typically, the batch of abnormal samples and the current set of reward rules can be filled into a prompt word template set for the rule optimization task, and then the complete prompt word obtained can be input into the third language model to obtain the optimization suggestions.

[0137] In one embodiment, the prompt word template may include a task description for the rule optimization task.

[0138] In one embodiment, the prompt word template may also include a difficulty parameter, which is used to adjust the overall strictness and emphasis of the reward rule set. For example, increasing the difficulty (such as the instruction "pursue high accuracy") will cause RDM to generate more stringent rules (e.g., adding a rule to code generation to "impose severe penalties on unverified user input"), driving the model to output more reliable results; decreasing the difficulty (such as the instruction "ensure basic accuracy") will cause RDM to produce a more lenient rule set, balancing output quality and efficiency. In this way, it can flexibly adapt to application scenarios with different complexity and quality requirements, from general consulting to high-end services.

[0139] For example, the prompt word template can be:

[0140] #Task Description

[0141] You are a reward mechanism design expert responsible for continuously improving the rule system used to evaluate the quality of policy model inference. The primary goal of this optimization is {difficulty parameter inserted here}. Based on the following input, analyze the shortcomings of the current reward rules and generate specific, actionable rule optimization suggestions.

[0142] {Place the current set of reward rules here}

[0143] The following are trajectory samples generated by the strategy model and identified as reward evaluation anomalies (each trajectory includes a question, answer, and corresponding failure type):

[0144] {This area is filled with a batch of abnormal samples}

[0145] #Output Requirements:

[0146] Output only structured optimization suggestions, without explanations or small talk;

[0147] Each suggestion must fall into one of the following three categories: adding a new rule, modifying an existing rule, or abolishing an invalid rule;

[0148] Each suggestion should include:

[0149] 1. Type: [Add / Modify / Deprecate]

[0150] 2. Target scenarios: (e.g., "multi-step reasoning", "tool invocation", "fact checking", etc.)

[0151] 3. Rule content: Use clear and unambiguous natural language to ensure that the process reward model can score the reasoning steps accordingly;

[0152] 4. Triggering conditions: Specify under what reasoning behavior the rule is activated (e.g., "when an API call that is not related to the context appears in the answer").

[0153] 5. Expected Results: Briefly describe the behaviors that this rule aims to suppress or encourage (e.g., "reducing redundant tool calls" or "improving logical coherence").

[0154] Furthermore, by using the prompts constructed using the aforementioned prompt template, optimization suggestions can be obtained for the current set of reward rules. It can be understood that these optimization suggestions include one or more of the following: adding new rules, modifying existing rules, and deleting existing rules.

[0155] For example, optimization suggestions for a particular output may include:

[0156] 1. Add new rules

[0157] Type: New

[0158] New rule number: R-108

[0159] Target scenario: Security checks during code generation

[0160] Rule content: If the generated code contains unverified user input directly concatenated to a system command (such as os.system("rm -rf " + user_input)), a significant negative reward will be applied to that inference step.

[0161] Triggering condition: High-risk function calls (such as exec, eval, os.system) and concatenation of external variables are detected in the code snippet.

[0162] Expected result: Suppress the generation of code with security vulnerabilities by the strategy model, and improve output security.

[0163] 2. Modify existing rules

[0164] Type: Modify

[0165] Original rule number: R-42

[0166] New rule number: R-42 (number retained, content updated)

[0167] Target scenario: Tool call chain

[0168] Original rule (R-42): [omitted here]

[0169] Revised rules: [omitted here]

[0170] Triggering conditions: [omitted here]

[0171] Expected results: [omitted here]

[0172] Note: Rule number R-42 remains unchanged, only the content is updated, and it is suitable for scenarios where the semantics of the rule continue but need to be refined.

[0173] 3. Delete existing rules (deprecated)

[0174] Type: Discarded

[0175] Original rule number: R-15

[0176] Target scenario: General dialogue generation

[0177] Original rule (R-15): [omitted here]

[0178] Obsolete operation: Remove R-15 from the current set of reward rules.

[0179] Alternative suggestion: Its functionality should be covered by the new rule R-109 (semantic redundancy detection).

[0180] Expected results: [omitted here].

[0181] The above provides an exemplary description of the prompt word template used in this step.

[0182] On the other hand, the sample portion of the reward design model input can include only one batch of outlier samples. Alternatively, it can include one or more "good samples" (such as samples with high reward scores and judged by the audited model as having "solid reasoning processes"). This design can achieve the following:

[0183] 1. Provide benchmark references for correction direction: Provide clear and positive behavioral examples for rule correction, ensuring that new or modified rules are not only used to "block" erroneous reasoning paths, but also to "consolidate and guide" correct reasoning patterns.

[0184] 2. Maintaining the balance and guidance of reward signals: By having the reward design model analyze both "why it's good" and "why it's bad," a more discriminative rule system can be generated. This system can clearly distinguish between "good reasoning that should be encouraged" and "speculative behavior that should be suppressed," thereby providing a more accurate and balanced optimization gradient for the strategy model.

[0185] 3. Enhance the generalization ability and robustness of new rules: By comparing positive and negative samples, reward design models can more profoundly abstract the essential characteristics of high-quality reasoning, rather than simply memorizing specific error patterns. This helps generate more universal rules, reduces overfitting to historical abnormal patterns, and enables the updated reward rule set to have stronger judgment and adaptability when facing new behaviors that the policy model may produce in the future.

[0186] In summary, mixing some high-quality positive samples into the abnormal samples can guide the reward design model to conduct comparative analysis, making its output rule optimization suggestions not only more targeted but also more constructive, ultimately driving the policy model and the reward rule set to achieve co-evolution at a higher quality equilibrium point.

[0187] Based on the above, we can derive optimization suggestions for the current set of reward rules.

[0188] Step S330: Update the current reward rule set based on the optimization suggestions.

[0189] It should be understood that this step involves structured parsing and automated updating of the rule set for the optimization suggestions generated in step S320. Furthermore, this step corresponds to... Figure 1 Step b4 is shown in the diagram.

[0190] Typically, the system receives structured optimization suggestions (including instructions to add, modify, or deprecate rules) output by the reward design model, and performs corresponding add, delete, and modify operations on the current set of reward rules based on the rule number, target scenario, and rule content explicitly specified in each suggestion:

[0191] For new rules, assign a new rule number (such as R-108) and add it to the rule set.

[0192] To modify a rule, locate the existing entry based on the original rule number (e.g., R-42) and replace the original rule text with the new content.

[0193] For obsolete rules, the corresponding entries are removed from the rule set according to the original rule number (e.g., R-15).

[0194] Furthermore, for example, after the update is completed, the new set of reward rules can be synchronized to the PRM rule cache server, so that the reward scoring model can call it in real time during subsequent reinforcement learning training, thereby realizing the closed-loop iterative optimization of the reward mechanism.

[0195] The above allows for the evolution and updating of the reward rule set.

[0196] Furthermore, Figure 4 The dual-loop collaborative architecture of the policy model training system is shown, consisting of a high-speed internal reinforcement learning loop and a low-speed asynchronous external rule evolution loop.

[0197] Within the inner loop, the policy model (denoted as π_θ) generates inference steps for the input problem and requests a reward from the dynamic process reward model (PRM_D). PRM_D queries the PRM rule cache server in real-time to obtain the latest reward rule set (R_t), combines the inference steps with R_t to form a reward hint, calculates the reward score, and then returns this score to the policy model. Based on this reward score, the policy model updates its parameters using reinforcement learning algorithms such as GRPO or PPO. This loop runs continuously at a high frequency, constituting a short-cycle policy optimization process.

[0198] In the outer loop, the system records trajectories identified as "bad samples" to the bad sample database (corresponding to the above-mentioned abnormal sample sub-database) with a certain probability during the training process. When the asynchronous triggering condition is met, the reward design model (RDM) periodically samples a batch of abnormal trajectories from the bad sample database, analyzes the failure modes in them, and synthesizes new or corrected rules (denoted as ∆R). Subsequently, the RDM pushes the updated rules to the PRM rule cache server to achieve iterative optimization of the reward rule set (R_t).

[0199] It should be noted that the training cycle of the policy model (π_θ) is shorter than the update cycle of the reward rule set (R_t). That is, the frequency of the external rule update cycle is much lower than that of the internal RL training cycle, constituting a typical asynchronous update mechanism. This design allows the policy model sufficient time to learn and converge in a stable reward environment, avoiding training oscillations or policy degradation caused by frequent or drastic changes in the reward function.

[0200] The two loops are linked by a bad sample database and a PRM rule cache server: bad samples generated by the inner loop drive the rule evolution of the outer loop, while the rules updated by the outer loop are fed back to the inner loop in real time to guide subsequent policy training, thus forming a stable, efficient, and self-evolving closed-loop training system.

[0201] In summary, the improved solutions disclosed in the embodiments of this specification can bring significant beneficial effects from both the perspectives of mechanism innovation and overall performance.

[0202] At the mechanism level, this plan achieves five core breakthroughs:

[0203] 1. Dynamic reward paradigm: It completely abandons the static PRM model of "train once, use forever" in RLHF, and instead generates reward signals in real time based on a set of dynamically evolving rules, which can respond to changes in the behavior of the policy model in real time.

[0204] 2. Co-evolution of strategy and reward: For the first time, the strategy model and the reward rule set (represented by the reward design model) are constructed as an asymmetric, game-like co-evolutionary system. The strategy model pursues reward maximization, while the reward rules continuously evolve to "block" its speculative behavior, and eventually they both tend to a more robust and aligned Nash equilibrium.

[0205] 3. Automated Reward Design and Repair: An independent Reward Design Model (RDM) is introduced, transforming its role from a passive "referee" to an active "rule designer." It can automatically synthesize higher-level reward rules based on failure cases, providing a scalable automated path to solve the problems of insufficient generalization of reward models and adversarial attacks.

[0206] 4. Decoupled dual-loop asynchronous architecture: By separating the high-speed internal RL training loop from the low-speed external rule update loop, the policy optimization and reward optimization are decoupled. While ensuring training efficiency, the system stability is improved by the time scale difference, effectively avoiding negative feedback crashes.

[0207] 5. System Stability and Convergence Guarantee: As a dynamic system, this solution effectively avoids the risk of negative feedback loop collapse that may occur in traditional methods (e.g., overly strict rules prevent the model from learning, or rule loopholes cause a sharp drop in performance) and ensures robust system convergence through the following points: ① Asynchronous Update Mechanism: The frequency of the external rule update loop is much lower than that of the internal RL training loop, giving the policy model sufficient adaptation time and avoiding training oscillations caused by drastic changes in the reward function. ② Global Perspective of Reward Design Model: RDM does not react to a single bad sample, but analyzes a batch of samples to extract general and robust rules. This rule generation method based on pattern recognition and with predictive capabilities fundamentally suppresses unstable negative feedback. ③ Clear Convergence Objective: The ideal convergence state of the system is a Nash equilibrium. In this ideal convergence state, neither side can benefit from unilaterally changing the policy: the policy model cannot find rule loopholes to obtain higher rewards, and the rule set has fully covered its various behaviors, achieving a dynamic balance.

[0208] The aforementioned mechanism innovations directly translate into significant comprehensive technological effects:

[0209] 1. Significantly improved training efficiency: Compared with random mixed training, the convergence speed is improved by 40-60%, and the number of training samples required to achieve the same performance is reduced by 30-40%.

[0210] 2. Comprehensive enhancement of model capabilities: The completion rate of complex tasks is increased by more than 50%, the accuracy of tool invocation is improved by 35-45%, and the multi-step reasoning ability is significantly improved.

[0211] 3. Enhanced system scalability: It can be easily migrated to new task scenarios, supports rapid integration of new tools, and the difficulty parameters can be flexibly adjusted according to application requirements.

[0212] 4. Significantly optimized cost-effectiveness: Data construction costs are reduced by approximately 80%, and training resource consumption is reduced by 30–40%.

[0213] Therefore, the embodiments in this specification not only break through the static reward bottleneck of traditional RLHF in principle, but also demonstrate the engineering advantages of high efficiency, robustness and low cost in actual deployment.

[0214] Corresponding to the rule update method and model training method described above, embodiments of this specification also disclose a rule update device and a model training device.

[0215] Figure 5 An updating device 500 for the reward rule set is shown, which includes the following functional units:

[0216] An abnormal sample sampling unit 510 is configured to sample a batch of abnormal samples whose reward scores meet preset abnormal conditions based on a training sample library. Each training sample includes a sample question, the answer output by a first language model for the sample question, and a reward score. The reward score is obtained by evaluating the reasoning steps in the sample answer using a second language model based on the current reward rule set. An optimization and prediction unit 520 is configured to process the batch of abnormal samples using a third language model to obtain optimization suggestions for the current reward rule set. A reward rule update unit 530 is configured to update the current reward rule set based on the optimization suggestions.

[0217] In one embodiment, the preset abnormal conditions include: the reward score is lower than the lower limit threshold, and / or the reward score is greater than the upper limit threshold.

[0218] In one embodiment, the preset abnormal condition includes: the probability of the second language model generating a reward score is less than a probability threshold.

[0219] In one embodiment, the preset abnormal condition includes: the variance among multiple reward scores corresponding to multiple reasoning steps in the answer is greater than a variance threshold.

[0220] In one embodiment, the preset abnormal condition includes: the answer is identified by the fourth language model as obtaining an excessively high reward score by exploiting a rule loophole.

[0221] In one embodiment, the abnormal sample sampling unit 510 is specifically configured to: sample a batch of abnormal samples based on the abnormal sample sub-library in the training sample library.

[0222] In one embodiment, the optimization prediction unit 520 is specifically configured to input the batch of abnormal samples, the current set of reward rules, and the task description for the rule optimization task into the third language model to obtain the optimization suggestions.

[0223] In one embodiment, the optimization suggestion includes one or more of the following: adding new rules, modifying existing rules, and deleting existing rules.

[0224] Figure 6 A reinforcement learning-based model training device 600 is shown, which includes the following functional units:

[0225] The reward rule query unit 610 is configured to query the current set of reward rules, and it adopts... Figure 5 The update device 500 shown is used to update the reward score. The reward score prediction unit 620 is configured to use a second language model to evaluate the reasoning steps in historical answers based on the current reward rule set to obtain the corresponding reward score; the historical answers are obtained by processing historical questions using a first language model. The policy model update unit 630 is configured to train the first language model based on the historical questions, historical answers, and reward scores.

[0226] In one embodiment, the training period of the first large language model is shorter than the update period of the current reward rule set.

[0227] In one embodiment, the reward score prediction unit 620 is specifically configured to: fill the current reward rule set, historical questions and historical answers into a preset prompt word template to obtain a complete prompt word; input the prompt word into the second large language model to obtain the reward score.

[0228] In one embodiment, the policy model update unit 630 is specifically configured to: use a policy gradient algorithm to update the parameters of the first large language model based on the training samples consisting of the historical questions, historical answers, and reward scores.

[0229] It should be noted that for a description of the above functional units, please refer to the relevant description of the process method in the foregoing embodiments.

[0230] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them outstanding generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0231] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0232] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0233] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform... Figure 2 or Figure 3 The method described.

[0234] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement... Figure 2 or Figure 3 The method described.

[0235] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0236] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for updating a set of reward rules, comprising: Based on the training sample library, a batch of abnormal samples whose reward scores meet the preset abnormality conditions are sampled. Each training sample includes a sample question, the answer output by the first language model for the sample question, and a reward score; the reward score is obtained by using the second language model to evaluate the reasoning steps in the sample answer according to the current reward rule set. The third language model is used to process the batch of abnormal samples to obtain optimization suggestions for the current set of reward rules; Based on the optimization suggestions, update the current set of reward rules.

2. The method according to claim 1, wherein, The preset abnormal conditions include: the reward score is lower than the lower limit threshold, and / or the reward score is greater than the upper limit threshold.

3. The method according to claim 1, wherein, The preset abnormal conditions include: the probability of the second language model generating a reward score is less than the probability threshold.

4. The method according to claim 1, wherein, The preset abnormal conditions include: the variance among multiple reward scores corresponding to multiple reasoning steps in the answer is greater than the variance threshold.

5. The method according to claim 1, wherein, The preset abnormal conditions include: the answer is identified by the fourth language model as obtaining an excessively high reward score by exploiting a rule loophole.

6. The method according to claim 1, wherein, Based on the training sample library, a batch of abnormal samples whose reward scores meet the preset anomaly conditions are sampled, including: Based on the abnormal sample sub-library in the training sample library, a batch of abnormal samples is sampled.

7. The method according to claim 1, wherein, The batch of abnormal samples is processed using a third language model to obtain optimization suggestions for the current reward rule set, including: The batch of abnormal samples, the current set of reward rules, and the task description for the rule optimization task are input into the third language model to obtain the optimization suggestions.

8. The method according to claim 1, wherein, The optimization suggestions include one or more of the following: adding new rules, modifying existing rules, and deleting existing rules.

9. A model training method based on reinforcement learning, comprising: The current set of reward rules is queried and updated using the method described in claim 1; Using the second language model, the reasoning steps in historical answers are evaluated based on the current reward rule set to obtain the corresponding reward score; the historical answers are obtained by processing historical questions using the first language model. The first large language model is trained based on the historical questions, historical answers, and reward scores.

10. The method according to claim 9, wherein, The training cycle of the first large language model is shorter than the update cycle of the current reward rule set.

11. The method according to claim 9, wherein, Using the second language model, the reasoning steps in historical responses are evaluated based on the set of reward rules to obtain the corresponding reward scores, including: The current set of reward rules, historical questions, and historical answers are filled into a preset prompt template to obtain complete prompts; The prompt word is input into the second large language model to obtain the reward score.

12. The method according to claim 9, wherein, Based on the historical questions, historical answers, and reward scores, the first large language model is trained, including: The policy gradient algorithm is used to update the parameters of the first large language model based on the training samples consisting of the historical questions, historical answers and reward scores.

13. An update device for a set of reward rules, comprising: An abnormal sample sampling unit is configured to sample a batch of abnormal samples whose reward scores meet preset abnormal conditions based on the training sample library; Each training sample includes a sample question, the answer output by the first language model for the sample question, and a reward score; the reward score is obtained by using the second language model to evaluate the reasoning steps in the sample answer according to the current reward rule set. The prediction unit is optimized and configured to use the third language model to process the batch of abnormal samples, so as to obtain optimization suggestions for the current reward rule set. The reward rule update unit is configured to update the current reward rule set based on the optimization suggestions.

14. A model training device based on reinforcement learning, comprising: The reward rule query unit is configured to query the current reward rule set, which is obtained by updating using the apparatus described in claim 13; The reward score prediction unit is configured to use a second language model to evaluate the reasoning steps in historical answers based on the current reward rule set, and obtain the corresponding reward score; the historical answers are obtained by processing historical questions using a first language model. The strategy model update unit is configured to train the first large language model based on the historical questions, historical answers, and reward scores.

15. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed in the computer, it causes the computer to perform the method according to any one of claims 1-12.

16. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-12.

Images