Task response determination method and device, storage medium and electronic equipment

Abstract

The invention discloses a task response determination method and device, a storage medium and electronic equipment, and the method comprises the steps: inputting a target cue word of a target task into a target model, so as to determine the task response of the target task through the target model; wherein the target model is obtained by training in the following mode: determining training prompt words included in a training task; inputting the training prompt words into a first model included in the initial model to obtain a plurality of first reasoning tracks; determining a target reward value based on a reference reasoning trajectory and the first reasoning trajectory, the reference reasoning trajectory being a trajectory obtained by reasoning the target cue word by a pre-trained reference model; and iteratively updating parameters of the initial model based on the target reward value to obtain a target model. According to the task output method and device, the problem that the response of the output task is inaccurate due to the fact that the complex task is not trained is solved, and then the effect of accurately outputting the response of the task is achieved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, storage medium, and electronic device for determining a task response. Background Technology

[0002] In related technologies, users can input task prompts into a model and obtain the model's response. However, during the model training process, complex tasks that the model cannot solve are often discarded, resulting in the model being unable to provide accurate responses to complex tasks during application.

[0003] This indicates that the relevant technologies suffer from inaccurate responses to output tasks due to a lack of training on complex tasks.

[0004] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for determining task responses, in order to at least solve the technical problem in the related art of inaccurate output task responses caused by the lack of training on complex tasks.

[0006] According to one aspect of the embodiments of this application, a method, apparatus, storage medium, and electronic device for determining a task response are provided, comprising: inputting target prompt words of a target task into a target model to determine the task response of the target task through the target model; wherein the target model is trained by: determining training prompt words included in a training task; inputting the training prompt words into a first model included in an initial model to obtain a plurality of first inference trajectories; determining a target reward value based on reference inference trajectories and the first inference trajectories, wherein the reference inference trajectories are trajectories obtained by inferring the target prompt words by a pre-trained reference model, and the inference accuracy of the reference model for the target prompt words satisfies a first preset threshold; and iteratively updating the parameters of the initial model based on the target reward value to obtain the target model.

[0007] In one exemplary embodiment, determining a target reward value based on a reference inference trajectory and a first inference trajectory includes: determining an attribute value for each first inference trajectory to obtain a plurality of first attribute values; determining an attribute value for the reference inference trajectory to obtain a second attribute value; determining an attribute average of the plurality of first attribute values ​​and the second attribute value; determining an attribute standard deviation based on the attribute average; determining a reward value for each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute values ​​to obtain a plurality of first reward values; determining a second reward value for the reference inference trajectory based on the attribute average, the attribute standard deviation, the plurality of first reward values, and the second attribute value; and determining the plurality of first reward values ​​and the second reward value as the target reward value.

[0008] In an exemplary embodiment, determining a first reward value for each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute value includes: performing the following operations for each first inference trajectory to determine the first reward value corresponding to the first inference trajectory: determining a first difference between the first attribute value of the first inference trajectory and the attribute average; and determining a first ratio of the first difference to the attribute standard deviation as the first reward value corresponding to the first inference trajectory.

[0009] In one exemplary embodiment, determining a second reward value for the reference inference trajectory based on the attribute average, the attribute standard deviation, a plurality of first reward values, and the second attribute value includes: determining the maximum reward value included among the plurality of first reward values; determining a first sum of the maximum reward value and a preset parameter; determining a second difference between the second attribute value and the attribute average; determining a second ratio between the second difference and the attribute standard deviation; and determining the minimum value between the second ratio and the first sum as the second reward value.

[0010] In an exemplary embodiment, iteratively updating the parameters of the initial model based on the target reward value to obtain the target model includes: determining a first loss value based on a plurality of first reward values ​​and a second reward value included in the target reward value; determining a second loss value based on the second reward value; determining the product of the second loss value and hyperparameters to obtain a first product; determining the sum of the first loss value and the first product to obtain a second sum; updating the parameters of the initial model using the second sum to obtain an updated model; and determining the updated model as the target model if the updated model meets preset conditions.

[0011] In an exemplary embodiment, determining a first loss value based on a first reward value and a second reward value included in the target reward value includes: determining the likelihood ratio of each first inference trajectory to obtain multiple first likelihood ratios, wherein the likelihood ratio represents the ratio of the probability of the initial model and the probability of the historical model responding to the preset task, and the historical model is the initial model that has not been trained and updated; determining the likelihood ratio of the reference inference trajectory to obtain a second likelihood ratio; determining the first loss value based on the multiple first likelihood ratios and the second likelihood ratio when the multiple first likelihood ratios are within a preset interval and the second likelihood ratio is within the preset interval; or, adjusting the first loss value based on the preset interval when there is a third likelihood ratio outside the preset interval among the multiple first likelihood ratios and the second likelihood ratio is within the preset interval. The first updated likelihood ratio is obtained by taking three likelihood ratios, and the first loss value is determined based on the first updated likelihood ratio, the second likelihood ratio, and other likelihood ratios, wherein the other likelihood ratios are likelihood ratios other than the third likelihood ratio included in the plurality of first likelihood ratios; or, if the plurality of first likelihood ratios are within a preset interval and the second likelihood ratio is outside the preset interval, the second likelihood ratio is adjusted based on the preset interval to obtain a second updated likelihood ratio, and the first loss value is determined based on the plurality of first likelihood ratios and the second updated likelihood ratio; or, if there is a third likelihood ratio outside the preset interval among the plurality of first likelihood ratios and the second likelihood ratio is outside the preset interval, the first loss value is determined based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio.

[0012] In an exemplary embodiment, determining the first loss value based on a plurality of first likelihood ratios and second likelihood ratios includes: determining the product of each first likelihood ratio and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a third sum of the plurality of first sub-loss values ​​and the second loss values; and determining the third sum as the first loss value.

[0013] In an exemplary embodiment, determining the first loss value based on the first updated likelihood ratio, the second likelihood ratio, and the other likelihood ratios includes: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a fourth sum of the third sub-loss value, the fourth sub-loss value, and the second sub-loss value; and determining the minimum value among the third sum and the fourth sum as the first loss value.

[0014] In an exemplary embodiment, determining the first loss value based on the plurality of first likelihood ratios and the second updated likelihood ratio includes: determining the product of the plurality of first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a fifth sum of the plurality of first sub-loss values ​​and the fifth sub-loss value; and determining the minimum value among the third sum and the fifth sum as the first loss value.

[0015] In one exemplary embodiment, based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio, the method includes: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a sixth sum of the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value; and determining the minimum value among the third sum and the sixth sum as the first loss value.

[0016] In an exemplary embodiment, determining a second loss value based on the second reward value includes: performing the following operations for each sub-inference trajectory included in the second inference trajectory to determine a sub-loss value for the sub-inference trajectory, wherein the second inference trajectory includes the first inference trajectory and the reference inference trajectory: inputting the sub-inference trajectory into a second model included in the target model to obtain multiple initial scores; determining an average score of the multiple initial scores; determining a binary cross-entropy loss value of the average score and the attribute value of the sub-inference trajectory; determining a fifth product of the binary cross-entropy loss value and the likelihood ratio of the sub-inference trajectory as the sub-loss value; determining a sum of multiple sub-loss values ​​to obtain a seventh sum; and determining a third ratio of the seventh sum to the number of sub-inference trajectories as the second loss value.

[0017] According to another aspect of the embodiments of this application, a task response determination apparatus is also provided, comprising: a determination module, configured to input target prompt words of a target task into a target model, so as to determine the task response of the target task through the target model; wherein the target model is trained by: determining training prompt words included in a training task; inputting the training prompt words into a first model included in an initial model to obtain a plurality of first inference trajectories; determining a target reward value based on a reference inference trajectory and the first inference trajectories, wherein the reference inference trajectory is a trajectory obtained by inferring the target prompt words by a pre-trained reference model, and the inference accuracy of the reference model for the target prompt words satisfies a first preset threshold; and iteratively updating the parameters of the initial model based on the target reward value to obtain the target model.

[0018] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed by a processor.

[0019] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the steps in any of the method embodiments described above.

[0020] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to perform the steps of any of the above method embodiments through the computer program.

[0021] This application allows inputting target prompts for a target task into a trained target model, which then outputs the inference trajectory and result (i.e., the task response). The target model is trained as follows: First, the training prompts for the training task are determined and input into the first model included in the initial model to be trained, resulting in a first inference trajectory generated by the first model. Next, a reference inference trajectory for the target prompts is determined. This reference trajectory is the correct inference trajectory obtained by inferring the target prompts using a pre-trained reference model whose inference accuracy for the target prompts meets a first preset threshold. The target reward value is determined using the first and reference inference trajectories, and the parameters of the initial model are iteratively updated using this target reward value to obtain the trained target parameters. Since training can be performed on difficult samples (i.e., the training samples) that the model could not solve during historical training (i.e., a high-quality reference inference trajectory is introduced for guidance), and a restricted dominance function (i.e., the determined target reward value) can be used to balance the relationship between the reference inference trajectory and the first training trajectory generated by the model itself, over-reliance on the reference inference trajectory can be avoided. This approach ensures that the model benefits from the instructions of the reference inference trajectory while maintaining its ability to explore the unknown policy space. It avoids the problem in related technologies where the reference inference trajectory often has an absolute advantage due to its high reward value, causing the model to quickly converge to the expert policy and stop trying to explore more paths. Therefore, it can solve the problem of inaccurate output task response caused by the lack of training on complex tasks in related technologies, and achieve the effect of accurately outputting the task response. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating an application scenario of a method for determining a task response according to an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating an optional task response determination method according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of the system structure for determining task responses according to an embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating the method for determining the task response in this optional example;

[0026] Figure 5 This is a structural block diagram of an optional task response determination device according to an embodiment of this application;

[0027] Figure 6This is a computer system architecture block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The following explains the technical terms that appear in this application:

[0031] Reinforcement Learning (RL): A machine learning paradigm in which an agent can learn how to take actions to maximize cumulative rewards by interacting with its environment and receiving rewards or penalties.

[0032] Large-scale language models (LLMs) can be understood as deep learning models with a huge number of parameters. They are trained on massive amounts of text data and are able to understand and generate natural language.

[0033] Proximal Policy Optimization (PPO): A mainstream reinforcement learning algorithm that limits the step size of policy updates by clipping the importance sampling ratio to ensure training stability.

[0034] Group Relative Policy Optimization (GRPO): A variant of PPO that estimates advantage by normalizing the rewards of a group of candidate responses without requiring a separate value network.

[0035] Online policy: In reinforcement learning, the policy used for evaluation and improvement is the same as the policy used to generate training data.

[0036] Off-Policy: This refers to a strategy used for improvement that differs from the strategy used to generate the training data. The improvement strategy typically utilizes historical data or external expert data.

[0037] Policy Entropy: A metric that measures the randomness of a policy's output action. High entropy represents high exploratory nature, while low entropy represents high determinism.

[0038] Importance Sampling (IS): In off-policy learning, it is a weighting technique used to correct biases caused by inconsistencies between the distributions of behavioral policies and target policies.

[0039] Process Reward Model (PRM): A model that scores the intermediate steps of the generated sequence to provide a more granular learning signal than the reward for the final result.

[0040] According to one aspect of the embodiments of this application, a method for determining a task response is provided. Optionally, in this embodiment, the above-described method for determining a task response may be applied, but is not limited to, to applications such as... Figure 1 The hardware environment shown includes terminal device 102 and server 104. Server 104 can be connected to terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) to terminal device 102 or clients installed on terminal device 102. A database can be set up on server 104 or independently of server 104 to provide data storage services for server 104.

[0041] The aforementioned network may include, but is not limited to, at least one of the following: wired network and wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network (WAN), metropolitan area network (MAN), and local area network (LAN). The aforementioned wireless network may include, but is not limited to, at least one of the following: Wireless Fidelity (WIFI) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), mobile phone, tablet computer, etc. Server 104 may be, but is not limited to, a cloud server, server cluster, or other server types.

[0042] The task response determination method of this application embodiment can be executed by server 104, terminal device 102, or jointly by server 104 and terminal device 102. Alternatively, the task response determination method of this application embodiment can be executed by a client installed on terminal device 102.

[0043] Figure 2 This is a flowchart illustrating an optional task response determination method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0044] Step S202: Input the target prompt words of the target task into the target model, so as to determine the task response of the target task through the target model;

[0045] The target model is trained as follows: training prompts are identified in the training task; the training prompts are input into a first model included in the initial model to obtain multiple first inference trajectories; a target reward value is determined based on a reference inference trajectory and the first inference trajectories, wherein the reference inference trajectory is a trajectory obtained by a pre-trained reference model inferring the target prompts, and the inference accuracy of the reference model for the target prompts meets a first preset threshold; the parameters of the initial model are iteratively updated based on the target reward value to obtain the target model.

[0046] The task response determination method in this embodiment can be applied to the fields of artificial intelligence and machine learning, with a particular focus on leveraging reinforcement learning techniques to enhance the reasoning capabilities of large-scale language models on complex tasks. Specifically, it can be applied to areas such as mathematical problem solving, code generation and debugging, and agent planning and decision-making. In mathematical proof and problem-solving scenarios, it can generate detailed reasoning logic. For example, when solving mathematical problems in competitions such as the American Invitational Mathematics Examination (AIME) and the International Mathematical Olympiad (IMO), the model can generate detailed and rigorous proofs and calculation processes. For programming tasks, the model can generate code, fix vulnerabilities, or optimize algorithms based on complex programming requirements in software engineering tasks that require long-term logical dependencies. In complex virtual or physical environments, it can train agents to perform task planning, decision-making, and execution.

[0047] However, in related technologies, the strategies for handling difficult samples limit the breakthrough of the model's upper limit. That is, although the traditional dynamic sampling mechanism can improve gradient efficiency by filtering out samples that are all correct or all wrong, it also discards difficult samples that the model cannot solve at all (with an on-policy accuracy of 0). Although it can stabilize training in the short term, it deprives the model of the opportunity to learn from the most challenging failures, making it difficult for the model to break through the current capability boundary and limiting the ultimate height of the model's ability to "emerge" through RL.

[0048] To at least partially address the aforementioned technical issues, a novel approach is taken to handle difficult samples. Unlike related technologies that directly discard difficult samples that the model cannot fully resolve, the dynamic batch construction module in this application intentionally retains and labels these samples. For difficult samples, the hybrid Rollout controller can conditionally introduce a high-quality off-policy trajectory from an external expert database for guidance. Finally, a restricted dominance function ensures that the external guidance of the off-policy trajectory is a gentle "pull" rather than a "dominant" one, providing the model with valuable signals to break through cognitive bottlenecks while avoiding completely suppressing its own exploratory attempts. In other words, through this precise learning, the model can learn efficiently from the most challenging failures, thereby effectively breaking through the capability ceiling faced by simple on-policy learning or simple data mixing.

[0049] In the above embodiments, the overall system structure diagram can be found here. Figure 3 , Figure 3 This is a schematic diagram of the system structure for determining the task response according to an embodiment of this application, as shown below. Figure 3As shown, the system includes a data management module in the data layer, a dynamic batch construction module in the control and execution layer, a hybrid Rollout controller, a unified inference model, and a model update module in the evaluation and update layer. Figure 3 The UpdateModule and the Rewards and Advantages Calculation Module (in the context of the UpdateModule) ... Figure 3 The system comprises the AdvantageModule, and its modules form a closed-loop, adaptive reinforcement learning training system. In practical applications, target prompts for the desired task can be input into the trained unified inference model (i.e., the target model mentioned above). The trained target model can then provide the task response, which may include the specific inference path and the answer result. For example, if the target task is to solve a mathematical problem, the target prompts can be specific mathematical problems, and the model can then provide the solution process and calculation results.

[0050] In the above embodiments, see again Figure 3 The target model is obtained through training on a large amount of data: the data management module can first prepare a dataset D containing a large number of task prompts (i.e., the training tasks mentioned above, such as mathematical problems), and sample batches of prompt words of size N from dataset D. traverse each The prompt word "prompt p" in the training algorithm identifies samples that the model couldn't solve during historical training. These samples are marked as difficult samples (i.e., the training prompt words mentioned above) and stored in the difficult sample database. For difficult samples, the policy model in the current unified inference model to be trained (i.e., the initial model mentioned above) can be used. Figure 3 (Strategy Header) (i.e., the first model mentioned above) generates G-1 on-policy inference trajectories. (i.e., the first inference trajectory mentioned above), after which it can be obtained from a pre-prepared high-quality offline expert trajectory dataset. Obtain high-quality expert trajectories corresponding to training prompts. (That is, the aforementioned reference inference trajectory) as the Gth trajectory can form a mixed group of G trajectories, including one reference inference trajectory and G-1 first inference trajectories. Among them, the offline expert trajectory dataset This includes all prompt words in dataset D, and the correct inference trajectory generated by a powerful teacher model (i.e., the aforementioned reference model, such as DeepSeek-R1) (i.e., the inference accuracy meets the first preset threshold). For mixed groups... Each trajectory included That is, a reward value can be calculated for each on-policy trajectory, and a reward value can also be calculated for each off-policy trajectory. The parameters of the initial model can be iteratively updated using the combined reward values ​​(i.e. the target reward value) to obtain the target model.

[0051] This application allows inputting target prompts for a target task into a trained target model, which then outputs the inference trajectory and result (i.e., the task response). The target model is trained as follows: First, the training prompts for the training task are determined and input into the first model included in the initial model to be trained, resulting in a first inference trajectory generated by the first model. Next, a reference inference trajectory for the target prompts is determined. This reference trajectory is the correct inference trajectory obtained by inferring the target prompts using a pre-trained reference model whose inference accuracy for the target prompts meets a first preset threshold. The target reward value is determined using the first and reference inference trajectories, and the parameters of the initial model are iteratively updated using this target reward value to obtain the trained target parameters. Since training can be performed on difficult samples (i.e., the training samples) that the model could not solve during historical training (i.e., a high-quality reference inference trajectory is introduced for guidance), and a restricted dominance function (i.e., the determined target reward value) can be used to balance the relationship between the reference inference trajectory and the first training trajectory generated by the model itself, over-reliance on the reference inference trajectory can be avoided. This approach ensures that the model benefits from the instructions of the reference inference trajectory while maintaining its ability to explore the unknown policy space. It avoids the problem in related technologies where the reference inference trajectory often has an absolute advantage due to its high reward value, causing the model to quickly converge to the expert policy and stop trying to explore more paths. Therefore, it can solve the problem of inaccurate output task response caused by the lack of training on complex tasks in related technologies, and achieve the effect of accurately outputting the task response.

[0052] Optionally, the entity performing the above steps can be a terminal, a client, or a server or other device with similar processing capabilities, but is not limited to these.

[0053] In one exemplary embodiment, determining a target reward value based on a reference inference trajectory and a first inference trajectory includes: determining an attribute value for each first inference trajectory to obtain a plurality of first attribute values; determining an attribute value for the reference inference trajectory to obtain a second attribute value; determining an attribute average of the plurality of first attribute values ​​and the second attribute value; determining an attribute standard deviation based on the attribute average; determining a reward value for each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute values ​​to obtain a plurality of first reward values; determining a second reward value for the reference inference trajectory based on the attribute average, the attribute standard deviation, the plurality of first reward values, and the second attribute value; and determining the plurality of first reward values ​​and the second reward value as the target reward value.

[0054] In the above embodiments, see again Figure 3 The reward and advantage calculation module is designed for Each reasoning trajectory in Each inference path can be evaluated by an external verifier to determine whether its final answer is correct, and the result reward (i.e., the attribute value mentioned above) can be obtained for each inference path. For example, if an external validator determines that the final answer of a certain reasoning trajectory is correct, the reward for the result is 1 if it is incorrect and 0 if it is incorrect. Alternatively, the reward could be 0 if it is correct and 1 if it is incorrect. Therefore, a first attribute value can be obtained for a first reasoning trajectory, and a second attribute value can be obtained for a reference reasoning trajectory.

[0055] In the above embodiments, after determining the attribute values ​​of each inference trajectory, the standard group normalization advantage can be calculated, which means that the calculation can be performed... The average reward of all reasoning trajectories (i.e., the average of the above attributes). The standard deviation of the attributes can also be determined by the average reward and the attribute values ​​of each inference trajectory. For each on-policy trajectory (i.e., the first inference trajectory mentioned above), the first reward value can be determined by the attribute average, attribute standard deviation, and its own first attribute value; for the Off-Policy trajectory The second reward value can be determined by the maximum value, the average value, the standard deviation of the attribute, and the value of the second attribute in the first reward value; finally, multiple first reward values ​​and second reward values ​​can constitute the target reward value.

[0056] By calculating the attribute mean and standard deviation of the first inference trajectory and the reference inference trajectory, a standardized evaluation system can be created. This system allows for detailed comparison and scoring of different inference paths, enabling the model to receive more accurate and targeted learning signals from each attempt. Compared to traditional single-outcome rewards, the introduction of attribute values ​​makes the reward signals richer and more multi-layered, helping the model learn more subtle and complex inference skills.

[0057] In an exemplary embodiment, determining a first reward value for each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute value includes: performing the following operations for each first inference trajectory to determine the first reward value corresponding to the first inference trajectory: determining a first difference between the first attribute value of the first inference trajectory and the attribute average; and determining a first ratio of the first difference to the attribute standard deviation as the first reward value corresponding to the first inference trajectory.

[0058] In the above embodiments, for each on-policy trajectory (i.e., the first reasoning trajectory mentioned above), its first reward value can be calculated using the following formula: ,in, That is, the first attribute value of a first inference trajectory. That is, the average value of the attribute. That is, the attribute standard deviation. This is the first difference mentioned above. This is the first ratio of the first difference to the attribute standard deviation, which is the first reward value of a first inference trajectory.

[0059] In this embodiment, combining attribute values ​​with attribute standard deviations encourages the model to try inference trajectories that, while not entirely correct, excel in certain attributes (such as creativity, diversity, and novelty). This allows the model to focus not only on the final result but also on the innovation and diversity of the process when searching the policy space, thereby stimulating its exploratory potential and leading to more efficient or innovative solutions. Furthermore, standardizing rewards using attribute standard deviations effectively reduces reward fluctuations, especially in the early training stages when model capabilities are still unstable and inference trajectories vary significantly. Standardized reward values ​​are also more stable, helping to prevent drastic policy fluctuations caused by sudden changes in rewards during training, ensuring the continuity and robustness of model training.

[0060] In one exemplary embodiment, determining a second reward value for the reference inference trajectory based on the attribute average, the attribute standard deviation, a plurality of first reward values, and the second attribute value includes: determining the maximum reward value included among the plurality of first reward values; determining a first sum of the maximum reward value and a preset parameter; determining a second difference between the second attribute value and the attribute average; determining a second ratio between the second difference and the attribute standard deviation; and determining the minimum value between the second ratio and the first sum as the second reward value.

[0061] In the above embodiments, a restricted advantage function can be applied to the Off-Policy trajectory (i.e., the aforementioned reference inference trajectory), and the formula for calculating the second reward value of the reference inference trajectory is as follows: ,in, This can be understood as the second attribute value of the reference reasoning trajectory. That is, the average value of the attribute. That is, the attribute standard deviation; (i.e., the aforementioned preset parameters) can be understood as a preset hyperparameter, such as 0.2, 0.3, etc., and this invention does not limit this; This can be understood as the maximum advantage value among all on-policy trajectories (i.e., the maximum reward value among the multiple first reward values ​​mentioned above); that is... The first sum mentioned above, The second difference mentioned above, The second ratio mentioned above, and the minimum value between the second ratio and the first sum, is the second reward value.

[0062] This embodiment effectively protects the model's exploratory capabilities while utilizing external expert knowledge (i.e., reference inference trajectories), achieving a better balance between imitation and exploration. Specifically, through a unique restricted advantage function, it fundamentally solves the problem in related technologies where external data may stifle exploration. In related technologies, correct expert trajectories typically yield much higher advantage values ​​than incorrect on-policy trajectories, causing the model's gradient updates to be completely dominated by "imitated" signals. However, in this embodiment, through... This approach dynamically sets a "ceiling" for the advantage value of expert trajectories. This ceiling is only slightly higher than the model's own best performance, ensuring that the core role of expert guidance is to "point out a better direction" rather than forcing the model to replicate its path with absolute advantage. This allows the model's policy update gradient to be a healthy hybrid, incorporating both expert guidance and consideration of its own suboptimal exploration paths. This maintains policy diversity and long-term exploration potential, ultimately achieving better overall performance than simple imitation or simple exploration.

[0063] In an exemplary embodiment, iteratively updating the parameters of the initial model based on the target reward value to obtain the target model includes: determining a first loss value based on a plurality of first reward values ​​and a second reward value included in the target reward value; determining a second loss value based on the second reward value; determining the product of the second loss value and hyperparameters to obtain a first product; determining the sum of the first loss value and the first product to obtain a second sum; updating the parameters of the initial model using the second sum to obtain an updated model; and determining the updated model as the target model if the updated model meets preset conditions.

[0064] In the above embodiments, the reward value calculated in the above embodiments can be used to simultaneously calculate two loss functions, and the two loss functions can be added together to update the parameters of the initial model. The policy loss can be determined by including the first reward value for each on-policy trajectory and the second reward value for each off-policy trajectory in the target reward function. (i.e., the first loss value mentioned above), the self-supervised process reward modeling (SPRM) loss can also be calculated using the second reward value of the off-policy trajectory. (That is, the second loss value mentioned above). The formula for calculating the total loss function (that is, the second sum value mentioned above) is as follows: ,in, This can be understood as preset hyperparameters. Then, the gradient can be calculated using the total loss value to update the parameters of the entire initial model (main network Backbone + first model Policy Head + second model SPRM Head), thus obtaining the updated model. When the number of iterations for updating the model reaches a preset number or the updated parameters reach a threshold (i.e., the aforementioned preset condition), the iteration process can end, and the resulting updated model is the target model after training. Through this embodiment, the independent calculation of the first and second loss values ​​allows the model to distinguish between the trajectory generated by the online policy (first inference trajectory) and the trajectory provided by the offline expert policy (reference inference trajectory) during the learning process. This ensures that while absorbing expert knowledge, the model also focuses on optimizing its own generated trajectories, avoiding confusion of learning signals and improving the targeting and efficiency of training.

[0065] In an exemplary embodiment, determining a first loss value based on a first reward value and a second reward value included in the target reward value includes: determining the likelihood ratio of each first inference trajectory to obtain multiple first likelihood ratios, wherein the likelihood ratio represents the ratio of the probability of the initial model and the probability of the historical model responding to the preset task, and the historical model is the initial model that has not been trained and updated; determining the likelihood ratio of the reference inference trajectory to obtain a second likelihood ratio; determining the first loss value based on the multiple first likelihood ratios and the second likelihood ratio when the multiple first likelihood ratios are within a preset interval and the second likelihood ratio is within the preset interval; or, adjusting the first loss value based on the preset interval when there is a third likelihood ratio outside the preset interval among the multiple first likelihood ratios and the second likelihood ratio is within the preset interval. The first updated likelihood ratio is obtained by taking three likelihood ratios, and the first loss value is determined based on the first updated likelihood ratio, the second likelihood ratio, and other likelihood ratios, wherein the other likelihood ratios are likelihood ratios other than the third likelihood ratio included in the plurality of first likelihood ratios; or, if the plurality of first likelihood ratios are within a preset interval and the second likelihood ratio is outside the preset interval, the second likelihood ratio is adjusted based on the preset interval to obtain a second updated likelihood ratio, and the first loss value is determined based on the plurality of first likelihood ratios and the second updated likelihood ratio; or, if there is a third likelihood ratio outside the preset interval among the plurality of first likelihood ratios and the second likelihood ratio is outside the preset interval, the first loss value is determined based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio.

[0066] In the above embodiments, the strategy loss (i.e., the first loss value mentioned above) is calculated. In this case, sequence-level importance sampling can be performed first, that is, targeting... Each trajectory in The likelihood ratios of each sequence can be calculated as importance weights, thus yielding the first likelihood ratio for each first inference trajectory and the second likelihood ratio for the reference inference trajectory. The formula for calculating the likelihood ratio is as follows: ,in, This can be understood as the generation probability of each trajectory under the current initial model; This can be understood as the generation probability of each trajectory under the historical model.

[0067] In the above embodiments, after sequence-level importance sampling is completed, token-level covariance filtering can also be performed. That is, when calculating the gradient during backpropagation, the Clip-Cov mechanism is applied to identify and ignore or reduce the gradient contribution of high-covariance tokens in the trajectory to stabilize the entropy. In other words, it is necessary to determine which of the first and second likelihood ratios exceeds a preset interval (…). The value of ) is controlled within a preset range, where These are preset parameters. When multiple first likelihood ratios and second likelihood ratios are all within the preset range, no adjustment is needed; the first loss value can be directly determined based on these multiple first likelihood ratios and second likelihood ratios. When there is a third likelihood ratio outside the preset range among the multiple first likelihood ratios, but the second likelihood ratio is within the preset range, the third likelihood ratio can be adjusted. That is, the third likelihood ratio less than [a certain value] can be adjusted. The value of the likelihood ratio is determined as follows: The third likelihood ratio is greater than The value of the likelihood ratio is determined as follows: This allows us to obtain the first updated likelihood ratio. Then, we can determine the first loss value using the first updated likelihood ratio, the second likelihood ratio, and other likelihood ratios (all likelihood ratios except the third likelihood ratio included in the multiple first likelihood ratios, i.e., those within a preset interval). When all first likelihood ratios are within the preset interval, but the second likelihood ratio is outside the preset interval, the second likelihood ratio can be adjusted. Specifically, values ​​smaller than [a certain value] in the second likelihood ratio can be adjusted. The value of the likelihood ratio is determined as follows: The second likelihood ratio is greater than The value of the likelihood ratio is determined as follows: This yields the second updated likelihood ratio, which, along with multiple first likelihood ratios, determines the first loss value. If a third likelihood ratio exists outside a preset interval among the multiple first likelihood ratios, and the second likelihood ratio also exists outside the preset interval, then both the third and second likelihood ratios can be adjusted simultaneously, reducing the third likelihood ratio smaller than a certain threshold. The value of the likelihood ratio is determined as follows: The third likelihood ratio is greater than The value of the likelihood ratio is determined as follows: The first updated likelihood ratio is obtained, and the second likelihood ratio is greater than 1. The value of the likelihood ratio is determined as follows: The second updated likelihood ratio is obtained, and the first loss value can be determined by the second updated likelihood ratio, the first updated likelihood ratio, and other likelihood ratios.

[0068] This embodiment, unlike the token-level IS of traditional GRPO, adopts the sequence-level importance sampling of GSPO, using the entire sequence as the basic unit for evaluating policy shift. This is perfectly aligned with the granularity of the final result reward at the sequence level, which can fundamentally avoid the high variance noise introduced by token-level IS, making the gradient signal smoother and more reliable.

[0069] In an exemplary embodiment, determining the first loss value based on a plurality of first likelihood ratios and second likelihood ratios includes: determining the product of each first likelihood ratio and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a third sum of the plurality of first sub-loss values ​​and the second loss values; and determining the third sum as the first loss value.

[0070] In the above embodiments, when multiple first likelihood ratios and second likelihood ratios are all within a preset range, the first loss value can be determined using multiple first likelihood ratios and second likelihood ratios. That is, the product of multiple first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio can be calculated to obtain multiple first sub-loss values. The product of the second likelihood ratio of the reference inference trajectory and the second reward value can also be calculated to obtain the second sub-loss value. The sum of multiple first sub-loss values ​​and second sub-loss values ​​is the first loss value.

[0071] This embodiment avoids drastic policy changes during model training by limiting the likelihood ratio update magnitude, thus maintaining the smoothness of policy updates and the stability of the learning process. Sequence-level clipping, compared to token-level clipping, also reduces the impact of high variance noise, especially when handling long sequence inference tasks, improving the accuracy and reliability of gradient estimation.

[0072] In an exemplary embodiment, determining the first loss value based on the first updated likelihood ratio, the second likelihood ratio, and the other likelihood ratios includes: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a fourth sum of the third sub-loss value, the fourth sub-loss value, and the second sub-loss value; and determining the minimum value among the third sum and the fourth sum as the first loss value.

[0073] In the above embodiments, when a third likelihood ratio exists outside a preset interval among multiple first likelihood ratios, but a second likelihood ratio is within the preset interval, sequence-level clipping can be introduced, that is, the weighted advantage can be... and The smaller of the two values ​​is used to form the final PPO-like loss (i.e., the first loss value mentioned above). This can be achieved by comparing the third sum calculated without adjusting the likelihood ratio with the loss value obtained after adjusting the third likelihood ratio outside the preset interval. First, the product of the first updated likelihood ratio obtained after adjusting the third likelihood ratio and the first reward value of the inference trajectory corresponding to the first updated likelihood ratio is determined to obtain the third sub-loss value. Then, the product of the other likelihood ratios in the first likelihood ratio that do not exceed the preset interval and their corresponding first reward values ​​is determined to obtain the fourth sub-loss value. Finally, the sum of the third sub-loss value, the fourth sub-loss value, and the second sub-loss value (the product of the second likelihood ratio and the second reward value of the reference inference trajectory) is compared with the third sum, and the minimum value between the two is determined as the first loss value. By introducing the clipping mechanism and comprehensively considering the likelihood ratio and reward value from different sources, we can effectively avoid over-adjustment and unstable states of the model when updating the policy. At the same time, we can also ensure that the model can obtain effective learning signals from various data sources, including on-policy and off-policy data, thereby improving learning efficiency and performance while maintaining exploratory nature.

[0074] In an exemplary embodiment, determining the first loss value based on the plurality of first likelihood ratios and the second updated likelihood ratio includes: determining the product of the plurality of first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a fifth sum of the plurality of first sub-loss values ​​and the fifth sub-loss value; and determining the minimum value among the third sum and the fifth sum as the first loss value.

[0075] In the above embodiments, when multiple first likelihood ratios are all within a preset interval, but the second likelihood ratio is outside the preset interval, sequence-level clipping can be introduced, that is, the weighted likelihood ratio can be adjusted. and By comparing the values ​​and taking the smaller one, the final PPO-like loss (i.e., the first loss value mentioned above) is formed. This can be achieved by comparing the third sum calculated without adjusting the likelihood ratio with the loss value obtained after adjusting the second likelihood ratio outside the preset interval. First, the product of multiple first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio can be determined to obtain multiple first sub-loss values. Then, the product of the second updated likelihood ratio obtained after adjusting the second likelihood ratio and the second reward value can be determined to obtain the fifth sub-loss value. Finally, the sum of multiple first sub-loss values ​​and the fifth sub-loss value (i.e., the fifth sum mentioned above) can be compared with the third sum, and the minimum value between the two can be determined as the first loss value.

[0076] In this embodiment, the Clip-Cov gradient filtering technique is used as an additional "stabilizer" to actively identify and suppress the gradient contribution of high covariance tokens that cause entropy collapse during backpropagation. Through the dual protection of "sequence-level IS + Clip-Cov", the risk of model collapse can be avoided and long-range dependencies can be learned more effectively.

[0077] In one exemplary embodiment, based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio, the method includes: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a sixth sum of the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value; and determining the minimum value among the third sum and the sixth sum as the first loss value.

[0078] In the above embodiments, when a third likelihood ratio exceeds a preset interval among multiple first likelihood ratios, and the second likelihood ratio is also outside the preset interval, sequence-level clipping can be introduced, that is, the weighted advantage can be... and The comparison is performed, and the smaller value is taken to form the final PPO-like loss (i.e., the first loss value mentioned above). This can be achieved by comparing the third sum calculated without adjusting the likelihood ratio with the loss value obtained after adjusting the third likelihood ratio outside the preset interval and the second likelihood ratio. First, the product of the first updated likelihood ratio obtained after adjusting the third likelihood ratio and the first reward value of the inference trajectory corresponding to the first updated likelihood ratio can be determined to obtain the third sub-loss value. Then, the product of the other likelihood ratios in the first likelihood ratio that do not exceed the preset interval and their corresponding first reward values ​​can be determined to obtain the fourth sub-loss value. Next, the product of the second updated likelihood ratio obtained after adjusting the second likelihood ratio and the second reward value can be determined to obtain the fifth sub-loss value. Finally, the sixth sum of the third, fourth, and fifth sub-loss values ​​can be compared with the third sum, and the minimum value between the two can be determined as the first loss value.

[0079] This embodiment addresses the gradient bias problem caused by missing importance sampling (IS) in related technologies by introducing an IS mechanism into the model update module. This involves reweighting using IS, enabling off-policy data to more accurately represent the true gradient under the target policy and eliminating gradient estimation bias. Secondly, the IS mechanism ensures that off-policy data samples can still be effectively utilized even with significant distribution differences, thereby improving training efficiency and sample utilization. Furthermore, combining sequence-level IS and clipping avoids the negative impact of single-token likelihood ratio fluctuations on overall training dynamics, improving model training stability.

[0080] In an exemplary embodiment, determining a second loss value based on the second reward value includes: performing the following operations for each sub-inference trajectory included in the second inference trajectory to determine a sub-loss value for the sub-inference trajectory, wherein the second inference trajectory includes the first inference trajectory and the reference inference trajectory: inputting the sub-inference trajectory into a second model included in the target model to obtain multiple initial scores; determining an average score of the multiple initial scores; determining a binary cross-entropy loss value of the average score and the attribute value of the sub-inference trajectory; determining a fifth product of the binary cross-entropy loss value and the likelihood ratio of the sub-inference trajectory as the sub-loss value; determining a sum of multiple sub-loss values ​​to obtain a seventh sum; and determining a third ratio of the seventh sum to the number of sub-inference trajectories as the second loss value.

[0081] In the above embodiments, SPR Loss can be used, which only uses the final result of the trajectory as a reward. As a supervisory signal, the parameters of the SPRM head are updated so that it can better distinguish between correct and incorrect reasoning steps; that is, the SPRM loss of the self-supervised process reward model. The following formula can be used for calculation: Here, G can be understood as the number of reasoning trajectories (i.e., the number of the aforementioned sub-reasoning trajectories), and i can be understood as the i-th reasoning trajectory. That is, the attribute value. This can be understood as the process score of the sub-inference trajectory (i.e., the average score mentioned above). This can be understood as a binary cross-entropy loss function. This is the likelihood ratio corresponding to the sub-inference trajectory. The process score of the sub-inference trajectory can be determined by the second model included in the target model: see [link to relevant documentation] Figure 3 It can Each trajectory (That is, the above sub-inference trajectories, whether on-policy or off-policy trajectories) are input into the first model. In the Self-Supervised Process Reward Model (SPRM) head of the shared backbone network (i.e., the second model mentioned above), the SPRM head can segment each sub-inference trajectory into steps and for each step... Generate a process score (i.e., the initial score mentioned above). Then, the geometric mean of the answer trajectory for the entire step can be calculated for each sub-inference trajectory. Calculate a total process score (i.e., the average score mentioned above). The total process score can be used to select the optimal trajectory during inference and to update the SPRM itself during training.

[0082] By using the lightweight SPRM head shared with the policy model in this embodiment, fine-grained scoring of inference steps can be learned without any manual annotation, improving learning efficiency and fine-grained final performance, thus achieving more efficient credit allocation. Unlike related technologies, the lightweight SPRM head in this embodiment can simultaneously evaluate trajectories from both on-policy and off-policy sources, enabling it to learn to judge the quality of inference steps from more diverse and higher-quality data. This process-level feedback provides richer and more timely guidance for policy learning than sparse final reward, accelerating the model's mastery of complex inference logic and achieving more efficient credit allocation, thereby achieving higher performance levels with the same training resources.

[0083] In an exemplary embodiment, determining training prompts included in a training task includes: for each sub-training prompt among the training prompts included in the training task, performing the following operations to determine the training prompt, wherein the training prompt includes the training prompt: determining a first number of historical inference trajectories generated by training a historical model using the sub-training prompt; determining a second number of correct inference trajectories included in the historical inference trajectories; determining a fourth ratio between the second number and the first number to obtain a fourth ratio; and determining the sub-training prompt as the training prompt if the fourth ratio is a first value.

[0084] In the above embodiments, a hard sample library can be initialized first. In the initial stage Empty. At the beginning of each training iteration, a pre-prepared dataset containing a large number of task-specific prompts can be used. Sample a batch of cue words of size N. (i.e., the above-mentioned prompts to be trained), traversal Each prompt in (i.e., the aforementioned sub-training prompt words), query this prompt. The on-policy success rate determined by the historical model during historical training. Among them, success rate This can be understood as the historical model's response to the prompt. The ratio of the number of correctly identified reasoning trajectories to the total number of reasoning trajectories (i.e., the fourth ratio mentioned above). If If the first value is 0, then prompt can be considered as... If a sample is entirely incorrect and considered a difficult sample, it can be retained, identified as a training prompt word, and labeled as a "difficult sample". Add to the hard sample library In this embodiment, by prioritizing the selection of sub-training prompts that the model performed poorly during historical training but are interpretable, the model can be encouraged to continuously challenge the boundaries of its capabilities and explore new strategies for solving problems.

[0085] In one exemplary embodiment, the method further includes: discarding the sub-training prompt word when the fourth ratio is a second value; and determining the sub-training prompt word as a retained prompt word when the fourth ratio is less than the second value and greater than the first value, wherein the retained prompt word is used to train the initial model.

[0086] In the above embodiments, if the determined success rate If the fourth ratio mentioned above is 1, then it can be considered that the model has fully mastered this prompt word in the historical process. Samples with all correct answers can be discarded directly; if the success rate is... That is, it lies between the first and second values, and can be considered as this prompt. Samples deemed valuable for learning are retained in the current batch. Repeat this process until the entire batch is traversed. Each prompt in It is possible to construct a containing Training batches of valid prompts .

[0087] In the above embodiments, due to training batches The model may contain both valuable and difficult samples. Therefore, before training the model using a prompt word, it's essential to determine whether the prompt word is labeled as a difficult sample. If it is a difficult sample, its corresponding reference inference trajectory can be determined, comparing it with the initial model's generated inference trajectory. -1 trajectories constitute a collection A mixture of trajectories If it is not a difficult sample, the initial model can be used to generate the next sample. A reasoning trajectory, and -1 trajectory also constitutes a containing A trajectory It should be noted here that the method of suppressing the prompt values, reward values, and loss values ​​of non-hard samples and hard samples will not be elaborated here.

[0088] This embodiment employs conditional data introduction: offline expert trajectories are only introduced into the training set when a task is determined to be a difficult sample (e.g., the model's historical success rate is below a threshold). Compared to continuously introducing offline data, conditional data introduction is more effective in avoiding entropy collapse and inhibiting exploration ability. This is because it ensures that the model will not suffer overwhelming advantages from expert trajectories before it has sufficiently explored the task, thus maintaining the model's inherent exploratory momentum. Offline data is only introduced as an auxiliary means, not a dominant force in the learning process, when the model encounters a genuine bottleneck on a specific task. In other words, the conditional data introduction mechanism ensures that the use of offline expert trajectories is efficient and targeted, avoiding the waste of expert data on tasks the model is already proficient in. Instead, these valuable resources are concentrated on improving the model's performance on the most difficult tasks, thereby achieving maximum learning efficiency with limited data sources.

[0089] The method for determining the task response in this application will be explained below with reference to specific embodiments.

[0090] Figure 4 This is a flowchart illustrating the method for determining the task response in this optional example, such as... Figure 4 As shown, the process of determining the task response may include the following steps:

[0091] Phase 1: Data Preparation

[0092] Step S402: Determine the task prompt library (D), the hard sample library (D_hard), and the offline expert trajectory library (D_expert).

[0093] Phase 2: Dynamic Batch Construction

[0094] Step S404: Sample the Prompt batch from D;

[0095] Step S406, sample classification and filtering, generating discarded samples, retained samples, and retained and labeled "difficult samples";

[0096] Phase 3: Hybrid Rollout and Process Evaluation

[0097] Step S408: Determine each Prompt in the batch;

[0098] Step S410: Determine whether it is a "hard sample". If yes, proceed to step S412; otherwise, proceed to step S414.

[0099] Step S412: Generate G-1 On-Policy trajectories and introduce 1 Off-Policy trajectory;

[0100] Step S414: Generate G On-Policy trajectories;

[0101] Step S416: Construct a mixed group Group_i;

[0102] Step S418, the SPRM head performs a process score S(τ) on each trajectory;

[0103] Phase 4: Advantage Calculation (Revised Version):

[0104] Step S420: Obtain the reward result R(τ);

[0105] Step S422, uniformly calculate the advantage: On-Policy: standard normalization, Off-Policy: apply the restrictive advantage function, A_off=min(A_std, max(A_on)+β);

[0106] Step S424: Output the final advantage A of all trajectories;

[0107] Phase 5: Model Update

[0108] Step S426, calculate the policy loss L_Policy (fusion sequence machine IS and Clip-Cov);

[0109] Step S428, calculate the SPRM loss L_SPRM (self-supervised learning);

[0110] Step S430, L_total = L_Policy + λ L_SPRM;

[0111] Step S432: Update the unified model parameters θ.

[0112] In the above embodiments, the method for determining the task response mainly includes five core stages: data preparation, batch construction, model inference and evaluation, advantage calculation, and model update. The data management module can be responsible for storing and managing the task prompt library. Expert trajectory database and difficult sample library The dynamic batch construction module can dynamically sample and construct effective training batches from the data management module based on historical training information. The unified inference model, as a core component, includes a shared backbone network, a policy head, and a SPRM head. The policy head is responsible for generating inference trajectories based on the input, and the SPRM head is responsible for procedural scoring of the trajectories generated by the policy head. The hybrid Rollout controller can control the unified inference model to generate on-policy trajectories and, based on the "hard sample" labels provided by the dynamic batch construction module, decide whether to introduce off-policy trajectories from the data management module. The reward and advantage calculation module can incorporate an external verifier based on RLVR (Reinforcement Learning with Verifiable Rewards) to calculate the result reward. It also includes a restricted advantage function to calculate the final advantage value for each trajectory. The model update module can include a policy loss calculation unit and a SPRM loss calculation unit. It integrates the loss values ​​of the two loss units and uses an optimizer (such as AdamW) to update the parameters of the unified inference model. The policy loss calculation unit has a built-in sequence-level IS calculator and a Clip-Cov gradient filter.

[0113] In the foregoing embodiments, this application achieves significant technical advantages in improving the performance ceiling, training stability, exploration efficiency, and sample utilization of large-scale language model reinforcement learning: First, conditional data is introduced: offline expert trajectories are only introduced into the training group when a task is determined to be a difficult sample (e.g., the model's historical success rate is below a threshold). Second, sequence-level policy optimization is utilized: importance sampling weights based on the likelihood ratio of the entire generated sequence are used to weight the advantage value obtained by the advantage function calculation method to update the policy model. Furthermore, a training method combining process reward modeling and sequence-level policy optimization is constructed: on a shared network backbone, a policy model for generating trajectories and a process reward model (SPRM) for evaluating inference steps are trained simultaneously. The policy model is updated using sequence-level importance sampling (IS); the process reward model is updated only based on the final reward of the trajectory, and this model is used to evaluate trajectories from all sources, including online trajectories and introduced offline trajectories. Compared to the core dilemma commonly encountered in existing technologies—how to leverage external expert knowledge (Off-Policy) to accelerate learning and overcome challenges while avoiding stifling the model's own exploratory spirit (On-Policy), and ensuring algorithmic stability and unlimited sample selection throughout the training process to systematically and efficiently break through the model's existing capability boundaries—this application intelligently and dynamically balances external guidance (imitation) and internal exploration. While ensuring training stability, it can efficiently utilize high-value, challenging samples to break through capability limits, and accelerate the learning process through fine-grained process feedback, thereby achieving comprehensive, efficient, and scalable optimization of the reasoning capabilities of large-scale language models.

[0114] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0116] According to another aspect of the embodiments of this application, a task response determination apparatus is also provided. This task response determination apparatus can be used to implement the task response determination method provided in the above embodiments, and details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0117] Figure 5 This is a structural block diagram of an optional task response determination device according to an embodiment of this application, such as... Figure 5 As shown, the device for determining the task response includes:

[0118] The determination module 52 is used to input the target prompt words of the target task into the target model, so as to determine the task response of the target task through the target model;

[0119] The target model is trained in the following manner:

[0120] Identify the training cue words included in the training task;

[0121] The training prompts are input into the first model included in the initial model to obtain multiple first inference trajectories;

[0122] The target reward value is determined based on the reference inference trajectory and the first inference trajectory, wherein the reference inference trajectory is the trajectory obtained by the pre-trained reference model inferring the target prompt word, and the inference accuracy of the reference model for the target prompt word meets a first preset threshold.

[0123] The parameters of the initial model are iteratively updated based on the target reward value to obtain the target model.

[0124] The determination module corresponds to the data management module, the dynamic batch construction module of the control and execution layer, the hybrid Rollout controller, the unified inference model, the model update module in the evaluation and update layer, and the reward and advantage calculation module.

[0125] In an exemplary embodiment, the determining module 52 can determine the target reward value based on the reference inference trajectory and the first inference trajectory in the following manner: determining the attribute value of each first inference trajectory to obtain a plurality of first attribute values; determining the attribute value of the reference inference trajectory to obtain a second attribute value; determining the attribute average of the plurality of first attribute values ​​and the second attribute value; determining the attribute standard deviation based on the attribute average; determining the reward value of each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute value to obtain a plurality of first reward values; determining the second reward value of the reference inference trajectory based on the attribute average, the attribute standard deviation, the plurality of first reward values, and the second attribute value; and determining the plurality of first reward values ​​and the second reward value as the target reward value.

[0126] In an exemplary embodiment, the determining module 52 may determine a first reward value for each first inference trajectory based on the attribute average, the attribute standard deviation, and the first attribute value in the following manner: for each first inference trajectory, perform the following operations to determine the first reward value corresponding to the first inference trajectory: determine a first difference between the first attribute value of the first inference trajectory and the attribute average; determine a first ratio of the first difference to the attribute standard deviation as the first reward value corresponding to the first inference trajectory.

[0127] In an exemplary embodiment, the determining module 52 may determine a second reward value for the reference inference trajectory based on the attribute average, the attribute standard deviation, a plurality of first reward values, and the second attribute value in the following manner: determining the maximum reward value included among the plurality of first reward values; determining a first sum of the maximum reward value and a preset parameter; determining a second difference between the second attribute value and the attribute average; determining a second ratio between the second difference and the attribute standard deviation; and determining the minimum value between the second ratio and the first sum as the second reward value.

[0128] In an exemplary embodiment, the determining module 52 can iteratively update the parameters of the initial model based on the target reward value to obtain the target model in the following manner: determining a first loss value based on a plurality of first reward values ​​and a second reward value included in the target reward value; determining a second loss value based on the second reward value; determining the product of the second loss value and the hyperparameters to obtain a first product; determining the sum of the first loss value and the first product to obtain a second sum; updating the parameters of the initial model using the second sum to obtain an updated model; and determining the updated model as the target model if the updated model meets preset conditions.

[0129] In an exemplary embodiment, the determining module 52 can determine a first loss value based on a first reward value and a second reward value included in the target reward value in the following manner: determining the likelihood ratio of each first inference trajectory to obtain multiple first likelihood ratios, wherein the likelihood ratio represents the ratio of the probability of the initial model and the probability of the historical model responding to the preset task, and the historical model is the initial model that has not been trained and updated; determining the likelihood ratio of the reference inference trajectory to obtain a second likelihood ratio; when multiple first likelihood ratios are within a preset interval and the second likelihood ratio is within the preset interval, determining the first loss value based on the multiple first likelihood ratios and the second likelihood ratio; or, when there is a third likelihood ratio outside the preset interval among the multiple first likelihood ratios and the second likelihood ratio is within the preset interval, determining the first loss value based on the first likelihood ratio and the second likelihood ratio. The method involves adjusting the third likelihood ratio within a given interval to obtain a first updated likelihood ratio, and determining the first loss value based on the first updated likelihood ratio, the second likelihood ratio, and other likelihood ratios, wherein the other likelihood ratios are likelihood ratios other than the third likelihood ratio included in the plurality of first likelihood ratios; or, if the plurality of first likelihood ratios are within a preset interval and the second likelihood ratio is outside the preset interval, the second likelihood ratio is adjusted based on the preset interval to obtain a second updated likelihood ratio, and the first loss value is determined based on the plurality of first likelihood ratios and the second updated likelihood ratio; or, if there is a third likelihood ratio outside the preset interval among the plurality of first likelihood ratios and the second likelihood ratio is outside the preset interval, the first loss value is determined based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio.

[0130] In an exemplary embodiment, the determining module 52 may determine the first loss value based on a plurality of first likelihood ratios and second likelihood ratios in the following manner: determining the product of each first likelihood ratio and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a third sum of the plurality of first sub-loss values ​​and the second loss value; and determining the third sum as the first loss value.

[0131] In an exemplary embodiment, the determining module 52 may determine the first loss value based on the first updated likelihood ratio, the second likelihood ratio, and the other likelihood ratios in the following manner: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a fourth sum of the third sub-loss value, the fourth sub-loss value, and the second sub-loss value; and determining the minimum value among the third sum and the fourth sum as the first loss value.

[0132] In an exemplary embodiment, the determining module 52 may determine the first loss value based on the plurality of first likelihood ratios and the second updated likelihood ratio in the following manner: determining the product of the plurality of first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a fifth sum of the plurality of first sub-loss values ​​and the fifth sub-loss value; and determining the minimum value among the third sum and the fifth sum as the first loss value.

[0133] In an exemplary embodiment, the determining module 52 may implement the determination based on the first updated likelihood ratio, the other likelihood ratios, and the second updated likelihood ratio in the following manner: determining the product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining the product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining the product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; determining a sixth sum of the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value; and determining the minimum value among the third sum and the sixth sum as the first loss value.

[0134] In an exemplary embodiment, the determining module 52 may determine the second loss value based on the second reward value by performing the following operations for each sub-inference trajectory included in the second inference trajectory to determine the sub-loss value of the sub-inference trajectory, wherein the second inference trajectory includes the first inference trajectory and the reference inference trajectory: inputting the sub-inference trajectory into the second model included in the target model to obtain multiple initial scores, determining the average score of the multiple initial scores, determining the binary cross-entropy loss value of the average score and the attribute value of the sub-inference trajectory, determining the fifth product of the binary cross-entropy loss value and the likelihood ratio of the sub-inference trajectory as the sub-loss value; determining the sum of the multiple sub-loss values ​​to obtain a seventh sum; and determining the third ratio of the seventh sum to the number of sub-inference trajectories as the second loss value.

[0135] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0136] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein the program executes the steps in any of the above method embodiments when it is run.

[0137] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0138] According to another aspect of the embodiments of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor is configured to perform the steps of any of the method embodiments described above via the computer program. In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0139] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.

[0140] According to another aspect of the embodiments of this application, a computer program product is also provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions provided in the embodiments of this application. The sequence numbers of the embodiments of this application above are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0141] Figure 6 A schematic block diagram of a computer system architecture for implementing embodiments of the present application is shown. Figure 6 As shown, the computer system 600 includes a Central Processing Unit (CPU) 601, which performs various appropriate actions and processes based on programs stored in ROM 602 or loaded into RAM 603 from storage section 608. Random access memory 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0142] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card, such as a local area network card or modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0143] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit 601, it performs various functions defined in the system of this application.

[0144] It should be noted that, Figure 6 The computer system 600 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0145] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0146] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a task response, characterized in that The method comprises the following steps: inputting a target prompt word of a target task into a target model to determine a task response of the target task through the target model; wherein the target model is trained in the following way: determining a training prompt word included in a training task; inputting the training prompt word into a first model included in an initial model to obtain a plurality of first inference tracks; determining a target reward value based on a reference inference track and the first inference tracks, wherein the reference inference track is a track obtained by a reference model pre-trained on the target prompt word, and the reference model has a first preset threshold of correct rate on the target prompt word; iteratively updating parameters of the initial model based on the target reward value to obtain the target model.

2. The method of claim 1, wherein, determining a target reward value based on a reference inference track and the first inference tracks, comprising: determining attribute values of each of the first inference tracks to obtain a plurality of first attribute values; determining an attribute value of the reference inference track to obtain a second attribute value; determining an attribute average value of the plurality of first attribute values and the second attribute value; determining an attribute standard deviation based on the attribute average value; determining a reward value of each of the first inference tracks based on the attribute average value, the attribute standard deviation, and the first attribute value to obtain a plurality of first reward values; determining a second reward value of the reference inference track based on the attribute average value, the attribute standard deviation, the plurality of first reward values, and the second attribute value; determining the plurality of first reward values and the second reward value as the target reward value.

3. The method of claim 2, wherein, determining a first reward value of each of the first inference tracks based on the attribute average value, the attribute standard deviation, and the first attribute value, comprising: for each of the first inference tracks, the following operations are performed to determine the first reward value corresponding to the first inference track: determining a first difference value between the first attribute value of the first inference track and the attribute average value; determining a first ratio of the first difference value to the attribute standard deviation as the first reward value corresponding to the first inference track.

4. The method of claim 2, wherein, determining a second reward value of the reference inference track based on the attribute average value, the attribute standard deviation, the plurality of first reward values, and the second attribute value, comprising: determining a maximum reward value included in the plurality of first reward values; determining a first sum of the maximum reward value and a preset parameter; determining a second difference value between the second attribute value and the attribute average value; determining a second ratio of the second difference value to the attribute standard deviation; determining the second ratio and the minimum value in the first sum as the second reward value.

5. The method of claim 1, wherein, iteratively updating parameters of the initial model based on the target reward value to obtain the target model, comprising: determining a first loss value based on a plurality of first reward values and a second reward value included in the target reward value; determining a second loss value based on the second reward value; determining a first product of the second loss value and a hyperparameter to obtain a first product; determining a second sum of the first loss value and the first product to obtain a second sum; updating parameters of the initial model by using the second sum value to obtain an updated model; in a case where the updated model meets a preset condition, determining the updated model as the target model.

6. The method of claim 5, wherein, determining a first loss value based on a first reward value and a second reward value included in the target reward value, including: determining a likelihood ratio of each first inference trajectory to obtain a plurality of first likelihood ratios, wherein the likelihood ratio represents a ratio of probabilities of the initial model and a historical model responding to a preset task, and the historical model is the initial model without being trained and updated; determining a likelihood ratio of the reference inference trajectory to obtain a second likelihood ratio; in a case where the plurality of first likelihood ratios are within a preset interval and the second likelihood ratio is within the preset interval, determining the first loss value based on the plurality of first likelihood ratios and the second likelihood ratio; or in a case where a third likelihood ratio of the plurality of first likelihood ratios is outside the preset interval and the second likelihood ratio is within the preset interval, adjusting the third likelihood ratio based on the preset interval to obtain a first updated likelihood ratio, and determining the first loss value based on the first updated likelihood ratio, the second likelihood ratio and other likelihood ratios, wherein the other likelihood ratios are likelihood ratios included in the plurality of first likelihood ratios except for the third likelihood ratio; or in a case where the plurality of first likelihood ratios are within a preset interval and the second likelihood ratio is outside the preset interval, adjusting the second likelihood ratio based on the preset interval to obtain a second updated likelihood ratio, and determining the first loss value based on the plurality of first likelihood ratios and the second updated likelihood ratio; or in a case where a third likelihood ratio of the plurality of first likelihood ratios is outside the preset interval and the second likelihood ratio is outside the preset interval, determining the first loss value based on the first updated likelihood ratio, the other likelihood ratios and the second updated likelihood ratio.

7. The method of claim 6, wherein, determining a first loss value based on a first reward value and a second reward value included in the target reward value, including: determining a product of each first likelihood ratio and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; determining a product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a third sum value of the plurality of first sub-loss values and the second loss value; determining the third sum value as the first loss value.

8. The method of claim 6, wherein, determining a first loss value based on a first updated likelihood ratio, a second likelihood ratio and other likelihood ratios, including: determining a product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; determining a product of the other likelihood ratios and the first reward value of the first inference trajectory corresponding to the other likelihood ratios to obtain a fourth sub-loss value; determining a product of the second likelihood ratio and the second reward value to obtain a second sub-loss value; determining a fourth sum value of the third sub-loss value, the fourth sub-loss value and the second sub-loss value; Determine the first loss value as the minimum of a third sum value and a fourth sum value.

9. The method of claim 6, wherein, Determining the first loss value based on the plurality of first likelihood ratios and the second updated likelihood ratio includes: Determining a product of the plurality of first likelihood ratios and the first reward value of the first inference trajectory corresponding to the first likelihood ratio to obtain a plurality of first sub-loss values; Determining a product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; Determining a fifth sum value of the plurality of first sub-loss values and the fifth sub-loss value; Determining the first loss value as the minimum of a third sum value and the fifth sum value.

10. The method of claim 6, wherein, Determining the first loss value based on the first updated likelihood ratio, the other likelihood ratio, and the second updated likelihood ratio includes: Determining a product of the first updated likelihood ratio and the first reward value of the first inference trajectory corresponding to the first updated likelihood ratio to obtain a third sub-loss value; Determining a product of the other likelihood ratio and the first reward value of the first inference trajectory corresponding to the other likelihood ratio to obtain a fourth sub-loss value; Determining a product of the second updated likelihood ratio and the second reward value to obtain a fifth sub-loss value; Determining a sixth sum value of the third sub-loss value, the fourth sub-loss value, and the fifth sub-loss value; Determining the first loss value as the minimum of a third sum value and the fifth sum value.

11. The method of claim 5, wherein, Determining a second loss value based on the second reward value includes: For each sub-inference trajectory included in a second inference trajectory, which includes the first inference trajectory and the reference inference trajectory, the following operations are performed to determine a sub-loss value of the sub-inference trajectory: inputting the sub-inference trajectory into a second model included in the target model to obtain a plurality of initial scores, determining an average score of the plurality of initial scores, determining a binary cross-entropy loss value of the average score and an attribute value of the sub-inference trajectory, and determining a fifth product of the binary cross-entropy loss value and a likelihood ratio of the sub-inference trajectory as the sub-loss value; Determining a sum value of the plurality of sub-loss values to obtain a seventh sum value; Determining a third ratio of the seventh sum value and the number of sub-inference trajectories as the second loss value.

12. A determination apparatus for task response, characterized by, includes: A determination module is configured to input a target prompt word of a target task into a target model to determine a task response of the target task by the target model. The target model is trained in the following manner: Determine a training prompt word included in a training task; Input the training prompt word into a first model included in an initial model to obtain a plurality of first inference trajectories; Determine a target reward value based on a reference inference trajectory and the first inference trajectory, wherein the reference inference trajectory is an inference trajectory obtained by a reference model trained in advance on the target prompt word, and the inference accuracy of the reference model on the target prompt word satisfies a first preset threshold; Iteratively update parameters of the initial model based on the target reward value to obtain the target model.

13. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1-11. The computer program / instructions are executed by the processor to implement the steps of the method of any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps of the method in any one of claims 1 to 11.

15. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 11.

Images