Method, device, equipment and medium for generating task execution results

Abstract

The present application relates to the field of artificial intelligence technology and provides a method, apparatus, device, and medium for generating task execution results. The method determines a set of reward evaluation agents based on a task description and determines a weight for each reward evaluation agent in the set; uses a pre-trained strategy model to obtain at least one sampling result based on the task description; uses each reward evaluation agent to evaluate each sampling result separately; and determines a strategy model reward value based on the weight and evaluation result of each reward evaluation agent. The strategy model is trained based on the reward value, and finally the task execution result is determined using the trained strategy model. This method automatically and accurately matches multiple suitable reward evaluation agents for non-deterministic tasks, and uses these multiple reward evaluation agents to jointly train the strategy model to improve the evaluation accuracy of the strategy model, thereby obtaining a task execution result that is more consistent with the task description and improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating task execution results. Background Art

[0002] Existing reinforcement learning-based strategy models have achieved good results in quantifiable tasks, such as math and coding problems, through regularized rewards. However, for non-deterministic tasks, such as creative copywriting and essay writing, where the results cannot be clearly determined and the evaluation is subjective, the difficulty in quantifying the reward rules in reinforcement learning algorithms means that the generated task execution results need to be improved.

[0003] Therefore, how to improve the accuracy of task execution results of non-deterministic tasks generated by the strategy model is a technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method, apparatus, device and medium for generating task execution results to solve the problem in the prior art of insufficient accuracy of strategy models for executing non-deterministic tasks due to inaccurate evaluation of such tasks.

[0005] A first aspect of an embodiment of the present application provides a method for generating a task execution result based on a policy model, comprising:

[0006] Obtain the task description, and the evaluation criteria of the task execution results change with the evaluation conditions;

[0007] Determine the set of reward evaluation agents and the weight of each reward evaluation agent in the set based on the task description;

[0008] Input the task description into the pre-trained strategy model and obtain at least one sampling result;

[0009] Use each reward evaluation agent to evaluate each sampling result separately, and determine the strategy model reward value corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent;

[0010] Train the pre-trained policy model based on the reward value to obtain a trained policy model;

[0011] Using the trained policy model, determine a task execution result in at least one sampled result.

[0012] A second aspect of an embodiment of the present application provides an apparatus for generating a task execution result based on a policy model, comprising:

[0013] The acquisition module is configured to acquire a task description, and the evaluation criteria of the task execution result changes as the evaluation conditions change;

[0014] A determination module is configured to determine a set of reward evaluation agents and a weight of each reward evaluation agent in the set of reward evaluation agents based on the task description;

[0015] a sampling module configured to input a task description into a pre-trained policy model and obtain at least one sampling result;

[0016] An evaluation module is configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine a policy model reward value corresponding to each sampling result based on the weight of each reward evaluation agent and the evaluation result;

[0017] A training module is configured to train the pre-trained policy model based on the reward value to obtain a trained policy model;

[0018] The generation module is configured to use the trained policy model to determine a task execution result in at least one sampling result.

[0019] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0020] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.

[0021] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: the embodiments of the present application determine a set of reward evaluation agents based on the task description, and determine a weight for each reward evaluation agent in the set; use a pre-trained strategy model to obtain at least one sampling result based on the task description, use each reward evaluation agent to evaluate each sampling result respectively, and determine the strategy model reward value corresponding to each sampling result based on the weight and evaluation result of each reward evaluation agent, train the strategy model based on the reward value, and finally use the trained strategy model to determine the task execution result in at least one sampling result, thereby achieving automatic and accurate matching of multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the strategy model, and using the reward value to train the strategy model to improve the evaluation accuracy of the strategy model, thereby obtaining a task execution result that is more in line with the task description, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0023] Figure 1 This is a flowchart of a method for generating task execution results based on a policy model provided in an embodiment of the present application.

[0024] Figure 2 It is a flowchart of a method for determining a set of reward evaluation agents based on a task description provided in an embodiment of the present application.

[0025] Figure 3 It is a flowchart of a method for determining the weight of each reward evaluation agent based on task description provided in an embodiment of the present application.

[0026] Figure 4 It is a flowchart of a method provided in an embodiment of the present application for determining the reward value of a strategy model corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent.

[0027] Figure 5 This is a flowchart of a method for determining a task execution result in at least one sampling result using a trained strategy model provided in an embodiment of the present application.

[0028] Figure 6 This is a flowchart of another method for generating task execution results based on a policy model provided in an embodiment of the present application.

[0029] Figure 7 This is a schematic diagram of an apparatus for generating task execution results based on a policy model provided in an embodiment of the present application.

[0030] Figure 8 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0032] A method and apparatus for generating task execution results based on a policy model according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0033] As mentioned above, for non-deterministic tasks, due to the difficulty in quantifying the reward rules in reinforcement learning algorithms, the generated task execution results need to be improved. Specifically, the following problems exist in using policy models to automatically generate task execution results for non-deterministic tasks:

[0034] 1) High data dependency, relying on large amounts of data containing human feedback, resulting in high data construction costs;

[0035] 2) For non-deterministic question scenarios such as creative and personalized Q&A, a question may correspond to multiple reasonable answers. Existing reward models can only assign a certainty score and output a single answer, making it difficult to cover a wide range of styles.

[0036] 3) Directly using human-aligned preference data for reward model training is prone to reward-hacking, which can cause the model to fail to train properly.

[0037] 4) For non-deterministic tasks, if the reward model is not robust enough, it may cause unstable reward signals, making it difficult to train the policy model.

[0038] In view of this, an embodiment of the present application provides a method for generating task execution results based on a strategy model, by determining a set of reward evaluation agents based on the task description, and determining a weight for each reward evaluation agent in the set; using a pre-trained strategy model to obtain at least one sampling result based on the task description, using each reward evaluation agent to evaluate each sampling result respectively, and determining the strategy model reward value corresponding to each sampling result based on the weight and evaluation result of each reward evaluation agent, training the strategy model based on the reward value, and finally using the trained strategy model to determine the task execution result in at least one sampling result, thereby achieving automatic and accurate matching of multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the strategy model, and using the reward value to train the strategy model to improve the evaluation accuracy of the strategy model, thereby obtaining a task execution result that is more in line with the task description, thereby improving the user experience.

[0039] Figure 1 This is a flow chart of a method for generating task execution results based on a policy model provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0040] In step S101 , a task description is obtained.

[0041] The evaluation criteria for the execution results of the tasks vary according to external conditions.

[0042] In step S102, a reward evaluation agent set and a weight of each reward evaluation agent in the reward evaluation agent set are determined based on the task description.

[0043] In step S103, the task description is input into the pre-trained strategy model to obtain at least one sampling result.

[0044] In step S104, each reward evaluation agent is used to evaluate each sampling result respectively, and the strategy model reward value corresponding to each sampling result is determined based on the weight and evaluation result of each reward evaluation agent.

[0045] In step S105 , the pre-trained policy model is trained based on the reward value to obtain a trained policy model.

[0046] In step S106 , the trained policy model is used to determine a task execution result from at least one sampling result.

[0047] In some embodiments of the present application, the method may be executed by a server, or by a terminal device with certain processing capabilities.

[0048] In some embodiments of the present application, a task description may be first obtained. The task description may be input by a user and include description information of at least one of a task scenario, a task goal, and other task conditions.

[0049] The evaluation criteria for the execution result of the task may change with changes in the evaluation conditions. Among them, the evaluation conditions may include any one or more of the task scenario, the task execution object, the task result object, the task evaluator, and other evaluation conditions. For example, if the task is to solve a math problem, since the answer to the math problem is certain and the same under different evaluation conditions, it is not a task whose evaluation criteria for the execution result changes with changes in the evaluation conditions. On the contrary, if the task is to write a short article with a specific subject, since under different evaluation conditions, such as a short article for publishing rules and a short article for publicity, or evaluated by user A or by user B, the evaluation results of the writing quality of the short article may vary greatly, so this task is a task whose evaluation criteria for the execution result changes with changes in the evaluation conditions.

[0050] In certain embodiments of the present application, a set of reward evaluation agents can be determined based on the task description, and the weight of each reward evaluation agent in the set can be determined. In other words, a specific set of reward evaluation agents can be determined for the task, and each reward evaluation agent in the set is compatible with the task and used to evaluate the task execution results.

[0051] At the same time, you can also set weights for each reward evaluation agent in the reward evaluation agent set. The specific method for determining the reward evaluation agent set and the method for setting weights for each reward evaluation agent will be described in detail later and will not be repeated here.

[0052] In some embodiments of the present application, the task description can be input into a pre-trained policy model to obtain at least one sampled result. In one example, the policy model can be a policy model based on a reinforcement learning algorithm, which receives the task description and automatically generates a corresponding task execution result based on the task description. The task execution result generated by the pre-trained policy model can be referred to as a sampled result.

[0053] Each reward evaluation agent in the reward evaluation agent set can be used to evaluate each sampling result. At the same time, for each sampling result, the policy model reward value corresponding to the sampling result can also be determined based on the evaluation results of each reward evaluation agent and the weight of each reward evaluation agent.

[0054] In some implementations, the policy model reward value corresponding to each sampling result can be fed back to the pre-trained policy model, and the pre-trained policy model can be trained based on the policy model reward value to obtain a trained policy model. Finally, the trained policy model is used to determine one or more optimal sampling results from the at least one sampling result as the execution result of the current task.

[0055] According to the technical solution provided in the embodiments of the present application, a set of reward evaluation agents is determined based on the task description, and a weight is determined for each reward evaluation agent in the set; a pre-trained strategy model is used to obtain at least one sampling result based on the task description, and each reward evaluation agent is used to evaluate each sampling result respectively, and the strategy model reward value corresponding to each sampling result is determined based on the weight and evaluation result of each reward evaluation agent, and the strategy model is trained based on the reward value. Finally, the task execution result is determined in at least one sampling result using the trained strategy model, thereby achieving automatic and accurate matching of multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the strategy model, and using the reward value to train the strategy model to improve the evaluation accuracy of the strategy model, thereby obtaining a task execution result that is more in line with the task description, thereby improving the user experience.

[0056] Figure 2 This is a flow chart of a method for determining a reward evaluation agent set based on a task description provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0057] In step S201, scenario analysis is performed on the task description to obtain scenario information of the task.

[0058] The scenario information includes at least one of the following: task domain, task audience, and task target object.

[0059] In step S202, the task description is parsed to obtain the task objective.

[0060] Among them, the task objectives include at least one of the following: publicity planning, user flow, and training.

[0061] In step S203 , the task evaluation role is determined based on the scenario information and the task goal.

[0062] In step S204, a reward evaluation agent is determined for each task evaluation role to obtain a reward evaluation agent set.

[0063] In some embodiments of the present application, a task description may first be scenario parsed to obtain the task's scenario information. This scenario information may include at least one of the following: task domain, task audience, and task target object. The task audience may be the task initiator, i.e., the task execution object in the task description; the task target object may be the target object of the task, i.e., the task result object in the task description.

[0064] For example, if the task description includes "Generate marketing copy for a specific brand for a certain product," then the specific brand is the task audience, and the potential consumer group of the product is the task target object. Another example is, if the task description includes "Generate writing examples on a certain topic for middle school teachers," then middle school teachers are the task audience, and middle school students are the task target object.

[0065] At the same time, you can also perform goal parsing on the task description to obtain the task objective. Task objectives include at least one of the following: publicity planning, user traffic generation, and training. In actual use, task objectives may also include other objectives, which are not limited here.

[0066] The task evaluation role can be determined based on the scenario information and the task objectives. Different task evaluation roles can correspond to different task evaluation dimensions to evaluate the task execution results. Among them, the task evaluation role can be obtained based on the scenario information or the matching of independent items in the task objectives, or it can be determined by combining multiple items in the scenario information, or by combining multiple items in the scenario information and the task objectives. For example, based on the task objective of publicity planning, a task evaluation role can be directly determined as a marketing expert; based on the task field and task target object in the scenario information, a task evaluation role can be determined as a brand user in that field.

[0067] In some implementations, implicit information can also be considered when determining task evaluation roles. In one example, implicit information can include compliance requirements. Even if the user does not specify the compliance requirements that the task must meet in the task description, the appropriate compliance evaluation role can be automatically matched based on the task scope, task target, and other factors.

[0068] After the task evaluation roles are determined, a reward evaluation agent can be determined for each task evaluation role, thereby obtaining the reward evaluation agent set.

[0069] Taking the task description of "generating high-quality marketing copy for a specific brand in a marketing scenario" as an example, after scenario analysis, goal analysis, and role adaptation, the following task evaluation roles can be automatically generated:

[0070] Role 1: Brand side, responsible for scoring the sampling results of the strategy model from the perspectives of brand tone fit, brand values, and marketing content and brand consistency;

[0071] Role 2: Marketing expert, responsible for scoring the sampling results of the strategy model based on factors such as marketing logic, copywriting, brand and platform fit, audience positioning, and conversion strategy.

[0072] Role 3: Data analysis expert, responsible for scoring the estimated conversion rate, historical success rate, and click-through rate of the sampling results of the strategy model;

[0073] Role 4: Creativity review expert, responsible for scoring the uniqueness, novelty, and creativity of the strategy model's sampling results;

[0074] Role 5: Audit Expert, responsible for scoring the compliance, sensitive words, and other dimensions of the sampling results of the strategy model;

[0075] Role 6: Brand user, responsible for scoring the sampling results of the strategy model based on dimensions such as interest, relevance, emotional resonance, and points of interest.

[0076] In this way, task evaluation roles can be automatically matched according to user descriptions, thereby improving the matching degree between the determined reward evaluation agent and the task, thereby improving the efficiency of policy model training, so that the trained policy model can better generate task execution results.

[0077] Figure 3 This is a flow chart of a method for determining the weight of each reward evaluation agent based on task description provided by an embodiment of the present application. Figure 3 As shown, the method includes the following steps:

[0078] In step S301 , the importance of each task evaluation role is determined based on at least the task goal in the task description.

[0079] In step S302, the weight of the reward evaluation agent is determined based on the importance of the task evaluation role.

[0080] In some embodiments of the present application, determining the weight of each reward evaluation agent based on the task description can be, first, determining the importance of each task evaluation role based at least on the task goal in the task description; and then determining the weight of the reward evaluation agent based on the importance of the task evaluation role.

[0081] For example, for the various task evaluation roles generated above for the task description "Generate high-quality marketing copy for a specific brand in a marketing scenario," the role of brand is naturally important, but the task description does not emphasize brand-related matters, so it can be matched with a medium weight, such as 0.2. As for the role of marketing expert, since the task description emphasizes the need to generate "high-quality marketing copy," it can be matched with a high weight, such as 0.3. At the same time, the role of creative review expert may also be related to "high-quality marketing copy," so it can also be matched with a medium weight, such as 0.2. For other roles, they can be matched with a low weight, such as 0.1.

[0082] In this way, weights can be automatically assigned according to the content of the task description, thereby better integrating the evaluation results of each reward evaluation agent and improving the fusion accuracy.

[0083] Figure 4 This is a flow chart of a method for determining the reward value of a strategy model corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent provided by an embodiment of the present application. Figure 4 As shown, the method includes the following steps:

[0084] In step S401, for each sampling result, the evaluation results of each reward evaluation agent are weighted and summed to obtain the weighted sum of the evaluation results.

[0085] In step S402 , in response to determining that the weighted sum is greater than or equal to the first value, the policy model reward value corresponding to the sampling result is determined to be a preset reward value.

[0086] In step S403 , in response to determining that the weighted sum is less than or equal to the second value, the policy model reward value corresponding to the sampling result is determined to be 0.

[0087] In step S404 , in response to determining that the weighted sum is greater than the second value and less than the first value, the sampling result is discarded.

[0088] In some embodiments of the present application, when determining the strategy model reward value corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent, the evaluation results of each reward evaluation agent can be weighted and summed for each sampling result to obtain the weighted sum of the evaluation results.

[0089] Taking the task evaluation roles generated for the task description "Generate high-quality marketing copy for a specific brand in a marketing scenario" as an example, if the evaluation results given by the reward evaluation agent corresponding to each task evaluation role for a certain sampling result are 0.8, 0.5, 0.1, 0.9, 1 and 0.2 respectively, then they can be weighted and summed up to obtain the weighted sum of the evaluation results as 0.2*0.8+0.3*0.5+0.1*0.1+0.2*0.9+0.1*1+0.1*0.2=0.62.

[0090] A first value and a second value can be preset. If it is determined that the weighted sum is greater than or equal to the first value, the strategy model reward value corresponding to the current sampling result is determined to be the preset reward value; if it is determined that the weighted sum is less than or equal to the second value, the strategy model reward value corresponding to the current sampling result is determined to be 0; if it is determined that the weighted sum is greater than the second value and less than the first value, the current sampling result is discarded.

[0091] In one example, the first value can be set to 0.7 and the second value can be set to 0.3. For sampling results with a weighted sum greater than or equal to 0.7, the sampling result can be considered a good result, and the corresponding policy model reward value is determined to be a preset reward value, such as 1. For sampling results with a weighted sum less than or equal to 0.3, the sampling result can be considered a poor result, and the corresponding policy model reward value is directly determined to be 0. For sampling results with a weighted sum greater than 0.3 and less than 0.7, the sampling result can be considered overly balanced and determined to be an invalid sample and discarded.

[0092] It is understandable that the specific values ​​of the first value and the second value can be set according to actual needs and are not limited here. At the same time, the sum of the weights of each reward evaluation agent is 1.

[0093] In this way, the evaluation results of each reward evaluation agent can be better integrated, so that the integrated reward value can better serve the training of the strategy model and improve the accuracy of the strategy model.

[0094] In some embodiments of the present application, if the number of discarded sampling results exceeds a first preset threshold, a first evaluation result distribution may be calculated. The first evaluation result distribution is the distribution of the results of each reward evaluation agent on the discarded sampling results. The weight of each reward evaluation agent is then modified based on the distribution.

[0095] In one example, if the number of discarded sampling results is greater than the first preset number threshold, the distribution of the evaluation results of each reward evaluation agent on these discarded sampling results can be counted. If it is determined that the evaluation result values ​​of a certain reward evaluation agent for each sampling result in these evaluation results are all high, and the weight of this reward evaluation agent is small, such as the evaluation results of the target reward evaluation agent for these discarded sampling results, the number of evaluation result values ​​higher than the preset evaluation threshold is greater than the fourth preset number threshold, and the weight of the reward evaluation agent corresponds to the low value weight, then the role function of the target reward evaluation agent can be matched with the task description again, and the weight of the target reward evaluation agent can be adjusted according to the matching result.

[0096] For example, if the matching degree between the role function of the target reward evaluation agent and the task description is higher than a preset matching threshold, the weight of the target reward evaluation agent can be increased. Otherwise, the weight of the target reward evaluation agent is not adjusted.

[0097] In some embodiments of the present application, if it is determined that the number of sampling results whose weighted sum is greater than or equal to the first value is less than a second preset number threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than a third preset number threshold, a second evaluation result distribution is calculated. The second evaluation result distribution is the distribution of evaluation results of each reward evaluation agent. Then, based on the evaluation result distribution, at least one reward evaluation agent in the set of reward evaluation agents is updated.

[0098] In one example, if the number of sampling results whose weighted sum is greater than or equal to the first value is less than the second preset number threshold, that is, there are fewer good sampling results at present, before directly adjusting the parameters of the strategy model, you can first try to adjust the weights of each reward evaluation agent to check whether the result is caused by unreasonable setting of the reward function.

[0099] In another example, if the number of sampling results whose weighted sum is less than or equal to the second value is greater than the third preset number threshold, that is, there are more poor sampling results at present, before directly adjusting the parameters of the strategy model, you can also first try to adjust the weights of each reward evaluation agent to check whether the result is caused by unreasonable setting of the reward function.

[0100] For example, the evaluation results of each reward evaluation agent set for each sampling result can be counted. For the reward evaluation agents with higher evaluation result values ​​and smaller weights, their role functions are matched with the task descriptions again, and the weight of the target reward evaluation agent is adjusted according to the matching results.

[0101] For example, if the matching degree between the role functions of one or more of these target reward evaluation agents and the task description is higher than a preset matching threshold, the weight of the reward evaluation agent can be increased. Otherwise, the weight of the reward evaluation agent is not adjusted.

[0102] After adjusting the weights of the reward evaluation agents, the adjusted set of reward evaluation agents can be used to re-evaluate each of the at least one sampled result. The policy model reward value corresponding to each sampled result is re-determined based on the adjusted weights of the reward evaluation agents and the re-evaluation results. The re-determined reward value is then used to train the pre-trained policy model to obtain a trained policy model, and the task execution result is determined based on the trained policy model.

[0103] If, when re-determining the reward value of the strategy model corresponding to each sampling result, the number of discarded sampling results is still greater than the first preset number threshold, or the number of sampling results whose weighted sum is greater than or equal to the first value is still less than the second preset number threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is still greater than the third preset number threshold, the weights of each reward evaluation agent will no longer be adjusted, but the parameters of the strategy model will be adjusted.

[0104] Among them, the values ​​of the first to fourth preset quantity thresholds are set according to actual needs and are not limited here.

[0105] In this way, when the policy model performs poorly, the weights of each reward evaluation agent can be adjusted first. If it still does not meet the requirements, the parameters of the policy model can be adjusted again to improve the efficiency of model training.

[0106] In some embodiments of the present application, training the pre-trained policy model based on the reward value may be to train the pre-trained policy model based on the reward value of each sampling result using a PPO (Proximal Policy Optimization) algorithm or a GRPO (Group Relative Policy Optimization) algorithm to obtain a trained policy model.

[0107] Figure 5 1 is a flow chart of a method for determining a task execution result in at least one sampling result using a trained strategy model provided by an embodiment of the present application. Figure 5 As shown, the method includes the following steps:

[0108] In step S501 , a target sampling result among at least one sampling result is determined based on a PPO algorithm or a GRPO algorithm.

[0109] In step S502 , it is determined that the target sampling result is a task execution result.

[0110] When determining a task execution result from at least one sampling result using the trained policy model, a target sampling result from the at least one sampling result may be determined based on a PPO algorithm or a GRPO algorithm. The target sampling result is a sampling result that meets preset performance requirements. The target sampling result is then determined as the task execution result.

[0111] Figure 6 This is a flow chart of another method for generating task execution results based on a policy model provided by an embodiment of the present application. Figure 6 As shown, the policy model can first parse the input task description (or input question) to initialize the reward evaluation agent. In one example, the reward evaluation agent role can be autonomously constructed based on the scenario system, setting the role and personality information for the generative reward evaluation agent.

[0112] Next, the pre-trained policy model samples the input task description and feeds the sampled results into each generative reward evaluation agent for evaluation and scoring. The reward score of each reward evaluation agent can be constrained to be between 0 and 1. The reward scores of each reward evaluation agent are then aggregated. The final reward score is the sum of the product of the reward scores of each reward evaluation agent and its weight.

[0113] Then the reward conversion can be performed. If the final reward score after aggregation is greater than or equal to the first value, the reward score corresponding to the sampling result is determined to be the preset reward value; if the final reward score after aggregation is less than or equal to the second value, the reward score corresponding to the sampling result is determined to be 0; if the final reward score after aggregation is greater than the second value and less than the first value, the sampling result is discarded.

[0114] Finally, the converted reward is fed back to the pre-trained policy model, and the pre-trained model is trained using the PPO algorithm or the GRPO algorithm to obtain a trained policy model. The trained policy model can determine the execution result of this task based on the sampling results.

[0115] The technical solutions provided by the embodiments of this application can provide reward signals with more dimensions for model training, improve the robustness of model training, reduce the probability of reward hacking in reinforcement learning, and provide better interpretability. Generative reward results can be used to analyze issues and biases in policy models. Furthermore, this method is more flexible, and the reward evaluation agent can be dynamically adjusted according to the needs of different task scenarios.

[0116] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0117] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0118] Figure 7 Schematic diagram of a device for generating task execution results based on a strategy model provided by an embodiment of the present application. Figure 7 As shown, the device includes:

[0119] The acquisition module 701 is configured to acquire a task description. The evaluation criteria of the task execution result changes with the evaluation conditions.

[0120] The determination module 702 is configured to determine a set of reward evaluation agents and a weight of each reward evaluation agent in the set of reward evaluation agents based on the task description.

[0121] The sampling module 703 is configured to input the task description into the pre-trained strategy model to obtain at least one sampling result.

[0122] The evaluation module 704 is configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weight of each reward evaluation agent and the evaluation result.

[0123] The training module 705 is configured to train the pre-trained policy model based on the reward value to obtain a trained policy model.

[0124] The generating module 706 is configured to use the trained policy model to determine the task execution result from at least one sampling result.

[0125] According to the technical solution provided in the embodiments of the present application, a set of reward evaluation agents is determined based on the task description, and a weight is determined for each reward evaluation agent in the set; a pre-trained strategy model is used to obtain at least one sampling result based on the task description, and each reward evaluation agent is used to evaluate each sampling result respectively, and the strategy model reward value corresponding to each sampling result is determined based on the weight and evaluation result of each reward evaluation agent, and the strategy model is trained based on the reward value. Finally, the task execution result is determined in at least one sampling result using the trained strategy model, thereby achieving automatic and accurate matching of multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the strategy model, and using the reward value to train the strategy model to improve the evaluation accuracy of the strategy model, thereby obtaining a task execution result that is more in line with the task description, thereby improving the user experience.

[0126] In some embodiments, determining a set of reward evaluation agents based on a task description includes: performing scenario analysis on the task description to obtain scenario information of the task; the scenario information includes at least one of the following: task domain, task audience, and task target object; performing goal analysis on the task description to obtain task goals; the task goals include at least one of the following: publicity planning, user flow, and training; determining task evaluation roles based on the scenario information and task goals; and determining a reward evaluation agent for each task evaluation role to obtain a set of reward evaluation agents.

[0127] In some embodiments, the weight of each reward evaluation agent is determined based on the task description, including: determining the importance of each task evaluation role based on at least the task goal in the task description; and determining the weight of the reward evaluation agent based on the importance of the task evaluation role.

[0128] In some embodiments, the strategy model reward value corresponding to each sampling result is determined based on the weight and evaluation result of each reward evaluation agent, including: for each sampling result, weighted summing the evaluation results of each reward evaluation agent to obtain the weighted sum of the evaluation results; in response to determining that the weighted sum is greater than or equal to a first value, determining that the strategy model reward value corresponding to the current sampling result is a preset reward value; in response to determining that the weighted sum is less than or equal to a second value, determining that the strategy model reward value corresponding to the current sampling result is 0; in response to determining that the weighted sum is greater than the second value and less than the first value, discarding the current sampling result.

[0129] In some embodiments, the step of training a pre-trained strategy model based on a reward value is to train the pre-trained strategy model based on the reward value of each sampling result using a proximal strategy optimization PPO algorithm or a group relative strategy optimization GRPO algorithm to obtain a trained strategy model; using the trained strategy model, determining the task execution result in at least one sampling result, including: determining a target sampling result in at least one sampling result based on the PPO algorithm or the GRPO algorithm; and determining the target sampling result as the task execution result.

[0130] In some embodiments, it also includes: in response to determining that the number of discarded sampling results is greater than a first preset number threshold, counting the distribution of the first evaluation results; the first evaluation result distribution is the result distribution of each reward evaluation agent for the discarded sampling results; and correcting the weight of each reward evaluation agent based on the distribution.

[0131] In some embodiments, it also includes: in response to determining that the number of sampling results whose weighted sum is greater than or equal to the first value is less than a second preset number threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than a third preset number threshold, counting the distribution of the second evaluation results; the second evaluation result distribution is the distribution of the evaluation results of each reward evaluation agent; and updating at least one reward evaluation agent in the reward evaluation agent set based on the evaluation result distribution.

[0132] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0133] Figure 8 Schematic diagram of an electronic device provided in an embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable by the processor 801. When the processor 801 executes the computer program 803, the steps of the above-described method embodiments are implemented. Alternatively, when the processor 801 executes the computer program 803, the functions of the modules / units in the above-described device embodiments are implemented.

[0134] The electronic device 8 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 8 may include but is not limited to a processor 801 and a memory 802. Those skilled in the art will appreciate that Figure 8 This is merely an example of the electronic device 8 and does not limit the electronic device 8 . The electronic device 8 may include more or fewer components than shown in the figure, or different components.

[0135] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0136] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard drive or memory of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. The memory 802 can also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0137] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0138] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0139] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for generating task execution results based on a strategy model, characterized in that: include: Obtaining a task description, wherein the evaluation criteria for the execution result of the task changes as the evaluation conditions change; Determining a reward evaluation agent set and a weight of each reward evaluation agent in the reward evaluation agent set based on the task description; Inputting the task description into a pre-trained strategy model to obtain at least one sampling result; Use each reward evaluation agent to evaluate each sampling result separately, and determine the strategy model reward value corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent; Training the pre-trained strategy model based on the reward value to obtain a trained strategy model; Determine a task execution result from the at least one sampled result using the trained policy model; Determining a reward evaluation agent set based on the task description includes: Performing scenario analysis on the task description to obtain scenario information of the task including at least one of the task domain, task audience, and task target object; Performing objective analysis on the task description to obtain a task objective including at least one of publicity planning, user flow generation, and training; Determining a task evaluation role based on the scenario information and the task goal; Determine a reward evaluation agent for each task evaluation role to obtain a set of reward evaluation agents; Determine the weight of each reward evaluation agent based on the task description, including: Determining the importance of each task evaluation role based at least on the task objectives in the task description; The weight of the reward evaluation agent is determined based on the importance of the task evaluation role, and the sum of the weights of the reward evaluation agents is 1; The reward value of the strategy model corresponding to each sampling result is determined based on the weight and evaluation results of each reward evaluation agent, including: For each sampling result, the evaluation results of each reward evaluation agent are weighted and summed to obtain the weighted sum of the evaluation results; In response to determining that the weighted sum is greater than the second value and less than the first value, discarding the sampling result; In response to determining that the number of discarded sampling results is greater than a first preset number threshold, calculating a first evaluation result distribution, and modifying the weight of each reward evaluation agent based on the first evaluation result distribution, wherein the first evaluation result distribution is a distribution of results of each reward evaluation agent for the discarded sampling results; In response to determining that the number of sampling results whose weighted sum is greater than or equal to the first value is less than a second preset number threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than a third preset number threshold, the second evaluation result distribution is counted, and at least one reward evaluation agent in the reward evaluation agent set is updated based on the second evaluation result distribution, where the second evaluation result distribution is the evaluation result distribution of each reward evaluation agent.

2. The method according to claim 1, characterized in that Determining the reward value of the strategy model corresponding to each sampling result based on the weight and evaluation results of each reward evaluation agent also includes: In response to determining that the weighted sum is greater than or equal to the first value, determining that the policy model reward value corresponding to the sampling result is a preset reward value; In response to determining that the weighted sum is less than or equal to the second value, the policy model reward value corresponding to the sampling result is determined to be 0.

3. The method according to claim 1, characterized in that The step of training the pre-trained strategy model based on the reward value is to train the pre-trained strategy model using a proximal strategy optimization (PPO) algorithm or a group relative strategy optimization (GRPO) algorithm based on the reward value of each sampling result to obtain a trained strategy model; Determining a task execution result from the at least one sampling result using the trained policy model includes: Determine a target sampling result among the at least one sampling result based on the PPO algorithm or the GRPO algorithm; Determine the target sampling result as the task execution result.

4. A device for generating task execution results based on a strategy model, characterized in that: include: an acquisition module configured to acquire a task description, wherein the evaluation criteria for the execution result of the task changes as the evaluation conditions change; a determination module configured to determine a set of reward evaluation agents and a weight of each reward evaluation agent in the set of reward evaluation agents based on the task description; a sampling module configured to input the task description into a pre-trained strategy model to obtain at least one sampling result; An evaluation module is configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine a policy model reward value corresponding to each sampling result based on the weight of each reward evaluation agent and the evaluation result; A training module is configured to train the pre-trained strategy model based on the reward value to obtain a trained strategy model; a generating module configured to determine a task execution result from the at least one sampling result using the trained policy model; The determination module is specifically configured to: perform scenario analysis on the task description to obtain scenario information of the task including at least one of the task domain, the task audience, and the task target object; determine the task evaluation role based on the scenario information and the task target; determine a reward evaluation agent for each task evaluation role to obtain the reward evaluation agent set; perform goal analysis on the task description to obtain the task target including at least one of publicity planning, user flow, and training; The determining module is further configured to: determine the importance of each task evaluation role based at least on the task objective in the task description; determine the weight of the reward evaluation agent based on the importance of the task evaluation role, wherein the sum of the weights of the reward evaluation agents is 1; The evaluation module is specifically configured to: for each sampling result, perform weighted summation of the evaluation results of each reward evaluation agent to obtain a weighted sum of the evaluation results; in response to determining that the weighted sum is greater than the second value and less than the first value, discard the sampling result; In response to determining that the number of discarded sampling results is greater than a first preset number threshold, the first evaluation result distribution is counted, and the weight of each reward evaluation agent is corrected based on the first evaluation result distribution, where the first evaluation result distribution is the result distribution of each reward evaluation agent for the discarded sampling results; in response to determining that the number of sampling results whose weighted sum is greater than or equal to the first value is less than a second preset number threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than a third preset number threshold, the second evaluation result distribution is counted, and at least one reward evaluation agent in the reward evaluation agent set is updated based on the second evaluation result distribution, where the second evaluation result distribution is the evaluation result distribution of each reward evaluation agent.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images