Method and device for generating task execution result, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a method and device for generating a task execution result, equipment and a medium. According to the method, a reward evaluation agent set is determined based on task description, and a weight is determined for each reward evaluation agent in the set; and obtaining at least one sampling result based on the task description by using a pre-training strategy model, evaluating each sampling result by using each reward evaluation agent, determining a strategy model reward value based on the weight of each reward evaluation agent and the evaluation result, training the strategy model based on the reward value, and obtaining the reward value of the reward evaluation agent. And finally, determining a task execution result by using the trained strategy model, so that a plurality of appropriate reward evaluation agents are automatically and accurately matched for a non-deterministic task, and the plurality of reward evaluation agents are used for jointly training the strategy model to improve the evaluation accuracy of the strategy model. And thus, a task execution result more conforming to the task description is obtained, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a method, apparatus, device, and medium for generating task execution results. Background Art

[0002] Existing policy models implemented based on reinforcement learning methods have achieved good results in quantifiable tasks, such as tasks related to mathematical problems and tasks related to code programming problems, by regularizing rewards. However, for non-deterministic tasks, such as copywriting creativity tasks and short essay writing tasks, where it is impossible to clearly give the quality of the results and the evaluation is subjective, since it is difficult to quantify the reward rules in the reinforcement learning algorithm, the generated task execution results still need to be improved.

[0003] Therefore, how to improve the accuracy of the task execution results generated by the policy model for non-deterministic tasks is a technical problem that needs to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, device, and medium for generating task execution results to solve the problem in the prior art that the accuracy of the policy model for executing such tasks is insufficient due to inaccurate evaluation of non-deterministic tasks.

[0005] In the first aspect of the embodiments of this application, a method for generating task execution results based on a policy model is provided, including: Obtain a task description, where the evaluation criteria for the execution result of the task change with the evaluation conditions; Determine a set of reward evaluation agents based on the task description, and the weights of each reward evaluation agent in the set of reward evaluation agents; Input the task description into a pre-trained policy model to obtain at least one sampling result; Use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results; Train the pre-trained policy model based on the reward value to obtain a trained policy model; Use the trained policy model to determine the task execution result among at least one sampling result.

[0006] In the second aspect of the embodiments of this application, a device for generating task execution results based on a policy model is provided, including: An obtaining module, configured to obtain a task description, where the evaluation criteria for the execution result of the task change with the evaluation conditions; A determining module, configured to determine a set of reward evaluation agents based on the task description, and the weights of each reward evaluation agent in the set of reward evaluation agents; A sampling module, configured to input a task description into a pre-trained policy model to obtain at least one sampling result; An evaluation module, configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights of the reward evaluation agents and the evaluation results; A training module, configured to train the pre-trained policy model based on the reward value to obtain a trained policy model; A generation module, configured to use the trained policy model to determine a task execution result from at least one sampling result.

[0007] In a third aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0008] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] The beneficial effects of the embodiments of the present application compared with the prior art are as follows: By determining a set of reward evaluation agents based on the task description and determining weights for each reward evaluation agent in the set in the embodiments of the present application; using the pre-trained policy model to obtain at least one sampling result based on the task description, using each reward evaluation agent to evaluate each sampling result respectively, and determining the policy model reward value corresponding to each sampling result based on the weights of the reward evaluation agents and the evaluation results, training the policy model based on the reward value, and finally using the trained policy model to determine the task execution result from at least one sampling result, it is realized to automatically and accurately match multiple suitable reward evaluation agents for non-deterministic tasks, and use the multiple reward evaluation agents to jointly determine the reward value of the policy model, and use the reward value to train the policy model to improve the evaluation accuracy of the policy model, thereby obtaining a task execution result that more conforms to the task description and improving the user experience. Description of the Drawings

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0011] Figure 1It is a schematic flowchart of a method for generating a task execution result based on a policy model provided by an embodiment of the present application. Figure 2 It is a schematic flowchart of a method for determining a set of reward evaluation agents based on a task description provided by an embodiment of the present application.

[0012] Figure 3 It is a schematic flowchart of a method for determining the weight of each reward evaluation agent based on a task description provided by an embodiment of the present application.

[0013] Figure 4 It is a schematic flowchart of a method for determining the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results provided by an embodiment of the present application.

[0014] Figure 5 It is a schematic flowchart of a method for determining a task execution result from at least one sampling result using a trained policy model provided by an embodiment of the present application.

[0015] Figure 6 It is a schematic flowchart of another method for generating a task execution result based on a policy model provided by an embodiment of the present application.

[0016] Figure 7 It is a schematic diagram of a device for generating a task execution result based on a policy model provided by an embodiment of the present application.

[0017] Figure 8 It is a schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0018] In the following description, specific details such as specific system architectures and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0019] A method and a device for generating a task execution result based on a policy model according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0020] As mentioned above, for non-deterministic tasks, since it is difficult to quantify the reward rules in the reinforcement learning algorithm, the generated task execution results still need to be improved. Specifically, there are currently the following problems in automatically generating task execution results using a policy model for non-deterministic tasks: 1) Strong data dependence, relying on a large amount of data containing human feedback, and the data construction cost is high; 2) For non-deterministic question scenarios such as creativity and personalized Q&A, a single question may correspond to multiple reasonable answers. Existing reward models can only give a single deterministic score and then output a single answer, making it difficult to cover multiple styles. 3) Directly using preference data aligned with humans to train the reward model is extremely prone to reward-hacking, resulting in the model being unable to be trained properly. 4) For non-deterministic tasks, if the reward model is not robust enough, it may cause unstable reward signals, making it difficult to train the policy model.

[0021] In view of this, the embodiments of the present application provide a method for generating a task execution result based on a policy model. By determining a set of reward evaluation agents based on the task description and determining weights for each reward evaluation agent in the set; using a pre-trained policy model to obtain at least one sampling result based on the task description, using each reward evaluation agent to evaluate each sampling result respectively, and determining the policy model reward value corresponding to each sampling result based on the weights and evaluation results of each reward evaluation agent, training the policy model based on the reward value, and finally using the trained policy model to determine the task execution result among at least one sampling result, it realizes automatically and accurately matching multiple appropriate reward evaluation agents for non-deterministic tasks, jointly determining the reward value of the policy model using the multiple reward evaluation agents, training the policy model using the reward value to improve the evaluation accuracy of the policy model, and thus obtaining a task execution result that better conforms to the task description, enhancing the user experience.

[0022] Figure 1 It is a schematic flowchart of a method for generating a task execution result based on a policy model provided by the embodiments of the present application. As Figure 1 shown, the method includes the following steps: In step S101, obtain the task description.

[0023] Among them, the evaluation criteria for the execution result of the task change according to external conditions.

[0024] In step S102, determine a set of reward evaluation agents based on the task description, and the weights of each reward evaluation agent in the set of reward evaluation agents.

[0025] In step S103, input the task description into the pre-trained policy model to obtain at least one sampling result.

[0026] In step S104, use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights and evaluation results of each reward evaluation agent.

[0027] In step S105, the pre-trained policy model is trained based on the reward value to obtain the trained policy model.

[0028] In step S106, the trained policy model is used to determine the task execution result among at least one sampling result.

[0029] In some embodiments of the present application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0030] In some embodiments of the present application, the task description may be obtained first. The task description may be input by a user and include at least one of the description information of the task scenario, task objective, and other task conditions.

[0031] The evaluation criteria for the execution result of the task may change with the change of evaluation conditions. Among them, the evaluation conditions may include any one or more of the task scenario, task execution object, task result target object, task evaluator, and other evaluation conditions. For example, if the task is to solve a math problem, since the answer to the math problem is certain and the same under different evaluation conditions, it is not a task whose evaluation criteria for the execution result change with the evaluation conditions. On the contrary, if the task is to write a short essay with a specific subject, since under different evaluation conditions, such as a short essay for publishing rules and a short essay for publicity, or evaluated by user A or user B, the evaluation results of the writing quality of the short essay may vary greatly, so this task is a task whose evaluation criteria for the execution result change with the evaluation conditions.

[0032] In certain embodiments of the present application, the reward evaluation agent set may be determined based on the task description, and the weights of each reward evaluation agent in the reward evaluation agent set may be determined. That is to say, a specific reward evaluation agent set for the task may be determined, and each reward evaluation agent in the reward evaluation agent set can match the task and be used to evaluate the task execution result of the task.

[0033] At the same time, weights may also be set for each reward evaluation agent in the reward evaluation agent set. The specific method for determining the reward evaluation agent set and the method for setting the weights of each reward evaluation agent are described in detail later and will not be elaborated here.

[0034] In some embodiments of the present application, the task description may be input into the pre-trained policy model to obtain at least one sampling result. In one example, the policy model may be a policy model based on the reinforcement learning algorithm, which receives the task description and automatically generates a corresponding task execution result based on the task description. The task execution result generated by the pre-trained policy model may be referred to as a sampling result.

[0035] Each reward evaluation agent in the reward evaluation agent set can be used to evaluate each sampling result. At the same time, for each sampling result, the policy model reward value corresponding to the sampling result can also be determined based on the evaluation results of each reward evaluation agent on it and the weights of each reward evaluation agent.

[0036] In some embodiments, the policy model reward values corresponding to the sampling results can be fed back to the pre-trained policy model, and the pre-trained policy model can be trained based on the policy model reward values to obtain a trained policy model. Finally, the trained policy model is used to determine one or more optimal sampling results from the at least one sampling result as the task execution result of this time.

[0037] According to the technical solution provided by the embodiments of the present application, by determining a reward evaluation agent set based on the task description, determining weights for each reward evaluation agent in the set; using a pre-trained policy model to obtain at least one sampling result based on the task description, using each reward evaluation agent to evaluate each sampling result respectively, and determining the policy model reward values corresponding to the sampling results based on the weights and evaluation results of each reward evaluation agent, training the policy model based on the reward values, and finally using the trained policy model to determine the task execution result from at least one sampling result, it realizes automatically and accurately matching multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the policy model, using the reward value to train the policy model to improve the evaluation accuracy of the policy model, and further obtaining a task execution result that more conforms to the task description, thus enhancing the user experience.

[0038] Figure 2 is a schematic flowchart of a method for determining a reward evaluation agent set based on a task description provided by an embodiment of the present application. As Figure 2 shown, the method includes the following steps: In step S201, the task description is parsed for the scenario to obtain the scenario information of the task.

[0039] Among them, the scenario information includes at least one of the following: task domain, task audience, task target object.

[0040] In step S202, the task description is parsed for the target to obtain the task target.

[0041] Among them, the task target includes at least one of the following: publicity planning, user drainage, training.

[0042] In step S203, the task evaluation role is determined based on the scenario information and the task target.

[0043] In step S204, a reward evaluation agent is determined for each task evaluation role, obtaining a set of reward evaluation agents.

[0044] In some embodiments of the present application, the task description may first be subjected to scenario analysis to obtain the scenario information of the task. The scenario information may include at least one of the following: task domain, task audience, task target object. Among them, the task audience may be the initiator of the task, that is, the task execution object in the task description; the task target object may be the target object of the task, that is, the object that the task result is oriented to in the task description.

[0045] For example, if the task description includes "generate marketing copy for a certain product for a specific brand", then the specific brand is the task audience, and the potential consumer group of the product is the task target object. Another example, if the task description includes "generate writing examples for a certain theme for middle school teachers", then middle school teachers are the task audience, and middle school students are the task target object.

[0046] At the same time, the task description can also be subjected to target analysis to obtain the task target. The task target includes at least one of the following: publicity planning, user drainage, training. In actual use, the task target may also include other targets, which are not limited here.

[0047] The task evaluation role can be determined based on the scenario information and the task target. Different task evaluation roles can correspond to different task evaluation dimensions to evaluate the task execution result. Among them, the task evaluation role can be obtained by matching an independent item in the scenario information or the task target, or by combining multiple items in the scenario information, or by combining multiple items in the scenario information and the task target. For example, a task evaluation role of a marketing expert can be directly determined based on the task target of publicity planning; a task evaluation role of a brand user in this field can also be determined based on the task domain and the task target object in the scenario information.

[0048] In some embodiments, implicit information can also be considered when determining the task evaluation role. In one example, the implicit information may include compliance requirement information. Although the user does not mention in the task description which compliance requirements the task needs to meet, appropriate task compliance evaluation roles can be automatically matched based on the task domain, task target object, etc.

[0049] After determining the task evaluation role, a reward evaluation agent can be determined for each task evaluation role, thereby obtaining the set of reward evaluation agents.

[0050] Taking the task description "generate high-quality marketing copy for a specific brand in a marketing scenario" as an example, after scenario analysis, target analysis, and role adaptation, the following task evaluation roles can be automatically generated: Role 1: Brand owner, responsible for scoring the sampling results of the strategy model from the perspectives of brand tone fit, brand values, and consistency between marketing content and the brand. Role 2: Marketing expert, responsible for scoring the sampling results of the strategy model in dimensions including marketing logic, matching degree of copywriting with the brand and the placement platform, audience positioning, and conversion strategy. Role 3: Data analysis expert, responsible for scoring the sampling results of the strategy model in dimensions such as estimated conversion rate, historical success rate, and click-through rate estimation. Role 4: Creative review expert, responsible for scoring the sampling results of the strategy model in dimensions such as uniqueness, novelty, and creativity. Role 5: Review expert, responsible for scoring the sampling results of the strategy model in dimensions such as compliance and sensitive words. Role 6: Brand user, responsible for scoring the sampling results of the strategy model in dimensions such as interest level, relevance, emotional resonance, and interest points.

[0051] By adopting this method, it is possible to automatically match task evaluation roles according to user descriptions, improve the matching degree between the determined reward evaluation agents and tasks, and further improve the training efficiency of the strategy model, enabling the trained strategy model to better generate task execution results.

[0052] Figure 3 It is a schematic flowchart of the method for determining the weights of each reward evaluation agent based on task descriptions provided by the embodiments of the present application. As Figure 3 shown, the method includes the following steps: In step S301, determine the importance of each task evaluation role based at least on the task objective in the task description.

[0053] In step S302, determine the weight of this reward evaluation agent based on the importance of the task evaluation role.

[0054] In some embodiments of the present application, determining the weights of each reward evaluation agent based on task descriptions may be that, first, determine the importance of each task evaluation role based at least on the task objective in the task description; then determine the weight of this reward evaluation agent based on the importance of the task evaluation role.

[0055] For example, for each task evaluation role generated above for the task description "generate high-quality marketing copy for a specific brand in a marketing scenario", among which the role of the brand side is of natural importance, but the brand-related matters are not emphasized in the task description. Therefore, a medium-weight value can be assigned to it, such as 0.2. For the role of the marketing expert, since the task description emphasizes the need to generate "high-quality marketing copy", a high-weight value can be assigned to it, such as 0.3. At the same time, the role of the creative review expert may also be related to "high-quality marketing copy", so a medium-weight value can also be assigned to it, such as 0.2. For other roles, a low-weight value can be assigned to them, such as 0.1.

[0056] In this way, the weights can be automatically assigned according to the content of the task description, so as to better integrate the evaluation results of each reward evaluation agent and improve the integration accuracy.

[0057] Figure 4 It is a schematic flowchart of a method for determining the reward value of the policy model corresponding to each sampling result based on the weights and evaluation results of each reward evaluation agent provided by an embodiment of the present application. As Figure 4 shown, the method includes the following steps: In step S401, for each sampling result, the evaluation results of each reward evaluation agent are weighted and summed to obtain the weighted sum of the evaluation results.

[0058] In step S402, in response to determining that the weighted sum is greater than or equal to the first value, it is determined that the reward value of the policy model corresponding to this sampling result is the preset reward value.

[0059] In step S403, in response to determining that the weighted sum is less than or equal to the second value, it is determined that the reward value of the policy model corresponding to this sampling result is 0.

[0060] In step S404, in response to determining that the weighted sum is greater than the second value and less than the first value, this sampling result is discarded.

[0061] In some embodiments of the present application, when determining the reward value of the policy model corresponding to each sampling result based on the weights and evaluation results of each reward evaluation agent, for each sampling result, the evaluation results of each reward evaluation agent can be weighted and summed to obtain the weighted sum of the evaluation results.

[0062] Taking the above-mentioned task evaluation roles generated for the task description of "generating high-quality marketing copy for a specific brand in a marketing scenario" as an example, if the evaluation results given by the reward evaluation agents corresponding to the task evaluation roles for a certain sampling result are 0.8, 0.5, 0.1, 0.9, 1, and 0.2 respectively, then their weighted sum can be calculated, and the weighted sum of the evaluation results is 0.2 * 0.8 + 0.3 * 0.5 + 0.1 * 0.1 + 0.2 * 0.9 + 0.1 * 1 + 0.1 * 0.2 = 0.62.

[0063] A first value and a second value can be preset. If it is determined that the weighted sum is greater than or equal to the first value, then the reward value of the policy model corresponding to this sampling result is determined to be the preset reward value; if it is determined that the weighted sum is less than or equal to the second value, then the reward value of the policy model corresponding to this sampling result is determined to be 0; if it is determined that the weighted sum is greater than the second value and less than the first value, then this sampling result is discarded.

[0064] In one example, the first value can be set to 0.7 and the second value can be set to 0.3. For the sampling results with a weighted sum greater than or equal to 0.7, it can be considered that this sampling result is a good result, so the reward value of the corresponding policy model is determined to be the preset reward value, for example, 1. For the sampling results with a weighted sum less than or equal to 0.3, it can be considered that this sampling result is a poor result, and then the reward value of the corresponding policy model is directly determined to be 0. For the sampling results with a weighted sum greater than 0.3 and less than 0.7, it can be considered that this sampling result is too balanced and is determined to be an invalid sample and discarded.

[0065] It can be understood that the specific values of the first value and the second value can be set according to actual needs and are not limited here. At the same time, the sum of the weights of each reward evaluation agent is 1.

[0066] Adopting this method can better integrate the evaluation results of each reward evaluation agent, so that the fused reward value can better serve the training of the policy model and improve the accuracy of the policy model.

[0067] In some embodiments of the present application, if the number of discarded sampling results is greater than the first preset quantity threshold, the first evaluation result distribution can also be statistically analyzed. Among them, the first evaluation result distribution is the result distribution of each reward evaluation agent for the discarded sampling results. Then, the weights of each reward evaluation agent are corrected based on the distribution.

[0068] In one example, if the number of discarded sampling results is greater than the first preset number threshold, the evaluation result distribution of each reward evaluation agent for these discarded sampling results can be counted. If it is determined that the evaluation result values of a certain reward evaluation agent for each sampling result in these evaluation results are all relatively high, and the weight of this reward evaluation agent is small. For example, in the evaluation results of the target reward evaluation agent for these discarded sampling results, the number of evaluation result values higher than the preset evaluation threshold is greater than the fourth preset number threshold, and the weight of this reward evaluation agent corresponds to a low-value weight, then the role function of the target reward evaluation agent can be rematched with the task description, and the weight of the target reward evaluation agent can be adjusted according to the matching result.

[0069] For example, if the matching degree between the role function of the target reward evaluation agent and the task description is higher than the preset matching threshold, the weight of the target reward evaluation agent can be increased. Otherwise, the weight of the target reward evaluation agent is not adjusted.

[0070] In some embodiments of the present application, if it is determined that the number of sampling results with a weighted sum greater than or equal to the first value is less than the second preset number threshold, or the number of sampling results with a weighted sum less than or equal to the second value is greater than the third preset number threshold, the second evaluation result distribution is counted. Wherein, the second evaluation result distribution is the evaluation result distribution of each reward evaluation agent. Then, at least one reward evaluation agent in the reward evaluation agent set is updated based on the evaluation result distribution.

[0071] In one example, if it is determined that the number of sampling results with a weighted sum greater than or equal to the first value is less than the second preset number threshold, that is, there are fewer current good sampling results. Before directly adjusting the parameters of the policy model, the weights of each reward evaluation agent can be first adjusted to check whether this result is caused by an unreasonable setting of the reward function.

[0072] In another example, if it is determined that the number of sampling results with a weighted sum less than or equal to the second value is greater than the third preset number threshold, that is, there are more current poor sampling results. Before directly adjusting the parameters of the policy model, the weights of each reward evaluation agent can also be first adjusted to check whether this result is caused by an unreasonable setting of the reward function.

[0073] For example, the evaluation results of each reward evaluation agent set for each sampling result can be counted. For the reward evaluation agent with a relatively high evaluation result value and a small weight, its role function is rematched with the task description, and the weight of the target reward evaluation agent is adjusted according to the matching result.

[0074] For example, if the matching degree between one or more corresponding role functions of these target reward evaluation agents and the task description is higher than a preset matching threshold, the weight of the reward evaluation agent can be increased. Otherwise, the weight of the reward evaluation agent is not adjusted.

[0075] After adjusting the weights of the reward evaluation agents, the adjusted set of reward evaluation agents can be used to evaluate each of the at least one sampling results again, and the policy model reward values corresponding to each sampling result are re-determined based on the adjusted weights of the reward evaluation agents and the re-evaluation results. Then, the pre-trained policy model is trained using the re-determined reward values to obtain a trained policy model, and the task execution result is determined based on the trained policy model.

[0076] If, when re-determining the policy model reward values corresponding to each sampling result, the number of discarded sampling results is still greater than a first preset quantity threshold, or the number of sampling results with a weighted sum greater than or equal to a first value is still less than a second preset quantity threshold, or the number of sampling results with a weighted sum less than or equal to a second value is still greater than a third preset quantity threshold, then the weights of the reward evaluation agents are not adjusted further, but the parameters of the policy model are adjusted.

[0077] Among them, the values of the first to fourth preset quantity thresholds are set according to actual needs and are not limited here.

[0078] In this way, when the execution effect of the policy model is not good, the weights of the reward evaluation agents can be adjusted first. If the requirements are still not met, the parameters of the policy model are adjusted, which can improve the model training efficiency.

[0079] In some embodiments of the present application, training the pre-trained policy model based on the reward value may be to train the pre-trained policy model using the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm based on the reward values of each sampling result to obtain a trained policy model.

[0080] Figure 5 It is a schematic flowchart of the method for determining the task execution result in at least one sampling result using the trained policy model provided by the embodiments of the present application. As Figure 5 shown, the method includes the following steps: In step S501, the target sampling result in at least one sampling result is determined based on the PPO algorithm or the GRPO algorithm.

[0081] In step S502, the target sampling result is determined as the task execution result.

[0082] When determining the task execution result from at least one sampling result using the trained policy model, the target sampling result in the at least one sampling result can be determined based on the PPO algorithm or the GRPO algorithm. The target sampling result is a sampling result that meets the preset performance requirements. Then, this target sampling result is determined as the task execution result.

[0083] Figure 6 It is a schematic flowchart of another method for generating a task execution result based on a policy model provided by an embodiment of the present application. As Figure 6 shown, the policy model can first parse the input task description (or input problem) to initialize the reward evaluation agent. In one example, the reward evaluation agent role can be autonomously constructed according to the scenario system, and role and character information can be set for the generative reward evaluation agent.

[0084] Next, the pre-trained policy model samples the input task description and sends the sampling results to each generative reward evaluation agent for evaluation and scoring. The reward score of each reward evaluation agent can be constrained between 0 and 1. Then, the reward scores of each reward evaluation agent are aggregated, and the final aggregated reward score is the sum of the products of the reward scores of each reward evaluation agent and their weights.

[0085] Then, reward conversion can be performed. If the final aggregated reward score is greater than or equal to the first value, the reward score corresponding to this sampling result is determined as the preset reward value; if the final aggregated reward score is less than or equal to the second value, the reward score corresponding to this sampling result is determined as 0; if the final aggregated reward score is greater than the second value and less than the first value, this sampling result is discarded.

[0086] Finally, the converted reward is fed back to the pre-trained policy model, and the pre-trained model is trained using the PPO algorithm or the GRPO algorithm to obtain a trained policy model. This trained policy model can determine the task execution result of this time from the sampling result.

[0087] By adopting the technical solution provided by the embodiment of the present application, it is possible to provide more dimensional reward signals for model training, improve the robustness of model training, reduce the probability of reward hacking in reinforcement learning, and have better interpretability. The problems and biases of the policy model can be analyzed based on the generative reward result. At the same time, this method is more flexible, and the reward evaluation agent can be dynamically adjusted according to different task scenarios.

[0088] All of the above optional technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be elaborated one by one here.

[0089] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the method embodiment of the present application.

[0090] Figure 7 It is a schematic diagram of an apparatus for generating a task execution result based on a policy model provided by an embodiment of the present application. As Figure 7 shown, the apparatus includes: An acquisition module 701, configured to acquire a task description, and the evaluation criteria for the execution result of the task change with the evaluation conditions.

[0091] A determination module 702, configured to determine a reward evaluation agent set based on the task description, and the weights of each reward evaluation agent in the reward evaluation agent set.

[0092] A sampling module 703, configured to input the task description into a pre-trained policy model to obtain at least one sampling result.

[0093] An evaluation module 704, configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results.

[0094] A training module 705, configured to train the pre-trained policy model based on the reward value to obtain a trained policy model.

[0095] A generation module 706, configured to use the trained policy model to determine the task execution result among at least one sampling result.

[0096] According to the technical solution provided by the embodiment of the present application, by determining a reward evaluation agent set based on the task description, determining the weights for each reward evaluation agent in the set; using the pre-trained policy model to obtain at least one sampling result based on the task description, using each reward evaluation agent to evaluate each sampling result respectively, and determining the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results, training the policy model based on the reward value, and finally using the trained policy model to determine the task execution result among at least one sampling result, it realizes automatically and accurately matching multiple suitable reward evaluation agents for non-deterministic tasks, and using the multiple reward evaluation agents to jointly determine the reward value of the policy model, using the reward value to train the policy model to improve the evaluation accuracy of the policy model, and thus obtaining a task execution result that better conforms to the task description, improving the user experience.

[0097] In some embodiments, determining a set of reward evaluation agents based on a task description includes: performing scenario parsing on the task description to obtain scenario information of the task; the scenario information includes at least one of the following: task domain, task audience, task target object; performing target parsing on the task description to obtain a task target; the task target includes at least one of the following: publicity planning, user drainage, training; determining a task evaluation role based on the scenario information and the task target; and determining a reward evaluation agent for each task evaluation role to obtain a set of reward evaluation agents.

[0098] In some embodiments, determining the weights of the respective reward evaluation agents based on the task description includes: determining the importance of each task evaluation role based at least on the task target in the task description; and determining the weight of this reward evaluation agent based on the importance of the task evaluation role.

[0099] In some embodiments, determining the policy model reward value corresponding to each sampling result based on the weights of the respective reward evaluation agents and the evaluation results includes: for each sampling result, performing a weighted sum of the evaluation results of the respective reward evaluation agents to obtain a weighted sum of the evaluation results; in response to determining that the weighted sum is greater than or equal to a first value, determining that the policy model reward value corresponding to this sampling result is a preset reward value; in response to determining that the weighted sum is less than or equal to a second value, determining that the policy model reward value corresponding to this sampling result is 0; and in response to determining that the weighted sum is greater than the second value and less than the first value, discarding this sampling result.

[0100] In some embodiments, the step of training a pre-trained policy model based on the reward value is to train the pre-trained policy model using the proximal policy optimization (PPO) algorithm or the group relative policy optimization (GRPO) algorithm based on the reward values of the respective sampling results to obtain a trained policy model; using the trained policy model to determine a task execution result among at least one sampling result includes: determining a target sampling result among at least one sampling result based on the PPO algorithm or the GRPO algorithm; and determining the target sampling result as the task execution result.

[0101] In some embodiments, it further includes: in response to determining that the number of discarded sampling results is greater than a first preset number threshold, counting the first evaluation result distribution; the first evaluation result distribution is the result distribution of each reward evaluation agent for the discarded sampling results; and correcting the weights of the respective reward evaluation agents based on the distribution.

[0102] In some embodiments, it further includes: in response to determining that the number of sampling results whose weighted sum is greater than or equal to the first value is less than the second preset quantity threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than the third preset quantity threshold, statistically analyzing the distribution of the second evaluation results; the distribution of the second evaluation results is the distribution of the evaluation results of each reward evaluation agent; updating at least one reward evaluation agent in the set of reward evaluation agents based on the distribution of the evaluation results.

[0103] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not imply the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0104] Figure 8 is a schematic diagram of the electronic device provided by the embodiments of the present application. As Figure 8 shown, the electronic device 8 of this embodiment includes: a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in each of the above method embodiments. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module / unit in each of the above device embodiments.

[0105] The electronic device 8 may be a desktop computer, a notebook, a palm computer, a cloud server, or other electronic devices. The electronic device 8 may include, but is not limited to, the processor 801 and the memory 802. Those skilled in the art can understand that Figure 8 this is only an example of the electronic device 8, and does not constitute a limitation to the electronic device 8. It may include more or fewer components than those shown in the figure, or different components.

[0106] The processor 801 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0107] The memory 802 may be an internal storage unit of the electronic device 8, for example, the hard disk or memory of the electronic device 8. The memory 802 may also be an external storage device of the electronic device 8, for example, a plug-in hard disk equipped on the electronic device 8, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. The memory 802 may also include both an internal storage unit and an external storage device of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.

[0108] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0109] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of the present application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above method embodiments can be implemented. The computer program may include computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disc, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0110] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for generating task execution results based on a policy model, characterized in that, Including: Obtain a task description, where the evaluation criteria for the execution result of the task vary with evaluation conditions; Determine a set of reward evaluation agents based on the task description, and the weights of each reward evaluation agent in the set of reward evaluation agents; Input the task description into a pre-trained policy model to obtain at least one sampling result; Use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results; Train the pre-trained policy model based on the reward value to obtain a trained policy model; Use the trained policy model to determine the task execution result among the at least one sampling result.

2. The method according to claim 1, characterized in that Determining a set of reward evaluation agents based on the task description includes: Perform scenario parsing on the task description to obtain the scenario information of the task; the scenario information includes at least one of the following: task domain, task audience, task target object; Perform target parsing on the task description to obtain the task target; the task target includes at least one of the following: publicity planning, user drainage, training; Determine the task evaluation role based on the scenario information and the task target; Determine a reward evaluation agent for each task evaluation role to obtain the set of reward evaluation agents.

3. The method according to claim 2, wherein Determining the weights of each reward evaluation agent based on the task description includes: Determine the importance of each task evaluation role based at least on the task target in the task description; Determine the weight of this reward evaluation agent based on the importance of the task evaluation role.

4. The method according to claim 1, wherein Determining the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results includes: For each sampling result, perform weighted summation on the evaluation results of each reward evaluation agent to obtain the weighted sum of the evaluation results; In response to determining that the weighted sum is greater than or equal to a first value, determine that the policy model reward value corresponding to this sampling result is a preset reward value; In response to determining that the weighted sum is less than or equal to a second value, determine that the policy model reward value corresponding to this sampling result is 0; In response to determining that the weighted sum is greater than the second value and less than the first value, discard this sampling result.

5. The method according to claim 1, characterized in that, The step of training the pre-trained policy model based on the reward value is to train the pre-trained policy model using the Proximal Policy Optimization (PPO) algorithm or the Group Relative Policy Optimization (GRPO) algorithm based on the reward values of each sampling result to obtain a trained policy model; Using the trained policy model to determine the task execution result among the at least one sampling result includes: Determine the target sampling result among the at least one sampling result based on the PPO algorithm or the GRPO algorithm; Determine the target sampling result as the task execution result.

6. The method according to claim 4, wherein The method further includes: In response to determining that the number of discarded sampling results is greater than a first preset quantity threshold, count the first evaluation result distribution; the first evaluation result distribution is the result distribution of each reward evaluation agent for the discarded sampling results; Correct the weights of each reward evaluation agent based on the distribution.

7. The method according to claim 4, characterized in that, The method further includes: In response to determining that the number of sampling results whose weighted sum is greater than or equal to the first value is less than the second preset quantity threshold, or the number of sampling results whose weighted sum is less than or equal to the second value is greater than the third preset quantity threshold, counting the distribution of the second evaluation results; the distribution of the second evaluation results is the distribution of the evaluation results of each reward evaluation agent; Updating at least one reward evaluation agent in the set of reward evaluation agents based on the distribution of the evaluation results.

8. An apparatus for generating a task execution result based on a policy model, characterized in that, It includes: An acquisition module, configured to acquire a task description, and the evaluation criteria for the execution result of the task change with the evaluation conditions; A determination module, configured to determine a set of reward evaluation agents and the weights of each reward evaluation agent in the set of reward evaluation agents based on the task description; A sampling module, configured to input the task description into a pre-trained policy model to obtain at least one sampling result; An evaluation module, configured to use each reward evaluation agent to evaluate each sampling result respectively, and determine the policy model reward value corresponding to each sampling result based on the weights of each reward evaluation agent and the evaluation results; A training module, configured to train the pre-trained policy model based on the reward value to obtain a trained policy model; A generation module, configured to use the trained policy model to determine a task execution result among the at least one sampling result.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images