Multi-task oriented agent training method, decision-making method and device

Abstract

Disclosed are a multi-task oriented agent training method, a decision-making method and a device. The training method includes: inputting an initial task state in a training sample into a hybrid encoder to obtain a predicted task feature; inputting the predicted task feature into a shared policy network to generate a predicted preliminary action; inputting the predicted task feature and the predicted preliminary action into an action correction policy network to generate a predicted correction action; inputting the predicted preliminary action and the predicted correction action into an action correction module to obtain a predicted next action; executing the predicted next action to obtain a predicted next task state; determining a sparse reward and a dense reward based on the initial task state, the predicted next task state and a target task state; taking the predicted next task state as the initial task state and returning to the step of obtaining the predicted task feature until the tasks in the training sample are completed; and updating the parameters of the shared policy network, the action correction policy network and the hybrid encoder based on all the sparse rewards and all the dense rewards of each training sample.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology, and more specifically, to methods and apparatuses for training an agent and making decisions for multi-tasks. Background Art

[0002] In recent years, reinforcement learning technology has made remarkable progress in the field of intelligent agent decision-making, especially showing excellent performance in single-task scenarios. However, most reinforcement learning studies mainly focus on specific problem scenarios, and prefer to master a single task by learning a single policy, which often comes at the cost of sacrificing generalization ability. When facing multi-task scenarios, existing methods still have obvious defects in terms of policy generalization and training efficiency, and it is difficult to form a general agent that supports multi-task scenarios. Traditional methods usually use independent policy networks to handle different tasks, resulting in the parameter scale growing linearly with the number of tasks, and it is difficult to achieve knowledge transfer across tasks. Although some studies have tried to improve the generalization ability by sharing network structures, these methods often ignore the differences in reward mechanisms between different tasks, especially lacking an effective mechanism for co-optimizing dense rewards and sparse rewards.

[0003] Currently, the following technical bottlenecks mainly exist: First, in a sparse reward environment, the agent has low training efficiency due to the lack of effective exploration signals, and conventional curriculum learning or reward reshaping methods are prone to sub-optimal policy convergence problems. Second, the multi-task shared policy often suffers from policy degradation due to task objective conflicts. Especially in long-term decision-making scenarios, it is difficult to effectively coordinate between the short-sighted immediate reward maximization policy and the far-sighted global goal achievement requirement. The task-specific knowledge obtained from one task may hinder the overall learning process of other tasks, resulting in over-focusing on the immediate rewards of a single task during multi-task learning, thus leading to myopia and hindering the ability to learn a general policy for effectively completing multiple tasks. In addition, existing dynamic weight adjustment methods mostly use fixed ratios or empirical parameter settings, and cannot adapt to changes in task difficulty and training stages, resulting in the gradients of policy modules moving in opposite directions, and the gradient conflict cannot be effectively alleviated. Summary of the Invention

[0004] Embodiments of the present disclosure provide a method and apparatus for training an agent and making decisions for multi-tasks, which can effectively solve at least one of the above problems.

[0005] In one general aspect, a multi-task oriented agent training method is provided. The agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The training method includes: obtaining a training sample set for one round, where each training sample in the training sample set includes an initial task state and a target task state of a task; for each training sample in the training sample set, perform the following processing: input the initial task state in the training sample into the hybrid encoder to obtain an estimated task feature; input the estimated task feature into the shared policy network to generate an estimated preliminary action for the initial task state; input the estimated task feature and the estimated preliminary action into the action correction policy network to generate an estimated corrected action for the initial task state; input the estimated preliminary action and the estimated corrected action into the action correction module to obtain an estimated next action for the initial task state; execute the estimated next action to obtain an estimated next task state of the task in the training sample; based on the estimated next task state and the target task state, determine a sparse reward; based on the initial task state, the estimated next task state, and the target task state, determine a dense reward; use the estimated next task state as the initial task state and return to the step of obtaining the estimated task feature until the task in the training sample is completed; in response to all training samples having completed the above processing, update the parameters of the shared policy network, the action correction policy network, and the hybrid encoder based on all sparse rewards and all dense rewards.

[0006] Optionally, determining a sparse reward based on the estimated next task state and the target task state includes: mapping the estimated next task state and the target task state to a first latent space; in the first latent space, determining a first distance between the estimated next task state and the target task state; based on the relationship between the first distance and a corresponding distance threshold, determining the sparse reward.

[0007] Optionally, determining a dense reward based on the initial task state, the estimated next task state, and the target task state includes: mapping the estimated next task state and the target task state to a second latent space; mapping the initial task state and a predetermined task state to a third latent space, where the predetermined task state is the target task state or the estimated next task state; in the second latent space, determining a second distance between a first predetermined information in the estimated next task state and the target task state, where the first predetermined information is determined based on the content of the task in the training sample; in the third latent space, determining a third distance between a second predetermined information in the initial task state and the predetermined task state, where the second predetermined information is determined based on the content of the task in the training sample; based on the relationship between the second distance, the third distance, and corresponding distance thresholds, determining the dense reward.

[0008] Optionally, the action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. The estimated preliminary action and the estimated correction action are input into the action correction module to obtain the estimated next action for the initial task state, including: inputting the estimated preliminary action and the estimated correction action into the weighted summation sub-module to obtain a weighted summation result; inputting the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and the first action boundary; inputting the maximization result into the minimization sub-module to obtain the estimated next action, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and the second action boundary; wherein, the estimated preliminary action, the estimated correction action, and the estimated next action are all within the range of the first action boundary and the second action boundary.

[0009] In another general aspect, a multi-task oriented decision-making method is provided, which is applied to an agent. The agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The decision-making method includes: inputting the initial task state of the target task into the hybrid encoder to obtain task features; inputting the task features into the shared policy network to generate a preliminary action for the initial task state; inputting the task features and the preliminary action into the action correction policy network to generate a correction action for the initial task state; inputting the preliminary action and the correction action into the action correction module to obtain the next action for the initial task state; executing the next action to obtain the next task state of the target task; taking the next task state as the initial task state and returning to the step of obtaining task features until the target task is completed; wherein, the agent is trained by the training method of any one of the above.

[0010] Optionally, the action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. The preliminary action and the correction action are input into the action correction module to obtain the next action for the initial task state, including: inputting the preliminary action and the correction action into the weighted summation sub-module to obtain a weighted summation result; inputting the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and the first action boundary; inputting the maximization result into the minimization sub-module to obtain the next action, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and the second action boundary; wherein, the preliminary action, the correction action, and the next action are all within the range of the first action boundary and the second action boundary.

[0011] In another general aspect, an agent training device for multitasking is provided. The agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The training device includes: a first acquisition unit configured to acquire a training sample set for one round, where each training sample in the training sample set includes an initial task state and a target task state of a task; a first execution unit configured to perform the following processing on each training sample in the training sample set: input the initial task state in the training sample into the hybrid encoder to obtain an estimated task feature; input the estimated task feature into the shared policy network to generate an estimated preliminary action for the initial task state; input the estimated task feature and the estimated preliminary action into the action correction policy network to generate an estimated corrected action for the initial task state; input the estimated preliminary action and the estimated corrected action into the action correction module to obtain an estimated next action for the initial task state; execute the estimated next action to obtain an estimated next task state of the task in the training sample; determine a sparse reward based on the estimated next task state and the target task state; determine a dense reward based on the initial task state, the estimated next task state, and the target task state; use the estimated next task state as the initial task state and return to the step of obtaining the estimated task feature until the task in the training sample is completed; an update unit configured to, in response to all the training samples having completed the above processing, update the parameters of the shared policy network, the action correction policy network, and the hybrid encoder based on all the sparse rewards and all the dense rewards.

[0012] Optionally, the first execution unit is further configured to map the estimated next task state and the target task state to a first latent space; in the first latent space, determine a first distance between the estimated next task state and the target task state; and determine the sparse reward based on the relationship between the first distance and a corresponding distance threshold.

[0013] Optionally, the first execution unit is further configured to map the estimated next task state and the target task state to a second latent space; map the initial task state and a predetermined task state to a third latent space, where the predetermined task state is the target task state or the estimated next task state; in the second latent space, determine a second distance between a first predetermined information in the estimated next task state and the target task state, where the first predetermined information is determined based on the content of the task in the training sample; in the third latent space, determine a third distance between a second predetermined information in the initial task state and the predetermined task state, where the second predetermined information is determined based on the content of the task in the training sample; and determine the dense reward based on the relationship between the second distance, the third distance, and corresponding distance thresholds.

[0014] Optionally, the action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. Among them, the first execution unit is further configured to input the predicted preliminary action and the predicted correction action into the weighted summation sub-module to obtain a weighted summation result; input the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to find the maximum value within the range of the weighted summation result and the first action boundary; input the maximization result into the minimization sub-module to obtain the predicted next action, where the minimization sub-module is used to find the minimum value within the range of the maximization result and the second action boundary; wherein, the predicted preliminary action, the predicted correction action, and the predicted next action are all within the range of the first action boundary and the second action boundary.

[0015] In another general aspect, a multi-task oriented decision-making device is provided, which is applied to an intelligent agent. The intelligent agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The decision-making device includes: a second acquisition unit configured to input the initial task state of the target task into the hybrid encoder to obtain task features; a first generation unit configured to input the task features into the shared policy network to generate a preliminary action for the initial task state; a second generation unit configured to input the task features into the action correction policy network to generate a correction action for the initial task state; a fusion unit configured to input the preliminary action and the correction action into the action correction module to obtain the next action for the initial task state; a second execution unit configured to execute the next action, obtain the next task state of the target task, use the next task state as the initial task state and return to the step of obtaining task features until the target task is completed; wherein, the intelligent agent is trained by the training method of any one of the above.

[0016] Optionally, the action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. Among them, the fusion unit is further configured to input the preliminary action and the correction action into the weighted summation sub-module to obtain a weighted summation result; input the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to find the maximum value within the range of the weighted summation result and the first action boundary; input the maximization result into the minimization sub-module to obtain the next action, where the minimization sub-module is used to find the minimum value within the range of the maximization result and the second action boundary; wherein, the preliminary action, the correction action, and the next action are all within the range of the first action boundary and the second action boundary.

[0017] In another general aspect, a computer-readable storage medium storing instructions is provided. When the instructions are run by at least one computing device, at least one computing device is prompted to execute any of the above multi-task oriented intelligent agent training methods and multi-task oriented decision-making methods.

[0018] In another general aspect, a system is provided that includes at least one computing device and at least one storage device storing instructions, where, when the instructions are run by the at least one computing device, the at least one computing device is caused to execute any one of the above multi-task oriented agent training methods and multi-task oriented decision-making methods.

[0019] In another general aspect, a computer program product is provided that includes computer instructions which, when executed by a processor, implement any one of the above multi-task oriented agent training methods and multi-task oriented decision-making methods.

[0020] For the multi-task oriented agent training method and decision-making method and device according to an embodiment of the present disclosure, the dense reward takes into account the initial task state and the predicted next task state, such that the dense reward is incorporated into the actions of the short-term perspective of each task, while the sparse reward takes into account the target task state and the predicted next task state, focusing on the overall completion of the task, enhancing the exploration ability of the agent in a multi-task scenario, enabling the agent to take a long-term perspective to correct the actions. Therefore, the agent trained by the present disclosure can generate the next action in a manner combining the short-term perspective and the long-term perspective, achieving cross-task transfer generalization.

[0021] Additional aspects and / or advantages of the general concept of the present disclosure will be set forth in part in the following description, and in part will be obvious from the description, or may be learned through the practice of the general concept of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Through the following description with reference to the drawings showing embodiments, the above and other objects and features of the embodiments of the present disclosure will become clearer, where:

[0023] Figure 1 is a flowchart showing the multi-task oriented agent training method of an embodiment of the present disclosure;

[0024] Figure 2 is a flowchart showing the multi-task oriented decision-making method of an embodiment of the present disclosure;

[0025] Figure 3 is a system flowchart showing the multi-task oriented decision-making method of an embodiment of the present disclosure;

[0026] Figure 4 is a system architecture diagram showing the multi-task oriented decision-making method of an embodiment of the present disclosure;

[0027] Figure 5 is a schematic diagram of an application scenario showing the multi-task oriented decision-making method of an embodiment of the present disclosure;

[0028] Figure 6It is a schematic diagram of the battle result showing the multi-task-oriented decision-making method of the embodiments of the present disclosure;

[0029] Figure 7 It is a block diagram showing the multi-task-oriented agent training device of the embodiments of the present disclosure;

[0030] Figure 8 It is a block diagram showing the multi-task-oriented decision-making device of the embodiments of the present disclosure. Detailed Embodiments

[0031] The following detailed embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices, and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be apparent. For example, the order of operations described herein is merely exemplary and is not limited to those set forth herein, but may be changed as will be apparent after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, descriptions of features known in the art may be omitted for greater clarity and conciseness.

[0032] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Instead, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, devices, and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0033] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0034] Although terms such as "first", "second", and "third" may be used herein to describe various components, components, regions, layers, or parts, these components, components, regions, layers, or parts should not be limited by these terms. Instead, these terms are only used to distinguish one component, component, region, layer, or part from another. Thus, the first component, first component, first region, first layer, or first part referred to in the examples described herein may also be referred to as the second component, second component, second region, second layer, or second part without departing from the teachings of the examples.

[0035] In the specification, when an element (such as, a layer, a region, or a substrate) is described as being "on" another element, "connected to" or "coupled to" another element, the element can be directly "on" the other element, directly "connected to" or "coupled to" the other element, or there can be one or more other elements therebetween. In contrast, when an element is described as being "directly on" another element, "directly connected to" or "directly coupled to" another element, there can be no other elements therebetween.

[0036] The terms used herein are for the purpose of describing various examples only and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular forms are also intended to include the plural forms. The terms "comprising", "including" and "having" specify the presence of the stated features, quantities, operations, components, elements and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains after understanding this disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) shall be construed to have a meaning consistent with their meaning in the context of the relevant art and this disclosure, and shall not be construed in an idealized or overly formal manner.

[0038] Furthermore, in the description of the examples, when a detailed description of a related structure or function that is considered to be well-known would cause an ambiguous interpretation of the disclosure, such a detailed description will be omitted.

[0039] The agent of the present disclosure can be applied to driverless vehicles, drones, humanoid robots, games (such as as a human-machine player), etc., and the present disclosure is not limited thereto. Assuming that the agent is applied to a humanoid robot, it can perform but is not limited to the following tasks: controlling the robotic arm to reach a target location to pick up or deliver a target object, controlling the robotic arm to open a drawer, and the present disclosure is not limited thereto; assuming that the agent is applied to a drone, it can perform but is not limited to the following tasks: delivering supplies to a target location, arranging in a fixed formation, and the present disclosure is not limited thereto; assuming that the agent is applied to a driverless vehicle, it can perform but is not limited to the following task: carrying supplies to a target location, and the present disclosure is not limited thereto. Assuming that the agent is a human-machine player in a game, it can perform but is not limited to the following tasks: competing with the enemy, capturing several enemy cities, etc., and the present disclosure is not limited thereto.

[0040] The multi-task oriented agent training method, decision method and device of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0041] The present disclosure proposes a multi-task oriented agent training method. Figure 1 It is a flowchart showing the multi-task oriented agent training method according to an embodiment of the present disclosure. Referring to Figure 1 , the agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The multi-task oriented agent training method includes the following steps:

[0042] In step S101, obtain a training sample set for one round. Among them, each training sample in the training sample set includes an initial task state and a target task state of a task.

[0043] As an example, a training sample set for one round can be randomly selected from the sample pool. The tasks included in two training samples in this training sample set can be different or the same, and the present disclosure does not limit this. However, the tasks included in all training samples in the training sample set are generally multiple tasks, which can improve the richness of the training samples.

[0044] As an example, after obtaining the training samples, the agent can be initialized before starting the training. For example, initialize the number of tasks of the agent, that is, the number of tasks that the agent can execute, including but not limited to the number of tasks that the agent executes simultaneously, the number of types of tasks that the agent can accept; another example is to initialize the virtual expected budget, that is, the expected value to be achieved in each step of each task; another example is to initialize the parameters of the hybrid encoder, the shared policy network, the action correction policy network, and the action correction module; it can also be any other information that needs to be initialized, and the present disclosure does not limit this.

[0045] In step S102, for each training sample in the training sample set, perform the following processing: input the initial task state in the training sample into the hybrid encoder to obtain a predicted task feature; input the predicted task feature into the shared policy network to generate a predicted preliminary action for the initial task state; input the predicted task feature and the predicted preliminary action into the action correction policy network to generate a predicted corrected action for the initial task state; input the predicted preliminary action and the predicted corrected action into the action correction module to obtain a predicted next action for the initial task state; execute the predicted next action to obtain the predicted next task state of the task in the training sample; determine a sparse reward based on the predicted next task state and the target task state; determine a dense reward based on the initial task state, the predicted next task state, and the target task state; use the predicted next task state as the initial task state and return to the step of obtaining the predicted task feature until the task in the training sample is completed.

[0046] As an example, after initializing the agent, a training sample can be randomly selected from the training sample set B. The initial task state in this training sample is input into the hybrid encoder to obtain the predicted task features. Then, the predicted task features are input into the shared policy network to generate a predicted preliminary action for the initial task state. The predicted task features and the predicted preliminary action are input into the action correction policy network to generate a predicted corrected action for the initial task state. Then, the predicted preliminary action and the predicted corrected action are input into the action correction module to obtain a predicted next action for the initial task state. After obtaining this predicted next action, execute the predicted next action to obtain the predicted next task state of the task in the training sample. Then, based on the predicted next task state and the target task state, determine the sparse reward. Based on the initial task state 、the predicted next task state and the target task state, determine the dense reward. Finally, add the new sample containing the initial task state (which is replaced by the corresponding next task state in each subsequent step), the predicted next action 、the predicted next task state 、the sparse reward and the dense reward to the training sample set, that is take the predicted next task state as the initial task state and return to the step of obtaining the predicted task features until the task in the training sample is completed.

[0047] It should be noted that 、 respectively represent the shared policy network and the action correction policy network. 、 respectively represent the learnable network parameters in the shared policy network and the action correction policy network. represents the function corresponding to the action correction module. represents the task in the training sample i 's sparse reward. represents the task in the training sample i 's dense reward.

[0048] As an example, the above-mentioned hybrid encoder may include 4 encoders, 1 multi-layer perceptron, and 1 attention network, but the present disclosure does not limit this. The structures of the above-mentioned shared policy network and action correction policy network may be the same. For example, they may be composed of 3 multi-layer perceptrons with 400 neurons. The activation function between layers is ReLu, and the parameters are initialized to a standard Gaussian distribution, but the present disclosure does not limit this.

[0049] According to an embodiment of the present disclosure, determining a sparse reward based on the predicted next task state and the target task state may include: mapping the predicted next task state and the target task state to a first latent space; in the first latent space, determining a first distance between the predicted next task state and the target task state; and determining the sparse reward based on the relationship between the first distance and a corresponding distance threshold. Through this embodiment, the distance between the next task state and the target task state is considered, so that the sparse reward can better focus on the overall completion of the task, and thus the action can be corrected better from a long-term perspective.

[0050] As an example, the above-mentioned sparse reward , that is, the sparse reward brought by the next task state in the training sample i of the task, can be specifically expressed as:

[0051]

[0052] wherein, represents the task state of the agent at the current moment (that is, the above-mentioned initial task state, which will be replaced by the next task state at each subsequent step), represents the target task state, represents a function that maps the target task state and the task state of the agent at the current moment to the latent space, and is used to calculate the distance between the target task state and the task state of the agent at the current moment in the latent space. At this time, it can be defined as , represents the reward value for reaching the target task state, represents the distance threshold, and the distance threshold can be set as needed.

[0053] As an example, the above-mentioned can generally take a constant 1 for formalizing the sparse reward. Only when the task is completed will a reward of 1 be obtained, that is, the reward value is incremented by 1.

[0054] According to an embodiment of the present disclosure, determining a dense reward based on an initial task state, a predicted next task state, and a target task state may include: mapping the predicted next task state and the target task state to a second latent space; mapping the initial task state and a predetermined task state to a third latent space, where the predetermined task state is the target task state or the predicted next task state; in the second latent space, determining a second distance between a first predetermined information in the predicted next task state and the target task state, where the first predetermined information is determined based on the content of the task in the training sample; in the third latent space, determining a third distance between a second predetermined information in the initial task state and the predetermined task state, where the second predetermined information is determined based on the content of the task in the training sample; determining the dense reward based on the relationship between the second distance, the third distance, and the corresponding distance thresholds. Through this embodiment, the distance between the next task state and the initial task state is considered, so that the dense reward can better focus on the immediate completion degree of the task, and thus more accurate actions can be obtained from a short-term perspective.

[0055] As an example, for different tasks, the relationship between the second distance, the third distance, and the corresponding distance thresholds is different. The following is an example:

[0056] Suppose the task is to control a robotic arm to reach a target location, then the dense reward can be expressed as:

[0057]

[0058]

[0059]

[0060] where represents the position of the target location, represents the task state of the robotic arm in, represents the initial position of the robotic arm, represents calculating the two-norm.

[0061] Suppose the task is to control the robotic arm to open a drawer, then the dense reward can be expressed as:

[0062]

[0063] where represents the initial state of the drawer, represents the task state of the drawer in, represents the target state of the drawer, represents the task state the position of the manipulator in represents the initial position of the manipulator.

[0064] According to an embodiment of the present disclosure, the motion correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. Among them, inputting the estimated preliminary motion and the estimated correction motion into the motion correction module to obtain the estimated next motion for the initial task state may include: inputting the estimated preliminary motion and the estimated correction motion into the weighted summation sub-module to obtain a weighted summation result; inputting the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to find the maximum value within the weighted summation result and the first motion boundary; inputting the maximization result into the minimization sub-module to obtain the estimated next motion, where the minimization sub-module is used to find the minimum value within the maximization result and the second motion boundary; where the estimated preliminary motion, the estimated correction motion, and the estimated next motion are all within the range of the first motion boundary and the second motion boundary. Through this embodiment, the estimated preliminary motion and the estimated correction motion are fused within the range of the first motion boundary and the second motion boundary, so that a relatively accurate next motion can be obtained.

[0065] As an example, a fixed value A can be selected as the motion boundary. The first motion boundary can be set to -A, and the second motion boundary can be set to A. The present disclosure does not limit this.

[0066] As an example, assuming that A is selected as the motion boundary, the next motion can be determined in the following manner :

[0067]

[0068] Among them, , represent finding the minimum value and the maximum value, represents the motion boundary, .

[0069] In step S103, in response to all training samples having completed the above processing, based on all sparse rewards and all dense rewards, update the parameters of the shared policy network, the motion correction policy network, and the hybrid encoder.

[0070] As an example, after obtaining all the estimated next motions, all the estimated next task states, all the sparse rewards, and all the dense rewards of all the training samples, the loss functions of the shared policy network and the motion correction policy network can be determined respectively, specifically as follows:

[0071]

[0072]

[0073] Among them, represents the loss function of the shared policy network; represents the loss function of the action correction policy network; represents the mathematical expectation of all possible states and actions, represents the mathematical expectation of the task state s under the probability distribution D, represents the estimated preliminary action under the policy probability of the shared policy network mathematical expectation, represents the task under the distribution probability of all tasks mathematical expectation, represents the estimated next task state under the task state transition probability mathematical expectation, represents the estimated correction action under the action correction policy policy probability; D represents the replay buffer; represents the th task; represents the distribution probability of all tasks; represents the discount factor; represents the task state transition probability; represents the state estimate value of the shared policy network, and the Euclidean distance can be used in this embodiment; represents the Lagrange multiplier, which is used to adjust the size of the loss function; represents the state-action estimate value of the action correction module; is the distance function; represents the learning rate.

[0074] Then, the gradient of the Lagrange multiplier is estimated using sparse rewards and virtual expected budgets, which can be specifically as follows:

[0075] ;

[0076] Among them, represents the value of the sparse reward after the th step action, represents the number of action steps required to reach the target task state from the initial task state, represents the virtual expected budget per step. It should be noted that for a task, the virtual expected budget usually needs to be set as a fixed value, which is mainly used to balance the training intensity of the shared policy network and the action correction policy network. For different tasks, different virtual expected budgets can be set.

[0077] Then, the shared policy network and the action correction policy network can be updated in the following manner:

[0078] , ,

[0079] Among them, , respectively represent , 's gradients. In this embodiment, the gradient descent method is used for calculation, represents the learning rate.

[0080] It should be noted that after the training using the current training sample set is completed, another training sample can be selected to continue the training until the termination state is reached or the maximum number of iterations is completed. Furthermore, the hybrid encoder can be updated together with the updates of the shared policy network and the action correction policy network. The specific update method is not limited in this disclosure.

[0081] This disclosure also proposes a multi-task oriented decision-making method, Figure 2 is a flowchart showing the multi-task oriented decision-making method of the embodiments of this disclosure. Referring to Figure 2 , this method is applied to an intelligent agent trained by the training method of any one of the above. The intelligent agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The multi-task oriented decision-making method includes the following steps:

[0082] In step S201, the initial task state of the target task is input into the hybrid encoder to obtain the task feature;

[0083] In step S202, the task feature is input into the shared policy network to generate a preliminary action for the initial task state;

[0084] In step S203, the task feature and the preliminary action are input into the action correction policy network to generate a corrected action for the initial task state;

[0085] In step S204, the preliminary action and the corrected action are input into the action correction module to obtain the next action for the initial task state;

[0086] Optionally, the action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. The preliminary action and the correction action are input into the action correction module to obtain the next action for the initial task state, including: inputting the preliminary action and the correction action into the weighted summation sub-module to obtain a weighted summation result; inputting the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and the first action boundary; inputting the maximization result into the minimization sub-module to obtain the next action, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and the second action boundary; where the preliminary action, the correction action, and the next action are all within the range of the first action boundary and the second action boundary. Through this embodiment, the preliminary action and the correction action are fused within the range of the first action boundary and the second action boundary, so that a relatively accurate next action can be obtained.

[0087] In step S205, execute the next action to obtain the next task state of the target task;

[0088] In step S206, return to the task feature acquisition step and replace the initial task state with the next task state until the target task is completed.

[0089] For the convenience of understanding the above embodiments, the following is combined with Figure 3 and Figure 4 for a systematic description.

[0090] Figure 3 FIG. shows the system flow chart of the decision-making method for multi-tasks, Figure 4 FIG. shows the system architecture diagram of the decision-making method for multi-tasks, as shown in Figure 3 and Figure 4 shown, the system process includes the following steps:

[0091] S301, initialize the agent parameters, such as initializing the number of tasks n, virtual expected budget, etc.;

[0092] S302, input the current task state of the target task into the hybrid encoder of the agent to extract all task features in the current task state;

[0093] S303, based on the shared policy network, generate the preliminary action of the agent, that is, input the extracted task features into the shared policy network to obtain the preliminary action;

[0094] S304, based on the action correction policy network, generate the correction action of the agent, that is, input the extracted task features and the preliminary action into the action correction policy network to obtain the correction action;

[0095] S305, based on the action correction function (i.e., the above-mentioned action correction module), fuses the initial action and the corrected action, and outputs the next action of the agent. That is, the initial action and the corrected action are input into the action correction function to obtain the next action.

[0096] After obtaining the next task, the next action can be executed to obtain the next task state of the target task. This next task state is used as the new current task state, and the process returns to step S302 until the target task is completed.

[0097] It should be noted that the executor of the next action can be a third party. For example, when the agent is a game human-machine player and can execute a battle task, the agent can be loaded onto a computer device. At this time, the executor is the computer device, and the computer device will inform the agent of the game progress in real time. The agent will give the most accurate next action according to the game progress and return it to the game application on the computer device to control the corresponding character in the game application to execute the next action. As Figure 5 shown, Figure 5 in (A) shows the decision-making of 3 of our warriors against 5 enemy warriors, Figure 5 in (B) shows the decision-making of 2 of our warriors against 64 enemy warriors, Figure 5 in (C) shows 6 of our warriors against 24 enemy warriors. After receiving the task state, the agent will determine the next action of our warriors according to the task state, so as to control our warriors to perform the next action until the battle task is completed. The battle results are as Figure 6 shown. The agent trained by the training method of the present disclosure has a relatively high average winning rate. Another example is when the agent is applied to a humanoid robot. The agent can give the next action according to the state of the humanoid robot, and the humanoid robot receives and executes the next action.

[0098] In summary, the present disclosure proposes an intelligent decision-making method for an agent supporting multi-task migration, aiming to give full play to the advantages of short-term and long-term perspectives, that is, combining a shared policy network and an action correction network, and incorporating dense rewards into each task to generate actions from a short-term perspective, ignoring the overall completion of the task. At the same time, sparse rewards adopt a longer decision-making scope, focusing on the overall completion of the task to enhance the exploration ability of the agent in a multi-task scenario, enabling the agent to take a long-term perspective to generate correction actions, thus achieving the generalization of the agent across tasks, and further enabling the agent to quickly decide the best actions supporting multi-task scenarios. Specifically, first, the agent parameters are initialized. Then, the task state is input into the hybrid encoder of the agent to extract all task features. Next, based on the shared policy network, the initial actions of the agent are generated, and based on the action correction policy network, the correction actions of the agent are generated. Then, based on the action correction function, the initial actions and correction actions are fused to output the next action. It can be seen that the present disclosure combines the information of short-term and long-term perspectives, can support the decision of the best actions in a multi-task scenario, and enables the agent to have higher cross-task autonomy and policy transferability.

[0099] Figure 7 FIG. is a block diagram of a multi-task-oriented agent training device showing an embodiment of the present disclosure, as Figure 7 shown, the agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module, and the device includes a first acquisition unit 70, a first execution unit 72, and an update unit 74.

[0100] The first acquisition unit 70 is configured to acquire a training sample set for one round, where each training sample in the training sample set includes an initial task state and a target task state of a task; the first execution unit 72 is configured to perform the following processing on each training sample in the training sample set: input the initial task state in the training sample into the hybrid encoder to obtain an estimated task feature; input the estimated task feature into the shared policy network to generate an estimated preliminary action for the initial task state; input the estimated task feature and the estimated preliminary action into the action correction policy network to generate an estimated correction action for the initial task state; input the estimated preliminary action and the estimated correction action into the action correction module to obtain an estimated next action for the initial task state; execute the estimated next action to obtain the estimated next task state of the task in the training sample; determine a sparse reward based on the estimated next task state and the target task state; determine a dense reward based on the initial task state, the estimated next task state, and the target task state; use the estimated next task state as the initial task state and return to the step of obtaining the estimated task feature until the task in the training sample is completed; the update unit 74 is configured to, in response to all the training samples having completed the above processing, update the parameters of the shared policy network, the action correction policy network, and the hybrid encoder based on all the sparse rewards and all the dense rewards.

[0101] According to an embodiment of the present disclosure, the first execution unit 72 is further configured to map the estimated next task state and the target task state to a first latent space; in the first latent space, determine a first distance between the estimated next task state and the target task state; determine the sparse reward based on the relationship between the first distance and a corresponding distance threshold.

[0102] According to an embodiment of the present disclosure, the first execution unit 72 is further configured to map the estimated next task state and the target task state to a second latent space; map the initial task state and a predetermined task state to a third latent space, where the predetermined task state is the target task state or the estimated next task state; in the second latent space, determine a second distance between a first predetermined information in the estimated next task state and the target task state, where the first predetermined information is determined based on the content of the task in the training sample; in the third latent space, determine a third distance between a second predetermined information in the initial task state and the predetermined task state, where the second predetermined information is determined based on the content of the task in the training sample; determine the dense reward based on the relationship between the second distance, the third distance, and corresponding distance thresholds.

[0103] According to an embodiment of the present disclosure, the motion correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. Among them, the first execution unit 72 is further configured to input the estimated preliminary motion and the estimated correction motion into the weighted summation sub-module to obtain a weighted summation result; input the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and the first motion boundary; input the maximization result into the minimization sub-module to obtain the estimated next motion, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and the second motion boundary; where the estimated preliminary motion, the estimated correction motion, and the estimated next motion are all within the range of the first motion boundary and the second motion boundary.

[0104] Figure 8 is a block diagram showing a multi-task oriented decision-making device according to an embodiment of the present disclosure, as Figure 8 shown, the intelligent agent includes a hybrid encoder, a shared policy network, a motion correction policy network, and a motion correction module. The device includes a second acquisition unit 80, a first generation unit 82, a second generation unit 84, a fusion unit 86, and a second execution unit 88.

[0105] The second acquisition unit 80 is configured to input the initial task state of the target task into the hybrid encoder to obtain task features; the first generation unit 82 is configured to input the task features into the shared policy network to generate a preliminary motion for the initial task state; the second generation unit 84 is configured to input the task features into the motion correction policy network to generate a correction motion for the initial task state; the fusion unit 86 is configured to input the preliminary motion and the correction motion into the motion correction module to obtain the next motion for the initial task state; the second execution unit 88 is configured to execute the next motion, obtain the next task state of the target task, use the next task state as the initial task state and return to the step of obtaining task features until the target task is completed; where the intelligent agent is trained by the training method of any one of the above.

[0106] According to an embodiment of the present disclosure, the motion correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module. Among them, the fusion unit 86 is further configured to input the preliminary motion and the correction motion into the weighted summation sub-module to obtain a weighted summation result; input the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and the first motion boundary; input the maximization result into the minimization sub-module to obtain the next motion, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and the second motion boundary; where the preliminary motion, the correction motion, and the next motion are all within the range of the first motion boundary and the second motion boundary.

[0107] According to an embodiment of the present disclosure, there is provided a computer-readable storage medium storing instructions, wherein when the instructions are run by at least one computing device, the at least one computing device is caused to execute the multi-task oriented agent training method and the multi-task oriented decision-making method of any of the above embodiments.

[0108] According to an embodiment of the present disclosure, there is provided a system including at least one computing device and at least one storage device storing instructions, wherein when the instructions are run by at least one computing device, the at least one computing device is caused to execute the multi-task oriented agent training method and the multi-task oriented decision-making method of any of the above embodiments.

[0109] According to an embodiment of the present disclosure, there is provided a computer program product including computer instructions, which when executed by a processor, implement the multi-task oriented agent training method and the multi-task oriented decision-making method of any of the above.

[0110] Although some embodiments of the present disclosure have been shown and described, those skilled in the art should understand that these embodiments can be modified without departing from the principles and spirit of the present disclosure defined by the claims and their equivalents.

Claims

1. An intelligent agent training method for multi - tasks, characterized in that, The agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The training method includes: Obtain a training sample set for one round. Each training sample in the training sample set contains an initial task state and a target task state of a task. For each training sample in the training sample set, perform the following processing: Input the initial task state in the training sample into the hybrid encoder to obtain an estimated task feature. Input the estimated task feature into the shared policy network to generate an estimated preliminary action for the initial task state. Input the estimated task feature and the estimated preliminary action into the action correction policy network to generate an estimated corrected action for the initial task state. Input the estimated preliminary action and the estimated corrected action into the action correction module to obtain the estimated next action for the initial task state. Execute the estimated next action to obtain the estimated next task state of the task in the training sample. Based on the estimated next task state and the target task state, determine a sparse reward. Based on the initial task state, the estimated next task state, and the target task state, determine a dense reward. Use the estimated next task state as the initial task state and return to the step of obtaining the estimated task feature until the task in the training sample is completed. In response to all training samples having completed the above processing, update the parameters of the shared policy network, the action correction policy network, and the hybrid encoder based on all sparse rewards and all dense rewards. The agent is applied to an autonomous vehicle, a drone, or a humanoid robot. When the agent is applied to a humanoid robot, the task includes controlling the manipulator of the humanoid robot to reach a target location, pick up a target object, or deliver a target object, or controlling the manipulator of the humanoid robot to open a drawer. When the agent is applied to a drone, the task includes controlling the drone to deliver supplies to a target location, or controlling the drone to arrange in a fixed formation. When the agent is applied to an autonomous vehicle, the task includes controlling the autonomous vehicle to carry supplies to a target location.

2. The intelligent agent training method according to claim 1, wherein The determining the sparse reward based on the estimated next task state and the target task state includes: Map the estimated next task state and the target task state to a first latent space. In the first latent space, determine a first distance between the estimated next task state and the target task state. Based on the relationship between the first distance and a corresponding distance threshold, determine the sparse reward.

3. The intelligent agent training method according to claim 1, wherein The determining the dense reward based on the initial task state, the estimated next task state, and the target task state includes: Map the estimated next task state and the target task state to a second latent space. Map the initial task state and a predetermined task state to a third latent space, where the predetermined task state is the target task state or the estimated next task state. In the second latent space, determine a second distance between the predicted next task state and first predetermined information in the target task state, where the first predetermined information is determined based on the content of tasks in the training samples; In the third latent space, determine a third distance between the initial task state and second predetermined information in the predetermined task state, where the second predetermined information is determined based on the content of tasks in the training samples; Determine the dense reward based on the relationship between the second distance, the third distance, and corresponding distance thresholds; 4. The intelligent agent training method according to claim 1, wherein The action correction module includes a weighted summation sub-module, a maximization sub-module, and a minimization sub-module; wherein, inputting the predicted preliminary action and the predicted correction action into the action correction module to obtain the predicted next action for the initial task state includes: Input the predicted preliminary action and the predicted correction action into the weighted summation sub-module to obtain a weighted summation result; Input the weighted summation result into the maximization sub-module to obtain a maximization result, where the maximization sub-module is used to obtain the maximum value within the range of the weighted summation result and a first action boundary; Input the maximization result into the minimization sub-module to obtain the predicted next action, where the minimization sub-module is used to obtain the minimum value within the range of the maximization result and a second action boundary; wherein, the predicted preliminary action, the predicted correction action, and the predicted next action are all within the range of the first action boundary and the second action boundary; 5. A decision-making method for multi-tasks, characterized in that Applied to an intelligent agent, the intelligent agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module, and the decision-making method includes: Input the initial task state of the target task into the hybrid encoder to obtain task features; Input the task features into the shared policy network to generate a preliminary action for the initial task state; Input the task features and the preliminary action into the action correction policy network to generate a correction action for the initial task state; Input the preliminary action and the correction action into the action correction module to obtain the next action for the initial task state; Execute the next action to obtain the next task state of the target task; Use the next task state as the initial task state and return to the step of obtaining task features until the target task is completed; wherein, the intelligent agent is trained by the training method according to any one of claims 1 to 4 above; wherein, the intelligent agent is applied to an autonomous vehicle, a drone, or a humanoid robot; When the intelligent agent is applied to a humanoid robot, the task includes controlling the manipulator of the humanoid robot to reach a target location, pick up a target object, or deliver a target object, or controlling the manipulator of the humanoid robot to open a drawer; When the intelligent agent is applied to a drone, the task includes controlling the drone to deliver supplies to a target location, or controlling the drone to arrange in a fixed formation; When the agent is applied to an autonomous vehicle, the task includes controlling the autonomous vehicle to carry supplies to a target location.

6. An intelligent agent training device for multi - task, characterized in that, The agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The training device includes: A first acquisition unit configured to acquire a training sample set for one round. Each training sample in the training sample set includes an initial task state and a target task state of a task. A first execution unit configured to perform the following processing on each training sample in the training sample set: Input the initial task state in the training sample into the hybrid encoder to obtain a predicted task feature. Input the predicted task feature into the shared policy network to generate a predicted preliminary action for the initial task state. Input the predicted task feature and the predicted preliminary action into the action correction policy network to generate a predicted correction action for the initial task state. Input the predicted preliminary action and the predicted correction action into the action correction module to obtain the predicted next action for the initial task state. Execute the predicted next action to obtain the predicted next task state of the task in the training sample. Determine a sparse reward based on the predicted next task state and the target task state. Determine a dense reward based on the initial task state, the predicted next task state, and the target task state. Use the predicted next task state as the initial task state and return to the step of obtaining the predicted task feature until the task in the training sample is completed. An update unit configured to update the parameters of the shared policy network, the action correction policy network, and the hybrid encoder based on all sparse rewards and all dense rewards in response to all training samples having completed the above processing. Wherein, the agent is applied to an autonomous vehicle, a drone, or a humanoid robot. When the agent is applied to a humanoid robot, the task includes controlling the manipulator of the humanoid robot to reach a target location, pick up a target object, or deliver a target object, or controlling the manipulator of the humanoid robot to open a drawer. When the agent is applied to a drone, the task includes controlling the drone to deliver supplies to a target location, or controlling the drone to arrange in a fixed formation. When the agent is applied to an autonomous vehicle, the task includes controlling the autonomous vehicle to carry supplies to a target location.

7. A decision-making device for multitasking, characterized in that, Applied to an agent, the agent includes a hybrid encoder, a shared policy network, an action correction policy network, and an action correction module. The decision-making device includes: A second acquisition unit configured to input the initial task state of a target task into the hybrid encoder to obtain a task feature. A first generation unit configured to input the task feature into the shared policy network to generate a preliminary action for the initial task state. A second generation unit configured to input the task feature into the action correction policy network to generate a correction action for the initial task state. A fusion unit, configured to input the preliminary action and the correction action into the action correction module to obtain the next action for the initial task state; A second execution unit, configured to execute the next action, obtain the next task state of the target task, use the next task state as the initial task state, and return to the step of obtaining the task feature until the target task is completed; wherein, the agent is trained by the training method according to any one of claims 1 to 4 above; wherein, the agent is applied to an autonomous vehicle, a drone or a humanoid robot, when the agent is applied to a humanoid robot, the task includes controlling the manipulator of the humanoid robot to reach a target location, pick up a target object or deliver a target object, or controlling the manipulator of the humanoid robot to open a drawer; when the agent is applied to a drone, the task includes controlling the drone to deliver supplies to a target location, or controlling the drone to arrange in a fixed formation; when the agent is applied to an autonomous vehicle, the task includes controlling the autonomous vehicle to carry supplies to a target location.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is run by at least one computing device, it causes the at least one computing device to execute the multi-task oriented agent training method according to any one of claims 1 to 4 and the multi-task oriented decision method according to claim 5.

9. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that, When the instruction is run by the at least one computing device, it causes the at least one computing device to execute the multi-task oriented agent training method according to any one of claims 1 to 4 and the multi-task oriented decision method according to claim 5.

10. A computer program product comprising computer instructions, characterized in that, When the computer instruction is executed by a processor, it implements the multi-task oriented agent training method according to any one of claims 1 to 4 and the multi-task oriented decision method according to claim 5.

Images