Multi-task control method based on general planner and reinforcement learning

Abstract

The present invention discloses a multi-task control method based on a general planner and reinforcement learning, collects trajectory data of a single-task scene, and selects high-priority samples through priority trajectory playback; uses selected samples to minimize the difference between adjacent states to train the general planner, expecting to obtain two states and one action as accessibility information under certain conditions, that is, the required step distance; trains the action network by minimizing the step distance output by the planner as a goal to obtain the optimal action between the two states, and then combines it with the reinforcement learning strategy network to guide it, so as to optimize the action selection, and gradually reduce the guidance, and finally realizes efficient learning in a multi-task environment by sharing the strategy network obtained by reinforcement learning of multi-task samples. The method of the present invention can effectively deal with the strategy conflicts between different tasks, improve the stability and generalization ability of the strategy, and realize efficient control in a multi-task environment.

Description

Technical Field

[0001] The present invention belongs to the field of multi-task operation and relates to a multi-task control method based on a general planner and reinforcement learning. Background Art

[0002] The introduction of multi-task learning has significantly expanded the application boundaries of robot manipulation, enabling it to cope with increasingly complex operation tasks. However, the generalization ability and distribution breadth of current reinforcement learning-based methods in multi-task training are limited, and this limitation greatly affects its practical application effect in complex environments. Therefore, how to obtain a strategy with wide applicability and stability has become a core challenge that needs to be solved urgently.

[0003] At present, robots have made significant progress in manipulation tasks in various scenarios, such as opening bottle caps, writing, assembly, and manipulation of flexible objects. However, these tasks often rely on manually set observations, actions, and reward functions, and the robot finally obtains an effective manipulation model through multiple iterative explorations. However, the limitation of this method is that its action distribution range is narrow, the state transition probability is limited, and it is difficult to achieve effective generalization in multi-task scenarios. In practical applications, robots need to have flexible and diverse skills to adapt to various different operation scenarios and precise operation requirements. In order to enable robots to be more widely used in practical environments, reinforcement learning algorithms must be able to efficiently acquire multiple skills in a single learning process. However, robots face the challenges of various behavior patterns and complex transition models, making it difficult to achieve fully autonomous and efficient operation in multi-task scenarios.

[0004] Multi-task reinforcement learning is considered to be one of the effective ways to solve the above problems through parameter sharing and reuse. However, there are problems such as uneven action distribution, unstable training and incomplete transition probability in its training process, which restrict its effectiveness. Recent studies have introduced planning-guided reinforcement learning methods, which have demonstrated their potential for application in robot motion planning and indoor navigation by combining different output modes, such as the length of the visual field or the change of modeling methods. These methods significantly improve the stability and exploration efficiency of training by performing effective state space exploration in the early stages of reinforcement learning, providing new ideas for solving the bottleneck problem in multi-task learning.

[0005] At present, the main challenge in the field of robotic manipulation is how to effectively integrate the advantages of motion planning and reinforcement learning. Existing methods usually use hierarchical reinforcement learning (HRL) to decompose complex tasks into multiple subtasks and perform modular processing. Although this approach can improve the processability of tasks, it requires significant engineering investment and is limited in scalability in multi-task scenarios. Summary of the invention

[0006] In order to solve the above problems in the prior art, the present invention proposes a multi-task control method based on a general planner and reinforcement learning, and its specific technical solution is as follows:

[0007] A multi-task control method based on a general planner and reinforcement learning, comprising the following steps:

[0008] Step 1: In a multi-task operation environment, record each task sample according to the state, action and reward function in different task scenarios, select a single task scenario, collect trajectory samples of the scenario task, and perform batch optimization;

[0009] Step 2: Use the trajectory samples optimized in batches in step 1 to train and optimize the general planner. At the same time, build and train an action network to map the output of the general planner to the state and obtain the corresponding optimal action.

[0010] Step 3: Combine the general planner and action network into the reinforcement learning policy network, use the general planner and action network to guide the output action of the reinforcement learning policy network to optimize the action selection, and gradually reduce the guidance through training, so that the reinforcement learning policy network can make independent decisions in the end;

[0011] Step 4: In a multi-task learning environment, share the trained reinforcement learning policy network for all tasks.

[0012] Furthermore, in step 1, in a single-task scenario, when the task success rate reaches 85%, the sample selection process is started, and then the priority trajectory playback is used to select trajectory samples. The priority is calculated based on the generalized advantage estimate and distribution entropy. The specific expression is as follows:

[0013] ,

[0014] in, is the regulating factor, is the trajectory priority, is a measure of the entropy of a distribution.

[0015] Furthermore, in step 2, the training objectives of the general planner include:

[0016] Minimize the distance difference between adjacent states: ,in is the state at time t, is the state at time t+1, is the action selection at time t;

[0017] Optimize the reachability evaluation from the state to the goal: ,in The target final state;

[0018] Keep the planner stable across state swaps: .

[0019] Furthermore, in step 2, the training of the action network is achieved by minimizing the error function, and the loss function loss expression is:

[0020] ,

[0021] in, Output of the generic planner for delayed updates.

[0022] Furthermore, the optimization objective function of the action network is:

[0023] ,in represents the optimal action given the current state and the target state, is the initial state, is the target state.

[0024] Furthermore, the general planner and the action network are both multi-layer perceptrons, using three layers of 512-dimensional fully connected layers with ReLU activation function. The input of the general planner is the connection vector of two states and the action, and the output is the spatial distance between the two states and the corresponding actions, that is, the required number of steps. The input of the action network is the connection vector of the two states, and the output is the action corresponding to the shortest path between the two states.

[0025] Furthermore, in step 3, the output action of the reinforcement learning strategy network is guided by the general planner and the action network to optimize the action selection, specifically:

[0026] If the general planner determines that the reachability from the current state to the target state is lower than the reachability at the previous moment, the action network will provide alternative actions to optimize the path; otherwise, the reinforcement learning policy network outputs the original policy action in the current state, expressed as follows:

[0027] ,

[0028] in, is the target distance when the current action is not executed, is the discount factor, For the number of completed rounds, the discount factor and the number of training rounds Control the diminishing role of guidance.

[0029] Furthermore, the reinforcement learning strategy network adopts a soft actor-critic SAC algorithm.

[0030] Furthermore, the step 4 specifically includes:

[0031] Step 401, define the task: task set D Contains multiple tasks, each task Corresponds to a Markov decision process:

[0032] ,

[0033] in is the state space, is the action space, is the transition probability, is the reward function, is the initial state distribution, is the discount factor;

[0034] Step 402: Share the reinforcement learning strategy network: All tasks share one reinforcement learning strategy network , task identification is achieved through task ID encoding, and the goal is to maximize the cumulative discounted return:

[0035] ,

[0036] in represents the trajectory length, represents the probability distribution of the trajectory, Represents the immediate reward, indicating that at time step t, the state Take action The rewards obtained.

[0037] Furthermore, the general planner shares kinematic model information between different tasks, and the reinforcement learning strategy network dynamically adjusts the weight distribution of different tasks by monitoring the task completion rate and its convergence speed.

[0038] The present invention mainly solves the generalization and stability problems of robot manipulation in a multi-task environment, improves learning efficiency, reduces negative transfer between tasks, and comprehensively considers the complexity of multi-tasks and the behavior patterns of robots in different tasks, thereby giving robots the ability to perform multiple tasks in a complex task environment and reducing training time and resource costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a process principle diagram of a multi-task control method based on a general planner and reinforcement learning according to an embodiment of the present invention;

[0040] Figure 2It is a schematic diagram of a process in which a general planner and an action network of an embodiment of the present invention are combined into a reinforcement learning strategy network to guide the output action of the reinforcement learning strategy network to optimize action selection. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0042] like Figure 1 As shown, an embodiment of the present invention discloses a multi-task control method based on a general planner (Task-Evoked Planner, TEP) and reinforcement learning, comprising the following steps:

[0043] Step 1: In a multi-task operation environment, record each task sample according to the state, action and reward function in different task scenarios, select a single task scenario, collect trajectory samples of the scenario task, and perform batch optimization. Specifically include:

[0044] Step 101: In a single-task scenario, when the task success rate reaches 85%, the sample selection process is started, and then Prioritized Trajectory Replay (PTR) is used to select trajectory samples. The priority is calculated based on the generalized advantage estimation (GAE) and distribution entropy, as follows:

[0045] Track Priority: ,in is the generalized advantage estimate;

[0046] Distribution entropy term: ;

[0047] The comprehensive priority can be obtained: , where the regulatory factor ,

[0048] Step 102: Batch optimization is performed on the selected samples, with each batch containing 512 samples to ensure the effectiveness of the planner in a multi-task environment.

[0049] Step 2: Use the trajectory samples optimized in batches in step 1 to train and optimize the general planner so that it can provide reachability information from the current state to the target state in different task scenarios; at the same time, build and train an action network (AN) to map the output of the general planner to the state, so that it can generate the optimal action plan based on the current state and the target state, and obtain the corresponding optimal action. Specifically include:

[0050] Step 201: Set the training objectives of the general planner, including:

[0051] Minimize the distance difference between adjacent states: ,in is the state at time t, is the state at time t+1, is the action selection at time t;

[0052] Optimize the reachability evaluation from the state to the goal: ,in The target final state;

[0053] Keep the planner stable across state swaps: .

[0054] Step 202: According to the goal in step 201, the general planner is optimized by minimizing the loss function to obtain a planner TEP that can accurately reflect the environment kinematic model, ensuring that it has sufficient environment modeling capabilities to provide effective guidance for robot manipulation in a multi-task environment.

[0055] Step 203: Considering the unknowability of the next state of reinforcement learning and the high dimensionality of the action space, the knowledge in TEP is transformed into a form and an AN network is further trained to complete The conversion of represents the optimal action given the current state and the target state, is the initial state, is the target state; the loss function loss of AN network training is as follows:

[0056] ,

[0057] in, is the state at time t, is the state at time t+1, is the action choice at time t, TEP output for delayed updates.

[0058] The internal idea is to traverse all the results of TEP and select the smallest action as The true value of the output is used to guide the training of the AN network.

[0059] The AN network transfers the spatial distance knowledge of TEP to the action knowledge. Both are multi-layer perceptrons (MLPs), using three layers of 512-dimensional fully connected layers with ReLU activation functions. The input of TEP is the connection vector of two states and actions, and the output is the spatial distance between the two states and the corresponding actions, that is, the number of steps required. The input of AN is the connection vector of two states, and the output is the action corresponding to the shortest path between the two states.

[0060] Step 3: Combine the general planner and action network into the reinforcement learning policy network to construct a general sample-guided reinforcement learning (Task-Evoked Reinforcement Learning, TERL). TEP and AN receive inputs simultaneously with the SAC policy network during training, and provide real-time guidance for policy learning in a multi-task environment by comparing the reachability of the current state and the target state. The guidance is gradually reduced during the training process, and finally the reinforcement learning policy network makes independent decisions, so that the robot can complete various tasks more autonomously in a multi-task environment.

[0061] Specifically, the general planner and action network are used to guide the output actions of the reinforcement learning policy network to optimize the action selection, such as Figure 2 As shown in Figure 1, if TEP determines that the reachability from the current state to the target state is lower than the reachability at the previous moment, AN will provide an alternative action to optimize the path; otherwise, the policy network outputs the original policy action in the current state. The specific expression is as follows:

[0062] ,

[0063] in, is the target distance when the current action is not executed, is the discount factor, For the number of completed rounds, the discount factor and the number of training rounds Control the diminishing role of guidance.

[0064] The reinforcement learning policy network adopts the Soft Actor-Critic (SAC) algorithm.

[0065] Step 4: In a multi-task learning environment, share the trained reinforcement learning policy network for all tasks. Specifically include:

[0066] Step 401, define the task: task set D Contains multiple tasks, each task Corresponding to a specific Markov decision process (MDP):

[0067] ,

[0068] in is the state space, is the action space, is the transition probability, is the reward function, is the initial state distribution, is the discount factor.

[0069] Step 402: Share a policy network: All tasks share a reinforcement learning policy network. , task identification is achieved through task ID encoding, and the goal is to maximize the cumulative discounted return:

[0070] ,

[0071] in represents the trajectory length, represents the probability distribution of the trajectory, Represents the immediate reward, indicating that at time step t, the state Take action The rewards obtained.

[0072] Step 403, measures to reduce negative transfer: In order to reduce negative transfer between different tasks, the planner shares kinematic model information between different tasks to maintain the similarity of samples in the state space and enhance the robustness of the strategy. At the same time, combined with the reinforcement learning method, the weight distribution of task-related features is optimized in real time, thereby enhancing the robustness of the strategy and the generalization ability of the task.

[0073] Step 404, dynamic weight balance among multiple tasks: During the training process, the weight distribution of different tasks in the multi-task strategy network is dynamically adjusted by monitoring the task completion rate and strategy convergence speed. For certain tasks that are difficult to learn, their weights in the loss function are increased to promote focused learning of the strategy; for tasks that are easy to converge, the weights are reduced to release computing resources.

[0074] In summary, the method of the present invention collects trajectory data of a single-task scenario in a multi-task operation environment according to the task success rate, and selects high-priority samples through priority trajectory playback; uses the selected samples to minimize the difference between adjacent states to train the general planner, expecting to obtain two states and one action as the reachability information under certain conditions, that is, the required step distance; trains the action network by minimizing the step distance output by the planner as the goal to obtain the optimal action between the two states; then combines TEP and AN into the reinforcement learning strategy network to guide it, so as to optimize the action selection and gradually reduce the guidance. Finally, the strategy network obtained by reinforcement learning by sharing multi-task samples is used to achieve efficient learning in a multi-task environment. The robot multi-task control method of the present invention can effectively deal with the policy conflicts between different tasks, improve the stability and generalization ability of the strategy, significantly improve the learning efficiency and success rate, and achieve efficient control in a multi-task environment.

[0075] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task control method based on a general planner and reinforcement learning, characterized in that: The following steps are involved: Step 1: In a multi-task operation environment, record each task sample according to the state, action and reward function in different task scenarios, select a single task scenario, collect trajectory samples of the scenario task, and perform batch optimization; Step 2: Use the trajectory samples optimized in batches in step 1 to train and optimize the general planner. At the same time, build and train an action network to map the output of the general planner to the state and obtain the corresponding optimal action. Step 3: Combine the general planner and action network into the reinforcement learning policy network, use the general planner and action network to guide the output action of the reinforcement learning policy network to optimize the action selection, and gradually reduce the guidance through training, so that the reinforcement learning policy network can make independent decisions in the end; The general planner and action network are both multi-layer perceptrons, using three layers of 512-dimensional fully connected layers with ReLU activation function. The input of the general planner is the connection vector of two states and actions, and the output is the spatial distance between the two states and the corresponding actions, that is, the number of steps required. The input of the action network is the connection vector of two states, and the output is the action corresponding to the shortest path between the two states. Step 4: In a multi-task learning environment, share the trained reinforcement learning policy network for all tasks; In step 3, the general planner and the action network are used to guide the output action of the reinforcement learning strategy network to optimize the action selection, specifically: If the general planner determines that the reachability from the current state to the target state is lower than the reachability at the previous moment, the action network will provide alternative actions to optimize the path; otherwise, the reinforcement learning policy network outputs the original policy action in the current state, expressed as follows: ， in, is the target distance when the current action is not executed, is the discount factor, For the number of completed rounds, the discount factor and the number of training rounds Control the diminishing role of guidance.

2. The multi-task control method according to claim 1, characterized in that: In step 1, in a single-task scenario, when the task success rate reaches 85%, the sample selection process is started, and then the priority trajectory playback is used to select trajectory samples. The priority is calculated based on the generalized advantage estimate and distribution entropy. The specific expression is as follows: ， in, is the regulating factor, is the trajectory priority, is a measure of the entropy of a distribution.

3. The multi-task control method according to claim 1, characterized in that: In step 2, the training objectives of the general planner include: Minimize the distance difference between adjacent states: ,in is the state at time t, is the state at time t+1, is the action selection at time t; Optimize the reachability evaluation from the state to the goal: ,in is the state at time t, is the state at time t+1, is the action choice at time t, The target final state; Keep the planner stable across state swaps: ,in is the state at time t, is the state at time t+1, is the action choice at time t.

4. The multi-task control method according to claim 1, characterized in that: In step 2, the training of the action network is achieved by minimizing the error function, and its loss function loss expression is: ， in, is the state at time t, is the state at time t+1, is the action choice at time t, Output of the generic planner for delayed updates.

5. The multi-task control method according to claim 4, characterized in that: The optimization objective function of the action network is: ,in represents the optimal action given the current state and the target state, is the initial state, is the target state.

6. The multi-task control method according to claim 1, characterized in that: The reinforcement learning policy network adopts the soft actor-critic SAC algorithm.

7. The multi-task control method according to claim 1, characterized in that: The step 4 specifically includes: Step 401, define the task: task set D Contains multiple tasks, each task Corresponds to a Markov decision process: ， in is the state space, is the action space, is the transition probability, is the reward function, is the initial state distribution, is the discount factor; Step 402: Share the reinforcement learning strategy network: All tasks share one reinforcement learning strategy network , task identification is achieved through task ID encoding, and the goal is to maximize the cumulative discounted return: ， in represents the trajectory length, represents the probability distribution of the trajectory, Represents the immediate reward, indicating that at time step t, the state Take action The rewards obtained.

8. The multi-task control method according to claim 7, characterized in that: The general planner shares kinematic model information between different tasks, and the reinforcement learning strategy network dynamically adjusts the weight distribution of different tasks by monitoring the task completion rate and its convergence speed.

Images