A transferable behavior decision and state planning learning method based on teaching and policy decoupling
By decomposing the black-box policy model into planning and decision-making modules and using a teaching and policy decoupling approach for joint learning, the problem of policy transfer in different environments and action spaces is solved, achieving high generalization performance in planning module learning and policy transfer.
Patent Information
- Application Number
- CN202210826945.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-13
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2042-07-13
AI Technical Summary
Existing reinforcement learning methods struggle to effectively transfer policies across different environments and action spaces. In particular, black-box policy models fail to effectively decouple planning and decision-making modules, resulting in the inability to share planning capabilities among agents with different control methods.
The black-box policy model is decomposed into a planning module and a decision module. Joint learning is carried out through teaching and policy decoupling. The module parameters are updated using supervision and adversarial learning methods to achieve end-to-end optimization.
The planning module learning achieves high generalization performance, can be effectively transferred in different task scenarios, and improves the generalization ability and training efficiency of the policy.
Smart Images
Figure CN115222043B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of reinforcement learning, and more particularly to a transferable learning method for behavioral decision-making and state planning based on decoupling of teaching and policy. Background Technology
[0002] The basic approach to imitation learning can be supervised learning, which directly learns the state-to-action mapping of a policy from expert data. However, due to the limitations of data distribution, it cannot be extrapolated to the entire state space, and algorithms using supervised learning often suffer from the problem of accumulated error.
[0003] In their paper "Generative Adversarial Imitation Learning" presented at the 2016 Conference on Neural Information Processing Systems, Ho et al. proposed using generative adversarial networks for imitation learning. Specifically, they employed a discriminative module to learn to distinguish between expert data and learning data, and to provide reward signals to the agent for policy learning.
[0004] In their 2017 paper, "Imitating Driver Behavior with Generative Adversarial Networks," Kuefler et al. applied the method proposed by Ho et al. to autonomous driving simulation, using the NGSIM dataset to learn driving strategies under various road conditions and highly interactive scenarios.
[0005] The algorithms mentioned above all employ end-to-end control black-box decision-making models based on neural networks, and simply learn the mapping of control signals from state to action, without including planning-level learning. Limited to the same action space or environmental dynamics model, they cannot share similar planning capabilities across agents with different control methods.
[0006] Therefore, those skilled in the art are dedicated to developing a transferable learning method for behavioral decision-making and state planning based on decoupling of teaching and policy. This method decouples the black-box policy into motion planning and behavioral decision-making modules. It provides a feedback mechanism for teaching behavior and a decoupled policy gradient for learning. It can jointly learn the two modules of motion planning and behavioral decision-making end-to-end. It can be used to simulate teaching data with only state trajectories and no control signals, and can be transferred to agents with different action spaces and action transitions. Summary of the Invention
[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is the practical problem of the difficulty in transferring strategies in reinforcement learning.
[0008] To achieve the above objectives, this invention provides a transferable behavioral decision-making and state planning learning method based on decoupling of teaching and strategy. The method comprises a decision module, a planning module, and a decision-planning-discrimination module. The planning module predicts the target state based on the state of the environment and assigns it to the decision module for execution, utilizing supervised learning. The decision module, also known as the inverse environment module, outputs a decision signal (i.e., an action) based on the state of the environment and the target state predicted by the planning module. The decision-planning-discrimination module performs adversarial learning based on sampled state data from the environment and expert teaching data, providing a reward signal for the planning module's learning.
[0009] Furthermore, the strategy decoupling is divided into motion planning and behavior decision-making modules;
[0010] Furthermore, the learning method includes a learning process and a migration process.
[0011] Furthermore, the learning process includes the following steps:
[0012] Step a1: Obtain initial data;
[0013] Step a2: Based on the current environmental state, the planning module outputs the predicted state of the plan and hands it over to the decision module to output the decision signal. In this way, the state sequence is obtained by interacting with the environment at fixed time steps, and the state action behavior sequence is put into the experience playback pool.
[0014] Step a3: Sample data from the experience replay pool to learn the decision module, predict decision signals based on the state action behavior sequence, and update module parameters using data from the experience replay pool;
[0015] Step a4: Collect the state-action-behavior sequences from the teaching data and the experience replay pool to train the discrimination module, and update the module parameters according to the discrimination error;
[0016] Step a5: Sample data from the experience replay pool to learn the planning module, and input them into the discrimination module. The decision planning discrimination module determines whether the input state-action behavior sequence pair comes from the teaching data and gives a corresponding feedback signal as a reward.
[0017] Step a6: The planning module updates the module parameters based on the feedback signal.
[0018] Step a7: If the planning module and the decision module converge, the method ends; otherwise, proceed to step a2.
[0019] Furthermore, the initial data in step a1 includes teaching data without control signals, an experience playback pool, the initialized discrimination module, the decision planning discrimination module, and the decision module.
[0020] Furthermore, the update of the module parameters in step a3 is performed using supervised learning and gradient descent.
[0021] Furthermore, the update of the module parameters in step a4 is performed using adversarial learning and gradient descent.
[0022] Furthermore, in step a6, the target update adopts a decoupling strategy for gradient update.
[0023] Furthermore, the migration process includes the following steps:
[0024] Step b1: Obtain the pre-trained planning module and the initialized decision module;
[0025] Step b2: Based on the current environmental state, the planning module outputs the predicted state of the plan and hands it over to the decision module to output the decision signal. In this way, the state sequence is obtained by interacting with the environment at fixed time steps, and the state action behavior sequence is put into the experience playback pool.
[0026] Step b3: Sample data from the experience replay pool to learn the decision module, predict the decision signal based on the state action behavior sequence, update the module parameters using the data in the experience replay pool, and update the target using supervised learning, which can be done using gradient descent.
[0027] Step b4: If the decision module converges, the method ends; otherwise, proceed to step 2.
[0028] Furthermore, the update of the module parameters in step b3 is performed using supervised learning and gradient descent.
[0029] The beneficial effects of this invention are as follows: 1) This invention proposes to decompose the black-box policy model into a planning module and a decision module, and performs joint end-to-end optimization, which can learn a planning module with high generalization performance, and can be used to continuously plan a series of target states; 2) This invention utilizes an incremental online learning method, which can cope with different task scenarios and effectively improve the generalization ability of the policy; 3) The planning module learned by this invention can be further used for policy transfer based on pre-training, for example, pre-training on a simple action space and using it for transfer of complex and multiple control tasks, which improves the generalization and efficiency of training.
[0030] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description
[0031] Figure 1 This is a system overall module diagram of a preferred embodiment of the present invention. Detailed Implementation
[0032] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.
[0033] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.
[0034] Example 1: Planning Learning and Pre-training Transfer Based on Simulated Robots
[0035] This embodiment primarily addresses the control problem of the Mujoco robot within OpenAI's open-source library Gym. Specifically, we define the state as the position and velocity of each part of the robot, and the action as the force applied to each part. The goal is to maximize the distance the robot travels without tipping over, while minimizing the force required for control. Specifically, in this embodiment, we tested the learning process using this method and transferred the trained planning module to different environmental dynamics models, detailing the following:
[0036] 1. First, initialize the planning module, decision-making module, discrimination module, and replay experience pool, including the planning module;
[0037] 2. Based on the current environmental state, output the expected target state and hand it over to the decision module to output control signals, interact with the environment to obtain the corresponding real target state, and form a state-action-state sequence pair;
[0038] 3. Based on the state sequence pairs obtained from the robot's interaction with the environment, input them into the decision planning and discrimination module. This module determines whether the input sequence comes from the teaching data and outputs a feedback signal.
[0039] 4. The decision-making module uses the state sequence obtained from the robot's interaction with the environment to update the module parameters in a supervised learning manner.
[0040] 5. The motion planning module updates its parameters based on feedback signals. The update method combines the proposed decoupling strategy gradient with the supervision signals from the teaching data.
[0041] 6. A decision-making and planning module utilizes state-action / behavior sequence pairs formed by the taught trajectory and state sequence pairs obtained from the robot's interaction with the environment. Module parameters are updated based on the discrimination error.
[0042] 7. Iterate through steps 2-6 until the module converges. The resulting planning and decision modules can perform corresponding goal planning and strategy output based on the current environmental state during the application phase. They can also plan the optimal goal and the decision signal to achieve the goal based on any state. Furthermore, the goal planning module can be used for multi-step planning. When the action space or environmental dynamics change, efficient transfer can be performed on the new task. The following assumes that the module has been pre-trained to convergence and describes the transfer process.
[0043] 8. Initialize the decision-making module and the experience replay pool;
[0044] 9. Repeat step 2 to collect data on the new task;
[0045] 10. Repeat step 4 to update the decision module parameters;
[0046] 11. Iterate through steps 8-10 above until the module converges. The resulting decision module can be directly combined with the pre-trained planning module in the application stage to perform corresponding target planning and strategy output based on the current environmental state. It can also plan the corresponding optimal target and the decision signal to achieve the target based on any state.
[0047] Example 2: Traffic Flow Trajectory Planning, Learning, and Prediction Based on Driver Teaching Data
[0048] This embodiment is primarily used for traffic flow trajectory planning, learning, and prediction based on real traffic flow data. Specifically, in this embodiment, we tested the use of this method to learn and transfer the trained planning module to different environmental dynamics models, detailing the following aspects:
[0049] 1. First, initialize the planning module, decision-making module, discrimination module, and replay experience pool, including the planning module;
[0050] 2. The planning module outputs the expected target state of the vehicle (including the position, orientation, and speed of the vehicle and surrounding adjacent vehicles) based on the current environmental state (including the position, orientation, and speed of the vehicle). This target state is then passed to the decision module (reverse environment dynamics module) to output the expected control signals (acceleration and angular velocity). The system interacts with the environment to obtain the corresponding real state (including the position, orientation, and speed of the vehicle and surrounding adjacent vehicles), thus forming a state-action-state sequence pair.
[0051] 3. Based on the vehicle's state, action, and behavior sequence obtained from the interaction between the vehicle and the environment, input the sequence into the decision planning and discrimination module. This module determines whether the input sequence comes from the teaching data and outputs a feedback signal.
[0052] 4. The decision-making module uses the vehicle state sequence data obtained from interaction with the environment to update the module parameters in a supervised learning manner;
[0053] 5. The motion planning module updates its parameters based on feedback signals. The update method combines the proposed decoupling strategy gradient with the supervision signals from the teaching data.
[0054] 6. A decision-making and planning module utilizes state-action / behavior sequence pairs formed by the taught trajectory and state sequence pairs obtained from the robot's interaction with the environment. Module parameters are updated based on the discrimination error.
[0055] 7. Iterate through steps 2-6 above until the module converges. The resulting planning and decision modules can perform corresponding target planning and strategy output based on the current environmental state during the application phase. They can also plan the optimal target and the decision signal to achieve the target based on any state. At the same time, the target planning module can be used for multi-step planning. When the action space or environmental dynamics change, it can be efficiently transferred to the new task. The following assumes that the module has been pre-trained to convergence and describes the transfer process.
[0056] 8. Initialize the decision-making module and the experience replay pool;
[0057] 9. Repeat step 2 to collect data on the new task;
[0058] 10. Repeat step 4 to update the decision module parameters;
[0059] 11. Iterate through steps 8-10 above until the module converges. The resulting decision module can be directly combined with the pre-trained planning module in the application stage to perform corresponding target planning and strategy output based on the current environment state. It can also plan the corresponding optimal target and decision signal to achieve the target based on any state. At the same time, it can use the target planning module to perform multi-step planning.
[0060] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A transferable learning method for behavioral decision-making and state planning based on teaching and policy decoupling, characterized in that, This method is mainly used to solve the Mujoco robot control problem in OpenAI's open-source library Gym. First, the state is defined as the position and velocity of each part of the robot, and the action is the force applied to each part. The goal is to make the robot move as far as possible without falling over, while minimizing the force used for control. The method includes a decision module, a planning module, and a decision-planning-discrimination module. The planning module predicts the target state based on the state of the environment and then passes it to the decision module for execution, utilizing supervised learning. The decision-making module, also known as the inverse environment module, outputs a decision signal, i.e., an action, based on the state of the environment and the target state predicted by the planning module. The decision-planning and discrimination module performs adversarial learning based on the sampled state data in the environment and the expert teaching data, and provides a reward signal for the planning module to learn. The strategy decoupling is divided into motion planning and behavior decision-making modules; The learning method includes a learning process and a migration process; The learning process includes the following steps: Step a1: Obtain initial data; Step a2: Based on the current environmental state, the planning module outputs the predicted state of the plan and hands it over to the decision module to output the decision signal. In this way, the state sequence is obtained by interacting with the environment at fixed time steps, and the state action behavior sequence is put into the experience playback pool. Step a3: Sample data from the experience replay pool to learn the decision module, predict decision signals based on the state action behavior sequence, and update module parameters using data from the experience replay pool; Step a4: Collect the state-action-behavior sequences from the teaching data and the experience replay pool to train the discrimination module, and update the module parameters according to the discrimination error; Step a5: Sample data from the experience replay pool to learn the planning module, and input them into the discrimination module. The decision planning discrimination module determines whether the input state-action behavior sequence pair comes from the teaching data and gives a corresponding feedback signal as a reward. Step a6: The planning module updates the module parameters based on the feedback signal. Step a7: If the planning module and the decision module converge, the method ends; otherwise, proceed to step a2.
2. The transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 1, characterized in that, The initial data in step a1 includes teaching data without control signals, an experience playback pool, the initialized discrimination module, the decision planning discrimination module, and the decision module.
3. The transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 2, characterized in that, The update of the module parameters in step a3 is performed using supervised learning and gradient descent.
4. The transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 3, characterized in that, The module parameters in step a4 are updated using gradient descent.
5. The transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 4, characterized in that, In step a6, the target is updated using a decoupled strategy gradient and a supervisory signal from expert data.
6. The transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 5, characterized in that, The migration process includes the following steps: Step b1: Obtain the pre-trained planning module and the initialized decision module; Step b2: Based on the current environmental state, the planning module outputs the predicted state of the plan and hands it over to the decision module to output the decision signal. In this way, the state sequence is obtained by interacting with the environment at fixed time steps, and the state action behavior sequence is put into the experience playback pool. Step b3: Sample data from the experience replay pool to learn the decision module, predict the decision signal based on the state action behavior sequence, update the module parameters using the data in the experience replay pool, and update the target using supervised learning, which can be done using gradient descent. Step b4: If the decision module converges, the method ends; otherwise, proceed to step 2.
7. A transferable behavioral decision-making and state planning learning method based on teaching and policy decoupling as described in claim 6, characterized in that, The update of the module parameters in step b3 is performed using supervised learning and gradient descent.