Robot path planning model and method based on two-stage decoupling offline reinforcement learning
Through the two-stage decoupling offline reinforcement learning method, the Transformer model is used to make sequence decisions of state and action, which solves the external distribution generalization problem of offline reinforcement learning in the existing technology, and improves the generalization ability and learning performance of robot path planning .
Patent Information
- Application Number
- CN202411100707.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-17
AI Technical Summary
The existing offline reinforcement learning methods have challenges in the problem of external distribution generalization (OOD), which leads to incorrect generalization during strategy optimization, inability to effectively utilize data set information, and high computing resources and poor generalization capabilities of model, making it easy to fall into the local area. excellent.
The offline reinforcement learning method with two-stage decoupling is adopted. Through the state prediction network and the execution policy network, state selection and action selection are processed respectively, and sequence decisions are made using the Transformer model to avoid out-of-distribution action value estimation and improve trajectory splicing capabilities.
It improves the generalization ability of robot path planning, reduces the demand for computing resources, avoids local optimal problems, and enhances the adaptability and performance of learning.
Smart Images
Figure CN119148702A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of machine intelligent decision-making and control, and specifically relates to a robot path planning model and method based on two-stage decoupled offline reinforcement learning. Background Art
[0002] At present, reinforcement learning is widely used in robot path planning tasks. Reinforcement learning has a certain degree of autonomy and adaptability; reinforcement learning enables robots to autonomously learn path planning strategies through interaction with the environment, rather than relying on pre-designed rules or models. Robots can continuously adjust and optimize their path planning strategies in dynamic and unknown environments, and have strong adaptability. At the same time, they have great advantages in continuous optimization, high-level state space processing, and handling uncertainty.
[0003] Standard online reinforcement learning is trained through real-time interaction between the agent and the environment. This method has extremely low sampling efficiency and requires frequent interaction with the environment to collect data to train the agent. In the process of interacting with the environment, a large amount of exploration behavior often means high costs in real scenarios. For example, using a robot to explore in a real scene may damage the robot itself or surrounding objects, and exploration in medical or autonomous driving scenarios may seriously endanger human safety. In contrast, offline reinforcement learning is a method of reinforcement learning using existing data sets. Compared with online reinforcement learning, it does not require real-time interaction with the environment, which can save sampling costs, improve data utilization, and reduce safety risks. It is applicable to many practical scenarios, such as medical and financial systems. The core problem of offline reinforcement learning is how to learn an effective strategy from a fixed data set so that it performs well in an unknown environment. This data set is usually generated by one or more behavior policies.
[0004] However, offline reinforcement learning also faces some challenges, the most important of which is the out-of-distribution generalization (OOD) problem. Traditional offline reinforcement learning methods, such as the BEAR algorithm (Bootstrapping Error Accumulation Reduction), the BRAC algorithm (Behavior Regularized Offline Reinforcement Learning), the temporal difference + action cloning algorithm, the CQL algorithm (Conservative Q-Learning), and the REM algorithm (Random Ensemble Mixture) all face this problem. The cause of this problem is distribution offset, that is, the state-action distribution in the offline dataset is inconsistent with the state-action distribution under the current policy, resulting in incorrect generalization during the policy optimization process. The OOD problem will make it impossible for the offline reinforcement learning algorithm to effectively utilize the information in the dataset, and may even cause the policy performance to deteriorate.
[0005] How to effectively solve the OOD problem has become a major difficulty in the development of offline reinforcement learning. The method based on the sequential decision model provides a new perspective for offline reinforcement learning. This method regards the reinforcement learning task as a sequential decision problem and uses Transformer for sequential decision making. Therefore, there is no need to estimate the value of out-of-distribution actions, which fundamentally avoids the OOD problem. The Transformer-based method has been applied to offline reinforcement learning tasks and has shown strong competitiveness, but there is still a lot of room for optimization in the trajectory splicing method. The present invention intends to improve the trajectory splicing ability of the Transformer-based offline reinforcement learning method and enhance the performance of the reinforcement learning agent through elastic history length and two-stage decoupling strategy. Summary of the invention
[0006] In view of the above-mentioned prior art, the present invention aims to propose a robot path planning model based on two-stage decoupled offline reinforcement learning, aiming to solve the problems of existing robot path planning methods, such as high computing resource requirements, poor model generalization ability, and easy to fall into local optimality.
[0007] In order to solve the above technical problems, the present invention adopts a technical solution of a robot path planning model based on two-stage decoupled offline reinforcement learning, including a state prediction network and an execution strategy network, and its construction method includes the following steps:
[0008] Step 1: Select training data and construct a training set; consider the robot path planning problem as a Markov process M = (S, A, P, R), and describe the robot's path through a Markov sequence; where S represents the state space, that is, the set of all possible states of the robot, represented by the two-dimensional coordinates of the robot; A represents the action space, that is, the set of actions that the robot can take; P(·|s, a) represents the state transition probability, that is, the probability of selecting action a in state s and transferring to a certain state s′; R(s, a) represents the reward; the training data includes an offline robot path planning dataset;
[0009] Step 2: Preprocess the training data to obtain the characteristics of the current state;
[0010] Step 3: The features are input into the state prediction network, and the state prediction network is used to predict the next optimal state;
[0011] Step 4: Construct a loss function to measure the gap between the state predicted by the model in step 3 and the actual next state; optimize and update the model parameters by minimizing the loss function during the training process;
[0012] Step 5: Build an execution strategy network and use it to predict the optimal action to execute from the current state to the next state;
[0013] Step 6: Optimize the execution strategy network;
[0014] Step 7: Use the training set to train the model and determine the hyperparameters; stop training when the prediction performance reaches the preset value, and the robot path planning model is completed.
[0015] Furthermore, the offline robot path planning dataset includes an optimal path set guided by experts and / or a robot path planning dataset completed by other high-performance robot path planning algorithms.
[0016] Furthermore, the preprocessing is specifically as follows: the last K time steps in the reinforcement learning are input into the decision converter, R, s, a are respectively mapped to an embedding and added to the position code to obtain tokens of a set length as the feature of the current state, which includes local features and global features.
[0017] Furthermore, the state prediction network is specifically a transformer model; the transformer model is composed of stacked self-attention layers and connection layers; the tokens of R, s, and a are input into the transformer model, the transformer model outputs a hidden state, and the hidden state is input into a linear layer to predict the next state;
[0018] Among them, the i-th output z of the self-attention layer i By query q i With other keys k j The normalized dot product weighted value v between j gives:
[0019]
[0020] Furthermore, the loss function in step 4 is specifically a cross entropy loss function, and the calculation formula is as follows:
[0021]
[0022] In the robot path planning process, p represents the actual state selection distribution, and q represents the probability distribution of the transformer model output.
[0023] Furthermore, the execution strategy network only takes the current state s and the next state s′ as input, selects the optimal action a in the current state, and uses the policy gradient method for training. The specific formula for constructing the execution strategy network is as follows:
[0024]
[0025]
[0026] Among them, π represents the control strategy of the model, that is, when the drone is in a certain state s, the model chooses to take control actions; Q(s, a) is the reward of the state-action pair, which represents the cumulative reward that can be obtained from the current state to the end of the task, that is, the robot takes control action a in the current state s, and the rewards and can be obtained in the future. The model selects the most appropriate control action by maximizing the Q value; represents the expectation, and Z represents the reward distribution of the current state-action pair (s, a).
[0027] Furthermore, the specific process of optimizing the execution strategy network is as follows:
[0028] Let the execution strategy be During the training process, the following objectives are obtained:
[0029]
[0030] During the evaluation process, given a state s, the final action Determined by both the guiding strategy and the execution strategy:
[0031]
[0032] In the robot path planning task, g ωRepresents the execution strategy network, and the optimization goal of g is:
[0033]
[0034] Where α represents the weight, V φ Represents the state-action-value function.
[0035] Furthermore, the hyperparameters include the learning rate of the state prediction network and the execution strategy network, the size of the experience replay pool, the number of exploration steps, the number of training steps, and the number of optimization steps.
[0036] The present invention also provides a robot path planning method based on two-stage decoupled offline reinforcement learning, comprising the following steps:
[0037] Step 1: For the robot to be controlled, the Markov decision process is used to describe the robot path planning process according to the possible states of the robot and the optional control actions. The state s is observed by the camera or other equipment carried by the robot, and the action a represents the action that the robot can perform;
[0038] Step 2: The current state of the robot to be controlled is preprocessed and input into the state prediction network. The state prediction network obtains the predicted state s′ based on the offline data set. The optimal state of the robot for the next step can be obtained through this network.
[0039] Step 3: Input the current state s and the next state s' into the execution strategy network, which selects the optimal action a by maximizing the reward * ;
[0040] Step 4: The robot performs the optimal action a * , after executing the optimal action, the robot enters the new state s; repeat steps 2-4 until the robot path planning process is completed.
[0041] The beneficial effect of the present invention is that the present invention proposes a two-stage decoupled offline reinforcement learning robot path planning method, which decouples state selection and action selection. The state selection is selected from the expert data set, while the action does not need to be selected from the expert data set. On the one hand, this reinforcement learning method improves the problem of traditional robot path planning methods requiring a large amount of computing resources and being prone to falling into local optimality; on the other hand, this two-stage decoupled offline reinforcement learning method can learn actions outside the offline data set, greatly increasing the generalization ability of robot learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flowchart of the robot path planning model training in Example 1 of the present invention.
[0043] Figure 2 This is a flow chart of the robot path planning method in Example 2 of the present invention. DETAILED DESCRIPTION
[0044] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.
[0045] In order to better understand the present invention for those skilled in the art, the present invention is further described in detail below in conjunction with the accompanying drawings and the following embodiments:
[0046] Example 1
[0047] This example provides a robot path planning model based on two-stage decoupled offline reinforcement learning, including a state prediction network and an execution strategy network. Figure 1 As shown, the following steps are included:
[0048] Step 1: Select training data and construct a training set. The robot path planning problem is regarded as a Markov process M = (S, A, P, R). Where S represents the state space, that is, the set of all possible states of the robot, represented by the two-dimensional coordinates of the robot; A represents the action space, that is, the set of actions that the robot can take, such as forward, backward, left, right, left front, left rear, right front, right rear; P (· | s, a) represents the state transition probability, that is, the probability of selecting action a in state s and transferring to a certain state s'; R (s, a) represents the reward, that is, the total reward obtained from the current state s to the end and the total reward obtained from the current state s to the end. Unlike traditional reinforcement learning, the reward calculation in the path planning task does not apply discounts. When the robot avoids the trap and successfully reaches the end, it receives a reward of +10; when the robot falls into the trap, it receives a reward of -10; the training data mainly consists of the optimal path set guided by experts. When expert data is missing, the robot path planning data set completed by other high-performance robot path planning algorithms can be used. For example, a human expert may formulate an optimal path for a robot in advance and use the sequence (s1, a1, R1, s2, a2, R2, …, s t , a t , R t ) means that for each moment s in the sequence i , a i , R i ,s i That is, the state of the drone at that moment, which can be obtained by the camera or positioning system on the drone. i is the control action taken at that moment, R i is the reward that can be obtained in this state. The training data set consists of multiple such sequences.
[0049] Step 2: Preprocess the input. Input the last K time steps in reinforcement learning into the decision transformer, map R, s, and a to an embedding respectively and add them to the position code to form a token with a length of 3,000. This token is used as the input of the transformer model. After preprocessing, the features of the current state can be obtained, which include the local features and global features of the current state.
[0050] Step 3: The state prediction network predicts the next state based on the transformer model.
[0051] The tokens of R, s, and a are input into the transformer model, and finally the transformer model outputs a hidden state, which is then input into a linear layer as a feature to predict the next state. The transformers model consists of stacked self-attention layers and connection layers. Each self-attention layer receives n fixed embeddings Output n fixed embeddings corresponding to unique input tokens Keep the input dimension unchanged. The i-th token is mapped to key k through a linear transformation i 、Query q i Sum value v i The i-th output z of the self-attention layer i By query q i With other keys k j The normalized dot product weighted v between j gives:
[0052]
[0053] Step 4: Construct a loss function to measure the gap between the state predicted by the model in step 3 and the actual next state. During the training process, the model parameters are optimized and updated by minimizing the loss function;
[0054] We sample mini-batches of sequence length J from the dataset. With the input state s t The corresponding prediction head is trained to predict action a t . For discrete actions, there is a cross entropy loss, while for continuous actions, the mean square error is used, and the losses of both errors at each time step are averaged. In the actual training process, discrete data sets are usually used. Therefore, the present invention mainly uses cross entropy loss to optimize the transformer model. The cross entropy loss function is as follows:
[0055]
[0056] In the robot path planning process, p represents the actual state selection distribution, and q represents the probability distribution of the transformer model output. During the training process, the results predicted by the transformer model are made as close as possible to the actual results. The transformer model can be optimized by minimizing the cross entropy loss.
[0057] Step 5: Build an execution policy network to predict actions.
[0058] The execution policy network is decoupled from the state prediction network and is trained separately. The state prediction network predicts the optimal next state, and the execution policy network learns what actions can be taken to get from s to s' given the current state s and the next state s'. Since the execution policy network only takes s and s' as input and is independent of the trajectory, it can theoretically learn good actions from bad trajectories, improving the problem that the transformer model only uses good trajectories.
[0059] First, initialize the optimizer's network parameters, collect initial environmental interaction data, and build an initial experience pool. Then use the output of the state prediction network to train the execution policy network. Based on the output of the state prediction network, the execution policy network selects the optimal action a in the current state to maximize the long-term cumulative return. The execution policy network is trained using the policy gradient method. The model formula is as follows:
[0060]
[0061]
[0062] In the robot path planning task, π represents the control strategy of the model, that is, when the drone is in a certain state s, the model chooses the control action to take; Q(s, a) is the reward of the state-action pair, which represents the cumulative reward that can be obtained from the current state to the end of the task, that is, the robot takes control action a in the current state s, and the rewards and can be obtained in the future. The model selects the most appropriate control action by maximizing the q value; represents the expectation, and Z represents the reward distribution of the current state-action pair (s, a).
[0063] Step 6: Optimize the execution strategy network
[0064] Since the task of executing the policy is to have strong generalization ability, we adopt the supervised learning RL framework and take the state s encountered in the dataset as the condition for executing the policy. Specifically, the execution policy is recorded as During training, π performs supervised learning by maximizing the likelihood of an action given a state and the next state, resulting in the following objective:
[0065]
[0066] During the evaluation process, given a state s, the final action Determined jointly by the guiding strategy and the execution strategy.
[0067]
[0068] In the robot path planning task, g ω Represents the execution strategy network, and the optimization goal of g is:
[0069]
[0070] where the weight α is the trade-off between guiding to the high-reward space and the space where the execution policy should have correct generalization, V φ Represents the state-action-value function.
[0071] Step 7: Use the training set to train the model and determine the hyperparameters, which include the learning rate of the two networks, the size of the experience replay pool, the number of exploration steps, the number of training steps, and the number of optimization steps; during the training process, the various hyperparameters of the network are continuously adjusted until the prediction performance reaches the preset value and then the training is stopped. A robot path planning model based on two-stage decoupled offline reinforcement learning is constructed.
[0072] Example 2
[0073] This example provides a robot path planning method based on two-stage decoupled offline reinforcement learning. Figure 2 As shown, the following steps are included:
[0074] Step 1: For the robot to be controlled, use the Markov process to describe the system according to the possible states of the robot and the selectable control actions; that is, use the Markov decision process to describe the robot path planning process, the state s is observed by the camera and other equipment carried by the robot, and the action a represents the action that the robot can perform (forward, backward, walk left, walk right, walk forward, walk left rear, walk right front, walk right rear).
[0075] Step 2: The current state of the robot to be controlled is preprocessed and input into the state prediction network. The state prediction network obtains the predicted state s′ based on the offline data set. Through this network, the optimal state of the robot in the next step, that is, the best position in the next step, can be obtained.
[0076] Step 3: Input state s and state s' into the execution strategy network, which selects the optimal action a by maximizing the reward * ; Through this network, we can get the action taken by the robot from the current state to the next state, such as executing the action to the right front.
[0077] Step 4: The robot performs the optimal action a * , after executing the optimal action, the robot enters a new state s; that is, the new position the robot reaches after executing the action.
[0078] Repeat steps 2-4 until the robot path planning process is complete.
[0079] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.
Claims
1. A robot path planning model based on two-stage decoupled offline reinforcement learning, characterized in that: The construction of the model includes the following steps: Step 1: Select training data and construct a training set; consider the robot path planning problem as a Markov process M = (S, A, P, R), and describe the robot's path through a Markov sequence; where S represents the state space, that is, the set of all possible states of the robot, represented by the two-dimensional coordinates of the robot; A represents the action space, that is, the set of actions that the robot can take; P(·|s, a) represents the state transition probability, that is, the probability of selecting action a in state s and transferring to a certain state s′; R(s, a) represents the reward; the training data includes an offline robot path planning dataset; Step 2: Preprocess the training data to obtain the characteristics of the current state; Step 3: The features are input into the state prediction network, and the state prediction network is used to predict the next optimal state; Step 4: Construct a loss function to measure the gap between the state predicted by the model in step 3 and the actual next state; optimize and update the model parameters by minimizing the loss function during the training process; Step 5: Build an execution strategy network and use it to predict the optimal action to execute from the current state to the next state; Step 6: Optimize the execution strategy network; Step 7: Use the training set to train the model and determine the hyperparameters; stop training when the prediction performance reaches the preset value, and the robot path planning model is completed.
2. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 1, characterized in that: The offline robot path planning dataset includes an optimal path set guided by experts and / or a robot path planning dataset completed by other high-performance robot path planning algorithms.
3. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 1, characterized in that: The preprocessing is specifically as follows: the last K time steps in the reinforcement learning are input into the decision converter, R, s, a are respectively mapped to an embedding and added to the position code to obtain tokens of a set length as the feature of the current state, which includes local features and global features.
4. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 3, characterized in that: The state prediction network is specifically a transformer model; the transformer model consists of stacked self-attention layers and connection layers; the tokens of R, s, and a are input into the transformer model, and the transformer model outputs a hidden state, which is input into a linear layer to predict the next state; Among them, the i-th output z of the self-attention layer i By query q i With other keys k j The normalized dot product weighted value v between j gives:
5. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 4, characterized in that: The loss function in step 4 is specifically a cross entropy loss function, and the calculation formula is as follows: In the robot path planning process, p represents the actual state selection distribution, and q represents the probability distribution of the transformer model output.
6. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 5, characterized in that: The execution strategy network only takes the current state s and the next state s′ as input, selects the optimal action a in the current state, and uses the policy gradient method for training. The specific formula for constructing the execution strategy network is as follows: Among them, π represents the control strategy of the model, that is, when the drone is in a certain state s, the model chooses to take control actions; Q(s, a) is the reward of the state-action pair, which represents the cumulative reward that can be obtained from the current state to the end of the task, that is, the robot takes control action a in the current state s, and the rewards and can be obtained in the future. The model selects the most appropriate control action by maximizing the Q value; represents the expectation, and Z represents the reward distribution of the current state-action pair (s, a).
7. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 6, characterized in that: The specific process of optimizing the execution strategy network is as follows: Let the execution policy be π θ (s,s ′ ): During the training process, the following objectives are obtained: During the evaluation process, given a state s, the final action Determined by both the guiding strategy and the execution strategy: In the robot path planning task, g ω Represents the execution strategy network, and the optimization goal of g is: Where α represents the weight, V φ Represents the state-action-value function.
8. A robot path planning model based on two-stage decoupled offline reinforcement learning according to claim 7, characterized in that: The hyperparameters include the learning rate of the state prediction network and the execution strategy network, the size of the experience replay pool, the number of exploration steps, the number of training steps, and the number of optimization steps.
9. A robot path planning method based on two-stage decoupled offline reinforcement learning, characterized in that: The following steps are involved: Step 1: For the robot to be controlled, the Markov decision process is used to describe the robot path planning process according to the possible states of the robot and the optional control actions. The state s is observed by the camera or other equipment carried by the robot, and the action a represents the action that the robot can perform; Step 2: The current state of the robot to be controlled is preprocessed and input into the state prediction network. The state prediction network obtains the predicted state s′ based on the offline data set. The optimal state of the robot for the next step can be obtained through this network. Step 3: Input the current state s and the next state s' into the execution strategy network, which selects the optimal action a by maximizing the reward * ; Step 4: The robot performs the optimal action a * , after executing the optimal action, the robot enters the new state s; repeat steps 2-4 until the robot path planning process is completed.