An end-to-end driving method based on hierarchical reinforcement learning
Through layered reinforcement learning and SAC algorithms, we model path follow-up and dynamic obstacle avoidance models, and directly output action decisions, solving the problems of low training efficiency and poor generalization in autonomous driving, and improving the accuracy and complexity of end-to-end driving Scene adaptability.
Patent Information
- Application Number
- CN202210179370.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-25
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-02-25
AI Technical Summary
The existing reinforcement learning solutions are inefficient in training in autonomous driving and are difficult to apply to complex and practical application scenarios, resulting in low accuracy and poor generalization of end-to-end driving decisions.
Using an end-to-end driving method based on hierarchical reinforcement learning, by obtaining pre-given vehicle driving paths, modeling the path following model and dynamic obstacle avoidance model, and using SAC algorithm to deploy the reinforcement learning network to find solutions, directly output the path following and Dynamic obstacle avoidance decisions, simplify state space and improve the microscopicity of decisions.
Improve the accuracy and generalization of end-to-end driving, and can effectively deal with unknown extremes, especially in dynamic obstacle avoidance scenarios, simplify the complexity of each subtask and enhance the stability and ability of dynamic obstacle avoidance decisions. Explanatory.
Smart Images

Figure CN114527759B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end driving method based on hierarchical reinforcement learning. Background Art
[0002] Autonomous driving is one of the core technologies of intelligent transportation. The core of autonomous driving includes perception prediction, decision planning and motion control. Currently, there are two solutions for autonomous driving: rule-based and end-to-end. The rule-based approach has the disadvantages of a large number of uncertainty boundary problems, complex rule construction for complex scenarios, cumulative deviation problems of multiple models, and the need to rely on high-precision maps.
[0003] End-to-end driving solutions are a new branch of autonomous driving, which is expected to simplify driving modeling by learning underlying mechanisms from data. End-to-end driving mainly consists of deep learning and reinforcement learning. Among them, reinforcement learning, as one of the practical technical solutions based on end-to-end, is suitable for solving time-series decision-making problems, which is consistent with the decision-making process of autonomous driving. Reinforcement learning has the ability to evolve in response to changes in the external environment. When unknown extreme situations occur, it can adapt and explore ways to solve the problem by changing its own driving strategy.
[0004] The applicant found that the rewards for actual autonomous driving problems using reinforcement learning are very sparse, and coupled with the huge combination of state space and action space, the training efficiency of reinforcement learning is low or even ineffective. In other words, some existing reinforcement learning schemes can only be used in set environmental variables or in conditions similar to the training environment, resulting in low accuracy and poor generalization of end-to-end driving decisions. Therefore, how to design a method that can be applied to complex application scenarios and actual application scenarios is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide an end-to-end driving method based on hierarchical reinforcement learning so as to be applicable to complex application scenarios and practical application scenarios, thereby improving the accuracy and generalization of end-to-end driving decisions.
[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0007] An end-to-end driving method based on hierarchical reinforcement learning comprises the following steps:
[0008] S1: Obtain a pre-given vehicle driving path;
[0009] S2: Based on reinforcement learning, the path following model and the dynamic obstacle avoidance model are modeled separately;
[0010] S3: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the path following model and obtain the corresponding path following decision;
[0011] S4: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the dynamic obstacle avoidance model in combination with the corresponding path following decision to obtain the corresponding dynamic obstacle avoidance decision;
[0012] S5: Realize end-to-end driving of the vehicle based on the vehicle's driving path, path following decision and dynamic obstacle avoidance decision.
[0013] Preferably, in step S1, a vehicle driving path from a starting point to a target point is generated by a rapid expansion random tree algorithm.
[0014] Preferably, in step S2, a path following model is modeled by the following steps:
[0015] S201: Initialize the state space and action space of the path following model and set the vehicle starting point posture;
[0016] S202: Taking the starting point of the vehicle as the origin of the coordinate system, moving along the vehicle's driving path, and forming a state space of the path following model through the state of the vehicle's surrounding environment and the vehicle's own state;
[0017] S203: forming an action space of a path following model through the speed and steering angle of the vehicle at the current moment;
[0018] S204: A reward function of the path following model is formed by speed rewards and penalties, rewards and penalties for approaching the target point, rewards and penalties for the driving direction, and rewards and penalties for reaching the target point and the end point.
[0019] Preferably, in step S2, the state space of the path following model is represented as S g =[V t ,S t ,y t ,d t ];
[0020] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment;
[0021] The action space of the path following model is represented as A g =[Vt ,S t ];
[0022] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the vehicle steering angle at the current moment;
[0023] The reward function of the path following model is denoted as R g =R v +R y +R c +R w ;
[0024] Among them, R v =(v s -v m / 2) / v m ;
[0025]
[0026] R c =(past_dist-dist);
[0027]
[0028] Where: R v Indicates speed reward and punishment; R c Indicates rewards and penalties for approaching the target point; R y Indicates driving direction reward and punishment; R w Indicates the rewards and penalties for reaching the target point and the end point; v s Indicates the vehicle speed at the current moment; v m It represents the speed set according to the task speed v0 in the set scenario; yaw represents the difference between the relative angle between the current vehicle and the next waypoint and the relative angle between the two waypoints in the local path; abs(yaw) represents the absolute value of the corresponding angle; dist represents the relative distance between the vehicle position and the waypoint at the current moment; past_dist represents the relative distance between the vehicle position and the waypoint at the previous moment; time_step represents the remaining time of the current training stage; waypoint and goal represent the arrival radius set for the waypoint and the end point respectively.
[0029] Preferably, in step S2, a dynamic obstacle avoidance model is modeled by the following steps:
[0030] S211: Initialize the state space and action space of the dynamic obstacle avoidance model;
[0031] S212: The distance information of the obstacle obtained by the laser radar, the state of the path following model and the output path following decision constitute the state space of the dynamic obstacle avoidance model;
[0032] S213: forming an action space of a dynamic obstacle avoidance model through the offset steering angle of the vehicle at the current moment;
[0033] S214: The reward function of the dynamic obstacle avoidance model is composed of the steering deviation reward and penalty and the collision reward and penalty.
[0034] Preferably, the state space of the dynamic obstacle avoidance model is represented as S b =[V t ,S t ,y t ,d t ,V out ,S out ,l1,l2,…,l 120 ];
[0035] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment; V out Indicates the speed of the path following the model output at the current moment; S out Indicates the speed steering angle output by the path following model at the current moment; l1,l2,...,l 120 Indicates the discretized information of the distance corresponding to the scanning angle of the laser radar;
[0036] The action space is represented as A b =Sct;
[0037] Where: Sct represents the current vehicle's offset steering angle, Sct = [-1, 1];
[0038] The reward function is denoted as R b =R z +R c ;
[0039] in,
[0040] Where: R s Indicates the reward and penalty for steering deviation; R c represents collision reward or penalty; abs(Sct) represents the absolute value of the vehicle's offset steering angle; b represents the coefficient for measuring the offset steering angle; C represents the coefficient for measuring the importance of the collision part.
[0041] Preferably, in steps S3 and S4, the SAC algorithm is used as the deployed reinforcement learning network to solve the path following model and the dynamic obstacle avoidance model respectively.
[0042] Preferably, the SAC algorithm is implemented based on an Actor-Critic network structure, including an Actor network and two VCritic networks and a Q Critic network; wherein the Actor network takes the corresponding state as input and the action probability density function as output; the V Critic network and the Q Critic network both take the corresponding state as input and the corresponding V value and Q value as output;
[0043] When solving the path following model and the dynamic obstacle avoidance model, the state space is used as the input of the SAC algorithm, and the corresponding action space is calculated by the SAC algorithm as the output.
[0044] Preferably, the policy function of the SAC algorithm is expressed as π(s,a)=P(A=a|S=s);
[0045] In the formula: s represents the state; a represents the action;
[0046] The objective function of the SAC algorithm is expressed as:
[0047]
[0048] Preferably, the update process of the SAC algorithm includes:
[0049] 1) Q Critic Network Update:
[0050] Use MSEloss as the loss function:
[0051]
[0052] Where: len(batch) represents the batch size; Rt+γV(s′) represents the true value estimate at the current moment; q(st,a) represents the predicted value estimate at the current moment;
[0053] 2) V Critic Network Update:
[0054] MSEloss is used as the loss function, and entropy is added to estimate the true value of V value δt:
[0055] δt=∑π(a′|s t ;θ)[min q(s t ,a′)-αlnπ(a′|s t ;θ)];
[0056]
[0057] Where: π(a′│st; θ) represents the action probability density function output by the Actor network; min q(st,a') represents the smaller Q value of the two Q Critic networks;
[0058] 3) Actor network update:
[0059] Calculate the loss function:
[0060]
[0061] Where: α represents the reward coefficient of entropy. The larger α is, the greater the importance of entropy is.
[0062] Compared with the prior art, the end-to-end driving method of the present invention has the following beneficial effects:
[0063] The present invention models a path following model and a dynamic obstacle avoidance model through reinforcement learning, and directly outputs the path following decision and the dynamic obstacle avoidance decision through the two models, that is, directly outputs the specific action behavior without performing path planning; and the output path following decision and dynamic obstacle avoidance decision are microscopic decisions of the vehicle, which are a direct mapping from sensor input to action output, so that unknown extreme situations can be effectively handled, and thus can be better applicable to complex application scenarios and actual application scenarios (especially dynamic obstacle avoidance scenarios), thereby improving the accuracy and generalization of end-to-end driving.
[0064] The present invention decomposes end-to-end driving into two layers of subtasks, namely, path following tasks and dynamic obstacle avoidance tasks, by modeling a path following model and a dynamic obstacle avoidance model. This simplifies the state space of end-to-end driving and enables each subtask to be solved in a smaller subproblem space, thereby decomposing and reducing the complexity of each individual task. At the same time, the present invention applies path following decisions to solving dynamic obstacle avoidance decisions, making the generated dynamic obstacle avoidance decisions more stable and interpretable, thereby further improving the accuracy of end-to-end driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0066] Figure 1 The logical block diagram of the end-to-end driving method based on hierarchical reinforcement learning. DETAILED DESCRIPTION
[0067] The following is a further detailed description through specific implementation methods:
[0068] Example:
[0069] This embodiment discloses an end-to-end driving method based on hierarchical reinforcement learning.
[0070] like Figure 1 As shown, the end-to-end driving method based on hierarchical reinforcement learning includes the following steps:
[0071] S1: Obtain a predetermined vehicle driving path; in this embodiment, a vehicle driving path from a starting point to a target point is generated by a rapidly expanding random tree algorithm (RRT).
[0072] S2: Based on reinforcement learning, the path following model and the dynamic obstacle avoidance model are modeled separately;
[0073] S3: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the path following model and obtain the corresponding path following decision;
[0074] S4: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the dynamic obstacle avoidance model in combination with the corresponding path following decision to obtain the corresponding dynamic obstacle avoidance decision;
[0075] S5: Realize end-to-end driving of the vehicle based on the vehicle's driving path, path following decision and dynamic obstacle avoidance decision.
[0076] The present invention models a path following model and a dynamic obstacle avoidance model through reinforcement learning, and directly outputs the path following decision and the dynamic obstacle avoidance decision through the two models, that is, directly outputs the specific action behavior without performing path planning on it; and, the output path following decision and the dynamic obstacle avoidance decision are microscopic decisions of the vehicle, which are a direct mapping from sensor input to action output, so that unknown extreme situations can be effectively handled, and thus can be better applied to complex application scenarios and practical application scenarios (especially dynamic obstacle avoidance scenarios), thereby improving the accuracy and generalization of end-to-end driving. Secondly, the present invention decomposes the end-to-end driving into two layers of subtasks, namely, the path following task and the dynamic obstacle avoidance task, by modeling the path following model and the dynamic obstacle avoidance model, which simplifies the state space of the end-to-end driving, so that each subtask can be solved in a smaller sub-problem space, thereby being able to decompose and reduce the complexity of each individual task; at the same time, the present invention applies the path following decision to solve the dynamic obstacle avoidance decision, so that the stability and interpretability of the generated dynamic obstacle avoidance decision are stronger, thereby further improving the accuracy of the end-to-end driving.
[0077] This embodiment uses Microsoft's Airsim as an environmental simulator. Airsim has the function of supporting user-designed scene creation, including scene changes (city, mountain, indoor, etc.), weather effects, time of day, traffic compliance (urban environment can track traffic violations), additional sensor accessories, etc. The scene selected in this embodiment is the LandscapeMountains map in Airsim, and the basic elements of reinforcement learning are used to model and encapsulate the environment, and then the Airsim related APIs are combined and encapsulated into a gym environment.
[0078] During the specific implementation process, the path following model is modeled through the following steps:
[0079] S201: Initialize the state space and action space of the path following model and set the vehicle starting point posture; it is the entrance of an eposide in the training process.
[0080] S202: Taking the starting point of the vehicle as the origin of the coordinate system, moving along the vehicle's driving path, and forming a state space of the path following model through the state of the vehicle's surrounding environment and the vehicle's own state;
[0081] S203: forming an action space of a path following model through the speed and steering angle of the vehicle at the current moment;
[0082] S204: A reward function of the path following model is formed by speed rewards and penalties, rewards and penalties for approaching the target point, rewards and penalties for the driving direction, and rewards and penalties for reaching the target point and the end point.
[0083] The state space of the path following model is represented as S g =[V t ,S t ,y t ,d t ];
[0084] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment;
[0085] In this embodiment, the vehicle speed and the vehicle steering angle are respectively acquired by corresponding sensors; the vertical distance d is the geometric distance between the vehicle and the RRT planning path; the angle α is calculated by the following formula:
[0086]
[0087] That is, calculate the difference between the angle between the current driving direction and the next target path point and the angle between the two path points generated by RRT planning.
[0088] The action space of the path following model is represented as A g =[V t ,S t ]
[0089] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; in this embodiment, V t and S t The value of V t =[0,V0],S t =[-1,1], where V0 represents the task speed in the set scenario, and V0=6m / s is selected.
[0090] The reward function of the path following model is denoted as R g =R v +R y +R c +R w ; The reward function is used to measure the pros and cons of taking action a in the current state s.
[0091] Among them, R v =(v s -v m / 2) / v m ;
[0092]
[0093] R c += move_dist / 2;
[0094] R c =(past_dist-dist);
[0095]
[0096] Where: R v Indicates speed reward and punishment; R c Indicates rewards and penalties for approaching the target point; R y Indicates driving direction reward and punishment; R w Indicates the rewards and penalties for reaching the target point and the end point; v s Indicates the vehicle speed at the current moment; v mIt represents the speed set according to the task speed v0 in the set scenario; yaw represents the difference between the relative angle between the current vehicle and the next waypoint and the relative angle between the two waypoints in the local path; abs(yaw) represents the absolute value of the corresponding angle; dist represents the relative distance between the vehicle position and the waypoint at the current moment; past_dist represents the relative distance between the vehicle position and the waypoint at the previous moment; time_step represents the remaining time of the current training stage; waypoint and goal represent the arrival radius set for the waypoint and the end point respectively.
[0097] In this embodiment, V0 is set to 6m / s, V m Set to 3m / s.
[0098] If dist<=waypoint, that is, the Euclidean distance from the current vehicle to the next waypoint is within the radius of the next waypoint, it is considered to have reached the waypoint and a reward is given; if dist<=goal, that is, the current vehicle is within the radius of the destination, the reward is 10*time_step, which is used to encourage the vehicle to reach the destination faster and better.
[0099] The present invention models a path following model through the above steps, so that end-to-end driving can be effectively decomposed into two layers of subtasks, namely, a path following task and a dynamic obstacle avoidance task. This simplifies the state space of end-to-end driving and enables each subtask to be solved in a smaller subproblem space, thereby decomposing and reducing the complexity of each individual task.
[0100] During the specific implementation process, the dynamic obstacle avoidance model is built through the following steps:
[0101] S211: Initialize the state space and action space of the dynamic obstacle avoidance model;
[0102] S212: The distance information of the obstacle obtained by the laser radar, the state of the path following model and the output path following decision constitute the state space of the dynamic obstacle avoidance model;
[0103] S213: forming an action space of a dynamic obstacle avoidance model through the offset steering angle of the vehicle at the current moment;
[0104] S214: The reward function of the dynamic obstacle avoidance model is composed of the steering deviation reward and penalty and the collision reward and penalty.
[0105] Specifically, the state space of the dynamic obstacle avoidance model is represented as S b =[V t ,S t ,y t ,d t ,V out ,S out,l1,l2,…,l 120 ];
[0106] Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment; V out Indicates the speed of the path following the model output at the current moment; S out Indicates the speed steering angle output by the path following model at the current moment; l1,l2,…,l 120 Indicates the discretized information of the distance corresponding to the scanning angle of the laser radar;
[0107] Among them, the point cloud information obtained by the laser radar is converted to obtain the distance information of the laser radar from the obstacle within the scanning radius; the obstacle state obtained by the laser radar is represented by 360 dimensions, and each dimension is represented by the obstacle within the 1-degree range of the laser scan. If there is data in the point cloud information, it is taken out and the Euclidean distance is calculated with the current laser radar position, and the position of the current point relative to the laser radar is calculated (expressed in angles), and the calculated angle is used as the subscript, and the calculated distance is used as the value corresponding to the subscript. The distance corresponding to other uncalculated angles is taken as the maximum perception range of Lidar 20m.
[0108] After that, discretization operation is performed to improve exploration efficiency. 360° is divided into 24 parts of 20°, and each part is divided into different areas according to [1, 3, 7, 12, 20], totaling 120 parts; then the obtained lidar distance information is judged to determine whether the division is in the corresponding area. If the corresponding area has the corresponding obstacle distance value, the value of the area is set to 1 (0 means no obstacle, 1 means there is an obstacle).
[0109] The action space is represented as A b =Sct;
[0110] Wherein: Sct represents the offset steering angle of the vehicle at the current moment, Sct = [-1, 1]; in this embodiment, the offset steering angle represents the corrected steering angle required to cope with obstacles based on the steering angle output by the previous path following task.
[0111] The reward function is denoted as R b =R z +R c ;
[0112] in, R c =C;
[0113] Where: R s Indicates the reward and penalty for steering deviation; R c Represents collision reward or penalty; abs(Sct) represents the absolute value of the vehicle's offset steering angle; b represents the coefficient for measuring the offset steering angle. b is set to 0.2, which means that no large penalty will be given if the steering angle does not exceed 0.2 steering angle to the left or right of the original steering angle; C represents the coefficient for measuring the importance of the collision part. C is set to -100, which is two orders of magnitude larger than the first part of the path following task and one order of magnitude larger than the steering offset task part. The purpose is to make the vehicle put obstacle avoidance as the primary task.
[0114] The present invention models a dynamic obstacle avoidance model through the above steps, so that end-to-end driving can be effectively decomposed into two layers of subtasks, namely, a path following task and a dynamic obstacle avoidance task, which simplifies the state space of end-to-end driving and enables each subtask to be solved in a smaller subproblem space, thereby decomposing and reducing the complexity of each individual task; at the same time, the present invention applies the path following decision to solve the dynamic obstacle avoidance decision, so that the stability and interpretability of the generated dynamic obstacle avoidance decision are stronger, thereby further improving the accuracy of end-to-end driving.
[0115] During the specific implementation process, the SAC algorithm is used as the deployed reinforcement learning network to solve the path following model and the dynamic obstacle avoidance model respectively.
[0116] Specific:
[0117] Set up the interaction module (): The logic used for the interaction between the vehicle and the environment. It accepts the input of the action, gives the state of the next moment, the reward of the current action, whether to end the current episode, and debugging information, etc.
[0118] Training SAC algorithm deployment: Build a deep reinforcement learning structure based on the SAC algorithm framework.
[0119] Based on the Stable Baselines3 library, combined with a custom gym environment, training can be quickly deployed. The present invention selects the SAC algorithm as the training algorithm for learning. The SAC algorithm is a non-strategy maximum entropy deep reinforcement learning algorithm that uses random actors to introduce robustness and avoids falling into suboptimal behavior by repeatedly selecting the same action.
[0120] The SAC algorithm adds the use of maximum entropy, which enables the network to output multiple actions with similar Q values instead of a single action, encouraging exploration and avoiding falling into local optimality. In order to improve the utilization of data, the SAC algorithm uses experience replay, which puts a trajectory (st, at, rt, st+1) generated by a state through the output of an actor into the experience buffer. In addition to using previous experience to avoid waste, experience replay is also used to break up the sequence and eliminate correlation.
[0121] The SAC algorithm is implemented based on the Actor-Critic network structure, which includes an Actor network and two V Critic networks and a Q Critic network. The Actor network takes the corresponding state as input and the action probability density function as output. The V Critic network and the Q Critic network both take the corresponding state as input and the corresponding V value (state value estimate) and Q value (state value pair estimate) as output.
[0122] When solving the path following model and the dynamic obstacle avoidance model, the state space is used as the input of the SAC algorithm, and the corresponding action space is calculated by the SAC algorithm as the output.
[0123] Through the SAC algorithm framework, a SAC algorithm based on a custom network structure is constructed. The size of the policy neural network is composed of 256 neurons in 2 fully connected layers.
[0124] The policy function of the SAC algorithm is expressed as π(s,a)=P(A=a|S=s);
[0125] In the formula: s represents the state; a represents the action;
[0126] The objective function of the SAC algorithm is expressed as:
[0127]
[0128] During the specific implementation process, the strategy function and the objective function may also adopt the strategy function and the objective function of the existing SAC algorithm.
[0129] During the specific implementation, the update process of the SAC algorithm includes:
[0130] 1) Q Critic Network Update:
[0131] Use MSEloss as the loss function:
[0132]
[0133] Where: len(batch) represents the batch size; Rt+γV(s′) represents the true value estimate at the current moment; q(st,a) represents the predicted value estimate at the current moment;
[0134] 2) V Critic Network Update:
[0135] MSEloss is used as the loss function, and entropy is added to estimate the true value of V value δt:
[0136] δt=∑π(a′|s t ;θ)[min q(s t ,a′)-αln π(a′|s t ;θ)];
[0137]
[0138] Where: π(a′│st; θ) represents the action probability density function output by the Actor network; min q(st,a') represents the smaller Q value of the two Q Critic networks;
[0139] 3) Actor network update:
[0140] Calculate the loss function:
[0141]
[0142] Where: α represents the reward coefficient of entropy. The larger α is, the greater the importance of entropy is.
[0143] Specifically, the training process for the path following model is as follows:
[0144] Set the training to 500 eposides, each eposide is set to 10,000 time steps, and the maximum training time step of each training process is set to 500 time steps.
[0145] During the network training process, as the state s is input, the Actor network in the SAC algorithm is responsible for outputting the probability density of the action. The Q critic and V critic evaluate and value the action in combination with the maximum entropy. After taking action a, the reward is obtained and the next state s' is entered. This cycle repeats until a sampling data of a driving trajectory is obtained.
[0146] At the same time, the network is trained iteratively, where the Target V Critic network has fixed parameters during the training process, and the other four networks are dynamically updated. After a period of time, the Target V Critic will copy its network parameters to another V Critic network.
[0147] After a large number of training iterations, the network converges and the network parameters no longer change. The trained network can calculate the corresponding action behavior based on the current vehicle state and environmental state, that is, a path following model that is adapted to the given vehicle driving path is obtained.
[0148] The training process for the dynamic obstacle avoidance model is as follows:
[0149] Set 1000 eposides to be trained, each eposide is set to 10000 time steps, and set the maximum training time step of each training process to 500 time steps. The training process is consistent with the path following task. The state space is used as the input part of the network, and the action output is calculated by the network, and the output is the offset steering angle.
[0150] As the network training iterates, the vehicle continuously interacts with the environment, collecting multiple collision experiences while following the path. After the network converges, the network parameters no longer change. Through the evaluation of the Ctitic network, the vehicle will gradually perform reasonable obstacle avoidance for dynamic obstacles based on path following and make correct decisions.
[0151] The present invention uses the SAC algorithm as a deployed reinforcement learning network to solve the path following model and the dynamic obstacle avoidance model. Compared with the existing deterministic strategy, the SAC algorithm adopted by the present invention can avoid falling into suboptimal state due to repeatedly selecting the same action. It has stronger exploration ability and is more robust. It can make adjustments more easily when facing interference, thereby improving the accuracy and generalization of end-to-end driving decisions.
[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.
Claims
1. An end-to-end driving method based on hierarchical reinforcement learning, characterized in that: The following steps are involved: S1: Obtain a pre-given vehicle driving path; S2: Based on reinforcement learning, the path following model and the dynamic obstacle avoidance model are modeled separately; In step S2, a path following model is built by the following steps: S201: Initialize the state space and action space of the path following model and set the vehicle starting point posture; S202: Taking the starting point of the vehicle as the origin of the coordinate system, moving along the vehicle's driving path, and forming a state space of the path following model through the state of the vehicle's surrounding environment and the vehicle's own state; S203: forming an action space of a path following model through the speed and steering angle of the vehicle at the current moment; S204: A reward function of the path following model is formed by speed rewards and penalties, rewards and penalties for approaching the target point, rewards and penalties for the driving direction, and rewards and penalties for reaching the target point and the end point; The state space of the path following model is represented as S g =[V t ,S t ,y t ,d t ]; Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment; The action space of the path following model is represented as A g =[V t ,S t ]; Where: V t Indicates the vehicle speed at the current moment; S t Indicates the vehicle steering angle at the current moment; The reward function of the path following model is denoted as R g =R v +R y +R c +R w ; Among them, R v =(v s -v m / 2) / v m ; R c =(past_dist-dist); Where: R v Indicates speed reward and punishment; R c Indicates rewards and penalties for approaching the target point; R y Indicates driving direction reward and punishment; R w Indicates the rewards and penalties for reaching the target point and the end point; v s Indicates the vehicle speed at the current moment; v m It represents the speed set according to the task speed v0 in the set scenario; yaw represents the difference between the relative angle between the current vehicle and the next path point and the relative angle between the two path points in the local path; abs(yaw) represents the absolute value of the corresponding angle; dist represents the relative distance between the vehicle position and the path point at the current moment; past_dist represents the relative distance between the vehicle position and the path point at the previous moment; time_step represents the remaining time of the current training phase; waypoint and goal represent the arrival radius set for the path point and the destination respectively; S3: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the path following model and obtain the corresponding path following decision; S4: Based on the vehicle's driving path, a reinforcement learning network is deployed to solve the dynamic obstacle avoidance model in combination with the corresponding path following decision to obtain the corresponding dynamic obstacle avoidance decision; S5: Realize end-to-end driving of the vehicle based on the vehicle's driving path, path following decision and dynamic obstacle avoidance decision.
2. The end-to-end driving method based on hierarchical reinforcement learning according to claim 1, characterized in that: In step S1, a vehicle driving path from a starting point to a target point is generated by a fast expanding random tree algorithm.
3. The end-to-end driving method based on hierarchical reinforcement learning according to claim 1, characterized in that: In step S2, a dynamic obstacle avoidance model is built through the following steps: S211: Initialize the state space and action space of the dynamic obstacle avoidance model; S212: The distance information of the obstacle obtained by the laser radar, the state of the path following model and the output path following decision constitute the state space of the dynamic obstacle avoidance model; S213: forming an action space of a dynamic obstacle avoidance model through the offset steering angle of the vehicle at the current moment; S214: The reward function of the dynamic obstacle avoidance model is composed of the steering deviation reward and penalty and the collision reward and penalty.
4. The end-to-end driving method based on hierarchical reinforcement learning as claimed in claim 3, characterized in that: The state space of the dynamic obstacle avoidance model is represented as S b =[V t ,S t ,y t ,d t ,V out ,S out ,l1,l2,...,l 120 ]; Where: V t Indicates the vehicle speed at the current moment; S t Indicates the current vehicle steering angle; y t Indicates the angle between the current driving direction and the target point direction; d t Indicates the vertical distance between the vehicle and the vehicle's driving path at the current moment; V out Indicates the speed of the path following the model output at the current moment; S out Indicates the speed steering angle output by the path following model at the current moment; l1,l2,...,l 120 Indicates the discretized information of the distance corresponding to the scanning angle of the laser radar; The action space is represented as A b =Sct; Where: Sct represents the current vehicle's offset steering angle, Sct = [-1, 1]; The reward function is denoted as R b =R z +R c ; in, R c =C; Where: R s Indicates the reward and penalty for steering deviation; R c represents collision reward or penalty; abs(Sct) represents the absolute value of the vehicle's offset steering angle; b represents the coefficient for measuring the offset steering angle; C represents the coefficient for measuring the importance of the collision part.
5. The end-to-end driving method based on hierarchical reinforcement learning as claimed in claim 3, characterized in that: In steps S3 and S4, the SAC algorithm is used as the deployed reinforcement learning network to solve the path following model and the dynamic obstacle avoidance model respectively.
6. The end-to-end driving method based on hierarchical reinforcement learning according to claim 5, characterized in that: The SAC algorithm is implemented based on the Actor-Critic network structure, which includes an Actor network and two V Critic networks and a Q Critic network. The Actor network takes the corresponding state as input and the action probability density function as output. The V Critic network and the Q Critic network both take the corresponding state as input and the corresponding V value and Q value as output. When solving the path following model and the dynamic obstacle avoidance model, the state space is used as the input of the SAC algorithm, and the corresponding action space is calculated by the SAC algorithm as the output.
7. The end-to-end driving method based on hierarchical reinforcement learning according to claim 6, characterized in that: The policy function of the SAC algorithm is expressed as π(s,a)=P(A=a|S=s); In the formula: s represents the state; a represents the action; The objective function of the SAC algorithm is expressed as:
8. The end-to-end driving method based on hierarchical reinforcement learning according to claim 7, characterized in that: The update process of the SAC algorithm includes: 1) Q Critic Network Update: Use MSEloss as the loss function: Where: len(batch) represents the batch size; Rt+γV(s') represents the true value estimate at the current moment; q(st,a) represents the predicted value estimate at the current moment; 2) V Critic Network Update: MSEloss is used as the loss function, and entropy is added to estimate the true value of V value δt: δt=∑π(a′|s t ;θ)[minq(s t ,a′)-αInπ(a′|s t ;i)]; Where: π(a'│st; θ) represents the action probability density function output by the Actor network; min q(st,a') represents the smaller Q value of the two QCritic networks; 3) Actor network update: Calculate the loss function: Where: α represents the reward coefficient of entropy. The larger α is, the greater the importance of entropy is.
