Novel automatic driving human-like trajectory planning method
By changing the calculation process of normalized functions from integral to summing and sampling key trajectories, the problem of low training and reasoning efficiency in deep inverse reinforcement learning human trajectory planning technology is solved, and more efficient autonomous driving behavior planning is achieved .
Patent Information
- Application Number
- CN202411223604.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing deep inverse reinforcement learning human trajectory planning technology has great challenges in training and reasoning efficiency, resulting in high computational complexity and difficulty, and it is difficult to effectively train and infer driving behavior.
By changing the calculation process of the normalized function from the integral process to the summing process, and sampling the key trajectory represents all trajectories, the reward network parameters are updated using the polynomial trajectory sampler and gradient rise method to improve training and inference efficiency.
The training and inference efficiency of the autonomous driving human-like trajectory planning model is improved, the computational complexity is reduced, and more efficient driving behavior planning is achieved.
Smart Images

Figure CN119088027A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of autonomous driving, and in particular to a novel humanoid trajectory planning method for autonomous driving. Background Art
[0002] Although autonomous driving technology has made breakthrough progress, there are still obvious differences in driving styles between autonomous vehicles and human drivers at this stage, which has exacerbated the rejection of autonomous vehicles by passengers and other traffic participants. Specifically, this difference in driving style has exacerbated passengers' distrust of autonomous vehicles and made it difficult for other traffic participants to predict the behavior of autonomous vehicles, which is more likely to cause traffic chaos. Therefore, it is necessary to develop humanoid autonomous driving so that autonomous vehicles can perform driving behaviors similar to those of human drivers. The trajectory of an autonomous vehicle is the most intuitive manifestation of its behavior, so humanoid autonomous driving is actually planning a humanoid trajectory.
[0003] The existing human-like trajectory planning technologies for autonomous driving mainly include behavioral cloning and inverse reinforcement learning. Compared with behavioral cloning technology, inverse reinforcement learning technology can infer the root cause of driving behavior, thus avoiding causal confusion and having better generalization, making it the most promising human-like trajectory planning technology. Among inverse reinforcement learning technologies, deep inverse reinforcement learning technology has obvious advantages in feature extraction, scene adaptability, and fitting capabilities, and is a more promising human-like trajectory planning technology.
[0004] Nevertheless, due to the use of artificial intelligence models to represent the driver's reward function, humanoid trajectory planning technology based on deep inverse reinforcement learning still faces major challenges in model training and reasoning efficiency. When training an inverse reinforcement learning model, it is necessary to solve the optimal strategy corresponding to the reward function. Since a deep network is used to represent the reward function, existing deep inverse reinforcement learning technology needs to apply reinforcement learning to solve the corresponding optimal strategy. This means that a complete reinforcement learning model needs to be trained every time the parameters are updated. Since the state and action space is large and continuous, the computational complexity and difficulty of solving the optimal strategy through reinforcement learning are high, making the existing deep inverse reinforcement learning model difficult to train. Similarly, the computational complexity and difficulty of its reasoning stage are also high.
[0005] Therefore, how to improve the training and reasoning efficiency of deep inverse reinforcement learning humanoid trajectory planning technology is an urgent problem to be solved. Summary of the invention
[0006] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.
[0007] In view of the problems existing in the above-mentioned existing novel human-like trajectory planning method for autonomous driving, the present invention breaks away from the conventional usage bias of the prior art and innovatively changes the calculation process of the normalized function from an integration process to a summation process.
[0008] Therefore, the object of the present invention is to provide a novel humanoid trajectory planning method for autonomous driving, which improves the training efficiency and reasoning efficiency of the humanoid trajectory planning model for autonomous driving.
[0009] In order to solve the above technical problems, the present invention provides the following technical solutions: a novel human-like trajectory planning method for autonomous driving, comprising the following steps:
[0010] Step 1: Initialize the parameters of the reward network;
[0011] Step 2: Sample a corresponding key trajectory set for each demonstration trajectory of a human driver, and the trajectory in each key trajectory set and the corresponding demonstration trajectory have the same initial state;
[0012] Step 3: Use the current reward network to calculate the reward of the demonstration trajectory;
[0013] Step 4: Use the current reward network to calculate the reward of the key trajectory;
[0014] Step 5: Combine the reward of the demonstration trajectory and the reward of the key trajectory to calculate the gradient of the updated parameters;
[0015] Step 6: Based on the gradient, update the parameters of the reward network using the gradient ascent algorithm;
[0016] Step 7: Repeat steps 3 to 6 until the reward network converges.
[0017] As a preferred solution of the novel human-like trajectory planning method for autonomous driving described in the present invention, the method for sampling key trajectories in step 2 includes:
[0018] Change the calculation process of the normalized function from an integration process to a summation process:
[0019] The normalization function is expressed as:
[0020]
[0021] Where φ represents the set of all key trajectories sampled, R(τ|θ) represents the reward of trajectory τ; θ represents the parameters of the reward network;
[0022] The training objective function of the human-like trajectory planning model for autonomous driving is reformulated as:
[0023]
[0024] Among them, τ e represents the demonstration trajectory of human drivers; E represents the set of demonstration trajectories of human drivers.
[0025] As a preferred solution of the novel human-like trajectory planning method for autonomous driving described in the present invention, in which: in the step 2, it is assumed that the time length of the planning horizon is L, and a polynomial trajectory sampler considering long-term goals is used to sample candidate trajectories. The trajectory sampler only samples the speed v of the autonomous driving vehicle at time L. e The velocity (L) and acceleration a(L) are sampled, and the state of the autonomous vehicle at the initial moment and the moment L is used as the boundary conditions to calculate the entire trajectory. The boundary conditions of the polynomial of the key trajectory are:
[0026]
[0027] Where v0 represents the speed of the autonomous vehicle at the initial moment; a0 represents the acceleration of the autonomous vehicle at the initial moment; v L represents the z speed at time L; a L represents the acceleration of the autonomous vehicle at time L. According to the boundary conditions, the speed of the key trajectory is fitted with a cubic polynomial as shown below:
[0028] v e (t) = b3t 3 +b2t 2 +b1t+b0
[0029] In the critical trajectory, the acceleration of the autonomous vehicle is the first derivative of the autonomous vehicle velocity, which is fitted by a quadratic polynomial as follows:
[0030] a e (t) = 3b3t 2 +2b2t+b1
[0031] Combining the above formula, the coefficients of the key trajectory polynomial are obtained as follows:
[0032]
[0033] Determine the speed v of the autonomous vehicle at time L e(L) and acceleration a(L), the entire following vehicle trajectory is calculated according to the above formula. Therefore, after determining the speed v of the autonomous driving car at time L, e After determining the sampling interval and sampling interval of the velocity (L) and acceleration a(L), the trajectory sampler can sample all candidate following trajectories.
[0034] As a preferred solution of the novel human-like trajectory planning method for autonomous driving described in the present invention, wherein: in the step three, after the trajectory point is input into the trained reward network, the reward value of the trajectory point is obtained; all trajectory points of the demonstration trajectory are input into the reward network in sequence, and their respective reward values are obtained respectively, and these reward values are added together to obtain the reward of the demonstration trajectory.
[0035] As a preferred solution of the novel human-like trajectory planning method for autonomous driving described in the present invention, wherein: in the step 4, after the trajectory point is input into the trained reward network, the reward value of the trajectory point is obtained; all trajectory points of the key trajectory are input into the reward network in sequence, and their respective reward values are obtained respectively, and these reward values are added together to obtain the reward of the key trajectory.
[0036] As a preferred solution of the novel human-like trajectory planning method for autonomous driving described in the present invention, wherein: in step 5, the objective function is derived to obtain the gradient of the parameter, and the specific calculation process is:
[0037]
[0038] As a preferred solution of the novel autonomous driving humanoid trajectory planning method of the present invention, after the autonomous driving humanoid trajectory planning model is trained, it can be deployed in an autonomous driving humanoid trajectory planning electronic device. During operation, the specific steps of the reasoning process of the autonomous driving humanoid trajectory planning model are as follows:
[0039] S1, sampling key trajectories according to the state of the autonomous vehicle;
[0040] S2, calculate the reward for each key trajectory;
[0041] S3. Select the key trajectory with the largest reward as the planning result output.
[0042] Beneficial effects of the present invention: The present invention breaks away from the conventional use bias of the prior art and innovatively changes the calculation process of the normalized function from an integral process to a summation process. The present invention no longer integrates the infinite trajectories in the solution space, but samples the key trajectories according to the particularity of the autonomous driving trajectory planning problem, and these key trajectories represent all trajectories. In fact, these trajectories are also the trajectories with the highest probability of being adopted. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:
[0044] Figure 1 Schematic diagram of the training process of the novel humanoid trajectory planning method for autonomous driving of the present invention.
[0045] Figure 2 Schematic diagram of the reasoning process of the novel humanoid trajectory planning method for autonomous driving of the present invention. DETAILED DESCRIPTION
[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0048] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0049] Secondly, the present invention is described in detail with reference to the schematic diagram. When describing the embodiments of the present invention in detail, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.
[0050] Reference Figure 1-2 , a novel humanoid trajectory planning method for autonomous driving is provided, comprising the following steps:
[0051] Step 1: Initialize the parameters of the reward network;
[0052] Step 2: Sample a corresponding key trajectory set for each demonstration trajectory of a human driver, and the trajectory in each key trajectory set and the corresponding demonstration trajectory have the same initial state;
[0053] Step 3: Use the current reward network to calculate the reward of the demonstration trajectory;
[0054] Step 4: Use the current reward network to calculate the reward of the key trajectory;
[0055] Step 5: Combine the reward of the demonstration trajectory and the reward of the key trajectory to calculate the gradient of the updated parameters;
[0056] Step 6: Based on the gradient, update the parameters of the reward network using the gradient ascent algorithm;
[0057] Step 7: Repeat steps 3 to 6 until the reward network converges.
[0058] The reasoning process of the new human-like trajectory planning method for autonomous driving is divided into three stages: sampling, evaluation, and selection. Specifically, first, the trajectory sampler samples a set of candidate trajectories based on the driving scenario; then, the reward network of the deep inverse reinforcement learning algorithm evaluates the reward value of each candidate trajectory; finally, the trajectory with the highest reward is selected as the planning result.
[0059] Specifically, the method for sampling key trajectories in step 2 includes:
[0060] Change the calculation process of the normalized function from an integration process to a summation process:
[0061] The normalization function is expressed as:
[0062]
[0063] Where φ represents the set of all key trajectories sampled, R(τ|θ) represents the reward of trajectory τ; θ represents the parameters of the reward network;
[0064] The training objective function of the human-like trajectory planning model for autonomous driving is reformulated as:
[0065]
[0066] Among them, τ e represents the demonstration trajectory of human drivers; E represents the set of demonstration trajectories of human drivers.
[0067] Furthermore, in step 2, it is assumed that the time length of the planning horizon is L, and a polynomial trajectory sampler considering long-term goals is used to sample candidate trajectories. The trajectory sampler only samples the speed v of the autonomous driving vehicle at time L. e The velocity (L) and acceleration a(L) are sampled, and the state of the autonomous vehicle at the initial moment and the moment L is used as the boundary conditions to calculate the entire trajectory. The boundary conditions of the polynomial of the key trajectory are:
[0068]
[0069] Where v0 represents the speed of the autonomous vehicle at the initial moment; a0 represents the acceleration of the autonomous vehicle at the initial moment; v L represents the z speed at time L; a L represents the acceleration of the autonomous vehicle at time L. According to the boundary conditions, the speed of the key trajectory is fitted with a cubic polynomial as shown below:
[0070] v e (t) = b3t 3 +b2t 2 +b1t+b0
[0071] In the critical trajectory, the acceleration of the autonomous vehicle is the first derivative of the autonomous vehicle velocity, which is fitted by a quadratic polynomial as follows:
[0072] a e (t) = 3b3t 2 +2b2t+b1
[0073] Combining the above formula, the coefficients of the key trajectory polynomial are obtained as follows:
[0074]
[0075] Determine the speed v of the autonomous vehicle at time L e (L) and acceleration a(L), the entire following vehicle trajectory is calculated according to the above formula. Therefore, after determining the speed v of the autonomous driving car at time L, e After determining the sampling interval and sampling interval of the velocity (L) and acceleration a(L), the trajectory sampler can sample all candidate following trajectories.
[0076] Among them, in the step three, after the trajectory point is input into the trained reward network, the reward value of the trajectory point is obtained; all the trajectory points of the demonstration trajectory are input into the reward network in sequence, and their respective reward values are obtained, and these reward values are added to obtain the reward of the demonstration trajectory; specifically, in the step four, after the trajectory point is input into the trained reward network, the reward value of the trajectory point is obtained; all the trajectory points of the key trajectory are input into the reward network in sequence, and their respective reward values are obtained, and these reward values are added to obtain the reward of the key trajectory.
[0077] Among them, in the step 5, the objective function is derived to obtain the gradient of the parameter, and the specific calculation process is:
[0078]
[0079] Among them, in step six, the existing Adam optimizer is used to update the parameters according to the gradient ascent method.
[0080] Furthermore, after the autonomous driving humanoid trajectory planning model is trained, it can be deployed in an autonomous driving humanoid trajectory planning electronic device. During operation, the specific steps of the reasoning process of the autonomous driving humanoid trajectory planning model are as follows:
[0081] S1, sampling key trajectories according to the state of the autonomous vehicle;
[0082] S2, calculate the reward for each key trajectory;
[0083] S3. Select the key trajectory with the largest reward as the planning result output.
[0084] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A novel humanoid trajectory planning method for autonomous driving, characterized in that: The following steps are involved: Step 1: Initialize the parameters of the reward network; Step 2: Sample a corresponding key trajectory set for each demonstration trajectory of a human driver, and the trajectory in each key trajectory set and the corresponding demonstration trajectory have the same initial state; Step 3: Use the current reward network to calculate the reward of the demonstration trajectory; Step 4: Use the current reward network to calculate the reward of the key trajectory; Step 5: Combine the reward of the demonstration trajectory and the reward of the key trajectory to calculate the gradient of the updated parameters; Step 6: Based on the gradient, update the parameters of the reward network using the gradient ascent algorithm; Step 7: Repeat steps 3 to 6 until the reward network converges.
2. The novel human-like trajectory planning method for autonomous driving according to claim 1 is characterized in that: The method for sampling key trajectories in step 2 includes: Change the calculation process of the normalized function from an integration process to a summation process: The normalization function is expressed as: Where φ represents the set of all key trajectories sampled, R(τ|θ) represents the reward of trajectory τ; θ represents the parameters of the reward network; The training objective function of the human-like trajectory planning model for autonomous driving is reformulated as: Among them, τ e represents the demonstration trajectory of human drivers; E represents the set of demonstration trajectories of human drivers.
3. The novel human-like trajectory planning method for autonomous driving according to claim 2 is characterized in that: In step 2, it is assumed that the time length of the planning horizon is L, and a polynomial trajectory sampler considering long-term goals is used to sample candidate trajectories. The trajectory sampler only samples the speed v of the autonomous driving vehicle at time L. e The velocity (L) and acceleration a(L) are sampled, and the state of the autonomous vehicle at the initial moment and the moment L is used as the boundary conditions to calculate the entire trajectory. The boundary conditions of the polynomial of the key trajectory are: Where v0 represents the speed of the autonomous vehicle at the initial moment; a0 represents the acceleration of the autonomous vehicle at the initial moment; v L represents the z speed at time L; a L represents the acceleration of the autonomous vehicle at time L. According to the boundary conditions, the speed of the key trajectory is fitted with a cubic polynomial as shown below: <h2 style=";text-align:left;direction:ltr">v<h2 style=";text-align:left;direction:ltr"> e <h2 style=";text-align:left;direction:ltr"> (t)=b3t<h2 style=";text-align:left;direction:ltr"> 3 <h2 style=";text-align:left;direction:ltr"> +b2t<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +b1t+b0 In the critical trajectory, the acceleration of the autonomous vehicle is the first derivative of the autonomous vehicle velocity, which is fitted by a quadratic polynomial as follows: a e (t)=3b3t 2 +2b2t+b1 Combining the above formula, the coefficients of the key trajectory polynomial are obtained as follows: Determine the speed v of the autonomous vehicle at time L e (L) and acceleration a(L), the entire following vehicle trajectory is calculated according to the above formula. Therefore, after determining the speed v of the autonomous driving car at time L, e After determining the sampling interval and sampling interval of the velocity (L) and acceleration a(L), the trajectory sampler can sample all candidate following trajectories.
4. The novel human-like trajectory planning method for autonomous driving according to claim 3 is characterized in that: After the trajectory point is input into the trained reward network in step three, the reward value of the trajectory point is obtained; all trajectory points of the demonstration trajectory are input into the reward network in sequence, their respective reward values are obtained, and these reward values are added together to obtain the reward of the demonstration trajectory.
5. The novel human-like trajectory planning method for autonomous driving according to claim 4 is characterized in that: In the step 4, after the trajectory point is input into the trained reward network, the reward value of the trajectory point is obtained; all the trajectory points of the key trajectory are input into the reward network in sequence, their respective reward values are obtained, and these reward values are added together to obtain the reward of the key trajectory.
6. The novel human-like trajectory planning method for autonomous driving according to claim 2 is characterized in that: In step 5, the objective function is derived to obtain the gradient of the parameter, and the specific calculation process is as follows:
7. The novel human-like trajectory planning method for autonomous driving according to claim 3 is characterized by: After the autonomous driving humanoid trajectory planning model is trained, it can be deployed in the autonomous driving humanoid trajectory planning electronic device. During operation, the specific steps of the reasoning process of the autonomous driving humanoid trajectory planning model are as follows: S1, sampling key trajectories according to the state of the autonomous vehicle; S2, calculate the reward for each key trajectory; S3. Select the key trajectory with the largest reward as the planning result output.
