A structured road vehicle trajectory planning method and system based on reinforcement learning

Abstract

The present application discloses a structured road vehicle trajectory planning method based on reinforcement learning. The trajectory planning method based on reinforcement learning improves the safety of trajectory planning by constructing a driving risk field and incorporating risk indicators into the optimization function; solves the initial trajectory through an intelligent agent, thereby improving the computational efficiency of trajectory planning in the time and space domain; and optimizes the trajectory based on the initial trajectory solution through a trajectory optimization model, thereby ensuring the optimality and feasibility of the generated trajectory.

Description

Technical Field

[0001] The present application belongs to the technical field of vehicle trajectory planning, and specifically relates to a structured road vehicle trajectory planning method and system based on reinforcement learning. Background Art

[0002] Autonomous vehicles have the ability to navigate autonomously in complex environments, significantly reducing the probability of traffic accidents caused by human intervention, improving road safety and utilization, and alleviating traffic congestion. The autonomous driving system of an autonomous vehicle comprises five modules: environmental perception, mapping and positioning, trajectory prediction, decision-making and planning, and motion control. The decision-making and planning module is the core of autonomous driving. It combines high-precision maps with environmental information obtained through perception technology to plan a feasible, collision-free trajectory within a limited future time domain.

[0003] In related technologies, trajectory planning methods for structured roads are mainly divided into two types: sampling-based and optimization-based. Sampling-based methods decouple the trajectory into path-velocity components, planning them separately and then synthesizing them to obtain the trajectory. However, this method is difficult to apply to complex scenarios with dynamic obstacles. Optimization-based methods model the trajectory planning problem as a nonlinear programming problem and solve it using an optimization algorithm. However, this method relies on the quality of the initial solution and is prone to falling into local optimal solutions.

[0004] Therefore, it is necessary to provide a structured road vehicle trajectory planning method and system based on reinforcement learning. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a structured road vehicle trajectory planning method and system based on reinforcement learning. By constructing a driving risk field and incorporating risk indicators into the optimization function, the safety of trajectory planning is improved; by solving the initial trajectory through an intelligent agent, the computational efficiency of trajectory planning in the time and space domain is improved; and by using a trajectory optimization model, trajectory optimization is performed based on the initial trajectory solution, ensuring the optimality and feasibility of the generated trajectory.

[0006] In order to solve the above technical problems, this application is implemented as follows:

[0007] A structured road vehicle trajectory planning method based on reinforcement learning, characterized by comprising the following steps:

[0008] S1: Collecting status information, including the current status information of the target vehicle, the status information of obstacles, and the road boundary information;

[0009] S2: Construct an intelligent agent, which includes a state space, an action space, and an action network. The state space converts the collected state information into a state vector, which includes the target vehicle state vector, the obstacle state vector, and the road boundary vector. The action network is a multi-layer perceptron model. The state vector is used as the input of the action network. Based on the output vector of the action network, multiple normal distributions are constructed, and action vectors are generated by sampling in the normal distribution. The action space is based on the elements in the action vector and plans the target vehicle in the future t according to the vehicle kinematic model. e Trajectory sequences in time;

[0010] S3: Construct a trajectory optimization model, which includes an objective function and constraints, based on the target vehicle's future t in step S2. e The trajectory sequence within time is used as the initial solution of the trajectory optimization model to optimize the planned trajectory. The trajectory optimization problem is solved using a nonlinear optimizer, and the generated optimized trajectory is tracked using a model predictive control method. The objective function is a weighted sum of three indicators: road risk level, comfort level, and trajectory point deviation level. The constraints include vehicle kinematic constraints, control variable range constraints, and collision-free constraints.

[0011] S4: defining a reward function and training the agent using a reinforcement learning algorithm in a simulation environment; after the training is completed, using the agent and the trajectory optimization model together for vehicle trajectory planning.

[0012] Preferably, the state vector is expressed as: s=[s ego ,s V ,s bound ], where s ego represents the state vector of the target vehicle, s V Represents the state vector of the obstacle, s bound Represents the environment boundary information; the state vector of the target vehicle Among them, x, y, ω, v, and u represent the lateral position, longitudinal position, heading angle, yaw rate, velocity, and acceleration of the target vehicle, respectively; the state vector of the obstacle is in Respectively represent the horizontal relative distance, vertical relative distance, horizontal relative speed and vertical relative speed between the target and the obstacle; environmental boundary information s bound =[d l ,d r ], where d l d r Respectively represent the distance between the target vehicle and the left and right boundary lines of the lane; the state vector of the obstacle The dimension of is K, where K is the preset maximum number of observable obstacles. If the number of obstacles actually observed is less than K, the remaining bits are padded with 0.

[0013] Preferably, the action space is used to plan the target vehicle in the future t e The trajectory in time, the process of trajectory planning is:

[0014] Set time t e Divide into N time nodes to obtain time series Where, t j represents the jth time node, t j =jΔt, Δt represents the interval between two adjacent time nodes; T represents vector transposition;

[0015] Calculate the target vehicle at time node t j+1 The horizontal and vertical positions at the time of calculation are expressed as:

[0016]

[0017] Where, v j and They represent the target vehicle at time node t j Speed and heading angle at time x j+1 and y j+1 They represent the target vehicle at time node t j+1 Horizontal and vertical positions when x j and y j They represent the target vehicle at time node t j Horizontal and vertical positions when

[0018] Calculate the target vehicle at time node t j+1 The speed and heading angle at time , the calculation process is expressed as:

[0019] v j+1 =v j +ε j u max Δt;

[0020] Where, v j+1 and They represent the target vehicle at time node t j+1 The speed and heading angle at time , ε j and ∈ j They represent the control coefficients of acceleration and yaw rate at the jth time node respectively; u max Indicates the maximum control value of the target vehicle acceleration; ω max Indicates the maximum control value of the target vehicle's yaw rate;

[0021] The target vehicle's trajectory points are characterized by its lateral position, longitudinal position, speed, and heading angle at any time node. The trajectory points at all time nodes are combined to form a trajectory sequence, which is expressed as:

[0022]

[0023] Where x1, y1, v1, represents the trajectory point of the target vehicle at time node t1; x j ,y j , v j , Indicates the target vehicle at time node t j The trajectory point at time x N ,y N , v N , Indicates the target vehicle at time node t e The trajectory point of time.

[0024] Preferably, the action network is a multi-layer perceptron model, including 1 input layer, 2 hidden layers and 1 output layer, and the network weight is denoted as w θ The number of neurons in each layer is 64. The input layer and hidden layer are activated by ReLU function, and the output layer is activated by tanh function. The input vector of the action network is the state vector s, whose dimension is 4K+8, and the dimension of the output vector is 2N. The value of the output vector is taken as the mean. by For the standard deviation σ, construct 2N normal distributions in turn Finally, the action vector is obtained by sampling in the normal distribution, which is recorded as a=[ε1,∈1,ε2,∈2,...,ε j ,∈ j ,...,ε N ,∈ N ], where ε j and ∈ j They represent the control coefficients of the target vehicle acceleration and yaw rate at the jth time node respectively.

[0025] Preferably, the intelligent agent also includes a value network, which is a multi-layer perceptron model, including 1 input layer, 2 hidden layers, and 1 output layer. The network weight is denoted as w v ; The number of neurons in each layer is 64, the input layer and the hidden layer are activated by the ReLU function, the input vector of the value network is the state vector s, and the output vector is the state value V(s) with a dimension of 1. The state value V(s) is used to evaluate the current state and reflects the contribution of the current state to completing the trajectory planning task.

[0026] Preferably, the objective function is expressed as:

[0027]

[0028] Where, J e Indicates the road risk index. Indicates the weight of the road risk index; J r Indicates comfort, Indicates the weight of comfort; J o Indicates the degree of trajectory deviation, The weight of the trajectory deviation degree;

[0029] Road risk index J e Using static obstacle risk field and dynamic obstacle risk field To evaluate, it is expressed as:

[0030]

[0031] Where K O Indicates the number of static obstacles, K V represents the number of dynamic obstacles, K O +K V =K;

[0032] represents the weight of the static obstacle risk field, represents the weight of the dynamic obstacle risk field;

[0033]

[0034]

[0035] Where, Indicates that the target vehicle is at position (x j ,y j ) and static obstacle O i collision risk; G represents a constant coefficient, which is greater than 0; R represents a road environment condition coefficient, which is greater than 0. The greater the road curvature and slope, the greater the value; r j Indicates that the target vehicle is at position (x j ,y j ) and the static obstacle position At the distance; Indicates that the target vehicle is at position (x j ,y j ) with the i-th dynamic obstacle V i The collision risk; η is the time decay coefficient between 0 and 1; r i，j Target vehicle position (xj ,y j ) and the distance between the position of the i-th dynamic obstacle around it;

[0036] Comfort is evaluated using acceleration and yaw rate, expressed as:

[0037]

[0038] The trajectory deviation degree refers to the deviation between the optimized trajectory ζ′ and the initial trajectory solution ζ, which is expressed as:

[0039]

[0040] Where ζ′ j and ζ j Represents the time node t j The trajectory sequences before and after optimization.

[0041] Preferably, the vehicle kinematic constraints are expressed as:

[0042]

[0043] v j+1 =v j +u j Δt;

[0044] The control variable range constraint is expressed as:

[0045] u min ≤u j ≤umax,ω min ≤ω j ≤ω max ;

[0046] Where u min 、u max is the minimum and maximum control value of acceleration; ω min 、ω max are the minimum and maximum control values of the yaw angular velocity;

[0047] The longitudinal position of the vehicle requires the left and right edges of the road (P l , P r ) and keep a safe distance safe :

[0048] P l ≤y j ≤P r ;||y j -P l ||2≥e safe ;||y j -P r||2≥e safe ;

[0049] The no-collision constraint is expressed as:

[0050]

[0051] Where, Represents obstacles V i At time node t j Horizontal and vertical positions at the time of α safe Indicates a safety factor greater than 1; using N c The target vehicle is completely covered by a unit circle. represents the coordinates of the center of the cth unit circle, d c represents the radius of the cth unit circle;

[0052]

[0053]

[0054]

[0055] Where, L b ,L,L w 、L f They are the wheelbase, length, width and front overhang of the target vehicle respectively.

[0056] Preferably, the reinforcement learning algorithm is selected from one of a proximal policy optimization algorithm, a soft actor-critic algorithm, and a deep deterministic policy gradient algorithm.

[0057] Preferably, the reward function is based on the collision penalty index r c (s, ζ′), arrival target point index r g (s,ζ′) and driving speed index r V (s, ζ′) is established and expressed as:

[0058] r(s,ζ′)=r c (s,ζ′)+r g (s,ζ′)+r v (s,ζ′);

[0059]

[0060]

[0061]

[0062] Where, v max is the maximum control value of the target vehicle speed, is the average speed of the planned trajectory of the target vehicle.

[0063] Preferably, during the training process of the agent, the network weights of the action network and the value network need to be updated. The specific steps are: first, the action network weight w θ and the value network weight w v Normal initialization is performed to make the weight distribution between each layer in each network a standard normal distribution; then the current state vector s is input into the action network to obtain the initial solution of the planned trajectory, and then trajectory optimization is performed and trajectory tracking is performed at a trajectory tracking frequency of 100 Hz. After the vehicle travels for 0.1 s, the state transfer is performed, and the next state vector s1 is constructed, and the trajectory planning is repeated until the vehicle reaches the end point. If the vehicle reaches the end point, the end point identifier d = 1 is recorded, otherwise it is recorded d = 0; among them, each time a state transfer is completed, the reward function and the state value are calculated, and the number of interactions is increased.

[0064] This application provides a structured road vehicle trajectory planning method based on reinforcement learning. The reinforcement learning-based trajectory planning method improves the safety of trajectory planning by constructing a driving risk field and incorporating risk indicators into the optimization function; solves the initial trajectory through an intelligent agent, thereby improving the computational efficiency of trajectory planning in the spatiotemporal domain; and optimizes the trajectory based on the initial trajectory solution through a trajectory optimization model, ensuring the optimality and feasibility of the generated trajectory. DETAILED DESCRIPTION

[0065] The following is a clear and complete description of the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0066] The present invention provides a structured road vehicle trajectory planning method based on reinforcement learning, comprising the following steps:

[0067] S1: Collecting status information, which includes the current status information of the target vehicle, the status information of obstacles, and the road boundary information.

[0068] The target vehicle is an autonomous vehicle, and the status information is collected by environmental perception sensors configured on the target vehicle. The environmental perception sensors mainly include visual cameras (including monocular, binocular stereo vision, panoramic vision and infrared cameras) and radar ranging sensors (lidar, millimeter wave radar, ultrasonic radar, etc.).

[0069] Obstacles include static obstacles and dynamic obstacles. Static obstacles and dynamic obstacles are distinguished based on their current state. For example, a moving vehicle is a dynamic obstacle, while a broken-down vehicle is a static obstacle. Road boundary information is the location of the left and right lane boundary lines.

[0070] S2: Construct an intelligent agent, which includes a state space, an action space, and an action network. The state space converts the collected state information into a state vector, which includes the target vehicle state vector, the obstacle state vector, and the road boundary vector. The action network is a multi-layer perceptron model. The state vector is used as the input of the action network. Based on the output vector of the action network, multiple normal distributions are constructed, and action vectors are generated by sampling in the normal distribution. The action space is based on the elements in the action vector and plans the target vehicle in the future t according to the vehicle kinematic model. e A sequence of trajectories over time.

[0071] The state vector is expressed as: s=[s ego , s V , s bound ], where s ego represents the state vector of the target vehicle, s V Represents the state vector of the obstacle, s bound Represents the environment boundary information. Taking the current position of the target vehicle as the origin of the Cartesian coordinate system, the state vector of the target vehicle Among them, x, y, ω, v, and u represent the lateral position, longitudinal position, heading angle, yaw rate, velocity, and acceleration of the target vehicle, respectively; the state vector of the obstacle is in Respectively represent the horizontal relative distance, vertical relative distance, horizontal relative speed and vertical relative speed between the target and the obstacle; environmental boundary information s bound =[d l , d r ], where d l d r Respectively represent the distance between the target vehicle and the left and right boundary lines of the lane.

[0072] During the actual driving process of the target vehicle, the number of obstacles within the observable range is uncertain. However, for the convenience of calculation, the dimension of the state vector of the obstacle should be a constant. Therefore, it is necessary to set a maximum number of observable obstacles, denoted as K. Then the state vector sV The dimension of is K. If the number of obstacles actually observed is less than K, the remaining bits are filled with 0. Take κ as an example, if the number of obstacles actually observed is κ (0≤κ≤K), then The elements from κ+1th to Kth are padded with 0.

[0073] The action network is a multi-layer perceptron model, including 1 input layer, 2 hidden layers and 1 output layer. The network weight is denoted as w θ The number of neurons in each layer is 64. The input layer and hidden layer are activated by ReLU function, and the output layer is activated by tanh function to ensure that the value of the output vector is between -1 and 1. The input vector of the action network is the state vector s, whose dimension is 4K+8, and the dimension of the output vector is 2N. The value of the output vector is taken as the mean by For the standard deviation σ, construct 2N normal distributions in turn Finally, the action vector is obtained by sampling in the normal distribution, which is recorded as a=[ε1,∈1,ε2,∈2,...,ε j ,∈ j ,...,ε N ,∈ N ], where ε j and ∈ j Denote the control coefficients of acceleration and yaw rate at the jth time node respectively. And the probability vector corresponding to the selected action vector is recorded as

[0074] The action space is used to plan the target vehicle's actions in the future t e The trajectory in time, the process of trajectory planning is:

[0075] Set time t e Divide into N time nodes to obtain time series Where, t j represents the jth time node, t j =jΔt, Δt represents the interval between two adjacent time nodes; T represents vector transposition;

[0076] According to the vehicle kinematic model, the target vehicle is calculated at time node t j+1 The horizontal and vertical positions at the time of calculation are expressed as:

[0077]

[0078] Where, v j and They represent the target vehicle at time node t j Speed and heading angle at time x j+1 and y j+1 They represent the target vehicle at time node t j+1 Horizontal and vertical positions when xj and y j They represent the target vehicle at time node t j The horizontal and vertical positions of the

[0079] In the trajectory planning of the target vehicle, in order to ensure the stability of the target vehicle, the speed and heading angle change between two adjacent time nodes, while the acceleration and yaw rate remain constant. Calculate the target vehicle at time node t j+1 The speed and heading angle at time , the calculation process is expressed as:

[0080] v j+1 =v j +ε j u max Δt; Where, v j+1 and They represent the target vehicle at time node t j+1 The speed and heading angle at time ε j and ∈ j They represent the control coefficients of acceleration and yaw rate at the jth time node, and their values are between -1 and 1; u max Indicates the maximum control value of the target vehicle acceleration; ω max Indicates the maximum control value of the target vehicle's yaw rate.

[0081] The target vehicle's trajectory points are characterized by its lateral position, longitudinal position, speed, and heading angle at any time node. The trajectory points at all time nodes are combined to form a trajectory sequence, which is expressed as:

[0082]

[0083] Where x1, y1, v1, represents the trajectory point of the target vehicle at time node t1; x j , v j , v j , Indicates the target vehicle at time node t j The trajectory point at time x N ,y N , v N , Indicates the target vehicle at time node t e The trajectory point of time.

[0084] The agent also includes a value network, which is a multi-layer perceptron model, including 1 input layer, 2 hidden layers, and 1 output layer. The network weight is denoted as w vThe number of neurons in each layer is 64. The input layer and hidden layer are activated using the ReLU function. The input vector of the value network is the state vector s, and the output vector is the state value V(s) with a dimension of 1. The state value V(s) is used to evaluate the current state and reflects the contribution of the current state to completing the trajectory planning task. The greater the state value, the higher the contribution. When performing trajectory planning, the agent has a greater probability of reaching a state with a greater state value.

[0085] S3: Construct a trajectory optimization model, which includes an objective function and constraints, based on the target vehicle's future t in step S2. e The trajectory sequence within time is used as the initial solution of the trajectory optimization model to optimize the planned trajectory. The trajectory optimization problem is solved using a nonlinear optimizer, and the generated optimized trajectory is tracked using a model predictive control method. The objective function is a weighted sum of three indicators: road risk level, comfort level, and trajectory point deviation level. The constraints include vehicle kinematic constraints, control variable range constraints, and collision-free constraints.

[0086] The objective function is expressed as:

[0087]

[0088] Where, J e Indicates the road risk index. Indicates the weight of the road risk index; J r Indicates comfort, Indicates the weight of comfort; J o Indicates the degree of trajectory deviation, The weight of the trajectory deviation degree.

[0089] When the target vehicle is traveling, the closer it is to the obstacle, the higher the risk of collision. The road risk index J e Using static obstacle risk field and dynamic obstacle risk field To evaluate, it is expressed as:

[0090]

[0091] Where K O Indicates the number of static obstacles, K V represents the number of dynamic obstacles, K O +K V =K;

[0092] represents the weight of the static obstacle risk field, represents the weight of the dynamic obstacle risk field;

[0093]

[0094]

[0095] Where, Indicates that the target vehicle is at position (x j ,y j ) and static obstacle O i collision risk; G represents a constant coefficient, which is greater than 0; R represents a road environment condition coefficient, which is greater than 0. The greater the road curvature and slope, the greater the value; r j Indicates that the target vehicle is at position (x j ,y j ) and the static obstacle position At the distance; Indicates that the target vehicle is at position (x j ,y j ) with the i-th dynamic obstacle V i The collision risk; η is the time decay coefficient between 0 and 1; r i，j Target vehicle position (x j ,y j ) and the distance between the position of the i-th dynamic obstacle around it;

[0096] Comfort is evaluated using acceleration and yaw rate, expressed as:

[0097]

[0098] The trajectory deviation degree refers to the deviation between the optimized trajectory ζ′ and the initial trajectory solution ζ, which is expressed as:

[0099]

[0100] Where ζ′ j and ζ j Represents the time node t j The trajectory sequences before and after optimization.

[0101] In trajectory planning, certain constraints also need to be met, including vehicle kinematic constraints, control variable range constraints, and collision-free constraints.

[0102] Construct kinematic constraints based on the vehicle kinematic model:

[0103]

[0104] v j+1 =v j +u j Δt;

[0105] The acceleration and yaw rate of the target vehicle also need to be within the range of the control values:

[0106] u min ≤u j ≤u max ,ω min ≤ω j ≤ω max .

[0107] Where u min 、u max is the minimum and maximum control value of acceleration; ω min 、ω max are the minimum and maximum control values of the yaw angular velocity.

[0108] To ensure the safety of the planned trajectory, the vehicle needs to be driven within the lane range, so the longitudinal position of the vehicle needs to be within the left and right boundaries of the road (P l , P r ) and keep a safe distance safe :

[0109] P l ≤y j ≤P r ;||y j -P l ||2≥e safe ;||y j -P r ||2≥e safe .

[0110] The planned trajectory of the target vehicle must also satisfy the non-collision constraint with obstacles. c The target vehicle is completely covered by unit circles, and the coordinates of the center of the cth unit circle are And the radius dc is obtained using the following formula:

[0111]

[0112]

[0113]

[0114] Where, L b ,L,L w 、L f They are the wheelbase, length, width and front overhang of the target vehicle respectively.

[0115] Using a safety factor α greater than 1 safe , establish the following collision-free inequality constraints:

[0116]

[0117] Where, Represents obstacles V i At time node t j The horizontal and vertical positions of the time.

[0118] It should be noted that for a static obstacle, it remains stationary, so the static obstacle is at time node t j The position at the time t is the same as the current position; for dynamic obstacles, they keep moving. j The position at the time is different from the current position, and can be predicted according to conventional prediction methods in the field.

[0119] The nonlinear optimizer is an IPOPT solver.

[0120] S4: defining a reward function and training the agent using a reinforcement learning algorithm in a simulation environment; after the training is completed, using the agent and the trajectory optimization model together for vehicle trajectory planning.

[0121] The reinforcement learning algorithm is selected from one of the Proximal Policy Optimization (PPO) algorithm, the Soft Actor Critic (SAC) algorithm and the Deep Deterministic Policy Gradient (DDPG) algorithm.

[0122] The reward function is based on the collision penalty index r c (s, ζ′), arrival target point index r g (s,ζ′) and driving speed index r v (s, ζ′) is established and expressed as:

[0123] r(s,ζ′)=r c (s,ζ′)+r g (s,ζ′)+r v (s,ζ′);

[0124]

[0125]

[0126]

[0127] Where, v max is the maximum control value of the target vehicle speed, is the average velocity of the planned trajectory.

[0128] During the training process of the agent, the network weights of the action network and the value network need to be updated. The specific steps are: first, update the action network weight w θ and the value network weight w v Normal initialization is performed to ensure that the weight distribution between each layer in each network is standard normal. The current state vector s is then input into the action network to obtain the initial solution for the planned trajectory. Trajectory optimization is then performed and the trajectory tracking is performed at a trajectory tracking frequency of 100Hz. Since the trajectory planning frequency is set to 10Hz, the vehicle transitions to a new state vector s1 after 0.1s of travel, and trajectory planning is repeated based on this state vector until the vehicle reaches the destination. If the vehicle reaches the destination, the destination identifier d = 1; otherwise, d = 0.

[0129] Each time a state transition is completed, the reward function and state value are calculated and the relevant data is stored, and the number of interactions is increased. The next state vector s′, reward value r, state value V(s) and end point identifier d are stored in a buffer D of size B, denoted as When the buffer is full, the neural network weights are updated using the data in the buffer. The action network and value network are updated 80 times each. After the update is complete, the data in the buffer is cleared and the agent continues to interact with the simulation environment.

[0130] Among them, the action network loss function calculation formula is as follows:

[0131]

[0132] Where, ρ t (w θ ) is the probability ratio of the new and old action networks to generate actions The same batch of data will be updated multiple times. is the action network weight parameter before updating. is the advantage function, and its calculation formula is γ and λ are the discount factor and advantage factor, respectively, which are selected as 0.99 and 0.97; μ is the update range threshold, which is 0.2; the clip function indicates that if the probability ratio ρ t (w θ ) is greater than 1.2, returns 1.2, if less than 0.8, returns 0.8, and if between 0.8 and 1.2, remains unchanged.

[0133] Among them, the calculation formula of the value network loss function is as follows:

[0134]

[0135] The above describes the embodiments of the present application, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A structured road vehicle trajectory planning method based on reinforcement learning, characterized in that: The steps include: S1: Collecting status information, including the current status information of the target vehicle, the status information of obstacles, and the road boundary information; S2: Construct an intelligent agent, which includes a state space, an action space, and an action network. The state space converts the collected state information into a state vector, which includes the target vehicle state vector, the obstacle state vector, and the road boundary vector. The action network is a multi-layer perceptron model. The state vector is used as the input of the action network. Based on the output vector of the action network, multiple normal distributions are constructed, and action vectors are generated by sampling in the normal distribution. The action space is based on the elements in the action vector and plans the target vehicle in the future t according to the vehicle kinematic model. e Trajectory sequences in time; S3: Construct a trajectory optimization model, which includes an objective function and constraints, based on the target vehicle's future t in step S2. e The trajectory sequence within time is used as the initial solution of the trajectory optimization model to optimize the planned trajectory. The trajectory optimization problem is solved using a nonlinear optimizer, and the generated optimized trajectory is tracked using a model predictive control method. The objective function is a weighted sum of three indicators: road risk level, comfort level, and trajectory point deviation level. The constraints include vehicle kinematic constraints, control variable range constraints, and collision-free constraints. S4: defining a reward function and training the agent using a reinforcement learning algorithm in a simulation environment; after the training is completed, using the agent and the trajectory optimization model together for vehicle trajectory planning.

2. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 1 is characterized in that: The state vector is expressed as: s=[s ego ,s V ,s bound ], where s ego represents the state vector of the target vehicle, s V Represents the state vector of the obstacle, s bound Represents the environment boundary information; the state vector of the target vehicle Among them, x, y, ω, v, and u represent the lateral position, longitudinal position, heading angle, yaw rate, velocity, and acceleration of the target vehicle, respectively; the state vector of the obstacle is in Respectively represent the horizontal relative distance, vertical relative distance, horizontal relative speed and vertical relative speed between the target and the obstacle; environmental boundary information s bound =[d l ,d r ], where d l d r Respectively represent the distance between the target vehicle and the left and right boundary lines of the lane; the state vector of the obstacle The dimension of is K, where K is the preset maximum number of observable obstacles. If the number of obstacles actually observed is less than K, the remaining bits are padded with 0.

3. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 2, characterized in that: The action space is used to plan the target vehicle's actions in the future t e The trajectory in time, the process of trajectory planning is: Set time t e Divide into N time nodes to obtain time series Where, t j represents the jth time node, t j =jΔt, Δt represents the interval between two adjacent time nodes; T represents vector transposition; Calculate the target vehicle at time node t j+1 The horizontal and vertical positions at the time of calculation are expressed as: Where, v j and They represent the target vehicle at time node t j Speed and heading angle at time x j+1 and y j+1 They represent the target vehicle at time node t j+1 Horizontal and vertical positions when x j and y j They represent the target vehicle at time node t j Horizontal and vertical positions when Calculate the target vehicle at time node t j+1 The speed and heading angle at time , the calculation process is expressed as: v j+1 =v j +ε j u max Δt; Where, v j+1 and They represent the target vehicle at time node t j+1 The speed and heading angle at time , ε j and 𝜖 j They represent the control coefficients of acceleration and yaw rate at the jth time node respectively; u max Indicates the maximum control value of the target vehicle acceleration; ω max Indicates the maximum control value of the target vehicle's yaw rate; The target vehicle's trajectory points are characterized by its lateral position, longitudinal position, speed, and heading angle at any time node. The trajectory points at all time nodes are combined to form a trajectory sequence, which is expressed as: Where x1, y1, v1, represents the trajectory point of the target vehicle at time node t1; x j ,y j ,v j , Indicates the target vehicle at time node t j The trajectory point at time x N ,y N ,v N , Indicates the target vehicle at time node t e The trajectory point of time.

4. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 3 is characterized in that: The action network is a multi-layer perceptron model, including 1 input layer, 2 hidden layers and 1 output layer. The network weight is denoted as w θ The number of neurons in each layer is 64. The input layer and hidden layer are activated by ReLU function, and the output layer is activated by tanh function. The input vector of the action network is the state vector s, whose dimension is 4K+8, and the dimension of the output vector is 2N. The value of the output vector is taken as the mean. by For the standard deviation σ, construct 2N normal distributions in turn Finally, the action vector is obtained by sampling in the normal distribution, which is recorded as a=[ε1,𝜖1,ε2,𝜖2,…,ε j ,𝜖 j ,…,ε N ,𝜖 N ], where ε j and 𝜖 j They represent the control coefficients of the target vehicle acceleration and yaw rate at the jth time node respectively.

5. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 4 is characterized in that: The intelligent agent also includes a value network, which is a multi-layer perceptron model, including 1 input layer, 2 hidden layers, and 1 output layer. The network weight is denoted as w v ; The number of neurons in each layer is 64, the input layer and the hidden layer are activated by the ReLU function, the input vector of the value network is the state vector s, and the output vector is the state value V(s) with a dimension of 1. The state value V(s) is used to evaluate the current state and reflects the contribution of the current state to completing the trajectory planning task.

6. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 5, characterized in that: The objective function is expressed as: Where, J e Indicates the road risk index. Indicates the weight of the road risk index; J r Indicates comfort, Indicates the weight of comfort; J o Indicates the degree of trajectory deviation, The weight of the trajectory deviation degree; Road risk index J e Using static obstacle risk field and dynamic obstacle risk field To evaluate, it is expressed as: Where K O Indicates the number of static obstacles, K V represents the number of dynamic obstacles, K O +K V =K; represents the weight of the static obstacle risk field, represents the weight of the dynamic obstacle risk field; Where, Indicates that the target vehicle is at position (x j ,y j ) and static obstacle O i collision risk; G represents a constant coefficient, which is greater than 0; R represents a road environment condition coefficient, which is greater than 0. The greater the road curvature and slope, the greater the value; r j Indicates that the target vehicle is at position (x j ,y j ) and the static obstacle position At the distance; Indicates that the target vehicle is at position (x j ,y j ) with the i-th dynamic obstacle V i the risk of collision; η is the time decay coefficient between 0 and 1; r i,j Target vehicle position (x j ,y j ) and the distance between the position of the i-th dynamic obstacle around it; Comfort is evaluated using acceleration and yaw rate, expressed as: The trajectory deviation degree refers to the deviation between the optimized trajectory ζ′ and the initial trajectory solution ζ, which is expressed as: Where ζ′ j and ζ j Represents the time node t j The trajectory sequences before and after optimization.

7. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 6, characterized in that: The vehicle kinematic constraints are expressed as: v j+1 =v j +u j Δt; The control variable range constraint is expressed as: you min ≤u j ≤u max Oh, oh min ≤ω j ≤ω max ; Where u min 、u max is the minimum and maximum control value of acceleration; ω min 、ω max are the minimum and maximum control values of the yaw angular velocity; The longitudinal position of the vehicle requires the left and right edges of the road (P l ,P r ) and keep a safe distance safe : P l ≤y j ≤P r ;||y j -P l ||2≥e safe ;||y j -P r ||2≥e safe ; The no-collision constraint is expressed as: Where, Represents obstacles V i At time node t j Horizontal and vertical position at the time; α safe Indicates a safety factor greater than 1; using N c The target vehicle is completely covered by a unit circle. represents the coordinates of the center of the cth unit circle, d c represents the radius of the cth unit circle; Where, L b ,L,L w , L f They are the wheelbase, length, width and front overhang of the target vehicle respectively.

8. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm is selected from a proximal policy optimization algorithm, a soft actor-critic algorithm, and a deep deterministic policy gradient algorithm.

9. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 1, characterized in that: The reward function is based on the collision penalty index r c (s,ζ′), reaching the target point index r g (s,ζ′) and driving speed index r v (s,ζ′) is established and expressed as: r(s,ζ′)=r c (s,ζ′)+r g (s,ζ′)+r v (s,ζ′); Where, v max is the maximum control value of the target vehicle speed, is the average speed of the planned trajectory of the target vehicle.

10. The structured road vehicle trajectory planning method based on reinforcement learning according to claim 7, characterized in that: During the training process of the intelligent agent, the network weights of the action network and the value network need to be updated. The specific steps are as follows: first, the action network weight w θ and the value network weight w v Normal initialization is performed to make the weight distribution between each layer in each network a standard normal distribution; then the current state vector s is input into the action network to obtain the initial solution of the planned trajectory, and then trajectory optimization is performed and trajectory tracking is performed at a trajectory tracking frequency of 100 Hz. After the vehicle travels for 0.1 s, the state transfer is performed, and the next state vector s1 is constructed, and the trajectory planning is repeated until the vehicle reaches the end point. If the vehicle reaches the end point, the end point identifier d = 1 is recorded, otherwise it is recorded d = 0; among them, each time a state transfer is completed, the reward function and the state value are calculated, and the number of interactions is increased.

Images