Markov decision process model training method and trajectory planning method
By adopting Markov decision-making process model and DQN neural network in the autonomous driving system, the problems of high computational complexity and poor real-time performance in the prior art are solved, and efficient trajectory planning and path control in complex traffic scenarios are realized, reducing model training costs and improving interpretability.
Patent Information
- Application Number
- CN202411926036.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-18
AI Technical Summary
When facing complex traffic scenarios, the existing autonomous driving trajectory planning methods have high computational complexity, poor real-time performance, and require a large amount of labeled data and difficult-to-explain models.
The Markov decision-making process model is used to combine the DQN neural network, and by obtaining the road traffic information around the vehicle, building state space and action space, and using the DQN neural network shared by the multi-vehicle network for training, a strategy for trajectory planning is obtained.
It has achieved strong adaptability in complex traffic scenarios, can timely plan the approximate optimal trajectory, and ensure the continuity of path tracking control, reducing the computational complexity and model training cost, and improving the interpretability of the model.
Smart Images
Figure CN119849588A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of unmanned driving technology, and in particular to a Markov decision process model training method and a trajectory planning method. Background Art
[0002] With the advancement of science and technology and the increase in traffic demand, autonomous vehicles have gradually become an important development direction in the future transportation field. As a key component of autonomous driving technology, dynamic vehicle trajectory planning is of great significance for improving traffic safety, reducing congestion, and improving traffic efficiency. Therefore, how to achieve accurate and efficient trajectory planning has become the focus of autonomous driving technology research. Most of the current methods use optimization-based and learning-based methods for autonomous driving trajectory planning. The optimization-based method finds the optimal trajectory by establishing an optimization model. This method usually transforms the trajectory planning problem into an optimization problem, considers factors such as vehicle dynamic constraints, road constraints, and traffic rules, and obtains the optimal driving path by solving the optimization model. The advantage is that it can handle complex traffic scenes and obtain the global optimal or approximately optimal trajectory. The disadvantage is that the computational complexity is high and the real-time performance is poor. The learning-based method uses machine learning, deep learning and other technologies to learn driving experience from a large amount of data to achieve trajectory planning. This type of method trains the model so that the model can predict the behavior of other traffic participants and plan a suitable driving path based on the prediction results. The advantage is that it can handle complex and changeable traffic scenes, has strong adaptability, and good real-time performance. The disadvantage is that it requires a large amount of labeled training data, and the training cost of the model is high; at the same time, the model has poor interpretability and it is difficult to directly understand its decision-making process. Summary of the invention
[0003] The purpose of the present invention is to provide a Markov decision process model training method and a trajectory planning method, so that when facing complex traffic scenes, the model has strong adaptability, can timely plan a near-optimal trajectory, and at the same time ensure the continuity of path tracking control.
[0004] A first aspect of the present invention provides a Markov decision process model training method for vehicle trajectory planning, comprising:
[0005] Acquire road traffic information around the vehicle, and determine the driving state of the vehicle according to the road traffic information;
[0006] According to the driving state, a Markov decision process model is constructed, wherein the Markov decision process model includes a state space, an action space, a DQN neural network shared by a multi-vehicle network, and a reward function, wherein the state space and the action space are determined based on the driving state, the DQN neural network is used to select an optimal state-action value from the action space based on the state space to control the vehicle, and obtain a trajectory planning strategy for the vehicle, and the reward function is used to quantify the performance of the trajectory planning strategy;
[0007] By putting the experiences collected by multiple vehicles into a network experience pool, when the experiences in the network experience pool reach a certain amount, the DQN neural network samples from the experience pool by random sampling to perform experience playback and update the neural network parameters, so as to train the Markov decision process model and obtain a Markov decision process model after training;
[0008] Among them, the Markov decision process model after training can output the optimal trajectory planning strategy based on the real-time driving status of any input vehicle.
[0009] Further, obtaining road traffic information around the vehicle and determining the driving state of the vehicle according to the road traffic information includes:
[0010] By integrating the vehicle's onboard cameras, ultrasonic radars, millimeter-wave radars and lidars with high-precision maps, the vehicle's surrounding roads and other traffic participant information can be obtained, including the position, speed and acceleration of surrounding vehicles, as well as road lane lines, traffic lights and congestion conditions. The vehicle can decide whether to follow or change lanes based on the road and other traffic participant information around the vehicle.
[0011] Furthermore, the Markov decision process model is a tuple (S, A, P, R, γ), where:
[0012] S represents the set of state spaces, that is, the set of all possible states of the vehicle;
[0013] A represents the set of action spaces, that is, the set of all possible actions that the vehicle may take;
[0014] P represents the state transition probability matrix, that is, the probability of transitioning from one state to another;
[0015] R represents the reward function, that is, the reward obtained by the vehicle after taking action a in state s;
[0016] γ represents the discount factor of cumulative returns, γ∈[0,1];
[0017] The agent selects action a from state s and executes the optimal strategy π*. The optimal state action value function Q*(s,a) is:
[0018]
[0019] Furthermore, the state space and action space are determined based on the driving state, specifically:
[0020] When the vehicle is in a following state, the state space is the distance between the vehicle and the current lane, and the action space is acceleration, deceleration or maintaining constant speed; when the vehicle is in a lane changing state, the state space of the vehicle is the longitudinal distance between the vehicle in front of the current lane and the vehicle in front of the target lane, and the action space is the vehicle acceleration and the coefficients of the quintic polynomial used to characterize the lane changing curve equation.
[0021] Furthermore, the DQN neural network is used to select the optimal state-action value from the action space based on the state space to control the vehicle, and obtain the trajectory planning strategy of the vehicle. The reward function is used to quantify the performance of the trajectory planning strategy, specifically:
[0022] The learning parameter α is set to 0.01, the discount factor γ is set to 0.9, and the greed factor ε is set to 0.1;
[0023] When the vehicle is in the following driving state, the reward function is:
[0024]
[0025] Where D represents the distance between the vehicle and the vehicle in front;
[0026] When the vehicle is in lane change, the formula s(t) = a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 represents the lane-changing trajectory of the vehicle, where a0, a1, a2, a3, a4, and a5 all represent constant coefficients, and t represents time;
[0027] When the vehicle is changing lanes, the reward function is:
[0028]
[0029] Among them, D1 represents the longitudinal distance between the vehicle and the vehicle in front of the current lane during the lane changing process; D2 represents the longitudinal distance between the vehicle and the vehicle in front of the target lane during the lane changing process.
[0030] Furthermore, the training of the Markov decision process model is specifically as follows:
[0031] The agent calculates the current state of the vehicle, including the speed and position of the vehicle, and the distance from the vehicle in front. The agent uses the vehicle's own state as the input of the Q network, performs a forward propagation on the Q network, and obtains the state action value Q of all possible actions of the vehicle. The action with the largest Q value is selected according to the ε-greedy strategy. The update method of the state action value Q is:
[0032] Q(S t , A t )←Q(S t , A t )+α(R t+1 +γmax a Q(S t+1 , a)-Q(S t , A t ))
[0033] Among them, α∈(0,1) represents the learning parameter, γ represents the discount factor, S represents the state reached by the vehicle, t represents the time, S t+1 represents the state at time t+1, A represents the action performed by the vehicle, and R represents the reward obtained.
[0034] The agent calculates the reward and the vehicle's next state according to the reward principle, and stores them in the experience pool for experience playback;
[0035] A record is randomly selected from the experience pool to calculate the target value of the current Q network feedforward learning and the gradient of the network parameter ω. The network parameter ω is updated by the gradient to obtain the ω target, and this process is repeated until the termination state is reached or the time slice is used up.
[0036] A second aspect of the present invention provides a method for dynamic trajectory planning for autonomous driving, comprising:
[0037] Acquire road traffic information around the vehicle, and determine the real-time driving status of the vehicle based on the road traffic information;
[0038] Inputting the real-time driving state of the vehicle into a trained Markov decision process model to obtain a trajectory planning strategy, wherein the Markov decision process model is trained according to any one of the above methods;
[0039] The vehicle is controlled according to the trajectory planning strategy.
[0040] The present invention has at least the following beneficial effects:
[0041] The present invention establishes a Markov decision process model of vehicle driving and uses a DQN neural network to train the Markov decision process model, so that the model has strong adaptability when facing complex traffic scenes, can timely plan a near-optimal trajectory, and at the same time ensure the continuity of path tracking control. Furthermore, this solution uses the DQN algorithm to learn path planning, which reduces the computational complexity and does not require a large amount of training data, thereby reducing the cost of model training and improving the interpretability of the model.
[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 A flowchart of a Markov decision process model training method for vehicle trajectory planning provided by the present invention;
[0044] Figure 2 A flowchart of a method for dynamic trajectory planning of an autonomous driving system provided by the present invention;
[0045] Figure 3 A schematic diagram of vehicle lane change trajectory planning provided by the present invention;
[0046] Figure 4 A schematic diagram of a DQN learning method for multi-vehicle network sharing provided by the present invention;
[0047] Figure 5 The cumulative reward graph of the model training provided by the present invention;
[0048] Figure 6 This is a schematic diagram of the reinforcement learning interaction provided by the present invention. DETAILED DESCRIPTION
[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0050] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.
[0051] Example 1: Combination Figure 1-6 This embodiment is described.
[0052] An embodiment of the present invention provides a Markov decision process model training method for vehicle trajectory planning, comprising:
[0053] S100: Acquire road traffic information around a vehicle, and determine a driving state of the vehicle according to the road traffic information.
[0054] In some embodiments, the road traffic information around the vehicle can be obtained by fusing a high-precision map with one or more of the vehicle-mounted camera, ultrasonic radar, millimeter-wave radar, and laser radar. The road traffic information may include information about the road itself and information about other traffic participants, such as the position, speed, and acceleration of surrounding vehicles, as well as road lane lines, traffic lights, and congestion conditions. The decision of the vehicle's driving state in the above step S100 may specifically include deciding whether the vehicle is following or changing lanes based on the road and other traffic participant information around the vehicle, that is, the above driving state may include two states: following or changing lanes.
[0055] S101. Construct a Markov decision process model according to the driving state, wherein the Markov decision process model includes a state space, an action space, a DQN neural network shared by a multi-vehicle network, and a reward function, wherein the state space and the action space are determined based on the driving state, the DQN neural network is used to select the optimal state-action value from the action space based on the state space to control the vehicle, and obtain a trajectory planning strategy for the vehicle, and the reward function is used to quantify the performance of the trajectory planning strategy.
[0056] It is worth noting that the vehicle has an acceleration choice at every moment during the driving process. When the vehicle is driving on the road, it may have different driving states. It may be far away from the vehicle in front, or it may be too close to the vehicle in front, with the risk of collision. In order to improve driving efficiency, each vehicle selects the acceleration as the target acceleration for the next moment according to the current state, and then accelerates, brakes or maintains the current speed according to the target acceleration. After the vehicle completes the action, it enters the next moment, and the vehicle selects the target acceleration according to the state, and repeats this process. Based on this, the acceleration selection process of the autonomous driving vehicle can be regarded as a Markov decision process, because the vehicle only considers the current state when making decisions, but not the previous state. The transfer of the vehicle and the surrounding traffic state is also only related to the state of the previous moment, and does not involve the historical state.
[0057] The Markov decision process model is a tuple (S, A, P, R, γ), where
[0058] S represents the set of state spaces, that is, the set of all possible states of the vehicle;
[0059] A represents the set of action spaces, that is, the set of all possible actions that the vehicle may take;
[0060] P represents the state transition probability matrix, that is, the probability of transitioning from one state to another;
[0061] R represents the reward function, that is, the reward obtained by the vehicle after taking action a in state s;
[0062] γ represents the discount factor of cumulative returns, γ∈[0,1];
[0063] For any Markov process, if the agent selects action a from state s and finally executes the optimal strategy π*, the optimal state action value function Q*(s,a) is:
[0064]
[0065] S102, by putting the experiences collected by multiple vehicles into a network experience pool, when the experiences in the network experience pool reach a certain amount, the DQN neural network samples from the experience pool by random sampling to perform experience playback and update the neural network parameters, so as to train the Markov decision process model and obtain a trained Markov decision process model.
[0066] Among them, the Markov decision process model after training can output the optimal trajectory planning strategy based on the real-time driving status of any input vehicle.
[0067] The DQN neural network is a DQN learning method based on multi-vehicle network sharing, which means that the surrounding vehicles share the same DQN reinforcement learning neural network, which makes action decisions. During the intelligent agent learning process, the vehicle puts the collected experience into the network experience pool. When the experience in the network experience pool reaches a certain amount, the DQN neural network can sample from the experience pool through random sampling to replay the experience and update the neural network parameters. During training, the vehicle continuously learns the optimal trajectory planning strategy through the DQN algorithm. After training, a Markov decision process model that is approximately optimal is obtained. The optimal result is reflected in the convergence of the reward value obtained by the intelligent agent in the later stage of model training. Figure 4 As shown in Figure 1, the convergence of the reward function established by model training has been given. By inputting the real-time driving state of any vehicle into the trained Markov decision process model, the optimal trajectory planning strategy can be obtained.
[0068] The present invention establishes a Markov decision process model of vehicle driving and uses a DQN neural network to train the Markov decision process model, so that the model has strong adaptability when facing complex traffic scenes, can timely plan a near-optimal trajectory, and at the same time ensure the continuity of path tracking control. Furthermore, this solution uses the DQN algorithm to learn path planning, which reduces the computational complexity and does not require a large amount of training data, thereby reducing the cost of model training and improving the interpretability of the model.
[0069] In order to enable those skilled in the art to better understand the technical solution provided by the present disclosure, the training method provided by the embodiment of the present disclosure is described in detail below.
[0070] In one implementation, the above-mentioned acquiring road traffic information around the vehicle and determining the driving state of the vehicle according to the road traffic information may specifically include:
[0071] By integrating the vehicle's onboard cameras, ultrasonic radars, millimeter-wave radars and lidars with high-precision maps, the vehicle's surrounding roads and other traffic participant information can be obtained, including the position, speed and acceleration of surrounding vehicles, as well as road lane lines, traffic lights and congestion conditions. The vehicle can decide whether to follow or change lanes based on the road and other traffic participant information around the vehicle.
[0072] Furthermore, the Markov decision process model is a tuple (S, A, P, R, γ), where:
[0073] S represents the set of state spaces, that is, the set of all possible states of the vehicle;
[0074] A represents the set of action spaces, that is, the set of all possible actions that the vehicle may take;
[0075] P represents the state transition probability matrix, that is, the probability of transitioning from one state to another;
[0076] R represents the reward function, that is, the reward obtained by the vehicle after taking action a in state s;
[0077] γ represents the discount factor of cumulative returns, γ∈[0,1];
[0078] The agent selects action a from state s, executes the optimal strategy π*, and the optimal state action value function Q * (s,a) is: Q * (s,a)=r(s,a)+γ∑ s' p(s'|s,a)maxq * (s',a'),s∈S,a∈A
[0079] Furthermore, the state space and action space are determined based on the driving state, specifically:
[0080] When the vehicle is in a following state, the state space is the distance between the vehicle and the current lane, and the action space is acceleration, deceleration or maintaining constant speed; when the vehicle is in a lane changing state, the state space of the vehicle is the longitudinal distance between the vehicle in front of the current lane and the vehicle in front of the target lane, and the action space is the vehicle acceleration and the coefficients of the quintic polynomial used to characterize the lane changing curve equation.
[0081] First, when the vehicle is in a straight-following driving state, the state space is the distance between the vehicle and the vehicle in front of the lane; when the vehicle changes lanes, the state space is the distance in the longitudinal direction between the vehicle in front of the lane and the vehicle in front of the target lane; when the vehicle is driving in a straight line, the action space is the vehicle acceleration; and when the vehicle is changing lanes, the action space is the vehicle acceleration and the coefficient of the lane changing curve represented by a fifth-order polynomial.
[0082] Furthermore, the DQN neural network is used to select the optimal state-action value from the action space based on the state space to control the vehicle, and obtain the trajectory planning strategy of the vehicle. The reward function is used to quantify the performance of the trajectory planning strategy, specifically:
[0083] The learning parameter α is set to 0.01, the discount factor γ is set to 0.9, and the greed factor ε is set to 0.1;
[0084] When the vehicle is in the following driving state, the reward function is:
[0085]
[0086] Among them, D represents the distance between the vehicle and the front vehicle; the smaller the distance between the vehicle and the front vehicle, the greater the reward. This reward function will only get corresponding feedback when the rear vehicle and the front vehicle maintain a safe distance and no collision occurs. If the vehicle collides with the front vehicle, the vehicle will get a large negative feedback, so that the intelligent agent can continuously learn in the direction of maximum reward during the learning process, thereby optimizing the action strategy of each step.
[0087] like Figure 3 As shown, when the vehicle is in lane change, the formula s(t) = a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 represents the lane-changing trajectory of the vehicle, where a0, a1, a2, a3, a4, and a5 all represent constant coefficients, and t represents time;
[0088] When the vehicle is changing lanes, the reward function is:
[0089]
[0090] Among them, D1 represents the longitudinal distance between the vehicle and the vehicle in front of the current lane during the lane changing process; D2 represents the longitudinal distance between the vehicle and the vehicle in front of the target lane during the lane changing process.
[0091] Furthermore, the training of the Markov decision process model is specifically as follows:
[0092] The agent calculates the current state of the vehicle, including the speed and position of the vehicle, and the distance from the vehicle in front. The agent uses the vehicle's own state as the input of the Q network, performs a forward propagation on the Q network, and obtains the state action value Q of all possible actions of the vehicle. The action with the largest Q value is selected according to the ε-greedy strategy. The update method of the state action value Q is:
[0093] Q(S t , A t )←Q(S t , A t )+α(R t+1 +γmax a Q(S t+1 , a)-Q(S t , A t ))
[0094] Among them, α∈(0,1) represents the learning parameter, γ represents the discount factor, S represents the state reached by the vehicle, t represents the time, S t+1 represents the state at time t+1, A represents the action performed by the vehicle, R represents the reward obtained,
[0095] The agent calculates the reward and the vehicle's next state according to the reward principle, and stores them in the experience pool for experience playback;
[0096] A record is randomly selected from the experience pool to calculate the target value of the current Q network feedforward learning and the gradient of the network parameter ω. The network parameter ω is updated by the gradient to obtain the ω target, and this process is repeated until the termination state is reached or the time slice is used up.
[0097] like Figure 3As shown in the figure, the DQN neural network is a DQN learning method based on multi-vehicle network sharing, which means that the surrounding vehicles share the same DQN reinforcement learning neural network, and the network makes action decisions. During the learning process of the intelligent agent, the vehicle puts the collected experience into the network experience pool. When the experience in the network experience pool reaches a certain amount, the DQN neural network can sample from the experience pool through random sampling to replay the experience and update the neural network parameters. At each moment, the intelligent agent calculates the data of its own vehicle at the current moment, including the speed and position of its own vehicle, and reads the state of the vehicle in front to obtain the distance between the two vehicles, and then forms the state of the vehicle at the current moment, takes the state of its own vehicle as the input of the Q network, and performs a forward propagation on the Q network to obtain the state action Q value of all possible actions. According to the heuristic information, a suitable greedy strategy is selected to select the action with the largest Q value. The ε-greedy strategy is adopted in this scheme. After selecting the action, the intelligent agent calculates the reward and the state of the vehicle at the next moment according to the reward principle, and stores them in the experience pool for later use as experience playback. Then, a record is randomly selected from the experience pool to calculate the target value of the Q network feedforward learning, and then the gradient of the network parameter ω is calculated, and this gradient is used to update the parameter ω to obtain the ω target, and this is repeated until the termination state is reached or the time slice is used up.
[0098] Example 2
[0099] The embodiment of the present invention also provides a method for dynamic trajectory planning of automatic driving, including:
[0100] S200: Acquire road traffic information around the vehicle, and determine the real-time driving status of the vehicle according to the road traffic information.
[0101] S201, inputting the real-time driving state of the vehicle into the trained Markov decision process model to obtain a trajectory planning strategy.
[0102] The Markov decision process model is based on Figure 1 The method steps shown in are obtained by training. For details, please refer to the detailed description in Example 1, which will not be repeated here.
[0103] S203: Control the vehicle according to the trajectory planning strategy.
[0104] By integrating the vehicle's onboard cameras, ultrasonic radars, millimeter-wave radars and lidars with high-precision maps, the road and other traffic participant information around the vehicle can be obtained, including the position, speed and acceleration of surrounding vehicles, as well as road lane lines, traffic lights and congestion conditions. The real-time driving status of the vehicle is determined based on the road and other traffic participant information around the vehicle. The optimal trajectory planning strategy can be obtained by inputting the real-time driving status of the vehicle into the trained Markov decision process model, and the optimal trajectory planning strategy is sent to the vehicle domain controller to control the vehicle to drive according to the planned optimal trajectory.
[0105] It should also be noted that the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "includes one..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements. The words "first", "second" and the like are used to indicate names, but do not indicate any specific order. The above schematically describes the invention and its implementation methods, which is not restrictive. The invention can be implemented in other specific forms without departing from the spirit or basic features of the invention. What is shown in the accompanying drawings is only one of the implementation methods of the invention, and the actual structure is not limited thereto. Any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it, and does not deviate from the purpose of the invention, and designs a structural mode and an embodiment similar to the technical solution without creativity, they should all belong to the scope of protection of this patent.
Claims
1. A Markov decision process model training method for vehicle trajectory planning, characterized in that: include: Acquire road traffic information around the vehicle, and determine the driving state of the vehicle according to the road traffic information; According to the driving state, a Markov decision process model is constructed, wherein the Markov decision process model includes a state space, an action space, a DQN neural network shared by a multi-vehicle network, and a reward function, wherein the state space and the action space are determined based on the driving state, the DQN neural network is used to select an optimal state-action value from the action space based on the state space to control the vehicle, and obtain a trajectory planning strategy for the vehicle, and the reward function is used to quantify the performance of the trajectory planning strategy; By putting the experiences collected by multiple vehicles into a network experience pool, when the experiences in the network experience pool reach a certain amount, the DQN neural network samples from the experience pool by random sampling to perform experience playback and update the neural network parameters, so as to train the Markov decision process model and obtain a Markov decision process model after training; Among them, the Markov decision process model after training can output the optimal trajectory planning strategy based on the real-time driving status of any input vehicle.
2. A Markov decision process model training method for vehicle trajectory planning according to claim 1, characterized in that: Acquiring road traffic information around the vehicle and determining the driving state of the vehicle according to the road traffic information, including: By integrating the vehicle's onboard cameras, ultrasonic radars, millimeter-wave radars and lidars with high-precision maps, the vehicle's surrounding roads and other traffic participant information can be obtained, including the position, speed and acceleration of surrounding vehicles, as well as road lane lines, traffic lights and congestion conditions. The vehicle can decide whether to follow or change lanes based on the road and other traffic participant information around the vehicle.
3. A Markov decision process model training method for vehicle trajectory planning according to claim 1, characterized in that: The Markov decision process model is a tuple (S, A, P, R, γ), where S represents the set of state spaces, that is, the set of all possible states of the vehicle; A represents the set of action spaces, that is, the set of all possible actions that the vehicle may take; P represents the state transition probability matrix, that is, the probability of transitioning from one state to another; R represents the reward function, that is, the reward obtained by the vehicle after taking action a in state s; γ represents the discount factor of cumulative returns, γ∈[0,1]; The agent selects action a from state s and executes the optimal strategy π*. The optimal state action value function Q*(s,a) is:
4. A Markov decision process model training method for vehicle trajectory planning according to claim 1, characterized in that: The state space and action space are determined based on the driving state, specifically: When the vehicle is in a following state, the state space is the distance between the vehicle and the current lane, and the action space is acceleration, deceleration or maintaining constant speed; when the vehicle is in a lane changing state, the state space of the vehicle is the longitudinal distance between the vehicle in front of the current lane and the vehicle in front of the target lane, and the action space is the vehicle acceleration and the coefficients of the quintic polynomial used to characterize the lane changing curve equation.
5. A Markov decision process model training method for vehicle trajectory planning according to claim 1, characterized in that: The DQN neural network is used to select the optimal state action value from the action space based on the state space to control the vehicle, and obtain the trajectory planning strategy of the vehicle. The reward function is used to quantify the performance of the trajectory planning strategy, specifically: The learning parameter α is set to 0.01, the discount factor γ is set to 0.9, and the greed factor ε is set to 0.1; When the vehicle is in the following driving state, the reward function is: Where D represents the distance between the vehicle and the vehicle in front; When the vehicle is changing lanes, the formula s(t) = a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5 represents the lane-changing trajectory of the vehicle, where a0, a1, a2, a3, a4, and a5 all represent constant coefficients, and t represents time; When the vehicle is changing lanes, the reward function is: Among them, D1 represents the longitudinal distance between the vehicle and the vehicle in front of the current lane during the lane changing process; D2 represents the longitudinal distance between the vehicle and the vehicle in front of the target lane during the lane changing process.
6. A Markov decision process model training method for vehicle trajectory planning according to claim 1, characterized in that: The training of the Markov decision process model is specifically as follows: The agent calculates the current state of the vehicle, including the speed and position of the vehicle, and the distance from the vehicle in front. The agent uses the vehicle's own state as the input of the Q network, performs a forward propagation on the Q network, and obtains the state action value Q of all possible actions of the vehicle. The action with the largest Q value is selected according to the ε-greedy strategy. The update method of the state action value Q is: Q(S t ,A t )←Q(S t ,A t )+α(R t+1 +γmax a Q(S t+1 ,a)-Q(S t ,A t )) Among them, α∈(0,1) represents the learning parameter, γ represents the discount factor, S represents the state reached by the vehicle, t represents the time, S t+1 represents the state at time t+1, A represents the action performed by the vehicle, and R represents the reward obtained. The agent calculates the reward and the vehicle's next state according to the reward principle, and stores them in the experience pool for experience playback; A record is randomly selected from the experience pool to calculate the target value of the current Q network feedforward learning and the gradient of the network parameter ω. The network parameter ω is updated by the gradient to obtain the ω target, and this process is repeated until the termination state is reached or the time slice is used up.
7. A method for dynamic trajectory planning for autonomous driving, characterized in that: include: Acquire road traffic information around the vehicle, and determine the real-time driving status of the vehicle based on the road traffic information; Inputting the real-time driving state of the vehicle into a trained Markov decision process model to obtain a trajectory planning strategy, wherein the Markov decision process model is trained according to any one of claims 1 to 6; The vehicle is controlled according to the trajectory planning strategy.