A path planning method considering dynamic constraints
By using a standardized advantage function path planning method combined with dynamic constraints, the problems of high trial-and-error times and slow speed in autonomous vehicle path planning are solved, achieving more efficient path planning and improved passenger comfort.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2024-01-18
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies for autonomous vehicle path planning suffer from high trial-and-error times and slow path planning, and fail to effectively consider dynamic constraints.
The standardized advantage function path planning method is adopted. Through pre-training and dynamic constraints, the action value function is estimated iteratively using the Bellman equation. The vehicle speed, acceleration and jerk are set as constraints. The method is combined with the Q neural network for training to finally obtain the optimal planned path.
It reduces the number of trial and error attempts, improves the real-time performance of route planning and passenger comfort, and enhances the compatibility of route planning results with real-vehicle applications.
Smart Images
Figure CN117906629B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a path planning method for autonomous vehicles, and more particularly to a path planning method that takes into account dynamic constraints. Background Technology
[0002] Path planning is an indispensable and crucial part of autonomous vehicles, providing a tracking target for vehicle motion control. Path planning refers to the process by which an autonomous vehicle, based on its own intentions, current traffic conditions, and the intentions of surrounding vehicles, uses a series of methods to obtain a safe and efficient future trajectory. Current path planning methods can be divided into two types: local path planning and global path planning. Global path planning involves planning the entire path from the starting point to the destination, while local path planning refines a small segment of the global path plan.
[0003] A search revealed Chinese invention patent publication number CN 111273668 B, which discloses a motion trajectory planning system and method for autonomous vehicles on structured roads. The system includes a perception module, a positioning module, a lane change decision module, a motion planning module, and a trajectory tracking module. The lane change decision module outputs a decision action based on data collected by the perception and positioning modules. The motion planning module outputs the optimal trajectory to the trajectory tracking module based on the decision action. Specifically, the system includes the following steps:
[0004] Step 1: Collect real-time driving data;
[0005] Step 2: Perform coordinate transformation on the collected driving data, converting the vehicle data in the Frenet coordinate system to the vehicle data in the global coordinate system;
[0006] Step 3: Determine the state space, action space, and action value reward function;
[0007] Step 4: Construct a deep reinforcement learning model. This existing patent suffers from high trial-and-error rates and slow path planning.
[0008] How to implement vehicle path planning that takes into account dynamic constraints has become a technical problem that needs to be solved. Summary of the Invention
[0009] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a path planning method that considers dynamic constraints.
[0010] The objective of this invention can be achieved through the following technical solutions:
[0011] According to one aspect of the present invention, a path planning method considering dynamic constraints is provided, the method comprising the following steps:
[0012] Step S1: Pre-train the standardized advantage function path planning and save the network parameter values;
[0013] Step S2: Set dynamic constraints for the standardized advantage function path planning;
[0014] Step S3: Perform formal training on the standardized advantage function path planning to obtain the optimal parameters of the policy network;
[0015] Step S4: Utilize the results of the trained policy network to obtain the optimal planned speed of the vehicle in each iteration, and connect them in chronological order to obtain the optimal planned path of the vehicle.
[0016] Preferably, the pre-training of the standardized advantage function path planning specifically involves:
[0017] The action-value function is estimated iteratively using the Bellman equation, as shown in the following formula:
[0018] Q i+1 (s,a)=E s′ [r+γmaxQ i (s ′ ,a′)|s,a]
[0019] Where i represents the iteration step, r represents the reward value, γ represents the discount factor, s represents the state, i.e., the traffic conditions around the vehicle in the current planning step, a represents the action, i.e., the vehicle's speed along the road and perpendicular to the road in the current planning step, s′ represents the state of the next iteration step, a′ represents the action of the next iteration step, Q is the Q-value that reinforcement learning methods focus on; the higher the Q-value, the better the action. E s′ Q is the expected value. i Let Q be the Q value at the i-th iteration step. i+1 Let Q be the value at the (i+1)th iteration step;
[0020] The standardized advantage function method uses a neural network Q to estimate the action-value function, as shown in the following equation:
[0021] Q(s,a;θ)≈Q * (s,a)
[0022] Where θ represents the parameters of the Q-neural network, Q * This represents the maximum value of Q;
[0023] As the number of iterations i increases, Q i It gradually converges to the maximum value of Q. * According to the maximum value Q * To obtain the optimal action;
[0024] During pre-training, the vehicle speed is set to a constant value [v]. x0,v y0 By training the network, the following loss function is minimized:
[0025] L(θ)=(v x -v x0 ) 2 +(v y -v y0 ) 2
[0026] Where v x v is the speed of the vehicle along the road. y v is the speed of the vehicle along the direction perpendicular to the road. x0 Let v be the initial velocity of the vehicle along the road direction. y0 Let be the initial velocity of the vehicle along the direction perpendicular to the road.
[0027] When the pre-training termination condition is reached, the parameters θ of the Q-network are recorded and saved as θ. p .
[0028] Preferably, the dynamic constraints include limiting the vehicle's speed, acceleration, and the minimum and maximum values of jerk.
[0029] Preferably, the process of setting dynamic constraints for the standardized advantage function path planning includes the following steps:
[0030] Step S101: Constrain the speed according to the set speed range;
[0031] Step S102: Calculate acceleration using the finite difference method;
[0032] Step S103: Constrain the acceleration according to the set acceleration range;
[0033] Step S104: Randomly generate jerk according to the set jerk range;
[0034] Step S105: Calculate the demonstration speed.
[0035] More preferably, constraining the speed according to the set speed range specifically means:
[0036] Based on vehicle speed limits and traffic conditions, the optimal speeds for vehicles along the road and perpendicular to the road are respectively limited to [v]. x-min v x-max ] and [v y-min v y-max That is, to perform the following operations:
[0037] v x =min(v x-max v x) or max(v x-min v x )
[0038] v y =min(v y-max v y ) or r max(v y-min v y )
[0039] Among them, v x-min and v x-m Let v be the minimum and maximum values of the optimal speed of the vehicle along the road direction. y-m and v y-ma These are the minimum and maximum values of the optimal speed of the vehicle along the direction perpendicular to the road, respectively.
[0040] More preferably, the calculation of acceleration using the finite difference method specifically involves:
[0041] The vehicle acceleration is calculated as follows:
[0042]
[0043]
[0044] Among them, a x,i Let a be the acceleration of the vehicle along the road direction in the i-th iteration. y,i Let v be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, T represent the time period between adjacent iterations, and v x,i-1 v represents the vehicle's velocity along the road direction in the (i-1)th iteration. x,i v represents the velocity of the vehicle along the road direction in the i-th iteration; y,i-1 v represents the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. y,i This represents the speed of the vehicle along the direction perpendicular to the road in the i-th iteration.
[0045] More preferably, constraining the acceleration according to the set acceleration range specifically means:
[0046] The vehicle acceleration is subject to the following constraints:
[0047] a x =min(a x-max a x )or max(a x-min a x )
[0048] a y =min(a y-max a y )or max(a y-mina y )
[0049] Among them, a x and a y Let a represent the vehicle's acceleration along the road and along the direction perpendicular to the road, respectively. x-min and a x-m Let a be the minimum and maximum values of the vehicle's acceleration along the road direction. y-m and a y-m These are the minimum and maximum values of the vehicle's acceleration along the direction perpendicular to the road, respectively.
[0050] More preferably, the step of randomly generating jerk according to the set jerk range specifically involves:
[0051] To balance the continuity of vehicle speed and the jerk constraint, the jerk in each iteration step is set as follows:
[0052] aa x = (rand(0,1)-0.5) / 10
[0053] aa y = (rand(0,1)-0.5) / 10
[0054] Among them, aa x and aa y These represent the vehicle's acceleration along the road and along the direction perpendicular to the road, respectively, and rand(0,1) is a random number between 0 and 1.
[0055] More preferably, the calculation demonstration speed is specifically:
[0056] Set the demonstration action for the current output action of the Q network according to the following formula:
[0057] v x-e =v x,i-1 +a x,i *T+0.5*aa x,i-1 *T*T
[0058] v y-e =v y,i-1 +a y,i *T+0.5*aa y,i-1 *T*T
[0059] Where v x-e and v y-e v represents the demonstration speed of the vehicle along the road and along the direction perpendicular to the road, respectively. x,i-1 v represents the vehicle's velocity along the road direction in the (i-1)th iteration. y,i-1 Let a represent the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. x,iLet a be the acceleration of the vehicle along the road direction in the i-th iteration. y,i Let a be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, and T be the time period between adjacent iterations. x,i-1 and aa y,i-1 These are the accelerations of the vehicle along the road and along the direction perpendicular to the road, respectively, in the (i-1)th iteration.
[0060] Preferably, the formal training of the standardized advantage function path planning to obtain the optimal parameters of the target network and the policy network specifically involves: at the start of formal training, the initial value θ0 of the Q network parameters is the same as the Q network parameter value θ at the end of pre-training. p The standardized advantage function divides the Q-network of the current state and current action into a policy network Q and a target network Q. ‘ Where the policy network Q is a network trained in real time, and the target network Q is... ‘ The parameters are the historical values of the parameters of the policy network Q;
[0061] The structure of the policy network Q is as follows:
[0062] Q(s,a|θ Q )=A(s,a|θ Q )+V(s|θ V )
[0063]
[0064] Wherein, P(s|θ) P Let P(s|θ) be a positive definite square matrix related to the state, and let P(s|θ) be a positive definite square matrix related to the state. P )=L(s|θ P )L(s|θ P V(s|θ) V L(s|θ) P ) and μ(s|θ μ θ represents the outputs of neural networks V, L, and μ, respectively. A θ V θ P and θ μ Let L(s|θ) be the parameters of neural networks A, V, P, and μ, respectively. P ) is a lower triangular matrix with the same dimension as the action, μ(s|θ) μ () is a vector with the same dimension as the action;
[0065] In a=μ(s|θ) μ When ), the Q function reaches its maximum value;
[0066] In the standardized advantage function path planning process, the reward function r of the standardized advantage function is set as follows:
[0067] r = w o *r o +w g *r g
[0068] r o =(x n -x) 2 +(y n -y) 2
[0069] r g =-|xX|
[0070] Where, r o w represents the reward function related to the obstacle avoidance target. o r represents the weighting factor related to the obstacle avoidance target. g w represents the reward function related to reaching the destination. g x represents the weighting factor related to reaching the destination. n x and y represent the coordinates of the nearest vehicle and the coordinates of the vehicle along the road, respectively. n y and y represent the coordinates of the nearest vehicle and the coordinates of this vehicle in the direction perpendicular to the road, respectively;
[0071] After setting the reward function, the loss function for standardizing the advantage function is defined and calculated during the formal training process. The loss function is as follows:
[0072] L i (θ i )=(r+γ*Q t -Q p ) 2 +w x (v x -v x-e ) 2 +w y (v y -v y-e ) 2
[0073] Where L i Let θ represent the loss function for the i-th iteration. i Q is the parameter of the policy network in the i-th iteration, where γ is the discount factor, taking a value between 0 and 1; t The Q-value represents the target network. p The Q-value represents the policy network; w x Represents the speed v of vehicles along the road direction x With demonstration speed v x-e The weighting coefficient of the difference between them, w y Represents the vehicle speed v in the direction perpendicular to the roady With demonstration speed v y-e Weighting coefficients for the differences between them;
[0074] During training, the loss function is minimized using stochastic gradient descent. After a given number of iterations, the optimal parameters for the target network and the policy network are obtained.
[0075] Compared with the prior art, the present invention has the following beneficial effects:
[0076] 1. The standardized advantage function pre-training scheme used in this invention reduces the number of trial and error attempts and improves training speed and real-time performance of path planning compared to the traditional standardized advantage function path planning method;
[0077] 2. This invention adds dynamic constraints to the standardized advantage function path planning, improving passenger comfort and making the path planning results more compatible with real-vehicle applications. Attached Figure Description
[0078] Figure 1 This is a schematic diagram of the vehicle path planning scenario in this invention;
[0079] Figure 2 This is a flowchart of the standardized advantage function path planning process in this invention;
[0080] Figure 3 This is a flowchart of the dynamic constraint process in this invention. Detailed Implementation
[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0082] This embodiment relates to a path planning method considering dynamic constraints. This method takes into account the good adaptability of the standardized advantage function (SEM) method to changes in the environment. This invention performs global and local path planning based on the SEM method. To address the problem that the SEM method requires a large amount of trial and error, this invention designs a pre-training scheme to perform a warm start for the SEM. To address the problem that the SEM method cannot consider vehicle jerk constraints, acceleration constraints, and velocity constraints, this invention incorporates dynamic constraints during the SEM path planning training process.
[0083] like Figure 1 As shown, this demonstrates route planning for vehicle A in a traffic scenario.
[0084] like Figure 2 The diagram shows a flowchart of the vehicle path planning method considering dynamic constraints according to the present invention, which includes the following steps:
[0085] Step S201: Set the path planning status and actions.
[0086] First, the standardized advantage function path planning is pre-trained. The standardized advantage function is a type of reinforcement learning method. The core idea of reinforcement learning algorithms is to iteratively estimate the action-value function using the following Bellman equation: [Formula follows].
[0087] Q i+1 (s,a)=E s′ [r+γmaxQ i (s ′ ,a′)|s,a]
[0088] Where i represents the iteration step, r represents the reward value, γ represents the discount factor, s represents the state, i.e., the traffic conditions around the vehicle in the current planning step, a represents the action, i.e., the vehicle's speed along the road and perpendicular to the road in the current planning step, s′ represents the state of the next iteration step, a′ represents the action of the next iteration step, and E s′ Q represents the expected value. Q is the Q-value that reinforcement learning methods focus on; actions with higher Q-values are considered better. i Let Q be the Q value at the i-th iteration step. i+1 Let Q be the Q value at the (i+1)th iteration step.
[0089] The standardized advantage function method uses a neural network Q to estimate the action-value function, called a Q-network, as shown in the following equation:
[0090] Q(s,a;θ)≈Q * (s,a)
[0091] Where θ represents the parameters of the Q-neural network, Q * This represents the maximum value of Q;
[0092] As the number of iterations i increases, Q i It gradually converges to the maximum value of Q. * According to the maximum value Q * To obtain the optimal action.
[0093] Step S202: Set the pre-training loss function.
[0094] During pre-training, the vehicle speed is set to a constant value [v]. x0 ,v y0 Therefore, by training the network, the following loss function is minimized:
[0095] L(θ)=(v x -vx0 ) 2 +(v y -v y0 ) 2
[0096] Where v x v is the speed of the vehicle along the road. y v is the speed of the vehicle along the direction perpendicular to the road. x0 Let v be the initial velocity of the vehicle along the road direction. y0 Let be the initial velocity of the vehicle along the direction perpendicular to the road.
[0097] Step S203: Complete pre-training and retain network parameters.
[0098] When the pre-training termination condition is reached, the parameters θ of the Q-network are recorded and saved as θ. p .
[0099] Step S204: Constrain the velocity, acceleration, and jerk.
[0100] like Figure 3 As shown, the process of setting dynamic constraints for standardized advantage function path planning includes the following steps:
[0101] Step S101: Constrain the speed according to the set speed range;
[0102] Step S102: Calculate acceleration using the finite difference method;
[0103] Step S103: Constrain the acceleration according to the set acceleration range;
[0104] Step S104: Randomly generate jerk according to the set jerk range;
[0105] Step S105: Calculate the demonstration speed.
[0106] Based on vehicle speed limits and traffic conditions, the optimal speeds for vehicles along the road and perpendicular to the road are respectively limited to [v]. x-min v x-max ] and [v y-min v y-ma That is, to perform the following operations:
[0107] v x =min(v x-max v x ) or max(v x-min v x )
[0108] v y =min(v y-max vy ) or r max(v y-min v y )
[0109] Among them, v x-min and v x-max Let v be the minimum and maximum values of the optimal speed of the vehicle along the road direction. y-min and v y-max These are the minimum and maximum values of the optimal speed of the vehicle along the direction perpendicular to the road, respectively.
[0110] The vehicle acceleration is calculated as follows:
[0111]
[0112]
[0113] Among them, a x,i Let a be the acceleration of the vehicle along the road direction in the i-th iteration. y,i Let v be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, T represent the time period between adjacent iterations, and v x,i-1 v represents the vehicle's velocity along the road direction in the (i-1)th iteration. x,i v represents the velocity of the vehicle along the road direction in the i-th iteration; y,i-1 v represents the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. y,i This represents the speed of the vehicle along the direction perpendicular to the road in the i-th iteration.
[0114] For the sake of passenger comfort and considering the constraints of the vehicle's engine power source and braking system, the vehicle acceleration is subject to the following constraints:
[0115] a x =min(a x-max a x )or max(a x-min a x )
[0116] a y =min(a y-max a y )or max(a y-min a y )
[0117] Among them, a x and a y Let a represent the vehicle's acceleration along the road and along the direction perpendicular to the road, respectively. x-min and a x-max Let a be the minimum and maximum values of the vehicle's acceleration along the road direction. y-min and ay-max These are the minimum and maximum values of the vehicle's acceleration along the direction perpendicular to the road, respectively.
[0118] The vehicle's acceleration affects passenger comfort; the closer the acceleration is to zero, the more comfortable the passengers. Considering both the continuity of vehicle speed and the acceleration constraint, the acceleration in each iteration step is set as follows:
[0119] aa x = (rand(0,1)-0.5) / 10
[0120] aa y = (rand(0,1)-0.5) / 10
[0121] Among them, aa x and aa y These are the vehicle's acceleration along the road and along the direction perpendicular to the road, respectively. rand(0,1) is a random number between 0 and 1, which ensures that the vehicle's acceleration in both directions is between [-0.05, 0.05].
[0122] In addition, to ensure the continuity of vehicle speed, during formal training, the demonstration action for the current output action of the Q network is set according to the following formula:
[0123] v x-e =v x,i-1 +a x,i *T+0.5*aa x,i-1 *T*T
[0124] v y-e =v y,i-1 +a y,i *T+0.5*aa y,i-1 *T*T
[0125] Where v x-e and v y-e v represents the demonstration speed of the vehicle along the road and along the direction perpendicular to the road, respectively. x,i-1 v represents the vehicle's velocity along the road direction in the (i-1)th iteration. y,i-1 Let a represent the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. x,i Let a be the acceleration of the vehicle along the road direction in the i-th iteration. y,i Let a be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, and T be the time period between adjacent iterations. x,i-1 and aa y,i-1 These are the accelerations of the vehicle along the road and along the direction perpendicular to the road, respectively, in the (i-1)th iteration.
[0126] Step S205: Use the pre-trained and saved parameters as initial parameters for formal training.
[0127] At the start of formal training, the initial value of the Q-network parameters θ0 is the same as the value of the Q-network parameters θ at the end of pre-training. p The standardized advantage function divides the Q-network of the current state and current action into a policy network Q and a target network Q', where the policy network Q is a network trained in real time, and the parameters of the target network Q' are the historical values of the parameters of the policy network Q.
[0128] The structure of the policy network Q is as follows:
[0129] Q(s,a|θ Q )=A(s,a|θ Q )+V(s|θ V )
[0130]
[0131] Wherein, P(s|θ) P Let P(s|θ) be a positive definite square matrix related to the state, and let P(s|θ) be a positive definite square matrix related to the state. P )=L(s|θ P )L(s|θ P V(s|θ) V L(s|θ) P ) and μ(s|θ μ θ represents the outputs of neural networks V, L, and μ, respectively. A θ V θ P θ μ Let L(s|θ) be the parameters of neural networks A, V, P, and μ, respectively. P ) is a lower triangular matrix with the same dimension as the action, μ(s|θ) μ () is a vector with the same dimension as the action;
[0132] In a=μ(s|θ) μ When ), the Q function reaches its maximum value;
[0133] The maximum value of the Q function is at a = μ(s|θ) μ (Obtained at that time)
[0134] During the route planning process, the planned route for this vehicle should avoid collisions with other surrounding vehicles and should follow the direction of the road.
[0135] In the standardized advantage function path planning process, the reward function r of the standardized advantage function is set as follows:
[0136] r = w o *r o+w g *r g
[0137] r o =(x n -x) 2 +(y n -y) 2
[0138] r g =-|xX|
[0139] Where, r o w represents the reward function related to the obstacle avoidance target. o r represents the weighting factor related to the obstacle avoidance target. g w represents the reward function related to reaching the destination. g x represents the weighting factor related to reaching the destination. n x and y represent the coordinates of the nearest vehicle and the coordinates of the vehicle along the road, respectively. n y and y represent the nearest vehicle to this vehicle and the coordinates of this vehicle in the direction perpendicular to the road, respectively.
[0140] After setting the reward function, the loss function for standardizing the advantage function is defined and calculated during the formal training process. The loss function is as follows:
[0141] L i (θ i )=(r+γ*Q t -Q p ) 2 +w x (v x -v x-e ) 2 +w y (v y -v y-e ) 2
[0142] Where L i Let θ represent the loss function for the i-th iteration. i Q is the parameter of the policy network in the i-th iteration, where γ is the discount factor, taking a value between 0 and 1; t The Q-value represents the target network. p The Q-value represents the policy network; w x Represents the speed v of vehicles along the road direction x With demonstration speed v x-e The weighting coefficient of the difference between them, w y Represents the vehicle speed v in the direction perpendicular to the road y With demonstration speed v y-e Weighting coefficients for the differences between them;
[0143] During training, the loss function is minimized using stochastic gradient descent.
[0144] Step S206: Determine whether the set number of iterations has been reached. If yes, proceed to step S207; otherwise, return to step S204.
[0145] Step S207: Save network parameters.
[0146] After reaching a given number of iterations, the optimal parameters for the target network and the policy network are obtained.
[0147] Step S208 yields the optimal planned path.
[0148] After training, the optimal action, which is the vehicle's best planned speed in each iteration step, is obtained by using the calculation results of the μ network in the policy network. By connecting all iteration steps in chronological order, the optimal path of the vehicle can be obtained.
[0149] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A path planning method considering dynamic constraints, characterized in that, The method includes the following steps: Step S1: Pre-train the standardized advantage function path planning and save the network parameter values; Step S2: Set dynamic constraints for the standardized advantage function path planning; Step S3: Perform formal training on the standardized advantage function path planning to obtain the optimal parameters of the policy network; Step S4: Utilize the results of the trained policy network to obtain the optimal planned speed of the vehicle in each iteration, and connect them in chronological order to obtain the optimal planned path of the vehicle. The process of setting dynamic constraints for the standardized advantage function path planning includes the following steps: Step S101: Constrain the speed according to the set speed range; Step S102: Calculate acceleration using the finite difference method; Step S103: Constrain the acceleration according to the set acceleration range; Step S104: Randomly generate jerk according to the set jerk range; Step S105: Calculate the demonstration speed; The specific steps for constraining the speed according to the set speed range are as follows: Based on vehicle speed limits and traffic conditions, the optimal speeds for vehicles in the road direction and perpendicular to the road direction are respectively limited to […]. , ]and[ That is, to perform the following operations: in, and These are the minimum and maximum values of the optimal speed for the vehicle along the road direction, respectively. and These are the minimum and maximum values of the optimal speed of the vehicle along the direction perpendicular to the road, respectively. The speed of the vehicle along the road direction. The speed of the vehicle in the direction perpendicular to the road; The specific method for calculating acceleration using the finite difference method is as follows: The vehicle acceleration is calculated as follows: in, Let be the acceleration of the vehicle along the road direction in the i-th iteration. Let be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, and T represent the time period between adjacent iterations. This represents the vehicle's speed along the road direction in the (i-1)th iteration. This represents the speed of the vehicle along the road direction in the i-th iteration; This represents the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. This represents the vehicle's velocity along the direction perpendicular to the road in the i-th iteration; The specific steps for constraining acceleration according to the set acceleration range are as follows: The vehicle acceleration is subject to the following constraints: in, and These represent the vehicle's acceleration along the road and along the direction perpendicular to the road, respectively. These are the minimum and maximum values of the vehicle's acceleration along the road direction, respectively. These are the minimum and maximum values of the vehicle's acceleration along the direction perpendicular to the road, respectively. The specific method for randomly generating jerk according to the set jerk range is as follows: To balance the continuity of vehicle speed and the jerk constraint, the jerk in each iteration step is set as follows: in, and These are the vehicle's accelerations along the road and perpendicular to the road, respectively. A random number between 0 and 1; The specific calculation demonstration speed is as follows: Set the current output action of the Q network as follows: in and These represent the demonstration speeds of the vehicle along the road and along the direction perpendicular to the road, respectively. This represents the vehicle's speed along the road direction in the (i-1)th iteration. This represents the vehicle's velocity along the direction perpendicular to the road in the (i-1)th iteration. Let be the acceleration of the vehicle along the road direction in the i-th iteration. Let be the acceleration of the vehicle along the direction perpendicular to the road in the i-th iteration, and T be the time period between adjacent iterations. and These are the accelerations of the vehicle along the road and along the direction perpendicular to the road, respectively, in the (i-1)th iteration.
2. The path planning method considering dynamic constraints according to claim 1, characterized in that, The pre-training of the standardized advantage function path planning specifically involves: The action-value function is estimated iteratively using the Bellman equation, as shown in the following formula: in Represents the number of iterations. Represents the reward value. Represents the discount factor. The representative state refers to the traffic conditions around the vehicle in the current planned area. The representative action refers to the vehicle's speed along the road and perpendicular to the road during the current planned step. Represents the state of the next iteration step. This represents the action for the next iteration step. To enhance the learning method's focus on the Q-value, actions with higher Q-values are considered superior. Expected value Let Q be the value of the i-th iteration step. Let Q be the value at the (i+1)th iteration step; The standardized advantage function method uses a neural network Q to estimate the action-value function, as shown in the following equation: in The parameters representing the Q-neural network, This represents the maximum value of Q; With the number of iterations The increase, Gradually converges to the maximum value of Q. According to the maximum value To obtain the optimal action; During pre-training, the vehicle speed is set to a constant value. By training the network, the following loss function is minimized: in The speed of the vehicle along the road direction. The speed of the vehicle in the direction perpendicular to the road. Let be the initial velocity of the vehicle along the road direction. Let be the initial velocity of the vehicle along the direction perpendicular to the road. When the pre-training termination condition is reached, the parameters of the Q-network are recorded and saved. for .
3. The path planning method considering dynamic constraints according to claim 1, characterized in that, The aforementioned dynamic constraints include limits on the vehicle's speed, acceleration, and minimum and maximum jerk.
4. The path planning method considering dynamic constraints according to claim 1, characterized in that, The formal training of the standardized advantage function path planning to obtain the optimal parameters of the target network and the policy network specifically involves: initial values of the Q-network parameters at the start of formal training. The Q-network parameter values at the end of pre-training The standardized advantage function divides the Q-network of the current state and current action into a policy network Q and a target network. Where the policy network Q is a network trained in real time, and the target network... The parameters are the historical values of the parameters of the policy network Q; The structure of the policy network Q is as follows: in, Let be a positive definite square matrix that is related to the state, and ; and These are neural networks V, L, and 3, respectively. The output, , , and These are neural networks A, V, P, and [other neural networks]. The parameters, A lower triangular matrix with the same dimension as the action. A vector with the same dimension as the action; exist = At that time, the Q function reaches its maximum value; In the standardized advantage function path planning process, the reward function r of the standardized advantage function is set as follows: in, This represents the reward function related to the obstacle avoidance objective. Represents the weighting factors related to the obstacle avoidance target. This represents the reward function related to reaching the destination. Represents the weighting factors related to reaching the destination. and These represent the coordinates of the nearest vehicle and the coordinates of this vehicle along the road, respectively. and These represent the coordinates of the nearest vehicle and the coordinates of this vehicle in the direction perpendicular to the road, respectively. After setting the reward function, the loss function for standardizing the advantage function is defined and calculated during the formal training process. The loss function is as follows: in This represents the loss function for the i-th iteration. These are the parameters of the policy network in the i-th iteration. This is the discount factor, with a value between 0 and 1; The Q-value represents the target network. The Q-value represents the policy network; Represents the speed of vehicles along the road direction With demonstration speed The weighting coefficient of the difference between them Represents vehicle speed in the direction perpendicular to the road With demonstration speed Weighting coefficients for the differences between them; During training, the loss function is minimized using stochastic gradient descent. After a given number of iterations, the optimal parameters for the target network and the policy network are obtained.