Crossroad multi-vehicle double-layer scheduling method

Abstract

The invention discloses a crossroad multi-vehicle double-layer scheduling method, and relates to the technical field of intelligent traffic, and the method comprises the steps: constructing a crossroad and vehicle motion model; an ATTD3 algorithm is established and trained, and an upper-layer scheduling strategy is designed by using the ATTD3 algorithm; after a feasible track of the networked autonomous vehicle is obtained according to an upper-layer scheduling strategy, track planning under a Frenet coordinate system is carried out, a human-driven vehicle or obstacle model is established by using a risk field, and a lower-layer local track re-planning strategy is designed; and carrying out crossroad multi-vehicle scheduling based on the upper-layer scheduling strategy and the lower-layer local trajectory re-planning strategy. According to the method, the ATTD3 algorithm is constructed when the upper-layer scheduling strategy is designed, the reasonable loss function is designed for the newly introduced attention network, the algorithm convergence speed is increased, meanwhile, the lower-layer local track re-planning strategy simulates human driving vehicles or obstacles through the risk field, and local re-planning can be accurately conducted on the vehicle driving track in real time.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and more specifically, to a multi-vehicle double-layer scheduling method at an intersection based on deep reinforcement learning and risk fields. Background Art

[0002] The intelligent transportation system introduces connected autonomous vehicles (CAVs) and vehicle-to-everything (V2X) technology, which can realize real-time information exchange between vehicles (V2V), vehicles and infrastructure (V2I), vehicles and pedestrians (V2P), and vehicles and networks (V2N). This provides a new solution for the dispatch of intersections without signal lights, and optimizes the travel path and travel time of vehicles by dynamically sensing traffic flow and environmental information, thereby improving traffic efficiency and traffic safety.

[0003] At present, most of the research on multi-vehicle scheduling at intersections without signal lights focuses on centralized scheduling methods, that is, using the global information of the entire intersection to centrally organize the movement of all approaching vehicles. In this type of scheduling method, the amount of computation usually increases rapidly with the increase in the number of vehicles, and needs to be carefully determined to ensure the feasibility of real-time implementation. In addition, most of the current research only considers connected autonomous vehicles, but at present, there is still a distance to achieve fully connected autonomous driving, so it is necessary to consider human-driven vehicles or other obstacles when implementing multi-vehicle scheduling.

[0004] The invention with application publication number CN117373249A discloses a vehicle dispatching method and system for a two-way two-lane unsignalized intersection, including: step S1: pre-processing the vehicle according to the given real two-way two-lane cross intersection vehicle driving conditions, judging whether there will be a collision between vehicles, and analyzing the conflict relationship between vehicles; step S2: constructing a vehicle dispatching model based on deep search of conflict relationships; step S3: using nonlinear programming to realize the action decision of the vehicle, and finding the optimal passing time and vehicle driving acceleration; according to the vehicle dispatching model, performing lower-level planning, and designing the unsignalized intersection vehicle dispatching implementation strategy; step S4: building a cross intersection simulation scene according to the given real cross intersection scene, and initializing the vehicle information of each vehicle; step S5: inputting the vehicle information into the vehicle dispatching model, making behavioral decisions, and obtaining the action output of each vehicle at the next moment. The invention can improve the traffic efficiency of networked autonomous driving vehicles and reduce the probability of accidents, but it cannot reduce the complexity and calculation amount of the dispatching algorithm, and ensure real-time dispatch while considering the influence of human-driven vehicles or obstacles to adjust the local trajectory planning. Summary of the invention

[0005] In order to overcome the defects of the above-mentioned prior art in reducing the complexity and calculation amount of the scheduling algorithm and ensuring real-time scheduling while considering the influence of human-driven vehicles or obstacles to adjust local trajectory planning, the present invention provides a double-layer scheduling method for multiple vehicles at an intersection.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention includes:

[0007] Construct intersection models and vehicle motion models;

[0008] Based on the intersection model and vehicle motion model, an ATTD3 algorithm model is established and trained, and the ATTD3 algorithm model is used to design an upper-layer multi-networked autonomous driving vehicle scheduling strategy;

[0009] After each connected autonomous driving vehicle obtains a feasible trajectory according to the upper-layer multi-connected autonomous driving vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in a Frenet coordinate system, and a human-driven vehicle model is established using a risk field, and a local trajectory replanning strategy for the lower-layer connected autonomous driving vehicle is designed based on the trajectory planning in the Frenet coordinate system and the human-driven vehicle model established using the risk field;

[0010] Multi-vehicle scheduling at an intersection is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0011] Furthermore, the intersection is a two-way two-lane intersection without traffic lights. The map of the intersection model is drawn based on the Lanelet2 framework. For the right lane, the vehicle can choose to turn right or go straight, and for the left lane, the vehicle can choose to turn left or go straight.

[0012] Furthermore, it is stipulated that each connected autonomous driving vehicle follows a predetermined path in the lane after entering the intersection. The movement of the vehicle can be decomposed into movement on the x-axis and the y-axis. The vehicle movement model is expressed as the following formula:

[0013] v i (t+1)=v i (t)+a i (t)τ

[0014] x i (t+1)=x i (t)+v i (t+1)cosθτ

[0015] y i (t+1)=y i (t)+v i (t+1)sinθτ

[0016] Among them, θ is the heading angle of the vehicle, which is calculated by the current position of the vehicle and the position of the target point, τ is the control time step, and v i and a i are the acceleration and velocity of the vehicle, x i and i It is the position of the vehicle in both the x-axis and y-axis directions.

[0017] Furthermore, the establishment of the ATTD3 algorithm model and training includes the following steps:

[0018] Based on the intersection model and the vehicle motion model, a Markov decision process MDP model is established;

[0019] According to the MDP model, a TD3 algorithm model is constructed;

[0020] The attention mechanism Self-Attention is introduced into the TD3 model, and the ATTD3 algorithm model is constructed and trained.

[0021] Furthermore, the Markov decision process MDP model consists of a tuple (S, A, P, E, γ), where S is the state space, A is the action space, P is the state transition probability, R is the reward function, and γ∈(0,1) is the discount factor;

[0022] Model-free reinforcement learning is used, and the state transition probability P is not set. The state space S includes the position and speed information of each vehicle. The networked autonomous driving vehicle travels on a predetermined path, and the action space A is the acceleration of each vehicle.

[0023] Considering the three aspects of vehicle driving efficiency, vehicle driving comfort and avoiding collisions between vehicles, the following reward function is set:

[0024]

[0025] Among them, V current is the current speed of the vehicle, V max is the maximum speed of the vehicle;

[0026]

[0027] Among them, a okd is the last acceleration, a new is the current acceleration;

[0028]

[0029] Among them, dist is the sum of the distances between the colliding vehicles when the vehicles collide;

[0030]

[0031] r total =ω eff *r eff +ω com *r com +ω col *r col +ω rea r rea

[0032] Among them, r total is the total reward, r eff is the reward function for vehicle driving efficiency, r com is the reward function for vehicle driving comfort, r col is the reward function for avoiding collisions between vehicles, r rea is the reward function for task completion, ω eff ,ω com ,ω col ,ω rea are the corresponding reward coefficients respectively.

[0033] Furthermore, the construction of the TD3 algorithm model according to the MDP model includes the following steps:

[0034] The TD3 algorithm consists of six networks: Actor network π(·|φπ ), Critic1 Network Critic2 Network Target Actor Networkπ′(·|φ π′ ), Target Critic1 Network and Target Critic2 network

[0035] At each time step, get the current agent state s, and select action a from the Actor network based on the current state s;

[0036] Execute the corresponding action to obtain the corresponding reward r and target state s′, and store the tuple (s, a, r, s′) in the experience replay buffer;

[0037] Randomly sample a small batch of experience (s, a, r, s′) in the experience replay buffer, with a batch size of N, and generate the target action a′ from the Target Actor network in turn;

[0038] Calculate the target Q value y from the Target Critic1 and Target Critic2 networks according to the Bellman formula;

[0039] Calculate TD-error as the loss function of Critic1 and Critic2 networks;

[0040] Update the Critic network parameters by minimizing the loss function of the Critic1 and Critic2 networks;

[0041] For the updates of the Actor network and the target network, the Actor network uses delayed updates, and the target network uses delayed updates and soft updates at the same time. After every n steps, the loss function of the Actor network is first calculated;

[0042] Update the Actor network parameters by minimizing the Actor network's loss function;

[0043] Furthermore, the construction of the ATTD3 algorithm model and training includes:

[0044] At each time step, get the current agent state s, and select action a from the Actor network based on the current state s b ;

[0045] Extract the features of the Actor network, Critic1 and Critic2 networks to obtain the Actor features, Critic1 features and Critic2 features;

[0046] The corresponding Critic feature is obtained by solving the average sum of Critic1 feature and Critic2 feature;

[0047] Use the Actor feature as the Query and Value in the attention mechanism, the Critic feature as the Key, and output the Attention Actor feature through the self-attention mechanism network;

[0048] Input the Attention Actor feature into the output layer of the Actor network to get a new action a t ;

[0049] According to the action a output by the normal Actor network b And the action a after the attention mechanism t , calculate the Q value of the two actions respectively;

[0050] Finally, the action corresponding to the larger Q value is selected as the corresponding output action a;

[0051] The subsequent training process is consistent with the TD3 algorithm model. A loss function is set for the new attention network layer introduced when the attention mechanism is introduced. The loss function is designed as follows:

[0052]

[0053] Among them, the attention mechanism network layer update is synchronized with the update of the Actor network and the target network.

[0054] Furthermore, the trajectory planning of the feasible trajectory in the Frenet coordinate system includes:

[0055] The trajectory planning in the Cartesian coordinate system is converted to the Frenet coordinate system. In the Frenet coordinate system, the direction along the reference trajectory is the longitudinal direction s, and the direction perpendicular to the reference trajectory is the lateral direction d. The motion state of the vehicle is

[0056] The trajectory planning in the Frenet coordinate system is divided into lateral planning and longitudinal planning, and the networked autonomous driving vehicle is required to travel along the predetermined path;

[0057] For lateral planning, the initial time configuration conditions, target time configuration conditions, sampling intervals, and lateral offsets are set to generate different lateral trajectories and obtain the cost function of the lateral trajectory;

[0058] For longitudinal planning, set the initial time configuration conditions, target time configuration conditions, sampling intervals, and speed intervals to generate different longitudinal trajectories and obtain the cost function of the longitudinal trajectory;

[0059] The total cost of the trajectory is as follows:

[0060] cost total =ω lon Cost s +ω lat Cost d

[0061] Among them, Cost s Cost d are the costs of vertical planning and horizontal planning respectively, ω lon ,ω lat is the corresponding coefficient;

[0062] After obtaining many lateral and longitudinal trajectories at each time step, determine whether the trajectory at each time step meets all the conditions that the speed is less than the maximum speed, the acceleration is less than the maximum acceleration, the curvature is less than the maximum curvature, and there is no collision with human-driven vehicles. Calculate the cost of each trajectory that meets all conditions, and select the trajectory with the smallest cost as the optimal trajectory for the current time step.

[0063] Further, the defining of the risk field and establishing a human driving vehicle model using the risk field includes:

[0064] The risk field function is defined as follows:

[0065]

[0066]

[0067] Among them, the subscripts x and y represent the x direction and y direction, the non-subscript x represents the x position of the calculation point, and the non-subscript y represents the y position of the calculation point. vehicle is the vehicle length, W vehicle is the vehicle width, β x , β y is the corresponding coefficient, α x , α y is the acceleration in the x and y directions, v x (t), v y (t) is the velocity in the x and y directions.

[0068] Setting risk threshold and risk field function R vehicle The values ​​of (x, y, t) are compared to determine whether the connected autonomous vehicle is likely to collide with a human-driven vehicle or an obstacle.

[0069] Furthermore, the method also includes a simulation step, by setting various parameters of the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy, simulation is performed in the intersection model to verify the reliability of the multi-vehicle double-layer scheduling method at the intersection.

[0070] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0071] The present invention uses the deep reinforcement learning algorithm TD3 and the attention mechanism when designing the upper-level multi-networked autonomous driving vehicle scheduling strategy, thereby constructing the ATTD3 algorithm and designing a reasonable loss function for the newly introduced attention network, thereby improving the convergence speed of the algorithm. At the same time, the local trajectory replanning strategy of the lower-level networked autonomous driving vehicle takes into account the influence of human-driven vehicles or obstacles, and uses the risk field to simulate human-driven vehicles or obstacles with different driving styles, which can accurately and real-timely replan the vehicle's driving trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0073] Figure 1 A framework diagram of the multi-vehicle double-layer scheduling method at an intersection of the present application;

[0074] Figure 2 A schematic diagram of a crossroads map in Lanelet2 format for this application;

[0075] Figure 3 The ATTD3 algorithm and TD3 algorithm training result diagram of Example 3 of the present application;

[0076] Figure 4 Outputting an acceleration graph for the upper strategy of Example 3 of the present application;

[0077] Figure 5 The upper layer strategy output speed map of Example 3 of the present application;

[0078] Figure 6 Schematic diagram of the risk field of human-driven vehicles for the lower-level strategy of Example 3 of the present application.

[0079] Figure 7 This is a result diagram of the lower-level strategy trajectory planning of Example 3 of the present application;

[0080] Figure 8 This is the trajectory replanning sd map of vehicle 1 and vehicle 3 in embodiment 3 of the present application. DETAILED DESCRIPTION

[0081] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0082] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0083] Example 1

[0084] See also Figure 1 The method framework diagram of this embodiment, a multi-vehicle double-layer scheduling method for an intersection in this embodiment of the application, includes the following steps:

[0085] S1: Construct intersection model and vehicle motion model;

[0086] S2: Based on the intersection model and vehicle motion model, the ATTD3 algorithm model is established and trained, and the ATTD3 algorithm model is used to design the upper-layer multi-network autonomous driving vehicle scheduling strategy;

[0087] S3: After each connected autonomous driving vehicle obtains a feasible trajectory according to the upper-layer multi-connected autonomous driving vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in a Frenet coordinate system, a risk field is defined and a human driving vehicle model is established using the risk field, and a local trajectory replanning strategy for the lower-layer connected autonomous driving vehicle is designed based on the trajectory planning in the Frenet coordinate system and the establishment of a human driving vehicle model using the risk field;

[0088] S4: Multi-vehicle dispatch at intersections based on the upper-layer multi-networked autonomous driving vehicle dispatch strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0089] In this embodiment 1, step S1 is specifically as follows:

[0090] Intersection Modeling: See Figure 2The scenario studied in this paper is a two-lane intersection without a signal light. For the right lane, the vehicle chooses to turn right or go straight, and for the left lane, the vehicle chooses to turn left or go straight. The map of the intersection is drawn according to the Lanelet2 framework. Since the centralized scheduling mode is adopted, there is a scheduling node, which is implemented on the road side unit (RSU) and collects all vehicle information entering the intersection through the vehicle-to-infrastructure V2I communication link. The real-time information of all networked autonomous driving vehicles CAV will be collected on the centralized coordinator and the coordinator will output the real-time instructions of each CAV.

[0091] Establishment of vehicle motion model: It is stipulated that each CAV will follow the established path of the lane after entering the intersection. The motion of the vehicle can be decomposed into the motion on the x-axis and the y-axis. The motion model of the vehicle can be simplified as follows:

[0092] v i (t+1)=v i (t)+a i (t)τ

[0093] x i (t+1)=x i (t)+v i (t+1)cosθτ

[0094] y i (t+1)=y i (t)+v i (t+1)sinθτ

[0095] Where θ is the heading angle of the vehicle, which can be calculated from the current position of the vehicle and the position of the target point, τ is the control time step, and v i and a i are the acceleration and velocity of the vehicle, x i and i is the position of the vehicle in both directions.

[0096] Step S2 specifically includes the following steps:

[0097] MDP model establishment:

[0098] The Markov decision process (MDP) consists of a tuple (S, A, P, R, γ), where S is the state space, A is the action space, P is the state transition probability, R is the reward function, and γ∈(0,1) is the discount factor. Since the present invention adopts model-free reinforcement learning, it is not necessary to set the state transition probability P.

[0099] The state space should include the position and velocity information of each vehicle, that is,

[0100] S=(x1,y1,v1,x2,y2,v2,…,x n ,y n ,v n )

[0101] For a given reference path, we only need to control the acceleration of the vehicle, which can greatly reduce the dimension of the space and reduce the training cost and complexity. Therefore, the action space should be the acceleration of each vehicle, that is,

[0102] A=(a1,a2,…,a n )

[0103] When setting the reward function R, we should consider the three aspects of vehicle driving efficiency, vehicle driving comfort, and avoiding collisions between vehicles. In addition, we should also set a sparse reward for task completion. So the reward function is set as follows:

[0104]

[0105] Among them, V current is the current speed of the vehicle, V max is the maximum speed of the vehicle;

[0106]

[0107] Among them, a old is the last acceleration, a new is the current acceleration;

[0108]

[0109] Among them, dist is the sum of the distances between the colliding vehicles when the vehicles collide;

[0110]

[0111] r total =ω eff *r eff +ω com *r com +ω col *r col +ω rea r rea

[0112] where r eff is a reward function for vehicle driving efficiency, which requires CAV to exit the intersection as soon as possible after entering the intersection to improve the efficiency of coordination; and r comIt is a function of vehicle driving comfort, which requires that CAV should not have too many acceleration mutations to maintain driving stability and riding comfort; col It is a function to avoid collision between vehicles. If a collision occurs, a large negative value is given, which is a fixed value plus the sum of the distance between the vehicles when the collision occurs. rea is the function of task completion, and the final total reward r total It is the weighted sum of the above four rewards.

[0113] According to the established MDP model, the ATTD3 algorithm model is established:

[0114] The ATTD3 (Attention Twin Delayed Deep Deterministic Policy Gradient) algorithm combines the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm with the Self-Attention mechanism. By introducing the attention mechanism, the agent can learn more valuable strategies faster. The TD3 algorithm is a deterministic deep reinforcement learning algorithm under the actor-critic (AC) framework. It combines the deep deterministic policy gradient algorithm and dual Q learning. The TD3 algorithm consists of six networks, namely, the Actor network π(·|φ π ), Critic1 network Critic2 Network Target Actor Networkπ′(·|φ π′ ), Target Critic1 network Target Critic2 Network

[0115] Training TD3:

[0116] In each time step, first obtain the current agent state s, and select action a from the Actor network according to the current state s, that is,

[0117] a=clip(π(s|φ π )+∈,-a max ,a max )

[0118] in To explore noise, the goal is to encourage the agent to take risks and try new behaviors that may bring higher rewards, rather than always choosing behaviors that are known to bring current high rewards. max is the maximum acceleration.

[0119] After executing the corresponding action, the corresponding reward r and target state s′ are obtained and the tuple (s, a, r, s′) is stored in the experience replay buffer.

[0120] A small batch of experiences (s, a, r, s′) is randomly sampled in the experience replay buffer, with a batch size of N, and the target action a′ is generated from the Target Actor network in turn, that is,

[0121] a′=clip(π′(s′|φ π′ )+∈′,-a max ,a max )

[0122] in c∈(0,1) is the noise clipping ratio, the purpose of this noise is to enhance the robustness of the network and improve the accuracy of the estimation, a max is the maximum acceleration.

[0123] Then the target Q value y is calculated from the Target Critic1 and Target Critic2 networks according to the Bellman formula, that is,

[0124]

[0125] Then calculate TD-error as the loss function of Critic1 and Critic2 networks, that is,

[0126]

[0127] Update the Critic network parameters by minimizing the loss function:

[0128]

[0129] For the update of the Actor network and the target network, the TD3 algorithm uses delayed update, so after every n steps, the loss function of the Actor network is first calculated, that is,

[0130]

[0131] Update the Actor network parameters by minimizing the loss function:

[0132]

[0133] The target network update also uses soft update, that is:

[0134] φ π′ ←τφ π +(1-τ)φ π′

[0135]

[0136] From the above TD3 training process, we can see that in the TD3 algorithm, the Actor network and the Critic network are independent. The Critic provides the Actor with a benchmark for evaluating actions based on the Q value, but the Actor cannot clearly learn why a certain action has a large Q value and a certain action has a small Q value. Considering that actions and values ​​exist in the form of feature vectors before generation, and feature vectors constitute different dimensions of actions and values, the introduction of the attention mechanism can correctly match the correlation of different vector dimensions, modify the features, and obtain better actions.

[0137] The principle of the attention mechanism is as follows:

[0138]

[0139] In TD3, each network can be decomposed into a feature layer and an output layer. The network receives input, passes through the feature layer, and outputs features to the output layer, which then outputs the final result. π ) can be expressed as

[0140]

[0141] In the Critic network, Critic1 and Critic2 can be expressed as

[0142]

[0143] By extracting the features of the Actor network and the Critic network and adding them to the attention mechanism. When selecting an action, the original Actor network Actor feature outputs an action, and then the AttentionActor feature obtained through the attention mechanism outputs another action, and the Critic network compares and selects the action with higher value. Then, by setting a suitable loss function for the attention network to guide the attention network to output better Actor features, the present invention adopts the self-attention mechanism to achieve this purpose.

[0144] After introducing the self-attention mechanism into the TD3 algorithm, the ATTD3 algorithm model is established and trained:

[0145] At each time step, the current agent state s is also obtained, and action a is selected from the Actor network according to the current state s b ,Right now

[0146] a b =clip(π(s|φ π )+∈,-a max ,amax )

[0147] Then extract the features of the Actor network, Critic1 and Critic2 networks to obtain the Actor features, Critic1 features and Critic2 features, and then obtain the corresponding Critic features by averaging the Critic1 features and Critic2 features. At this time, the Actor features are used as the Query and Value, the Critic features are used as the Key, and the Attention Actor features are output through the self-attention mechanism network, that is,

[0148]

[0149] in Represents the Attention Actor feature, is the Actor feature, is the averaged and summed critic feature. Finally, the Attention Actor feature is input into the output layer of the Actor network to obtain the new action a t ,Right now

[0150]

[0151] At this point we get an action a output by the normal Actor network b And the action a after the attention mechanism t , calculate the Q value of the two actions respectively, that is,

[0152]

[0153] Finally, the action corresponding to the larger Q value is selected as the corresponding output action a.

[0154] The subsequent training process is consistent with the above TD3, but because a new network layer is introduced when implementing the attention mechanism, it is necessary to set a loss function for the new attention network layer. In order to allow the attention mechanism to better guide the Actor to output actions with higher value, the loss function is designed as follows:

[0155]

[0156] Attention network layer updates are synchronized with the Actor network and the target network

[0157] Using the ATTD3 algorithm model, design the upper-level multi-networked autonomous driving vehicle scheduling strategy.

[0158] Through the upper-layer multi-networked autonomous driving vehicle scheduling strategy, each CAV can obtain a feasible trajectory. However, in real-world scenarios, in addition to autonomous vehicles, there are also human driving vehicles (HDVs) or obstacles at the intersection. Therefore, in some conflicting situations, it is necessary to replan the trajectory locally. Therefore, step S3 is specifically as follows:

[0159] Conversion between Frenet coordinate system and Cartesian coordinate system:

[0160] Since the Frenet coordinate system can project the trajectory of the vehicle at a certain moment onto the reference trajectory and decompose it into two dimensions of motion, horizontal and vertical, it can simplify the subsequent trajectory planning work. Therefore, it is necessary to convert the trajectory planning in the Cartesian coordinate system to the Frenet coordinate system. In the rectangular coordinate system, the position point on the trajectory can be p = (x, y), but the Frenet coordinate system is established based on the reference trajectory. In this reference system, the direction along the reference trajectory is the longitudinal axis s, and the direction perpendicular to the reference trajectory is the transverse axis d. The motion state of the vehicle in the Frenet coordinate system can be described as The conversion formula between Frenet coordinate system and Cartesian coordinate system is as follows:

[0161]

[0162] Perform trajectory planning in the Frenet coordinate system:

[0163] Trajectory planning in the Frenet coordinate system can be divided into lateral planning and longitudinal planning. The lateral planning is mainly responsible for tasks such as obstacle avoidance and lane change, while the longitudinal planning can be flexibly configured according to the actual driving scenario of the vehicle. The final output trajectory consists of s(t) and d(t), where s(t) and d(t) are both polynomials about time.

[0164] The polynomial of horizontal planning can be expressed by a 5th-order polynomial, namely:

[0165]

[0166] Set the condition to the initial time t0 configuration And the target time t1 configuration It is stipulated that the vehicle should follow the reference trajectory as much as possible, so there is By setting the sampling interval ΔT and the lateral offset Δd, different lateral trajectories are generated.

[0167] The cost function for the lateral trajectory is as follows:

[0168]

[0169] in, It is the third-order derivative in the d direction, which is used to measure the severity of the change in acceleration in the d direction. is a measure of the degree of deviation from the trajectory of the path, ω j and ω d is the corresponding coefficient.

[0170] The polynomial of longitudinal planning can be expressed by a 4-degree polynomial, namely:

[0171]

[0172] Set the condition to the initial time t0 configuration And the target time t1 configuration Generally speaking, the position along the reference line direction, configuration s(t1) does not need to be considered. Different longitudinal trajectories are generated by setting the sampling interval ΔT and the velocity interval Δv.

[0173] The cost function of the longitudinal trajectory is as follows:

[0174]

[0175] in, It is the third-order derivative of s, representing the acceleration change in the s direction, and is used to measure the driving comfort of the vehicle. It is the square of the difference between the first-order derivative of s and the reference velocity, which is used to measure the difference between the velocity in the s direction of the new trajectory and the velocity of the reference trajectory. j and ω d is the corresponding coefficient.

[0176] The total cost of the trajectory is calculated as follows:

[0177] cost total =ω lon Cost s +ω lat Cost d

[0178] Among them, ω lon ,ω lat is the corresponding coefficient.

[0179] So far, many lateral and longitudinal trajectories have been obtained at each time step. The optimal trajectory selection conditions for each time step include the following aspects: speed is less than the maximum speed, acceleration is less than the maximum acceleration, curvature is less than the maximum curvature, and there is no collision with human-driven vehicles. Among the trajectories that meet all the above conditions, the cost of each trajectory is calculated, and the one with the smallest cost is selected as the optimal trajectory for the current time step.

[0180] Define and use risk fields to model human-driven vehicles:

[0181] For collision detection with human-driven vehicles, considering that human-driven vehicles have different driving styles, the risk field function is used to model human-driven vehicles. First, the risk field function is defined as follows:

[0182]

[0183]

[0184] Among them, the subscripts x and y represent the x direction and y direction, the non-subscript x represents the x position of the calculation point, and the non-subscript y represents the y position of the calculation point. vehicle is the vehicle length, W vehicle is the vehicle width, β x , β y is the corresponding coefficient, α x , α y is the acceleration in the x and y directions, v x (t), v y (t) is the velocity in the x and y directions.

[0185] Vehicles with different driving styles will be reflected in the size of the speed. We can set the risk threshold and R vehicle The values ​​of (x, y, t) are compared to determine whether a collision with a human-driven vehicle is possible. In addition, the use of risk fields to simulate human vehicles with different driving styles can also be extended to simulate other static or dynamic obstacles.

[0186] Based on the trajectory planning in the Frenet coordinate system and the human driving vehicle model established using the risk field, the local trajectory replanning strategy of the lower-level networked autonomous driving vehicle is designed.

[0187] Multi-vehicle scheduling at intersections is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0188] Example 2

[0189] Based on Example 1, this Example 2 further provides a multi-vehicle double-layer dispatching system at an intersection, including the following modules:

[0190] Object module: used to build intersection model and vehicle motion model;

[0191] Algorithm module: used to establish a deep reinforcement learning algorithm model and train the deep reinforcement learning algorithm model based on the intersection model and vehicle motion model;

[0192] Upper-level planning module: includes an upper-level multi-networked autonomous driving vehicle scheduling strategy based on a deep reinforcement learning algorithm model, which is used to plan vehicle trajectories and obtain feasible trajectories;

[0193] Lower-level planning module: includes trajectory planning based on the Frenet coordinate system and the local trajectory replanning strategy of the lower-level networked autonomous driving vehicle based on the human-driven vehicle model established using the risk field, which is used to perform local replanning of feasible trajectories.

[0194] Example 3

[0195] Based on Example 1, this Example 3 adds a simulation step. By setting various parameters of the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy, simulation is performed in the intersection model to verify the reliability of the multi-vehicle double-layer scheduling method at the intersection. The specific steps include:

[0196] The simulation time step is set to 0.1s, the number of connected autonomous vehicles is 4, the number of human-driven vehicles is 2, and the size of the intersection conflict area is 20m long and 20m wide.

[0197] Set the ATTD3 algorithm parameters for the upper-layer multi-network autonomous driving vehicle scheduling strategy:

[0198] The initial learning rate lr=0.0003, the decay coefficient ε=0.95, the soft update rate Sr is 0.005, the batch size batchsize is 64, the discount factor γ=0.99, and the noise variance σ=0.1.

[0199] Set the CAV vehicle parameters for the upper-level multi-connected autonomous driving vehicle dispatch strategy:

[0200] Maximum acceleration max =5m / s 2 , maximum speed v max =15m / s, minimum speed v min =0m / s.

[0201] Set the parameters of the local trajectory replanning strategy for the lower-level connected autonomous driving vehicle:

[0202] Road width R width =5m, road curvature k=2, sampling interval ΔT=0.1s, sampling lateral offset Δd=0.5m, speed interval Δv=1m / s, risk threshold r thr =0.8.

[0203] The parameters of the human-driven vehicle in the local trajectory replanning strategy of the lower-level connected autonomous driving vehicle are set as follows:

[0204] Maximum acceleration max=6m / s 2 , maximum speed v max =16m / s, minimum speed v min =0m / s.

[0205] See also Figure 3 , 4 , 5, 6, 7, and 8 are partial simulation result diagrams of this embodiment 3. The upper-level strategy introduces the attention mechanism when using the deep reinforcement learning algorithm TD3, proposes ATTD3, ​​and designs a reasonable loss function for the newly introduced attention network, which improves the reward of the algorithm and thus improves the convergence speed of the algorithm. The lower-level strategy uses the risk field to simulate human-driven vehicles with different driving styles when considering the impact of human-driven vehicles, and can perform local trajectory replanning in real time to ensure traffic safety and efficiency.

[0206] The other steps of this embodiment 3 are the same as those of embodiment 1 and will not be repeated here.

[0207] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A multi-vehicle double-layer dispatching method at an intersection, characterized in that: The method comprises: Construct intersection models and vehicle motion models; Based on the intersection model and vehicle motion model, a deep reinforcement learning algorithm model is established and trained, and the deep reinforcement learning algorithm model is used to design an upper-layer multi-networked autonomous driving vehicle scheduling strategy; After each connected autonomous driving vehicle obtains a feasible trajectory according to the upper-layer multi-connected autonomous driving vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in a Frenet coordinate system, a risk field is defined and a human-driven vehicle model is established using the risk field, and a local trajectory replanning strategy for the lower-layer connected autonomous driving vehicle is designed based on the trajectory planning in the Frenet coordinate system and the establishment of a human-driven vehicle model using the risk field; Multi-vehicle scheduling at an intersection is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

2. A method for dispatching multiple vehicles at a double-deck intersection according to claim 1, characterized in that: The intersection is a two-way two-lane intersection without traffic lights. The map of the intersection model is drawn based on the Lanelet2 framework. For the right lane, the vehicle chooses to turn right or go straight, and for the left lane, the vehicle chooses to turn left or go straight.

3. A method for dispatching multiple vehicles at a double-deck intersection according to claim 1, characterized in that: In the intersection model, each connected autonomous driving vehicle follows a predetermined path in the lane after entering the intersection. The movement of the vehicle is decomposed into movement on the x-axis and the y-axis. The vehicle movement model is expressed as the following formula: v i (t+1)=v i (t)+a i (t)τ x i (t+1)=x i (t)+v i (t+1)cosθτ y i (t+1)=y i (t)+v i (t+1)sinθτ Among them, θ is the heading angle of the vehicle, which is calculated by the current position of the vehicle and the position of the target point, τ is the control time step, and v i and a i are the acceleration and velocity of the vehicle, x i and i It is the position of the vehicle in both the x-axis and y-axis directions.

4. A method for dispatching multiple vehicles in double layers at an intersection according to claim 1, characterized in that: The deep reinforcement learning algorithm model is an ATTD3 algorithm model. Establishing the ATTD3 algorithm model and training it include the following steps: Based on the intersection model and the vehicle motion model, a Markov decision process MDP model is established; According to the MDP model, a TD3 algorithm model is constructed; The attention mechanism Self-Attention is introduced into the TD3 model, and the ATTD3 algorithm model is constructed and trained.

5. A method for dispatching multiple vehicles in double layers at an intersection according to claim 4, characterized in that: The Markov decision process MDP model consists of a tuple (S, A, P, R, γ), where S is the state space, A is the action space, P is the state transition probability, R is the reward function, and γ∈(0,1) is the discount factor; Model-free reinforcement learning is used, and the state transition probability P is not set. The state space S includes the position and speed information of each vehicle. The networked autonomous driving vehicle travels on a predetermined path, and the action space is the acceleration of each vehicle A. Considering the three aspects of vehicle driving efficiency, vehicle driving comfort and avoiding collisions between vehicles, the following reward function is set: Among them, V current is the current speed of the vehicle, V max is the maximum speed of the vehicle; Among them, a old is the last acceleration, a new is the current acceleration; Among them, dist is the sum of the distances between the colliding vehicles when the vehicles collide; r total =ω eff *r eff +oh com *r com +oh col *r col +oh rea r rea Among them, r total is the total reward, r eff is the reward function for vehicle driving efficiency, r com is the reward function for vehicle driving comfort, r col is the reward function for avoiding collisions between vehicles, r rea is the reward function for task completion, ω eff ,ω com ,ω col ,ω rea are the corresponding reward coefficients respectively.

6. A method for dispatching multiple vehicles in double layers at an intersection according to claim 4, characterized in that: According to the MDP model, constructing the TD3 algorithm model includes the following steps: The TD3 algorithm consists of six networks: Actor network π(·|φπ), Critic1 network Critic2 Network Target Actor network π′(·|φπ′), Target Critic1 network Q1′(· and Target Critic2 network At each time step, get the current agent state s, and select action a from the Actor network based on the current state s; Execute the corresponding action to obtain the corresponding reward r and target state s′, and store the tuple (s, a, r, s′) in the experience replay buffer; Randomly sample a small batch of experience (s, a, r, s′) in the experience replay buffer, with a batch size of N, and generate the target action a′ from the TargetActor network in turn; Calculate the target Q value y from the Target Critic1 and Target Critic2 networks according to the Bellman formula; Calculate TD-error as the loss function of Critic1 and Critic2 networks; Update the Critic network parameters by minimizing the loss function of the Critic1 and Critic2 networks; For the updates of the Actor network and the target network, the Actor network uses delayed updates, and the target network uses delayed updates and soft updates at the same time. After every n steps, the loss function of the Actor network is first calculated; Update the Actor network parameters by minimizing the Actor network's loss function.

7. A method for dispatching multiple vehicles in two layers at an intersection according to any one of claims 4 or 6, characterized in that: The construction of the ATTD3 algorithm model and training includes: At each time step, get the current agent state s, and select action a from the Actor network based on the current state s b ; Extract the features of the Actor network, Critic1 and Critic2 networks to obtain the Actor features, Critic1 features and Critic2 features; The corresponding Critic feature is obtained by solving the average sum of Critic1 feature and Critic2 feature; Use the Actor feature as the Query and Value in the attention mechanism, the Critic feature as the Key, and output the Attention Actor feature through the self-attention mechanism network; Input the Attention Actor feature into the output layer of the Actor network to get a new action a t ; According to the action a output by the normal Actor network b And the action a after the attention mechanism t , calculate the Q value of the two actions respectively; Select the action corresponding to the larger Q value as the corresponding output action a; The subsequent training process is consistent with the TD3 algorithm model. A loss function is set for the new attention network layer introduced when the attention mechanism is introduced. The loss function is designed as follows: Among them, the attention mechanism network layer update is synchronized with the update of the Actor network and the target network.

8. A method for dispatching multiple vehicles in double layers at an intersection according to claim 1, characterized in that: The trajectory planning of the feasible trajectory in the Frenet coordinate system includes: The trajectory planning in the Cartesian coordinate system is converted to the Frenet coordinate system. In the Frenet coordinate system, the direction along the reference trajectory is the longitudinal direction s, and the direction perpendicular to the reference trajectory is the lateral direction d. The motion state of the vehicle is The trajectory planning in the Frenet coordinate system is divided into lateral planning and longitudinal planning; For lateral planning, the initial time configuration conditions, target time configuration conditions, sampling intervals, and lateral offsets are set to generate different lateral trajectories and obtain the cost function of the lateral trajectory; For longitudinal planning, set the initial time configuration conditions, target time configuration conditions, sampling intervals, and speed intervals to generate different longitudinal trajectories and obtain the cost function of the longitudinal trajectory; The total cost of the trajectory is as follows: cost total =ω lon Cost s +ω lat Cost d Among them, Cost s Cost d are the costs of vertical planning and horizontal planning respectively, ω lon ,ω lat is the corresponding coefficient; After obtaining many lateral and longitudinal trajectories at each time step, determine whether the trajectory at each time step meets all the conditions that the speed is less than the maximum speed, the acceleration is less than the maximum acceleration, the curvature is less than the maximum curvature, and there is no collision with human-driven vehicles. Calculate the cost of each trajectory that meets all conditions, and select the trajectory with the smallest cost as the optimal trajectory for the current time step.

9. A method for dispatching multiple vehicles in double layers at an intersection according to claim 1, characterized in that: The method of using the risk field to establish a human-driven vehicle or obstacle model includes: The risk field function is defined as follows: Among them, the subscripts x and y represent the x-direction and y-direction, the non-subscript x represents the x-direction position of the calculation point, and the non-subscript y represents the y-direction position of the calculation point. vehicle is the vehicle length, W vehicle is the vehicle width, β x , β y is the corresponding coefficient, α x , α y is the acceleration in the x and y directions, v x (t), v y (t) is the velocity in the x and y directions, t represents the time t; Setting risk threshold and risk field function R vehicle The values ​​of (x, y, t) are compared to determine whether the connected autonomous vehicle may potentially collide with a human-driven vehicle or obstacle.

10. A multi-vehicle double-layer dispatching system at an intersection, characterized in that: Includes the following modules: Object module: used to build intersection model and vehicle motion model; Algorithm module: used to establish a deep reinforcement learning algorithm model and train the deep reinforcement learning algorithm model based on the intersection model and the vehicle motion model; Upper-level planning module: including an upper-level multi-networked autonomous driving vehicle scheduling strategy based on the deep reinforcement learning algorithm model, used for vehicle trajectory planning and obtaining feasible trajectories of networked autonomous driving vehicles; Lower-level planning module: includes trajectory planning based on the Frenet coordinate system and a local trajectory replanning strategy for a lower-level networked autonomous driving vehicle based on a human-driven vehicle model established using a risk field, and is used to perform local replanning of the feasible trajectory.

Images