A multi-vehicle double-layer scheduling method at an intersection

Abstract

The present invention discloses a multi-vehicle double-layer scheduling method at an intersection, which relates to the field of intelligent transportation technology. The method comprises: constructing an intersection and vehicle motion model; establishing and training an ATTD3 algorithm, and designing an upper-layer scheduling strategy using the ATTD3 algorithm; after obtaining a feasible trajectory of a networked autonomous driving vehicle according to the upper-layer scheduling strategy, performing trajectory planning in a Frenet coordinate system, and using a risk field to establish a human-driven vehicle or obstacle model, and designing a lower-layer local trajectory replanning strategy; and performing multi-vehicle scheduling at an intersection based on the upper-layer scheduling strategy and the lower-layer local trajectory replanning strategy. The present invention improves the convergence speed of the algorithm by constructing the ATTD3 algorithm when designing the upper-layer scheduling strategy and designing a reasonable loss function for the newly introduced attention network. At the same time, the lower-layer local trajectory replanning strategy uses the risk field to simulate human-driven vehicles or obstacles, and can accurately and locally replan the vehicle driving trajectory in real time.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and more specifically, to a multi-vehicle double-layer scheduling method at an intersection based on deep reinforcement learning and risk fields. Background Art

[0002] Intelligent transportation systems incorporate connected autonomous vehicles (CAVs) and vehicle-to-everything (V2X) technologies, enabling real-time information exchange between vehicles (V2V), infrastructure (V2I), pedestrians (V2P), and networks (V2N). This provides a new solution for scheduling at intersections without traffic lights. By dynamically sensing traffic flow and environmental information, it optimizes vehicle routes and travel times, improving traffic efficiency and safety.

[0003] Currently, research on multi-vehicle scheduling at unsignaled intersections has largely focused on centralized scheduling methods, which leverage global information from the entire intersection to centrally organize the movement of all approaching vehicles. In these scheduling methods, the computational effort typically increases rapidly with the number of vehicles, requiring careful consideration to ensure real-time feasibility. Furthermore, most current research considers only connected autonomous vehicles, but fully connected autonomous driving is still a long way off. Therefore, it is necessary to consider human drivers and other obstacles when implementing multi-vehicle scheduling.

[0004] The invention, with application publication number CN117373249A, discloses a vehicle dispatching method and system for a two-way, two-lane, unsignalized intersection. The method comprises the following steps: Step S1: pre-processing vehicles based on a given real-world two-way, two-lane intersection to determine whether a collision will occur and analyzing the conflict relationships between vehicles; Step S2: constructing a vehicle dispatching model based on a deep search of conflict relationships; Step S3: utilizing nonlinear programming to implement vehicle action decisions and determine the optimal transit time and vehicle acceleration; performing lower-level planning based on the vehicle dispatching model to design a vehicle dispatching strategy for the unsignalized intersection; Step S4: constructing an intersection simulation scenario based on a given real-world intersection scenario and initializing vehicle information for each vehicle; and Step S5: inputting the vehicle information into the vehicle dispatching model to perform behavioral decisions and obtain the action output for each vehicle at the next moment. This invention can improve the traffic efficiency of connected autonomous vehicles and reduce the probability of accidents, but it cannot reduce the complexity and computational load of the dispatching algorithm, ensure real-time dispatching, and simultaneously consider the impact of human drivers or obstacles to adjust local trajectory planning. Summary of the Invention

[0005] In order to overcome the defects of the above-mentioned prior art in reducing the complexity and computational complexity of the scheduling algorithm and ensuring real-time scheduling while taking into account the influence of human-driven vehicles or obstacles to adjust local trajectory planning, the present invention provides a multi-vehicle double-layer scheduling method at an intersection.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention includes:

[0007] Construct intersection models and vehicle motion models;

[0008] Based on the intersection model and vehicle motion model, an ATTD3 algorithm model is established and trained, and the ATTD3 algorithm model is used to design an upper-layer multi-networked autonomous driving vehicle scheduling strategy;

[0009] After each connected autonomous vehicle obtains a feasible trajectory according to the upper-layer multi-connected autonomous vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in a Frenet coordinate system, and a human-driven vehicle model is established using a risk field. Based on the trajectory planning in the Frenet coordinate system and the human-driven vehicle model established using the risk field, a local trajectory replanning strategy for the lower-layer connected autonomous vehicle is designed;

[0010] Multi-vehicle scheduling at an intersection is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0011] Furthermore, the intersection is a two-way two-lane intersection without traffic lights. The map of the intersection model is drawn based on the Lanelet2 framework. For the right lane, vehicles can choose to turn right or go straight, and for the left lane, vehicles can choose to turn left or go straight.

[0012] Furthermore, it is stipulated that each connected autonomous driving vehicle follows a predetermined path in the lane after entering the intersection. The movement of the vehicle can be decomposed into movement on the x-axis and the y-axis. The vehicle motion model is expressed as the following formula:

[0013]

[0014]

[0015]

[0016] in, Is the heading angle of the vehicle, which is calculated by the current position of the vehicle and the position of the target point. is the control time step, and are the acceleration and speed of the vehicle, and It is the position of the vehicle in the x-axis and y-axis directions.

[0017] Furthermore, the establishment of the ATTD3 algorithm model and training includes the following steps:

[0018] Based on the intersection model and the vehicle motion model, a Markov decision process MDP model is established;

[0019] According to the MDP model, a TD3 algorithm model is constructed;

[0020] The attention mechanism Self-Attention is introduced into the TD3 model, and the ATTD3 algorithm model is constructed and trained.

[0021] Furthermore, the Markov decision process MDP model consists of the tuple where S is the state space, A is the action space, P is the state transition probability, R is the reward function, is the discount factor;

[0022] Model-free reinforcement learning is used, without setting the state transition probability P. The state space S includes the position and velocity information of each vehicle. The connected autonomous vehicles drive on a predetermined path, and the action space A is the acceleration of each vehicle.

[0023] Considering the three aspects of vehicle driving efficiency, vehicle driving comfort, and avoiding vehicle collisions, the following reward function is set:

[0024]

[0025] in, is the current speed of the vehicle, is the maximum speed of the vehicle;

[0026]

[0027] in, is the last acceleration, is the current acceleration;

[0028]

[0029] Among them, dist is the sum of the distances between the colliding vehicles when the vehicles collide;

[0030]

[0031]

[0032] in, For the total reward, is the reward function for vehicle driving efficiency, is the reward function for vehicle driving comfort, is the reward function for avoiding collisions between vehicles, is the reward function for task completion, 、 、 、 are the corresponding reward coefficients respectively.

[0033] Furthermore, the construction of the TD3 algorithm model based on the MDP model includes the following steps:

[0034] The TD3 algorithm consists of six networks: Actor network , Critic1 Network , Critic2 Network Target Actor Network Target Critic1 network and Target Critic2 network ;

[0035] At each time step, get the current agent state s and select an action from the Actor network based on the current state s ;

[0036] Execute the corresponding action to obtain the corresponding reward r and target state , and the tuple Store into the experience replay buffer;

[0037] Randomly sample a mini-batch of experience in the experience replay buffer , the batch size is N, and the target actions are generated from the Target Actor network in sequence ;

[0038] Calculate the target Q value from the Target Critic1 and Target Critirc2 networks according to the Bellman formula ;

[0039] Calculate TD-error as the loss function of Critic1 and Critic2 networks;

[0040] Update the Critic network parameters by minimizing the loss function of Critic1 and Critic2 networks;

[0041] For the updates of the Actor network and the target network, the Actor network adopts delayed update, and the target network adopts both delayed update and soft update. After every n steps, the loss function of the Actor network is first calculated;

[0042] Update the Actor network parameters by minimizing the loss function of the Actor network;

[0043] Furthermore, the construction and training of the ATTD3 algorithm model includes:

[0044] At each time step, get the current agent state s and select an action from the Actor network based on the current state s ;

[0045] Extract the features of the Actor network, Critic1 and Critic2 networks to obtain the Actor features, Critic1 features and Critic2 features;

[0046] The corresponding Critic feature is obtained by solving the average sum of Critic1 feature and Critic2 feature;

[0047] Use the Actor feature as the Query and Value in the attention mechanism, the Critic feature as the Key, and output the Attention Actor feature through the self-attention mechanism network;

[0048] Input the Attention Actor feature into the output layer of the Actor network to obtain a new action ;

[0049] Based on the actions output by the normal Actor network And the actions affected by the attention mechanism , calculate the Q value of the two actions respectively;

[0050] Finally, the action corresponding to the larger Q value is selected as the corresponding output action a;

[0051] The subsequent training process is consistent with the TD3 algorithm model. A loss function is set for the new attention network layer introduced when the attention mechanism is introduced. The loss function is designed as follows:

[0052]

[0053] Among them, the attention mechanism network layer update is synchronized with the update of the Actor network and the target network.

[0054] Furthermore, the trajectory planning of the feasible trajectory in the Frenet coordinate system includes:

[0055] The trajectory planning in the Cartesian coordinate system is converted to the Frenet coordinate system. In the Frenet coordinate system, the direction along the reference trajectory is the longitudinal direction s, and the direction perpendicular to the reference trajectory is the transverse direction d. The motion state of the vehicle is ;

[0056] The trajectory planning in the Frenet coordinate system is divided into lateral planning and longitudinal planning, and the connected autonomous vehicle is required to travel along the predetermined path;

[0057] For lateral planning, the initial time configuration conditions, target time configuration conditions, sampling intervals, and lateral offsets are set to generate different lateral trajectories and obtain the cost function of the lateral trajectory.

[0058] For longitudinal planning, the initial time configuration conditions, target time configuration conditions, sampling intervals, and speed intervals are set to generate different longitudinal trajectories and obtain the cost function of the longitudinal trajectory.

[0059] The total cost of the trajectory is as follows:

[0060] = +

[0061] in, 、 are the costs of vertical planning and horizontal planning respectively, 、 is the corresponding coefficient;

[0062] After obtaining many lateral and longitudinal trajectories at each time step, we determine whether the trajectory at each time step meets all the conditions: speed less than the maximum speed, acceleration less than the maximum acceleration, curvature less than the maximum curvature, and no collision with human-driven vehicles. We calculate the cost of each trajectory that meets all the conditions, and select the trajectory with the smallest cost as the optimal trajectory for the current time step.

[0063] Furthermore, defining a risk field and establishing a human driving vehicle model using the risk field includes:

[0064] The risk field function is defined as follows:

[0065]

[0066]

[0067]

[0068]

[0069] Among them, the subscripts x and y represent the x direction and y direction, the non-subscript x represents the x position of the calculation point, and the non-subscript y represents the y position of the calculation point. is the vehicle length, is the vehicle width, 、 is the corresponding coefficient, 、 are the accelerations in the x and y directions, 、 are the velocities in the x and y directions.

[0070] Setting risk threshold and risk field function The value of is compared to determine whether the connected autonomous vehicle is likely to collide with a human-driven vehicle or obstacle.

[0071] Furthermore, the method also includes a simulation step, by setting various parameters of the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy, simulation is performed in an intersection model to verify the reliability of the multi-vehicle double-layer scheduling method at the intersection.

[0072] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0073] This invention uses the deep reinforcement learning algorithm TD3 and the attention mechanism when designing the upper-layer multi-networked autonomous driving vehicle scheduling strategy, thereby constructing the ATTD3 algorithm and designing a reasonable loss function for the newly introduced attention network, thereby improving the convergence speed of the algorithm. At the same time, the local trajectory replanning strategy of the lower-layer networked autonomous driving vehicle takes into account the influence of human-driven vehicles or obstacles, and uses the risk field to simulate human-driven vehicles or obstacles with different driving styles, which can accurately and in real time perform local replanning of the vehicle's driving trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0075] Figure 1 This is a framework diagram of the multi-vehicle double-layer scheduling method at an intersection of this application;

[0076] Figure 2 A schematic diagram of the intersection map in Lanelet2 format for this application;

[0077] Figure 3 ATTD3 algorithm and TD3 algorithm training result diagram of Example 3 of the present application;

[0078] Figure 4 Outputting an acceleration graph for the upper layer strategy of Example 3 of the present application;

[0079] Figure 5 Output speed map of the upper layer strategy of Example 3 of the present application;

[0080] Figure 6 Schematic diagram of the risk field of human-driven vehicles for the lower-level strategy of Example 3 of the present application.

[0081] Figure 7 This is the result diagram of the lower-level strategy trajectory planning of Example 3 of the present application;

[0082] Figure 8 This is the trajectory replanning sd map of vehicle 1 and vehicle 3 in embodiment 3 of the present application. DETAILED DESCRIPTION

[0083] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0084] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0085] Example 1

[0086] See also Figure 1 The method framework diagram of this embodiment, a multi-vehicle double-layer scheduling method at an intersection according to the embodiment of the present application, includes the following steps:

[0087] S1: Construct intersection model and vehicle motion model;

[0088] S2: Based on the intersection model and vehicle motion model, the ATTD3 algorithm model is established and trained, and the ATTD3 algorithm model is used to design the upper-layer multi-network autonomous driving vehicle scheduling strategy;

[0089] S3: After each connected autonomous vehicle obtains a feasible trajectory according to the upper-layer multi-connected autonomous vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in a Frenet coordinate system, a risk field is defined, and a human-driven vehicle model is established using the risk field. Based on the trajectory planning in the Frenet coordinate system and the human-driven vehicle model established using the risk field, a local trajectory replanning strategy for the lower-layer connected autonomous vehicle is designed;

[0090] S4: Multi-vehicle scheduling at intersections based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0091] In this embodiment 1, step S1 is specifically as follows:

[0092] Intersection Modeling: See Figure 2 The scenario studied in this paper is a two-lane, two-way intersection without traffic lights. Vehicles in the right lane have the option of turning right or going straight, while those in the left lane have the option of turning left or going straight. The intersection map is drawn using the Lanelet2 framework. Because a centralized scheduling model is used, a scheduling node exists on the Roadside Unit (RSU). This node collects information about all vehicles entering the intersection through the vehicle-to-infrastructure (V2I) communication link. Real-time information from all connected autonomous vehicles (CAVs) is collected by a centralized coordinator, which then outputs real-time instructions to each CAV.

[0093] Establishing a vehicle motion model: Each CAV is required to follow a predetermined path in the lane after entering an intersection. The vehicle's motion can be decomposed into motion on the x-axis and the y-axis. The vehicle's motion model can be simplified as follows:

[0094]

[0095]

[0096]

[0097] in It is the heading angle of the vehicle, which can be calculated from the current position of the vehicle and the position of the target point. is the control time step, and are the acceleration and speed of the vehicle, and is the position of the vehicle in both directions.

[0098] Step S2 specifically includes the following steps:

[0099] MDP model establishment:

[0100] A Markov decision process (MDP) consists of a tuple where S is the state space, A is the action space, P is the state transition probability, R is the reward function, Is the discount factor. Since the present invention adopts model-free reinforcement learning, there is no need to set the state transition probability P.

[0101] The state space should include the position and velocity information of each vehicle, i.e.

[0102]

[0103] For a given reference path, we only need to control the acceleration of the vehicle, which can greatly reduce the dimension of the space and reduce the training cost and complexity. Therefore, the action space should be the acceleration of each vehicle, that is,

[0104]

[0105] When setting the reward function R, we should consider three aspects: vehicle driving efficiency, vehicle driving comfort, and avoiding collisions between vehicles. In addition, we should also set a sparse reward for task completion. Therefore, the reward function is set as follows:

[0106]

[0107] in, is the current speed of the vehicle, is the maximum speed of the vehicle;

[0108]

[0109] in, is the last acceleration, is the current acceleration;

[0110]

[0111] in, The sum of the distances between the colliding vehicles when the vehicles collide;

[0112]

[0113]

[0114] in It is a reward function for vehicle driving efficiency, which requires CAV to exit the intersection as quickly as possible after entering the intersection to improve the efficiency of coordination; It is a function of vehicle driving comfort, which requires that the CAV should not have excessive acceleration mutations and maintain driving stability and ride comfort; It is a function to avoid collisions between vehicles. If a collision occurs, a large negative value is given, which is a fixed value plus the sum of the distances between the vehicles when the collision occurs. Function for task completion, final total reward is the weighted sum of the above four rewards.

[0115] According to the established MDP model, the ATTD3 algorithm model is established:

[0116] The ATTD3 (Attention Twin Delayed Deep Deterministic Policy Gradient) algorithm combines the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm with the Self-Attention mechanism. By introducing the attention mechanism, the agent can learn more valuable strategies faster. The TD3 algorithm is a deterministic deep reinforcement learning algorithm under the actor-critic (AC) framework. It combines the deep deterministic policy gradient algorithm and double Q learning. The TD3 algorithm consists of six networks: the actor network , Critic1 network , Critic2 network , Target Actor Network , TargetCritic1 network , Target Critic2 network .

[0117] Training TD3:

[0118] In each time step, first obtain the current agent state s, and select action a from the Actor network according to the current state s, that is,

[0119]

[0120] in To explore noise, the purpose is to encourage the agent to take risks and try new behaviors that may bring higher rewards, rather than always choosing behaviors that are known to bring current high rewards. is the maximum acceleration.

[0121] After executing the corresponding action, you will get the corresponding reward r and target state And the tuple Stored to the experience replay buffer.

[0122] Randomly sample a mini-batch of experience in the experience replay buffer , the batch size is N, and the target actions are generated from the Target Actor network in sequence ,Right now

[0123]

[0124] in , is the noise clipping ratio, the purpose of the noise is to enhance the robustness of the network and improve the accuracy of the estimation, is the maximum acceleration.

[0125] Then the target Q value is calculated from the Target Critic1 and Target Critirc2 networks according to the Bellman formula ,Right now

[0126]

[0127] Then calculate TD-error as the loss function of Critic1 and Critic2 networks, that is,

[0128]

[0129]

[0130] Update the Critic network parameters by minimizing the loss function:

[0131]

[0132]

[0133] For the update of the Actor network and the target network, the TD3 algorithm uses delayed update, so after every n steps, the loss function of the Actor network is first calculated, that is,

[0134]

[0135] Update the Actor network parameters by minimizing the loss function:

[0136]

[0137] The target network update also uses soft update, that is:

[0138]

[0139]

[0140]

[0141] As can be seen from the TD3 training process above, the actor and critic networks in the TD3 algorithm are independent. The critic provides the actor with a benchmark for evaluating actions based on Q-values, but the actor cannot clearly learn why certain actions have high Q-values ​​and others have low Q-values. Considering that actions and values ​​exist as feature vectors before generation, which constitute the different dimensions of actions and values, the introduction of the attention mechanism can accurately match the correlations of different vector dimensions, allowing feature modification and obtaining better actions.

[0142] The principle of the attention mechanism is as follows:

[0143]

[0144] In TD3, each network can be decomposed into Feature layer and Output layer. The network receives input, outputs features to the Output layer after passing through the Feature layer, and the Output layer outputs the final result. It can be expressed as

[0145]

[0146] Critic1 and Critic2 in the Critic network can be expressed as

[0147] 、

[0148] This approach extracts features from the actor and critic networks and incorporates them into the attention mechanism. When selecting an action, the original actor network's Actor features output one action, while the AttentionActor features obtained through the attention mechanism output another action. The critic network then compares these features and selects the action with the higher value. This approach uses a self-attention mechanism to achieve this goal by setting an appropriate loss function for the attention network to guide it in outputting better actor features.

[0149] After introducing the self-attention mechanism into the TD3 algorithm, the ATTD3 algorithm model is established and trained:

[0150] At each time step, the current agent state s is also obtained, and the action is selected from the Actor network according to the current state s ,Right now

[0151]

[0152] Then, the features of the Actor network, Critic1 and Critic2 networks are extracted to obtain the Actor features, Critic1 features and Critic2 features. Then, the corresponding Critic features are obtained by averaging the Critic1 features and Critic2 features. At this time, the Actor features are used as the Query and Value, and the Critic features are used as the Key. The Attention Actor features are output through the self-attention mechanism network, that is,

[0153]

[0154] in Represents the Attention Actor feature, is the Actor feature, is the averaged and summed critic feature. Finally, the Attention Actor feature is input into the output layer of the Actor network to obtain the new action ,Right now

[0155]

[0156] At this point we get an action output by a normal Actor network And the actions affected by the attention mechanism , calculate the Q value of the two actions respectively, that is,

[0157]

[0158]

[0159] Finally, the action corresponding to the larger Q value is selected as the corresponding output action a.

[0160] The subsequent training process is consistent with the above TD3. However, since a new network layer is introduced when implementing the attention mechanism, a loss function needs to be set for the new attention network layer. In order to enable the attention mechanism to better guide the actor to output actions with higher value, the loss function is designed as follows:

[0161]

[0162] Attention mechanism network layer updates are synchronized with the Actor network and the target network

[0163] Using the ATTD3 algorithm model, design the upper-layer multi-network autonomous driving vehicle scheduling strategy.

[0164] Each CAV can obtain a feasible trajectory through the upper-layer multi-networked autonomous vehicle scheduling strategy. However, in real-world scenarios, in addition to autonomous vehicles, there may also be human-driven vehicles (HDVs) or obstacles at intersections. Therefore, in certain conflict situations, local trajectory replanning is required. Therefore, step S3 is specifically as follows:

[0165] Conversion between Frenet coordinate system and Cartesian coordinate system:

[0166] Since the Frenet coordinate system can project the vehicle's trajectory at a certain moment onto the reference trajectory and decompose it into two dimensions, horizontal and vertical, it can simplify the subsequent trajectory planning work. Therefore, it is necessary to convert the trajectory planning in the Cartesian coordinate system to the Frenet coordinate system. In the rectangular coordinate system, the position points on the trajectory can be , but the Frenet coordinate system is established based on the reference trajectory. In this reference system, the direction along the reference trajectory is the longitudinal axis s, and the direction perpendicular to the reference trajectory is the transverse axis d. The motion state of the vehicle in the Frenet coordinate system can be described as , the mutual conversion formula between Frenet coordinate system and Cartesian coordinate system is as follows:

[0167]

[0168]

[0169] Perform trajectory planning in the Frenet coordinate system:

[0170] Trajectory planning in the Frenet coordinate system can be divided into lateral planning and longitudinal planning. The lateral planning is mainly responsible for tasks such as obstacle avoidance and lane changing, while the longitudinal planning can be flexibly configured according to the actual driving scenario of the vehicle. The final output trajectory is as well as Composition, of which and They are all polynomials in time.

[0171] The polynomial of horizontal planning can be expressed by a 5th-order polynomial, namely:

[0172]

[0173] Set the condition to the initial time Configuration and target time Configuration . It is stipulated that the vehicle should follow the reference trajectory as much as possible, so there is , By setting the sampling interval and lateral offset , generating different lateral trajectories.

[0174] The cost function for the lateral trajectory is as follows:

[0175]

[0176] in, is the third-order derivative in the d direction, which is used to measure the intensity of the change in acceleration in the d direction. is a measure of the deviation from the path trajectory, and is the corresponding coefficient.

[0177] The polynomial of longitudinal planning can be expressed by a 4-degree polynomial, namely:

[0178]

[0179] Set the condition to the initial time Configuration and target time Configuration Generally speaking, the position along the reference line is configured By setting the sampling interval and speed intervals , generating different longitudinal trajectories.

[0180] The cost function of the longitudinal trajectory is as follows:

[0181]

[0182] in, It is the third-order derivative of s, representing the acceleration change in the s direction, and is used to measure the driving comfort of the vehicle. It is the square of the difference between the first-order derivative of s and the reference velocity, which is used to measure the gap between the velocity in the s direction of the new trajectory and the velocity of the reference trajectory. and is the corresponding coefficient.

[0183] The total cost of the trajectory is calculated as follows:

[0184] = +

[0185] in, 、 is the corresponding coefficient.

[0186] At this point, many lateral and longitudinal trajectories have been obtained at each time step. The optimal trajectory selection criteria for each time step include the following: velocity less than the maximum velocity, acceleration less than the maximum acceleration, curvature less than the maximum curvature, and no collision with human-driven vehicles. Among the trajectories that meet all of these conditions, the cost of each trajectory is calculated, and the one with the lowest cost is selected as the optimal trajectory for the current time step.

[0187] Define and utilize risk fields to model human-driven vehicles:

[0188] For collision detection with human-driven vehicles, considering that human-driven vehicles have different driving styles, a risk field function is used to model human-driven vehicles. First, the risk field function is defined as follows:

[0189]

[0190]

[0191]

[0192]

[0193] Among them, the subscripts x and y represent the x direction and y direction, the non-subscript x represents the x position of the calculation point, and the non-subscript y represents the y position of the calculation point. is the vehicle length, is the vehicle width, 、 is the corresponding coefficient, 、 are the accelerations in the x and y directions, 、 are the velocities in the x and y directions.

[0194] Vehicles with different driving styles will be reflected in the size of the speed, we can set the risk threshold and The value of is compared to determine whether a collision with a human-driven vehicle is possible. In addition, the use of risk fields to simulate human vehicles with different driving styles can also be extended to simulate other static or dynamic obstacles.

[0195] Based on trajectory planning in the Frenet coordinate system and the use of risk fields to establish a human-driven vehicle model, a local trajectory replanning strategy for the lower-level connected autonomous driving vehicle is designed.

[0196] Multi-vehicle scheduling at intersections is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

[0197] Example 2

[0198] Based on Example 1, this Example 2 further provides a multi-vehicle double-layer dispatching system at an intersection, including the following modules:

[0199] Object module: used to build intersection model and vehicle motion model;

[0200] Algorithm module: used to establish a deep reinforcement learning algorithm model and train the deep reinforcement learning algorithm model based on the intersection model and vehicle motion model;

[0201] Upper-level planning module: This module includes an upper-level multi-networked autonomous driving vehicle scheduling strategy based on a deep reinforcement learning algorithm model, which is used to plan vehicle trajectories and obtain feasible trajectories.

[0202] Lower-level planning module: This module includes trajectory planning based on the Frenet coordinate system and a local trajectory replanning strategy for lower-level connected autonomous driving vehicles using a human-driven vehicle model established using a risk field, which is used to locally replan feasible trajectories.

[0203] Example 3

[0204] Based on Example 1, this Example 3 adds a simulation step. By setting various parameters of the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy, simulation is performed in the intersection model to verify the reliability of the multi-vehicle double-layer scheduling method at the intersection. The specific steps include:

[0205] The simulation time step is set to 0.1s, the number of connected autonomous vehicles is 4, the number of human-driven vehicles is 2, and the size of the intersection conflict area is 20m long and 20m wide.

[0206] Set the ATTD3 algorithm parameters for the upper-layer multi-network autonomous driving vehicle scheduling strategy:

[0207] Initial learning rate lr=0.0003, decay coefficient , the soft update rate Sr is 0.005, the batch size batchsize is 64, and the discount factor , noise variance .

[0208] Set the CAV vehicle parameters for the upper-layer multi-connected autonomous driving vehicle scheduling strategy:

[0209] Maximum acceleration , maximum speed , minimum speed .

[0210] Set the parameters of the local trajectory replanning strategy for the lower-level connected autonomous vehicle:

[0211] Road width , road curvature k=2, sampling interval =0.1s, sampling lateral offset , speed interval , risk threshold .

[0212] The parameters of the human-driven vehicle in the local trajectory replanning strategy of the lower-level connected autonomous driving vehicle are set as follows:

[0213] Maximum acceleration , maximum speed , minimum speed .

[0214] See also Figure 3 、 4 Figures 5, 6, 7, and 8 are partial simulation results of Example 3. The upper-level strategy introduces an attention mechanism when using the deep reinforcement learning algorithm TD3, proposes ATTD3, ​​and designs a reasonable loss function for the newly introduced attention network, which improves the algorithm's reward and thus its convergence speed. The lower-level strategy uses a risk field to simulate human-driven vehicles with different driving styles when considering the impact of human-driven vehicles. This allows for real-time local trajectory replanning to ensure traffic safety and efficiency.

[0215] The other steps of this embodiment 3 are the same as those of embodiment 1 and will not be repeated here.

[0216] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-vehicle double-deck dispatching method at an intersection, characterized in that: The method comprises: Construct intersection models and vehicle motion models; Based on the intersection model and vehicle motion model, a Markov decision process (MDP) model is established. Based on the MDP model, a TD3 algorithm model is constructed. The Self-Attention mechanism is introduced into the TD3 algorithm model to guide the Actor network to output higher-value actions, thereby constructing and training the ATTD3 algorithm model. The ATTD3 algorithm model is then used to design an upper-layer multi-network autonomous driving vehicle scheduling strategy. The Markov decision process MDP model consists of a tuple where S is the state space, A is the action space, P is the state transition probability, R is the reward function, is the discount factor; Model-free reinforcement learning is used, without setting the state transition probability P. The state space S includes the position and velocity information of each vehicle. The connected autonomous vehicles drive on a predetermined path, and the action space A is the acceleration of each vehicle. Considering the three aspects of vehicle driving efficiency, vehicle driving comfort, and avoiding vehicle collisions, the following reward function is set: in, is the current speed of the vehicle, is the maximum speed of the vehicle; in, is the last acceleration, is the current acceleration; Among them, dist is the sum of the distances between the colliding vehicles when the vehicles collide; in, For the total reward, is the reward function for vehicle driving efficiency, is the reward function for vehicle driving comfort, is the reward function for avoiding collisions between vehicles, is the reward function for task completion, 、 、 、 are the corresponding reward coefficients respectively; After obtaining a feasible trajectory for each connected autonomous driving vehicle according to the upper-layer multi-connected autonomous driving vehicle scheduling strategy, trajectory planning is performed on the feasible trajectory in the Frenet coordinate system to define the risk field. The risk field function is as follows: Among them, the subscript x and y represent the x-direction and y-direction, the non-subscript x represents the x-direction position of the calculation point, and the non-subscript y represents the y-direction position of the calculation point. is the vehicle length, is the vehicle width, 、 are the coefficients corresponding to vehicle length and vehicle width, 、 are the accelerations in the x and y directions, 、 are the velocities in the x and y directions, and t represents the time instant; Using the risk field to establish a human-driven vehicle model or an obstacle model, and designing a local trajectory replanning strategy for the lower-layer connected autonomous driving vehicle based on the trajectory planning in the Frenet coordinate system and the human-driven vehicle model or obstacle model established by the risk field; Multi-vehicle scheduling at an intersection is performed based on the upper-layer multi-networked autonomous driving vehicle scheduling strategy and the lower-layer networked autonomous driving vehicle local trajectory replanning strategy.

2. A multi-vehicle double-deck dispatching method at an intersection according to claim 1, characterized in that: The intersection is a two-way two-lane intersection without traffic lights. The map of the intersection model is drawn based on the Lanelet2 framework. For the right lane, the vehicle chooses to turn right or go straight, and for the left lane, the vehicle chooses to turn left or go straight.

3. A method for dispatching multiple vehicles in a double-deck manner at an intersection according to claim 1, characterized in that: In the intersection model, each connected autonomous vehicle follows a predetermined path in the lane after entering the intersection. The vehicle's motion is decomposed into motion on the x-axis and the y-axis. The vehicle motion model is expressed as the following formula: in, Is the heading angle of the vehicle, which is calculated by the current position of the vehicle and the position of the target point. is the control time step, and are the acceleration and speed of the vehicle, and It is the position of the vehicle in the x-axis and y-axis directions.

4. A method for dispatching multiple vehicles in a double-deck manner at an intersection according to claim 1, characterized in that: The step of constructing the TD3 algorithm model according to the MDP model includes the following steps: The TD3 algorithm consists of six networks: Actor network , Critic1 Network , Critic2 Network Target Actor Network Target Critic1 network and TargetCritic2 network ; At each time step, get the current agent state s and select an action from the Actor network based on the current state s ,Right now in To explore noise, the purpose is to encourage the agent to take risks and try new behaviors that may bring higher rewards, rather than always choosing behaviors that are known to bring current high rewards. is the maximum acceleration; Execute the corresponding action to obtain the corresponding reward r and target state , and the tuple Store into the experience replay buffer; Randomly sample a mini-batch of experience in the experience replay buffer , the batch size is N, and the target actions are generated from the TargetActor network in sequence ,Right now in , is the noise clipping ratio, the purpose of the noise is to enhance the robustness of the network and improve the accuracy of the estimation, is the maximum acceleration; Calculate the target Q value from the Target Critic1 and Target Critirc2 networks according to the Bellman formula ,Right now Calculate TD-error as the loss function of Critic1 and Critic2 networks, that is, Update the Critic network parameters by minimizing the loss function of Critic1 and Critic2 networks: For the updates of the Actor network and the target network, the Actor network adopts delayed update, and the target network adopts delayed update and soft update at the same time. After every n steps, the loss function of the Actor network is first calculated, that is, Update the Actor network parameters by minimizing the Actor network's loss function: The target network update also uses soft update, that is: 。 5. A method for dispatching multiple vehicles in a double-deck manner at an intersection according to claim 4, characterized in that: The construction of the ATTD3 algorithm model and training includes: At each time step, get the current agent state s and select an action from the Actor network based on the current state s ,Right now Extract the features of the Actor network, Critic1 and Critic2 networks to obtain the Actor features, Critic1 features and Critic2 features; The corresponding Critic feature is obtained by solving the average sum of Critic1 feature and Critic2 feature; The Actor feature is used as the Query and Value in the attention mechanism, the Critic feature is used as the Key, and the Attention Actor feature is output through the self-attention mechanism network, that is in Represents the Attention Actor feature, is the Actor feature, is the averaged and summed critic feature; Input the Attention Actor feature into the output layer of the Actor network to obtain a new action ,Right now Based on the actions output by the normal Actor network And the actions affected by the attention mechanism , calculate the Q value of the two actions respectively, that is, Select the action with the larger Q value as the corresponding output action a; The subsequent training process is consistent with the TD3 algorithm model. A loss function is set for the new attention network layer introduced when the attention mechanism is introduced. The loss function is designed as follows: Among them, the attention mechanism network layer update is synchronized with the update of the Actor network and the target network.

6. A method for dispatching multiple vehicles in a double-deck manner at an intersection according to claim 1, characterized in that: The trajectory planning of the feasible trajectory in the Frenet coordinate system includes: The trajectory planning in the Cartesian coordinate system is converted to the Frenet coordinate system. In the Frenet coordinate system, the direction along the reference trajectory is the longitudinal direction s, and the direction perpendicular to the reference trajectory is the transverse direction d. The motion state of the vehicle is , the mutual conversion formula between Frenet coordinate system and Cartesian coordinate system is as follows: The trajectory planning in the Frenet coordinate system is divided into horizontal planning and longitudinal planning; The final output trajectory is given by as well as Composition, of which and They are all polynomials in time; The polynomial of horizontal planning is expressed as a 5th-degree polynomial, namely: For horizontal planning, set the condition to initial time Configuration and target time Configuration , which stipulates that the vehicle should follow the reference trajectory as much as possible, so there is , , by setting the sampling interval and lateral offset , generating different lateral trajectories; The cost function for the lateral trajectory is as follows: in, is the third-order derivative in the d direction, which is used to measure the intensity of the change in acceleration in the d direction. is a measure of the deviation from the path trajectory, and is the corresponding coefficient; The polynomial of longitudinal planning is expressed as a 4-degree polynomial, namely: Set the condition to the initial time Configuration and target time Configuration ; By setting the sampling interval and speed intervals , generating different longitudinal trajectories; The cost function of the longitudinal trajectory is as follows: in, It is the third-order derivative of s, representing the acceleration change in the s direction, and is used to measure the driving comfort of the vehicle. It is the square of the difference between the first-order derivative of s and the reference velocity, which is used to measure the gap between the velocity in the s direction of the new trajectory and the velocity of the reference trajectory. and is the corresponding coefficient; The total cost of the trajectory is as follows: = + in, 、 are the costs of vertical planning and horizontal planning respectively, 、 is the corresponding coefficient; After obtaining many lateral and longitudinal trajectories at each time step, we determine whether the trajectory at each time step meets all the conditions: speed less than the maximum speed, acceleration less than the maximum acceleration, curvature less than the maximum curvature, and no collision with human-driven vehicles. We calculate the cost of each trajectory that meets all the conditions, and select the trajectory with the smallest cost as the optimal trajectory for the current time step.

7. A method for dispatching multiple vehicles in a double-deck manner at an intersection according to claim 1, characterized in that: The method of using the risk field to establish a human-driven vehicle model or an obstacle model includes: Setting risk threshold and risk field function The value of is compared to determine whether the connected autonomous vehicle may have a potential collision with a human-driven vehicle or obstacle.

8. A multi-vehicle double-layer dispatching system at an intersection, used to implement a multi-vehicle double-layer dispatching method at an intersection as claimed in any one of claims 1 to 7, characterized in that: The system includes the following modules: Object module: used to build intersection model and vehicle motion model; Algorithm module: used to establish a deep reinforcement learning algorithm model and train the deep reinforcement learning algorithm model based on the intersection model and the vehicle motion model; Upper-level planning module: including an upper-level multi-networked autonomous driving vehicle scheduling strategy based on the deep reinforcement learning algorithm model, used to plan vehicle trajectories and obtain feasible trajectories for networked autonomous driving vehicles; Lower-level planning module: includes a lower-level networked autonomous driving vehicle local trajectory replanning strategy based on trajectory planning in the Frenet coordinate system and a human-driven vehicle model established using a risk field, which is used to locally replan the feasible trajectory.

Images