Cooperative guidance control method for mixed vehicle fleet at entrance lane of intersection

Abstract

The invention discloses an intersection entrance lane mixed vehicle team cooperative guidance control method, which overcomes the problems of large calculation amount and difficulty in achieving a global optimal state in the prior art, and comprises the following steps: 1) obtaining road information; 2) dividing intelligent agents; and 3) establishing a multi-agent ecological cooperative control model of the intersection entrance lane mixed vehicle fleet: firstly, establishing a single-agent speed trajectory planning model, calculating an optimal passing trajectory of a single agent, and then establishing a multi-agent cooperative control model by taking the overall operation efficiency of the vehicle fleet as an optimization target on the basis of the optimal passing trajectory of the single agent, so as to realize the multi-agent ecological cooperative control of the intersection entrance lane mixed vehicle fleet. Cooperative control among a plurality of agents is realized, and a model is trained through a reinforcement learning method after model construction is completed; and 4) the controllable agents execute the optimized trajectory: issuing the established and trained model to each controllable agent, adopting a rolling time domain updating strategy, calculating the speed trajectory of each agent passing through the intersection by using road traffic information, and inputting the speed trajectory to a CAV controller for execution.

Description

Technical Field

[0001] The present invention relates to a control method in the field of traffic information engineering and control, and more specifically, to a method for cooperative guidance control of mixed-traffic vehicle fleets at intersection entrances. Background Art

[0002] As a key node in the urban road network, the operation status of the intersection has a direct impact on the operation efficiency of the urban road network. With the development of vehicle-road cooperative technology and intelligent network technology, fine control of traffic flow is a common means to alleviate intersection congestion, reduce overall delays in the fleet, reduce traffic conflicts, and avoid unnecessary energy waste. Compared with the single-lane control scenario, the interweaving operation of vehicles with different needs in the multi-lane scenario increases the difficulty of traffic control. Vehicles not only have longitudinal following behavior, but also make lateral lane change decisions according to their driving goals. According to the current development process of intelligent network technology, connected and automated vehicles (CAV) will gradually replace manually driven vehicles (Regular Vehicle, RV), but in the short term, CAV cannot reach 100% penetration. The mixed fleet composed of CAV and RV will exist as a new traffic normal for a long time in the future. In the scenario of multi-lane entrance lanes at intersections, due to the constraints of fixed signal phase sequence, the random lane-changing behavior of RV driven by different driving purposes will cause traffic conflicts between vehicles, reducing the safety and stability of mixed-vehicle fleet operation. At the same time, the arrival distribution of vehicles with different driving goals and the degree of adaptation of intersection signal timing schemes will affect the fleet's traffic efficiency. Therefore, the research on the coordinated guidance and control method for mixed-vehicle fleets in multi-lane entrance lanes at intersections is of great significance to ensure the safety and stability of fleet operation and improve the traffic efficiency of intersections.

[0003] The multi-agent reinforcement learning method learns the optimal control strategy directly from the empirical data through trial and error game among multiple agents. Its computational burden mainly lies in the offline training process, while the learned driving strategy can be implemented quickly in real time. At present, some agent reinforcement methods have been applied to the control of fleets to improve the operational stability, energy economy and safety of the fleet. This method shows good control performance in complex traffic environments. The control problem of mixed-traffic fleets at intersections with multiple entrance lanes has high requirements on the real-time performance of the solution method, while the multi-agent reinforcement learning method has a relatively fast calculation speed and can provide the optimal solution while meeting the real-time requirements of fleet control. Existing research on the use of multi-agent reinforcement learning methods to achieve fleet ecological control is mostly real-time optimization of the running trajectories of all individual vehicles in the fleet, which requires a large amount of calculation and is difficult to reach the global optimal state. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the problems of large amount of calculation and difficulty in reaching the global optimal state in the prior art, and to provide a method for collaborative guidance and control of mixed-traffic fleets at intersection entrances.

[0005] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions: the method for cooperative guidance control of mixed traffic vehicles at the entrance of an intersection comprises the following steps:

[0006] 1) Obtain road information:

[0007] The acquisition of road information refers to determining the control scenario, defining the control scope, control object and control target;

[0008] 2) Divide the agents:

[0009] The division of intelligent agents is based on road traffic information and the division of different intelligent agents by the following behaviors between vehicles;

[0010] 3) Establish a multi-agent ecological collaborative control model for mixed traffic fleets at intersection entrances:

[0011] First, a single-agent speed trajectory planning model is established to calculate the optimal trajectory of the single agent. Then, based on this, a multi-agent collaborative control model is established with the overall operation efficiency of the fleet as the optimization goal to achieve collaborative control among multiple agents. After the model is built, the model is trained using reinforcement learning methods.

[0012] 4) The controllable agent executes the optimized trajectory:

[0013] The established and trained model is sent to each controllable intelligent agent. A rolling time domain update strategy is adopted to calculate the speed trajectory of each intelligent agent through the intersection using road traffic information and input it into the CAV controller for execution.

[0014] The steps of obtaining road information described in the technical solution are as follows:

[0015] 1) The control scenario of the present invention is a single three-lane entrance road at a conventional crossroad of an urban road. The lane attributes include left turn, straight ahead and right turn. The lane information is parameterized as follows:

[0016] L int ={l1,l2,l3}

[0017] Where: l1, l2 and l3 are lane attributes. Different values ​​represent different lane attributes. -1 represents the left turn lane, 0 represents the through lane, and 1 represents the right turn lane.

[0018] 2) In the control scenario of the present invention, the intersection signal control type is fixed four-phase signal timing, and the signal timing scheme is expressed as follows:

[0019]

[0020] Where: sig int Represents the intersection signal timing plan information set, It represents the timing information set of phase k (k is 1, 2, 3, 4); the specific expression is as follows:

[0021]

[0022] Where: The value of represents the phase type of the k phase, where the value 1 represents the north-south straight phase, -1 represents the east-west straight phase, 2 represents the north-south left turn phase, and -2 represents the east-west left turn phase; in the intersection signal control of the present invention, the right-turn traffic is not constrained by the signal timing scheme, T k phase Indicates the duration information of the red light and green light in phase k, specifically expressed as follows:

[0023]

[0024] Where: and Respectively represent the red light duration and green light duration of signal phase k.

[0025] 3) The control range of the mixed-vehicle convoy coordinated guidance control model for the entrance of a multi-lane intersection based on multi-agent reinforcement learning proposed in the present invention is from the intersection stop line to 800m away from the intersection stop line. The control object is the mixed-vehicle convoy composed of CAVs and RVs within the control range. The mixed-vehicle convoy information of each lane is expressed as follows:

[0026]

[0027] Where: F int Represents the set of traffic flow information within the control range, and It represents the mixed traffic flow information set in each lane, as shown in the following formula:

[0028]

[0029] Where: C i_type Indicates the vehicle type of vehicle i (i is 1, 2, ..., I) on lane L (L is 1, 2, 3); its value is 0 or 1, where 0 represents CAV and 1 represents RV.

[0030] 4) Acquire the operation information of CAV within the control range of the intersection, including the speed, acceleration and position of the vehicle. CAV can obtain the operation information of surrounding vehicles and the traffic status information of the intersection through V2V and V2I communication. The present invention realizes the control of mixed traffic flow in the above control scenario by utilizing this characteristic of CAV; the position, speed, acceleration and other information of each CAV are expressed as follows:

[0031]

[0032]

[0033] Where: Indicates l L The total set of CAV operation information on the lane, is the operation information set of CAVh (the value of h is 1, 2, ..., H); is the current location of the CAV; is the current speed of CAV; is the current acceleration of CAV.

[0034] The present invention aims to optimize the speed-space trajectory of CAVs in the traffic flow, thereby adjusting the overall distribution and operation status of the traffic flow, guiding different traffic flow arrival distributions to be more compatible with the fixed signal timing scheme of the intersection, so as to pass through the intersection in a safer, more stable, efficient and energy-saving manner.

[0035] The division of agents described in the technical solution includes the following steps:

[0036] 1) Based on the information obtained in the road information acquisition step, in order to reflect the guiding role of CAV on RV and facilitate the subsequent model building, according to different vehicle arrival distributions and the following behaviors between vehicles, the vehicles are divided into vehicle groups of different sizes and compositions, and further abstracted into intelligent agents;

[0037] Based on different vehicle group compositions, the present invention divides the fleet on each lane within the control range into the following three types of agents: a vehicle group consisting of one CAV or multiple CAVs is defined as a CAV agent CAV ;

[0038] The vehicle group consisting of a CAV as the lead vehicle and several RVs following behind is defined as the minimum ecological control unit. On this basis, this type of vehicle group is abstracted into an intelligent agent, defined as the minimum ecological control unit intelligent agent agent ECU ;

[0039] A group of one or more RVs that are independent and not affected by the vehicle in front is defined as an RV agent. RV ;

[0040] Among them, since there is a CAV in the vehicle group, the present invention uses the CAV agent CAV and ECU agent ECU Defined as a controllable agent, RV agent RV Defined as an uncontrollable agent.

[0041] When the distance between the RV and the front vehicle is in the indifference area When the present invention determines that the vehicle in front belongs to the same control unit, and Represents the maximum and minimum values ​​of the indifference area, and its calculation formula is as follows:

[0042]

[0043] Where: Bound = max or min; T bound is the expected time interval under Bound conditions; S0 is the parking distance; v i (t) is the speed of RV at time t; Δv i (t) is the speed difference between RV and the preceding vehicle at time t; α max is the maximum acceleration of the vehicle; β is the comfortable deceleration of the vehicle;

[0044] 2) Based on the information obtained in the step of obtaining road information, the possible lane changing behavior of the RV is predicted. In the control scenario of the present invention, the CAV enters the control range through its target lane, so there is no lane changing demand during operation. As an uncontrollable vehicle, the driving behavior of the RV has certain uncertainty. The present invention calculates the lane changing probability of the RV by the following formula:

[0045]

[0046] Where: To indicate the possibility of RV changing lanes, The larger the value, the greater the possibility that RV will perform lane change under the current traffic conditions: nece and safe They represent the necessity of lane change according to the driving goal of RV and the safety condition for lane change respectively; the expressions of the two parameters are:

[0047]

[0048]

[0049]

[0050]

[0051] Where: p line is the longitudinal critical position of RV lane change; p i (t) is the position of vehicle i at time t; λ i is the number of lane changes required by vehicle i; d i,j (t) is the distance between vehicle i and vehicle j at time t; D i,j (t) is the safe distance between vehicle i and vehicle j at time t; ρ is the safe headway; a i,max,brake is the maximum braking deceleration of vehicle i; a i,min,brake is the minimum braking deceleration of vehicle i; a i,brake (t) is the braking deceleration of vehicle i at time t; v i,max is the maximum speed that vehicle i can travel on the road section.

[0052] The technical solution for establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance includes the following steps:

[0053] 1) Based on the agent division step, the controllable agents obtained by dividing the fleet on each lane are divided, and a single-agent speed trajectory planning model is established with the optimization goal of minimizing the cumulative energy consumption of the speed trajectory of a single agent passing through the intersection;

[0054] 2) Considering the overall traffic efficiency of mixed traffic flow, a multi-agent collaborative control model is established;

[0055] 3) Multi-agent collaborative control model training;

[0056] The single-agent speed trajectory planning model established in step 1) and the multi-agent collaborative control model established in step 2) of the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance are trained through an improved twin-delayed deep deterministic policy gradient (TD3) algorithm. The purpose of the training is to simulate the fleet's multiple interactions with the environment based on the optimization results of the model established by the present invention, so as to obtain the overall operating benefits of the fleet corresponding to different decisions of the agents under different environments and vehicle operation information, obtain the optimal benefits through repeated interactions, and store the optimal benefits and corresponding decisions as known experience. In the actual operation of the fleet, if there is an operating condition that is consistent with the known experience, the corresponding optimal solution can be immediately obtained, thereby improving the optimization efficiency.

[0057] The controllable agents obtained by dividing the fleet on each lane based on the agent division step described in the technical solution are used to minimize the cumulative energy consumption of the speed trajectory of a single agent passing through the intersection, and a single agent speed trajectory planning model is established; the specific steps are as follows:

[0058] (1) Calculate the individual optimal speed of a single agent. Considering that a single agent may contain several fuel vehicles (FV) and several electric vehicles (EV), their energy consumption types are different and therefore need to be considered separately. The individual optimal speed of a single agent is to minimize the operating energy consumption of all vehicles within the agent. The calculation formula is as follows:

[0059] Z=minE agent

[0060]

[0061] Where: μ fv and μ ev are the unified coefficients of fuel consumption and electricity consumption respectively; m and n are the number of FV and EV in the intelligent body respectively; T0 is the starting time; T d_i is the time it takes for a vehicle to pass through the stop line at the intersection; v i (t) and a i (t) are the speed and acceleration of the vehicle, v i The value of (t) is at the minimum speed limit v min With maximum speed limit v max Between i The value of (t) is at the minimum acceleration a min With the maximum acceleration a max Between; F f_i and F e_i They are the instantaneous fuel consumption of fuel vehicles and the instantaneous power consumption of electric vehicles;

[0062] (2) Based on the ecological speed obtained in (1), in order to achieve smooth guidance of the CAV to the RV, the speed trajectory of the intelligent agent from the current speed to the ecological speed is calculated by trigonometric function. The calculation formula is as follows:

[0063]

[0064] Where: v opt (t) is the speed of the vehicle after optimization based on trigonometric function at time t; v aim is the average target speed of the vehicle; v dis is the difference between the average target speed of the vehicle and the initial speed v0; ω tr When the vehicle speed is between the current speed v con and the average target speed vaim The rate of change of acceleration / deceleration between The vehicle speed is lower than the average target speed v aim The rate of change of acceleration / deceleration at the time of f The target distance D int and the average target speed v aim The ratio of

[0065] (3) Considering the uncertainty of the driving behavior of the preceding vehicle, the speed trajectory of the single agent is constrained by a virtual trajectory. The specific steps are as follows:

[0066] a. If the front vehicle is in motion, a dynamic virtual trajectory is used to constrain the agent to ensure that the agent maintains a safe following distance from the front vehicle. The dynamic virtual trajectory calculation formula is as follows:

[0067]

[0068]

[0069]

[0070] Where: and is the intermediate quantity; τ is the driver’s reaction time;

[0071] b. If the vehicle in front is in a queue, the agent can only pass through the intersection after the queue of vehicles in front dissipates. The present invention uses a static virtual trajectory s p_i (t) constraining the agent;

[0072] (4) Calculate the lane-changing probability of the RVs inside and around the controllable agent by step 2) in the step of dividing the agents. According to the obtained lane-changing probability, analyze the influence of the lane-changing behaviors of the internal RV leaving the agent, the surrounding RV entering the agent, and the surrounding RV inserting in front of the agent on the overall ecology of the fleet and the operation efficiency of the intersection. If the lane-changing behavior has no positive effect, the relevant CAV suppresses the lane-changing behavior of the RV by compressing the possible lane-changing interval of the RV; otherwise, the relevant CAV gives it a safe lane-changing space to ensure the safe lane-changing of the RV. For the lane-changing behavior that causes the internal composition of the agent to change, return to step (1), step (2), and step (3) in step 1) in the step of establishing the multi-agent ecological collaborative control model of the mixed-traffic fleet at the entrance of the intersection to recalculate and generate the speed trajectory of the single agent;

[0073] For the RV that is inserted in front of the agent due to lane changing behavior, the calculation formula of the dynamic virtual trajectory described in (3) a is transformed into the following formula:

[0074]

[0075] Where: x and y are the longitudinal and lateral displacements of the agent at time t; p i and q j are coefficients that determine the trajectory shape; since the longitudinal and lateral displacements are represented using time-based functions, the velocity v at each time node can be easily obtained by taking the first and second order derivatives x and v y and acceleration a x and a y .

[0076] By establishing step 1) of the multi-agent ecological collaborative control model for mixed traffic fleets at intersection entrances, the optimal speed trajectory of each agent is calculated, and the speed trajectory serves as the initial state input of the subsequent multi-agent collaborative control model.

[0077] The technical solution takes into account the overall traffic efficiency of mixed traffic flow and establishes a multi-agent collaborative control model. The specific steps are as follows:

[0078] (1) In order to reduce the amount of calculation, the single-agent speed trajectory planning model is linked to the multi-agent collaborative control model, and the time it takes for the agent to pass through the intersection is used as the action space of the multi-agent collaborative control model:

[0079] a j =T d_j

[0080] The time it takes for each agent to pass the intersection is calculated using the ecological speed trajectory obtained from the single agent ecological speed trajectory planning model in 1). As the initial action of the multi-agent collaborative control model, after the action of the agent is updated, the speed trajectory corresponding to the new action can be output through the smoothness constraint obtained by the trigonometric function in the single-agent speed trajectory planning model;

[0081] (2) In the present invention, the state space of the agent is defined as the speed and position of the agent in the current action:

[0082] s j =(v j (t),p j (t))

[0083] Where: p j (t) is the position of the agent at the current moment;

[0084] (3) In order to ensure that the training result of the agent is as close to the optimal solution as possible, the present invention designs the reward function as follows. The specific steps are as follows:

[0085] a. First, calculate the reward function of a single agent. Considering the safety of the agent during movement, the maximum available deceleration is used as the safety indicator of the agent's operation. The safety reward function calculation formula is as follows:

[0086]

[0087] Where: a safe is the maximum available deceleration of the leading vehicle to avoid conflict with the preceding vehicle; safe For penalty items;

[0088] b. Establishing the multi-agent ecological collaborative control model for mixed-traffic vehicles on the entrance road of an intersection In step 1), the optimal ecological speed trajectory of a single agent is calculated. Therefore, the present invention believes that the closer the action is to the initial action of the agent, the smaller the overall energy consumption of the corresponding speed trajectory, so the calculation formula of the ecological reward function is obtained as follows:

[0089]

[0090] c. By combining the safety reward function and the ecological reward function with a certain weight, the reward value of a single agent can be obtained. The calculation formula is as follows:

[0091]

[0092] Where: w1 and w2 are weight coefficients;

[0093] d. The number of vehicles passing through the intersection stop line in each lane of the target entrance road within a cycle is used as the efficiency reward function of the intersection to reflect the traffic efficiency of the entire entrance road. The calculation formula of the entrance road efficiency reward function is as follows:

[0094]

[0095] Where: N pass_li is the lane l in a signal cycle i The number of vehicles passing through the intersection;

[0096] e. After adding up the single agent reward values ​​of all controllable agents within the control range, it is combined with the entrance efficiency reward function with a certain weight coefficient to obtain the reward value of the entire fleet. The calculation formula is:

[0097]

[0098] Where: w3 and w4 are weight coefficients.

[0099] The technical solution uses an improved Twin Delayed Deep Deterministic policy gradient (TD3) algorithm to train the single-agent speed trajectory planning model established in step 1) and the multi-agent collaborative control model established in step 2) of the step of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance. The purpose of the training is to simulate the fleet's optimization results based on the model established by the present invention and the environment for multiple interactions, thereby obtaining the overall operation benefits of the fleet corresponding to different decisions of the agents under different environments and vehicle operation information, and obtaining the optimal benefits through repeated interactions. The optimal benefits and corresponding decisions are stored as known experience. In the actual operation of the fleet, if there is an operating condition that is consistent with the known experience, the corresponding optimal solution can be immediately obtained, thereby improving the optimization efficiency. The specific steps are as follows:

[0100] (1) First, use the DQN solver to solve the single-agent velocity trajectory planning model. The specific steps are as follows:

[0101] a. Initialize the experience pool Its capacity N single , action-value function and the target network

[0102] b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M single between;

[0103] c. Select action And execute, in exploration mode, with ε single The probability of randomly selecting an action In experience mode, select Next The maximum movement is calculated as follows:

[0104]

[0105] d. Action Interact with the environment and get rewards And determine the next state Whether the termination condition is triggered;

[0106] e. Experience Deposit into experience pool

[0107] f. Repeat steps b to e until the experience pool is full;

[0108] g. Select a small batch of experience from the experience pool and calculate the target network The objective function is calculated as follows:

[0109]

[0110] Where: γ single is the discount factor;

[0111] h. Calculate the loss function:

[0112]

[0113] i. Use the gradient descent method to minimize the loss function of step h, thereby updating the target network Weight

[0114] j. Every T single Repeat steps g to i at fixed intervals to update the target network parameters;

[0115] k. Repeat the above steps until the target network converges or the total number of cycles M is reached. single , end the training; after completing the training of the single-agent speed trajectory planning model, solve the model, and the obtained speed trajectory is the optimal speed trajectory of the single agent itself. Calculate the time for each agent to pass the intersection based on the trajectory, and use it as the initial action when training the multi-agent collaborative control model;

[0116] (2) The present invention uses the improved TD3 algorithm to train the multi-agent collaborative control model. The specific steps are as follows:

[0117] a. Initialize the Actor network μ(θ μ ), Critic1 Network Critic2 Network Target Actor Network μ'(θ μ′ ), target Critic1 network Target Critic2 Network Its capacity N single ;

[0118] b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M multi between;

[0119] c. Select action To improve the efficiency of the action search of the agent during the training process, the present invention designs the following strategy: As a starting point for the search, that is:

[0120]

[0121] If the reward value obtained by the action shifting to the left has an upward trend, update the action to the left, otherwise update it to the right;

[0122] d. Action Interact with the environment to obtain single-agent rewards After that, calculate the overall reward value of the intersection And determine the next state Whether the termination condition is triggered;

[0123] e. Experience Deposit into experience pool

[0124] f. Repeat steps b to e of step (2) in the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance until the experience pool becomes full;

[0125] g. Select a small batch of experience from the experience pool Using the target Actor network μ'(θ μ' ) calculated The following actions are taken and noise is added based on the target strategy regularization, in order to limit the complexity of the model and avoid overfitting as much as possible:

[0126]

[0127] h. Based on the dual network idea, calculate the target value:

[0128]

[0129] Where: γ single is the discount factor.

[0130] i. Calculate the loss function:

[0131]

[0132] j. Use gradient descent to minimize the loss function of step i and update the Critic1 network and Critic1 network Parameters in ;

[0133] k.Critic1 Network and Critic2 Network Updated critic Then, using the target Actor network μ'(θ μ' )Calculation status Next action:

[0134]

[0135] l. Using the Critic1 network and Critic2 Network Calculate the pair Evaluated value:

[0136]

[0137] m. Use the gradient ascent method to maximize step l Complete the Actor network μ(θ μ ) updates;

[0138] n. The new and old target network parameters are weighted averaged through the learning rate κ to complete the update of the target network parameters:

[0139]

[0140] θ μ′ =κθ μ +(1-κ)θ μ′

[0141] o. Repeat the above steps until the target network converges or the total number of cycles M is reached multi , end the training;

[0142] Step 3) of the steps of establishing a multi-agent ecological collaborative control model for a mixed traffic fleet at an intersection entrance completes the training of the multi-agent collaborative control model proposed by the present invention.

[0143] The controllable intelligent agent described in the technical solution executes the optimized trajectory including the following steps:

[0144] 1) Send the trained multi-agent collaborative control model to the server in the intersection control area;

[0145] 2) Perform step 1 to obtain various information parameters of the intersection and upload them to the server. When a vehicle enters the control area, obtain the vehicle composition information of the mixed-traffic fleet and upload it to the fleet operation information storage;

[0146] 3) Based on the obtained information, the mixed traffic fleet on each lane in the control area is divided into agents of various sizes and types;

[0147] 4) The fleet operation information storage device uploads the mixed-traffic fleet information in the current control area to the intersection server, and performs step 1) of the steps of establishing the mixed-traffic fleet multi-agent ecological collaborative control model of the intersection entrance road, optimizes the initial ecological speed trajectory of the controllable agent entering the control area through the intersection, and sends it to the controller of the CAV in each controllable agent for execution;

[0148] 5) Perform step 2) of the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at the entrance of an intersection, calculate the time it takes for each agent to pass through the intersection under the current speed trajectory, and the intersection server performs secondary optimization on the speed trajectory of each controllable agent based on the operating conditions of each fleet at the current intersection, taking into account the energy consumption, exhaust emission level, and traffic efficiency of the fleet during operation, and outputs the optimization results to each CAV controller for execution;

[0149] 6) The fleet operation information storage device uploads the stored information to the historical database of the intersection server at regular intervals. The server uses the updated information to re-acquire road information steps, divide intelligent agents, and establish a multi-agent ecological collaborative control model for mixed-traffic fleets at intersection entrances, thereby ensuring the safety and stability of mixed-traffic fleet operations while ensuring the traffic efficiency of intersection entrances.

[0150] Compared with the prior art, the present invention has the following beneficial effects:

[0151] Aiming at the demand for safe and efficient operation of mixed traffic flow at multi-lane entrances of urban intersections, the present invention designs a multi-agent collaborative control method that takes into account the uncertainty of RV driving behavior.

[0152] 1. In order to facilitate the establishment of the model, reduce the amount of calculation and better reflect the guiding role of CAV on RV in the method for collaborative guidance and control of mixed-traffic fleets at the entrance lane of the intersection described in the present invention, the vehicle group formed by the following behavior in the fleet is defined as an intelligent agent. During the operation of the vehicle, CAV and RV have a certain probability of changing lanes due to their respective driving goals. The lane changing behavior of the vehicle involves the splitting and recombining of the intelligent agent before and after. According to different types of vehicle combinations, three types of intelligent agents are defined, including CAV agents, RV agents and minimum ecological control unit agents. The RV agent is regarded as an uncontrollable intelligent agent, and the CAV agent and the minimum ecological control unit agent are regarded as controllable intelligent agents. A multi-agent collaborative control optimization model is established.

[0153] 2. The method for collaborative guidance and control of mixed-vehicle fleets at intersection entrances described in the present invention first calculates the individual optimal speed for a controllable intelligent agent based on the power structure of the vehicles within the intelligent agent, outputs the optimal speed of the intelligent agent during the current optimization period, and then uses trigonometric functions to smoothly transition from the current speed to the target speed and pass the speed trajectory of the intersection. On this basis, the possibility of lane changing of surrounding related RVs is calculated and predicted in combination with traffic information, and its lane changing trajectory is used as a constraint on the trajectory of the controllable intelligent agent.

[0154] 3. The method for collaborative guidance and control of mixed traffic vehicles at the entrance of an intersection described in the present invention can effectively improve the safety and stability of mixed traffic flow at the entrance of a multi-lane intersection and reduce safety risks during the operation of traffic, while taking into account the vehicle traffic efficiency, avoiding unnecessary waiting of vehicles for red lights at intersections, and overcoming the large-scale congestion problem caused by queuing of vehicles at current urban road intersections, which is conducive to the realization and application of urban intelligent transportation systems under future intelligent network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0155] The present invention will be further described below in conjunction with the accompanying drawings:

[0156] Figure 1 It is a flowchart of the method for cooperative guidance control of mixed traffic vehicles at the entrance of an intersection according to the present invention;

[0157] Figure 2 A schematic diagram of an RV intelligent body in the method for cooperative guidance and control of a mixed vehicle fleet at an intersection entrance of the present invention;

[0158] Figure 3 It is a schematic diagram of the minimum ecological control unit intelligent body in the coordinated guidance control method for mixed traffic vehicles at the entrance of the intersection according to the present invention;

[0159] Figure 4 A schematic diagram of a CAV agent in the method for cooperatively guiding and controlling a mixed-vehicle fleet at an intersection entrance of the present invention;

[0160] Figure 5 A schematic diagram of intersection settings based on a traffic simulation platform in one embodiment of the method for cooperative guidance control of mixed-vehicle fleets at intersection entrances of the present invention;

[0161] Figure 6 It is a schematic diagram of the operation of a control model in a traffic simulation platform in one embodiment of the method for cooperative guidance control of mixed-vehicle fleets at intersection entrances of the present invention;

[0162] Figure 7 It is a vehicle running trajectory diagram under the action of the proposed control model in one embodiment of the coordinated guidance control method for mixed traffic vehicles at the entrance of an intersection described in the present invention. DETAILED DESCRIPTION

[0163] The present invention is described in detail below in conjunction with the accompanying drawings:

[0164] See also Figure 1 The method for cooperatively guiding and controlling a mixed-vehicle vehicle fleet at an intersection entrance of the present invention comprises the following steps:

[0165] 1. Obtain road information

[0166] The acquisition of road information refers to determining the control scenario, defining the control range, control object and control target; the steps include:

[0167] 1) The control scenario of the present invention is a single three-lane entrance road at a conventional crossroad of an urban road. The lane attributes include left turn, straight ahead and right turn. The lane information is parameterized as follows:

[0168] L int ={l1,l2,l3}

[0169] Where: l1, l2 and l3 are lane attributes. Different values ​​represent different lane attributes. -1 represents the left turn lane, 0 represents the through lane, and 1 represents the right turn lane.

[0170] 2) In the control scenario of the present invention, the intersection signal control type is fixed four-phase signal timing, and the signal timing scheme is expressed as follows:

[0171]

[0172] Where: sig int Represents the intersection signal timing plan information set, It represents the timing information set of phase k (k is 1, 2, 3, 4); the specific expression is as follows:

[0173]

[0174] Where: The value represents the phase type of the k phase, where the value 1 represents the north-south straight phase, -1 represents the east-west straight phase, 2 represents the north-south left turn phase, and -2 represents the east-west left turn phase; in the intersection signal control of the present invention, the right-turn traffic is not constrained by the signal timing scheme, Indicates the duration information of the red light and green light in phase k, specifically expressed as follows:

[0175]

[0176] Where: and Respectively represent the red light duration and green light duration of signal phase k.

[0177] 3) The control range of the mixed-vehicle convoy coordinated guidance control model for the entrance of a multi-lane intersection based on multi-agent reinforcement learning proposed in the present invention is from the intersection stop line to 800m away from the intersection stop line. The control object is the mixed-vehicle convoy composed of CAVs and RVs within the control range. The mixed-vehicle convoy information of each lane is expressed as follows:

[0178]

[0179] Where: F int Represents the set of traffic flow information within the control range, and It represents the mixed traffic flow information set in each lane, as shown in the following formula:

[0180]

[0181] Where: C i_type Indicates the vehicle type of vehicle i (i is 1, 2, ..., I) on lane L (L is 1, 2, 3); its value is 0 or 1, where 0 represents CAV and 1 represents RV.

[0182] 4) Acquire the operation information of CAV within the control range of the intersection, including the speed, acceleration and position of the vehicle. CAV can obtain the operation information of surrounding vehicles and the traffic status information of the intersection through V2V and V2I communication. The present invention realizes the control of mixed traffic flow in the above control scenario by utilizing this characteristic of CAV; the position, speed, acceleration and other information of each CAV are expressed as follows:

[0183]

[0184]

[0185] Where: Indicates l L The total set of CAV operation information on the lane, is the operation information set of CAVh (the value of h is 1, 2, ..., H); is the current location of the CAV; is the current speed of CAV; is the current acceleration of CAV;

[0186] The present invention aims to optimize the speed and space-time trajectory of CAVs in the traffic flow, thereby adjusting the overall distribution and operation status of the traffic flow, guiding different traffic flow arrival distributions to be more compatible with the fixed signal timing scheme of the intersection, so as to pass through the intersection in a safer, more stable, efficient and energy-saving manner;

[0187] 2. Divide Agents

[0188] See also Figure 2 , 3 As in 4, the division of intelligent agents is based on road traffic information, and the division of different intelligent agents is based on the following behaviors of vehicles; the steps are as follows:

[0189] 1) Based on the information obtained in step 1, i.e., obtaining road information, in order to reflect the guiding role of CAV on RV and facilitate the subsequent model building, according to different vehicle arrival distributions and the following behaviors between vehicles, the vehicles are divided into vehicle groups of different sizes and compositions, and further abstracted into intelligent agents;

[0190] Based on different vehicle group compositions, the present invention divides the fleet on each lane within the control range into the following three types of agents: a vehicle group consisting of one CAV or multiple CAVs is defined as a CAV agent CAV ;

[0191] The vehicle group consisting of a CAV as the lead vehicle and several RVs following behind is defined as the minimum ecological control unit. On this basis, this type of vehicle group is abstracted into an intelligent agent, defined as the minimum ecological control unit intelligent agent agent ECU ;

[0192] A group of one or more RVs that are independent and not affected by the vehicle in front is defined as an RV agent. RV ;

[0193] Among them, since there is a CAV in the vehicle group, the present invention uses the CAV agent CAV and ECU agent ECU Defined as a controllable agent, RV agent RV Defined as an uncontrollable agent.

[0194] When the distance between the RV and the front vehicle is in the indifference area When the present invention determines that the vehicle in front belongs to the same control unit, and Represents the maximum and minimum values ​​of the indifference area, and its calculation formula is as follows:

[0195]

[0196] Where: Bound = max or min; T bound is the expected time interval under Bound conditions; S0 is the parking distance; v i (t) is the speed of RV at time t; Δv i (t) is the speed difference between RV and the preceding vehicle at time t; α max is the maximum acceleration of the vehicle; β is the comfortable deceleration of the vehicle;

[0197] 2) Based on the information obtained in step 1, i.e., the step of obtaining road information, predict the possible lane-changing behavior of the RV. In the control scenario of the present invention, the CAV enters the control range through its target lane, so there is no lane-changing requirement during operation. As an uncontrollable vehicle, the driving behavior of the RV has certain uncertainty. The present invention calculates the lane-changing probability of the RV by the following formula:

[0198]

[0199] Where: To indicate the possibility of RV changing lanes, The larger the value, the greater the possibility that RV will perform lane change under the current traffic conditions: nece and safe They represent the necessity of lane change according to the driving goal of RV and the safety condition for lane change respectively; the expressions of the two parameters are:

[0200]

[0201]

[0202]

[0203]

[0204] Where: p line is the longitudinal critical position of RV lane change; p i (t) is the position of vehicle i at time t; λ i is the number of lane changes required by vehicle i; d i,j (t) is the distance between vehicle i and vehicle j at time t; D i,j (t) is the safe distance between vehicle i and vehicle j at time t; ρ is the safe headway; a i,max,brake is the maximum braking deceleration of vehicle i; a i,min,brake is the minimum braking deceleration of vehicle i; a i,brake (t) is the braking deceleration of vehicle i at time t; v i,max is the maximum speed that vehicle i can travel on the road section;

[0205] 3. Establish a multi-agent ecological collaborative control model for mixed traffic fleets at intersection entrances

[0206] First, a single-agent speed trajectory planning model is established to calculate the optimal trajectory of the single-agent itself. Then, based on this, the overall operation efficiency of the fleet is taken as the optimization goal, and a multi-agent collaborative control model is established to achieve collaborative control among multiple agents. After the model is built, the model is trained using the reinforcement learning method. The steps are as follows:

[0207] 1) Based on step 2, i.e. the agent division step, the controllable agents obtained by dividing the fleet on each lane are established, and the cumulative energy consumption of the speed trajectory of a single agent passing through the intersection is minimized as the optimization goal. A single-agent speed trajectory planning model is established. The specific steps are as follows:

[0208] (1) Calculate the individual optimal speed of a single agent. Considering that a single agent may contain several fuel vehicles (FV) and several electric vehicles (EV), their energy consumption types are different and therefore need to be considered separately. The individual optimal speed of a single agent is to minimize the operating energy consumption of all vehicles within the agent. The calculation formula is as follows:

[0209] Z=minE agent

[0210]

[0211] Where: μ fv and μ ev are the unified coefficients of fuel consumption and electricity consumption respectively; m and n are the number of FV and EV in the intelligent body respectively; T0 is the starting time; T d_i is the time it takes for a vehicle to pass the stop line at the intersection; v i (t) and a i (t) are the speed and acceleration of the vehicle, v i The value of (t) is at the minimum speed limit v min With maximum speed limit v max Between i The value of (t) is at the minimum acceleration a min With the maximum acceleration a max Between; F f_i and F e_i They are the instantaneous fuel consumption of fuel vehicles and the instantaneous power consumption of electric vehicles;

[0212] (2) Based on the ecological speed obtained in (1), in order to achieve smooth guidance of the CAV to the RV, the speed trajectory of the intelligent agent from the current speed to the ecological speed is calculated by trigonometric function. The calculation formula is as follows:

[0213]

[0214] Where: v opt (t) is the speed of the vehicle after optimization based on trigonometric function at time t; v aim is the average target speed of the vehicle; v dis is the difference between the average target speed of the vehicle and the initial speed v0; ω tr When the vehicle speed is between the current speed vcon and the average target speed v aim The rate of change of acceleration / deceleration between The vehicle speed is lower than the average target speed v aim The rate of change of acceleration / deceleration at the time of f The target distance D int and the average target speed v aim The ratio of

[0215] (3) Considering the uncertainty of the driving behavior of the preceding vehicle, the speed trajectory of the single agent is constrained by a virtual trajectory. The specific steps are as follows:

[0216] a. If the front vehicle is in motion, a dynamic virtual trajectory is used to constrain the agent to ensure that the agent maintains a safe following distance from the front vehicle. The dynamic virtual trajectory calculation formula is as follows:

[0217]

[0218]

[0219]

[0220] Where: and is the intermediate quantity; τ is the driver’s reaction time;

[0221] b. If the vehicle in front is in a queue, the agent can only pass through the intersection after the queue of vehicles in front dissipates. The present invention uses a static virtual trajectory s p_i (t) constraining the agent;

[0222] (4) Calculate the lane-changing probability of the RVs inside and around the controllable agent by step 2, i.e., step 2) in the step of dividing the agents. According to the obtained lane-changing probability, analyze the influence of the lane-changing behaviors of the internal RV leaving the agent, the surrounding RV entering the agent, and the surrounding RV inserting in front of the agent on the overall ecology of the fleet and the operation efficiency of the intersection. If the lane-changing behavior has no positive effect, the relevant CAV suppresses the lane-changing behavior of the RV by compressing the possible lane-changing interval of the RV; otherwise, the relevant CAV gives it a safe lane-changing space to ensure the safe lane-changing of the RV. For the lane-changing behavior that causes the internal composition of the agent to change, return to step 3, i.e., step 1) in the step of establishing the multi-agent ecological collaborative control model of the mixed-traffic fleet at the entrance of the intersection, and recalculate and generate the speed trajectory of the single agent;

[0223] For the RV that is inserted in front of the agent due to lane changing behavior, the calculation formula of the dynamic virtual trajectory described in (3) a is transformed into the following formula:

[0224]

[0225] Where: x and y are the longitudinal and lateral displacements of the agent at time t; p i and q j are coefficients that determine the trajectory shape; since the longitudinal and lateral displacements are represented using time-based functions, the velocity v at each time node can be easily obtained by taking the first and second order derivatives x and v y and acceleration a x and a y .

[0226] Through step 3, i.e., step 1) of the step of establishing the multi-agent ecological collaborative control model of the mixed traffic fleet at the entrance of the intersection, the optimal speed trajectory of each agent is calculated, and the speed trajectory is the initial state input of the subsequent multi-agent collaborative control model;

[0227] 2) Considering the overall traffic efficiency of mixed traffic, a multi-agent collaborative control model is established. The specific steps are as follows:

[0228] (1) In order to reduce the amount of calculation, the single-agent speed trajectory planning model is linked to the multi-agent collaborative control model, and the time it takes for the agent to pass through the intersection is used as the action space of the multi-agent collaborative control model:

[0229] a j =T d_j

[0230] The time it takes for each agent to pass the intersection is calculated using the ecological speed trajectory obtained from the single agent ecological speed trajectory planning model in 1). As the initial action of the multi-agent collaborative control model, after the action of the agent is updated, the speed trajectory corresponding to the new action can be output through the smoothness constraint obtained by the trigonometric function in the single-agent speed trajectory planning model;

[0231] (2) In the present invention, the state space of the agent is defined as the speed and position of the agent in the current action:

[0232] s j =(v j (t),p j (t))

[0233] Where: p j (t) is the position of the agent at the current moment;

[0234] (3) In order to ensure that the training result of the agent is as close to the optimal solution as possible, the present invention designs the reward function as follows. The specific steps are as follows:

[0235] a. First, calculate the reward function of a single agent. Considering the safety of the agent during movement, the maximum available deceleration is used as the safety indicator of the agent's operation. The safety reward function calculation formula is as follows:

[0236]

[0237] Where: a safe is the maximum available deceleration of the leading vehicle to avoid conflict with the preceding vehicle; safe For penalty items;

[0238] b. Step 3 is to establish a multi-agent ecological collaborative control model for a mixed-traffic fleet at the entrance of an intersection. In step 1), the optimal ecological speed trajectory of a single agent is calculated. Therefore, the present invention believes that the closer the action is to the initial action of the agent, the smaller the overall energy consumption of the corresponding speed trajectory, so the calculation formula of the ecological reward function is obtained as follows:

[0239]

[0240] c. By combining the safety reward function and the ecological reward function with a certain weight, the reward value of a single agent can be obtained. The calculation formula is as follows:

[0241]

[0242] Where: w1 and w2 are weight coefficients;

[0243] d. The number of vehicles passing through the intersection stop line in each lane of the target entrance road within a cycle is used as the efficiency reward function of the intersection to reflect the traffic efficiency of the entire entrance road. The calculation formula of the entrance road efficiency reward function is as follows:

[0244]

[0245] Where: is the lane l in a signal cycle i The number of vehicles passing through the intersection;

[0246] e. After adding up the single agent reward values ​​of all controllable agents within the control range, it is combined with the entrance efficiency reward function with a certain weight coefficient to obtain the reward value of the entire fleet. The calculation formula is:

[0247]

[0248] Where: w3 and w4 are weight coefficients;

[0249] 3) Multi-agent collaborative control model training

[0250] The single-agent speed trajectory planning model established in step 1) and the multi-agent collaborative control model established in step 2) in step 3, i.e., the step of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance, are trained by an improved twin-delayed deep deterministic policy gradient (TD3) algorithm. The purpose of the training is to simulate the fleet to interact with the environment multiple times based on the optimization results of the model established by the present invention, so as to obtain the overall operation benefit of the fleet corresponding to different decisions of the agents under different environments and vehicle operation information, obtain the optimal benefit through repeated interactions, and store the optimal benefit and the corresponding decision as known experience. In the actual operation of the fleet, if there is an operation condition that is consistent with the known experience, the corresponding optimal solution can be immediately obtained, thereby improving the optimization efficiency. The specific steps are as follows:

[0251] (1) First, use the DQN solver to solve the single-agent velocity trajectory planning model. The specific steps are as follows:

[0252] a. Initialize the experience pool and its capacity Nsingle , action-value function and the target network

[0253] b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M single between;

[0254] c. Select action And execute, in exploration mode, with ε single The probability of randomly selecting an action In experience mode, select Next The maximum movement is calculated as follows:

[0255]

[0256] d. Action Interact with the environment and get rewards And determine the next state Whether the termination condition is triggered;

[0257] e. Experience Deposit into experience pool

[0258] f. Repeat steps b to f until the experience pool is full;

[0259] g. Select a small batch of experience from the experience pool and calculate the target network The objective function is calculated as follows:

[0260]

[0261] Where: γ single is the discount factor;

[0262] h. Calculate the loss function:

[0263]

[0264] i. Use the gradient descent method to minimize the loss function of step h, thereby updating the target network Weight

[0265] j. Every T single Repeat steps g to i at fixed intervals to update the target network parameters;

[0266] k. Repeat the above steps until the target network converges or the total number of cycles M is reached. single , end the training; after completing the training of the single-agent speed trajectory planning model, solve the model, and the obtained speed trajectory is the optimal speed trajectory of the single agent itself. Calculate the time for each agent to pass the intersection based on the trajectory, and use it as the initial action when training the multi-agent collaborative control model;

[0267] (2) The present invention uses the improved TD3 algorithm to train the multi-agent collaborative control model. The specific steps are as follows:

[0268] a. Initialize the Actor network μ(θ μ ), Critic1 Network Critic2 Network Target Actor Network μ'(θ μ′ ), target Critic1 network Target Critic2 Network Its capacity N single ;

[0269] b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M multi between;

[0270] c. Select action To improve the efficiency of the action search of the agent during the training process, the present invention designs the following strategy: As a starting point for the search, that is:

[0271]

[0272] If the reward value obtained by the action shifting to the left has an upward trend, update the action to the left, otherwise update it to the right;

[0273] d. Action Interact with the environment to obtain single-agent rewards After that, calculate the overall reward value of the intersection And determine the next state Whether the termination condition is triggered;

[0274] e. Experience Deposit into experience pool

[0275] f. Repeat step 3, i.e., establish a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance, from step b to step e in step (2), until the experience pool becomes full;

[0276] g. Select a small batch of experience from the experience pool Using the target Actor network μ'(θ μ' ) calculated The following actions are taken and noise is added based on the target strategy regularization, in order to limit the complexity of the model and avoid overfitting as much as possible:

[0277]

[0278] h. Based on the dual network idea, calculate the target value:

[0279]

[0280] Where: γ single is the discount factor.

[0281] i. Calculate the loss function:

[0282]

[0283] j. Use gradient descent to minimize the loss function of step i and update the Critic1 network and Critic1 network Parameters in ;

[0284] k.Critic1 Network and Critic2 Network Updated critic Then, using the target Actor network μ'(θ μ' )Calculation status Next action:

[0285]

[0286] l. Using the Critic1 network and Critic2 Network Calculate the pair Evaluated value:

[0287]

[0288] m. Use the gradient ascent method to maximize step l Complete the Actor network μ(θ μ ) updates;

[0289] n. The new and old target network parameters are weighted averaged through the learning rate κ to complete the update of the target network parameters:

[0290]

[0291] θ μ =κθ μ +(1-κ)θ μ

[0292] o. Repeat the above steps until the target network converges or the total number of cycles M is reached multi , end the training;

[0293] Step 3 is to establish a multi-agent ecological collaborative control model for a mixed-traffic vehicle fleet at an intersection entrance. Step 3) completes the training of the multi-agent collaborative control model proposed by the present invention.

[0294] 4. Controllable Agent Executes Optimized Trajectory

[0295] The established and trained model is sent to each controllable agent, and a rolling time domain update strategy is adopted to calculate the speed trajectory of each agent passing through the intersection using road traffic information, and input it into the CAV controller for execution, including the following steps:

[0296] 1) Send the trained multi-agent collaborative control model to the server in the intersection control area;

[0297] 2) Perform step 1 to obtain various information parameters of the intersection and upload them to the server. When a vehicle enters the control area, obtain the vehicle composition information of the mixed-traffic fleet and upload it to the fleet operation information storage;

[0298] 3) Based on the obtained information, the mixed traffic fleet on each lane in the control area is divided into agents of various sizes and types;

[0299] 4) The fleet operation information storage device uploads the mixed-traffic fleet information in the current control area to the intersection server, and performs step 3, i.e., step 1) of the step of establishing the mixed-traffic fleet multi-agent ecological collaborative control model of the intersection entrance road, optimizes the initial ecological speed trajectory of the controllable agent entering the control area through the intersection, and sends it to the controller of the CAV in each controllable agent for execution;

[0300] 5) Perform step 2) of step 3, i.e., establish a multi-agent ecological collaborative control model for mixed-traffic fleets at the entrance of the intersection, calculate the time it takes for each agent to pass through the intersection under the current speed trajectory, and the intersection server performs secondary optimization on the speed trajectory of each controllable agent based on the operating conditions of each fleet at the current intersection, taking into account the energy consumption, exhaust emission level, and traffic efficiency of the fleet during operation, and outputs the optimization results to each CAV controller for execution;

[0301] 6) The fleet operation information storage device uploads the stored information to the historical database of the intersection server at regular intervals. The server uses the updated information to re-perform step 1, i.e., obtain road information, step 2, i.e., divide the intelligent agents, and step 3, i.e., establish a multi-agent ecological collaborative control model for the mixed-traffic fleet at the entrance of the intersection, thereby ensuring the safety and stability of the mixed-traffic fleet operation and the traffic efficiency of the entrance of the intersection.

[0302] Example 1

[0303] See also Figure 5 , Figure 6 and Figure 7 , the following combination Figures 5 to 7 The method for collaborative guidance and control of a mixed vehicle fleet at an intersection entrance lane of the present invention is described.

[0304] The implementation scenario is a city intersection set up in the traffic simulation platform. Each entrance and exit has three lanes. The signal timing adopts the traditional fixed signal timing, with a cycle time of 70s, a red light time of 50s, and a green light time of 20s. The intersection scene setting is as follows: Figure 5 As shown, the multi-agent ecological collaborative control model proposed in the present invention is implemented in MATLAB through the COM interface of the traffic simulation platform.

[0305] Figure 6 The operation diagram of the multi-agent cooperative control model provided by the present invention in the traffic simulation platform is given. In order to prove the effectiveness of the cooperative guidance control method for mixed traffic fleets at intersection entrances based on multi-agent reinforcement learning proposed by the present invention, the average delay, average travel time and average energy consumption of the fleet are used as evaluation indicators to evaluate the optimization results of the model. Let T 0_1 ,…,T 0_i ,...T0_n Indicates the time when the vehicle enters the control area, T d_1 ,…T d_ i ...T d_n Indicates the time when the vehicle leaves the stop line of the intersection, F f_i and F e_i are the instantaneous fuel consumption of FV and the instantaneous power consumption of EV, n1 represents the number of fuel vehicles passing through; n2 represents the number of fuel vehicles passing through, then:

[0306] Average travel time:

[0307] Average delay:

[0308] Average energy consumption:

[0309] To verify the effectiveness of the present invention, based on the above experimental scenario, the vehicle was released at 300m from the stop line of the intersection, and the above three evaluation indicators were used to compare with the uncontrolled method. The comparison results of each indicator are shown in the following table. Among them, the autonomous driving vehicle in the uncontrolled method has no networking function, and its driving is the same as that of a manually driven vehicle. The CAV under the multi-agent control method proposed in the present invention is an autonomous driving vehicle with networking function.

[0310] Table 1 Comparison of various indicators between the proposed method and the uncontrolled method under different CAV penetration rates and traffic volumes

[0311]

[0312] It can be seen from the above table that under the condition of fixed traffic volume, with the increase of CAV penetration rate, the average delay time and average energy consumption of the fleet at the intersection gradually decrease. Compared with the uncontrolled method, the maximum delay and average energy consumption of the method proposed in the present invention are reduced. The reason is that with the increase of CAV vehicles, the number of controllable intelligent agents in the fleet also increases, and the delay and energy consumption decrease accordingly. When the CAV penetration rate reaches 100% and the traffic volume is 1200pcu / h, the average delay reduction rate of the method proposed in the present invention reaches 16.31% compared with the uncontrolled method. The reason is that when the fleet is all CAV vehicles, all vehicles operate according to the optimization results of the method proposed in the present invention.

[0313] When the CAV penetration rate is fixed, the average delay time and average energy consumption of the fleet tend to decrease first and then decline. This is because under low traffic conditions, the mutual influence between vehicles is small. As the number of vehicles on the road increases, CAVs have a positive impact on the following RV vehicles, thereby reducing delays and energy consumption. When the traffic volume reaches 1000pcu / h, the above indicators reach the lowest value. Subsequently, the traffic volume increases further, and the queuing behavior of vehicles becomes increasingly obvious, further limiting the operating speed of vehicles. CAVs are unable to execute optimization strategies, which greatly reduces the control effect.

[0314] Figure 7 The running trajectory diagram of the vehicle under the control of the multi-agent cooperative control model provided by the present invention is given, in which the dark curve is the CAV running trajectory and the light curve is the RV running trajectory. It can be seen from the trajectory in the wireframe that the front intelligent body can completely pass through the intersection, but the intelligent body behind cannot pass through within the current green light time. If it continues to follow the intelligent body in front, it needs to stop at the intersection and wait for the green light to turn on. This way of passing will undoubtedly increase the overall energy consumption level of the fleet while reducing the vehicle traffic efficiency. However, under the multi-agent ecological cooperative control model proposed by the present invention, the intelligent body behind adjusts its own speed trajectory to avoid long-term idling waiting during the red light time, reaches the stop line and passes completely at the beginning of the next green light time, thereby improving the overall traffic efficiency of the fleet.

Claims

1. A method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection, characterized in that The method for cooperative guidance control of mixed traffic vehicles on the entrance road of an intersection includes the following steps: 1) Obtain road information: The acquisition of road information refers to determining the control scenario, defining the control scope, control object and control target; 2) Divide the agents: The division of intelligent agents is based on road traffic information and the division of different intelligent agents by the following behaviors between vehicles; 3) Establish a multi-agent ecological collaborative control model for mixed traffic fleets at intersection entrances: First, a single-agent speed trajectory planning model is established to calculate the optimal trajectory of the single agent. Then, based on this, a multi-agent collaborative control model is established with the overall operation efficiency of the fleet as the optimization goal to achieve collaborative control among multiple agents. After the model is built, the model is trained using reinforcement learning methods. 4) The controllable agent executes the optimized trajectory: The established and trained model is sent to each controllable intelligent agent. A rolling time domain update strategy is adopted to calculate the speed trajectory of each intelligent agent through the intersection using road traffic information and input it into the CAV controller for execution.

2. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 1 is characterized in that , the step of obtaining road information includes the following steps: 1) The control scenario of the present invention is a single three-lane entrance road at a conventional crossroad of an urban road. The lane attributes include left turn, straight ahead and right turn. The lane information is parameterized as follows: L int ＝{l1,l2,l3} Where: l1, l2 and l3 are lane attributes. Different values ​​represent different lane attributes. -1 represents the left turn lane, 0 represents the through lane, and 1 represents the right turn lane. 2) In the control scenario of the present invention, the intersection signal control type is fixed four-phase signal timing, and the signal timing scheme is expressed as follows: Where: sig int Represents the intersection signal timing plan information set, It represents the timing information set of phase k (k is 1, 2, 3, 4); the specific expression is as follows: Where: The value represents the phase type of the k phase, where the value 1 represents the north-south straight phase, -1 represents the east-west straight phase, 2 represents the north-south left turn phase, and -2 represents the east-west left turn phase; in the intersection signal control of the present invention, the right-turn traffic is not constrained by the signal timing scheme, Indicates the duration information of the red light and green light in phase k, specifically expressed as follows: Where: and They represent the red light duration and green light duration of signal phase k respectively. 3) The control range of the mixed-vehicle convoy coordinated guidance control model for the entrance of a multi-lane intersection based on multi-agent reinforcement learning proposed in the present invention is from the intersection stop line to 800m away from the intersection stop line. The control object is the mixed-vehicle convoy composed of CAVs and RVs within the control range. The mixed-vehicle convoy information of each lane is expressed as follows: Where: F int Represents the set of traffic flow information within the control range, and It represents the mixed traffic flow information set in each lane, as shown in the following formula: Where: C i_type Indicates the vehicle type of vehicle i (i is 1, 2, ..., I) on lane L (L is 1, 2, 3); its value is 0 or 1, where 0 represents CAV and 1 represents RV. 4) Acquire the operation information of CAV within the control range of the intersection, including the speed, acceleration and position of the vehicle. CAV can obtain the operation information of surrounding vehicles and the traffic status information of the intersection through V2V and V2I communication. The present invention realizes the control of mixed traffic flow in the above control scenario by utilizing this characteristic of CAV; the position, speed, acceleration and other information of each CAV are expressed as follows: Where: Indicates l L The total set of CAV operation information on the lane, is the operation information set of CAVh (the value of h is 1, 2, ..., H); is the current location of the CAV; is the current speed of CAV; is the current acceleration of CAV. The present invention aims to optimize the speed-space trajectory of CAVs in the traffic flow, thereby adjusting the overall distribution and operation status of the traffic flow, guiding different traffic flow arrival distributions to be more compatible with the fixed signal timing scheme of the intersection, so as to pass through the intersection in a safer, more stable, efficient and energy-saving manner.

3. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 1 is characterized in that , the division of agents includes the following steps: 1) Based on the information obtained in the road information acquisition step, in order to reflect the guiding role of CAV on RV and facilitate the subsequent model building, according to different vehicle arrival distributions and the following behaviors between vehicles, the vehicles are divided into vehicle groups of different sizes and compositions, and further abstracted into intelligent agents; Based on different vehicle group compositions, the present invention divides the fleet on each lane within the control range into the following three types of agents: a vehicle group consisting of one CAV or multiple CAVs is defined as a CAV agent CAV ; The vehicle group consisting of a CAV as the lead vehicle and several RVs following behind is defined as the minimum ecological control unit. On this basis, this type of vehicle group is abstracted into an intelligent agent, defined as the minimum ecological control unit intelligent agent agent ECU ; A group of one or more RVs that are independent of the vehicle in front is defined as an RV agent. RV ; Among them, since there is a CAV in the vehicle group, the present invention uses the CAV agent CAV and ECU agent ECU Defined as a controllable agent, RV agent RV Defined as an uncontrollable agent. When the distance between the RV and the front vehicle is in the indifference area When the present invention determines that the vehicle in front belongs to the same control unit, and Represents the maximum and minimum values ​​of the indifference area, and its calculation formula is as follows: Where: Bound = max or min; T bound For B ound The expected time interval under the condition; S0 is the parking interval; v i (t) is the speed of RV at time t; Δv i (t) is the speed difference between RV and the preceding vehicle at time t; α max is the maximum acceleration of the vehicle; β is the comfortable deceleration of the vehicle; 2) Based on the information obtained in the step of obtaining road information, the possible lane changing behavior of the RV is predicted. In the control scenario of the present invention, the CAV enters the control range through its target lane, so there is no lane changing demand during operation. As an uncontrollable vehicle, the driving behavior of the RV has certain uncertainty. The present invention calculates the lane changing probability of the RV by the following formula: Where: To indicate the possibility of RV changing lanes, The larger the value, the greater the possibility that RV will perform lane change under the current traffic conditions: nece and safe They represent the necessity of lane change according to the driving goal of RV and the safety condition for lane change respectively; the expressions of the two parameters are: Where: p line is the longitudinal critical position of RV lane change; p i (t) is the position of vehicle i at time t; λ i is the number of lane changes required by vehicle i; d i,j (t) is the distance between vehicle i and vehicle j at time t; D i,j (t) is the safe distance between vehicle i and vehicle j at time t; ρ is the safe headway; a i,max,brake is the maximum braking deceleration of vehicle i; a i,min,brake is the minimum braking deceleration of vehicle i; a i,brake (t) is the braking deceleration of vehicle i at time t; v i,max is the maximum speed that vehicle i can travel on the road section.

4. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 1 is characterized in that The establishment of a multi-agent ecological collaborative control model for a mixed-traffic vehicle fleet at an intersection entrance includes the following steps: 1) Based on the agent division step, the controllable agents obtained by dividing the fleet on each lane are divided, and a single-agent speed trajectory planning model is established with the optimization goal of minimizing the cumulative energy consumption of the speed trajectory of a single agent passing through the intersection; 2) Considering the overall traffic efficiency of mixed traffic flow, a multi-agent collaborative control model is established; 3) Multi-agent collaborative control model training; The single-agent speed trajectory planning model established in step 1) and the multi-agent collaborative control model established in step 2) of the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance are trained through an improved Twin Delayed Deep Deterministic policy gradient (TD3) algorithm. The purpose of the training is to simulate the fleet's multiple interactions with the environment based on the optimization results of the model established by the present invention, so as to obtain the overall operating benefit of the fleet corresponding to different decisions of the agents under different environments and vehicle operation information, obtain the optimal benefit through repeated interactions, and store the optimal benefit and the corresponding decision as known experience. In the actual operation of the fleet, if there is an operating condition consistent with the known experience, the corresponding optimal solution can be immediately obtained, thereby improving the optimization efficiency.

5. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 4 is characterized in that , the controllable agents obtained by dividing the fleet on each lane based on the agent division step are used to minimize the cumulative energy consumption of the speed trajectory of a single agent passing through the intersection, and a single agent speed trajectory planning model is established; the specific steps are as follows: (1) Calculate the individual optimal speed of a single agent. Considering that a single agent may contain several fuel vehicles (FV) and several electric vehicles (EV), their energy consumption types are different and therefore need to be considered separately. The individual optimal speed of a single agent is to minimize the operating energy consumption of all vehicles within the agent. The calculation formula is as follows: Z=minE agent Where: μ fv and μ ev are the unified coefficients of fuel consumption and electricity consumption respectively; m and n are the number of FV and EV in the intelligent body respectively; T0 is the starting time; T d_i is the time it takes for a vehicle to pass the stop line at the intersection; v i (t) and a i (t) are the speed and acceleration of the vehicle, v i The value of (t) is at the minimum speed limit v min With maximum speed limit v max Between i The value of (t) is at the minimum acceleration a min With the maximum acceleration a max Between; F f_i and F e_i They are the instantaneous fuel consumption of fuel vehicles and the instantaneous power consumption of electric vehicles; (2) Based on the ecological speed obtained in (1), in order to achieve smooth guidance of the CAV to the RV, the speed trajectory of the intelligent agent from the current speed to the ecological speed is calculated by trigonometric function. The calculation formula is as follows: Where: v opt (t) is the speed of the vehicle after optimization based on trigonometric function at time t; v aim is the average target speed of the vehicle; v dis is the difference between the average target speed of the vehicle and the initial speed v0; ω tr When the vehicle speed is between the current speed v con and the average target speed v aim The rate of change of acceleration / deceleration between The vehicle speed is lower than the average target speed v aim The rate of change of acceleration / deceleration at the time of f The target distance D int and the average target speed v aim The ratio of (3) Considering the uncertainty of the driving behavior of the preceding vehicle, the speed trajectory of the single agent is constrained by a virtual trajectory. The specific steps are as follows: a. If the front vehicle is in motion, a dynamic virtual trajectory is used to constrain the agent to ensure that the agent maintains a safe following distance from the front vehicle. The dynamic virtual trajectory calculation formula is as follows: Where: and is the intermediate quantity; τ is the driver’s reaction time; b. If the vehicle in front is in a queue, the agent can only pass through the intersection after the queue of vehicles in front dissipates. The present invention uses a static virtual trajectory s p_i (t) constraining the agent; (4) Calculate the lane-changing probabilities of the RVs inside and around the controllable agent according to step 2) in the step of dividing the agents. According to the obtained lane-changing probabilities, analyze the impact of the lane-changing behaviors of the internal RV leaving the agent, the surrounding RV entering the agent, and the surrounding RV inserting in front of the agent on the overall ecology of the fleet and the operation efficiency of the intersection. If the lane-changing behavior has no positive impact, the relevant CAV suppresses the lane-changing behavior of the RV by compressing the possible lane-changing interval of the RV. Otherwise, the relevant CAV will give it a safe lane-changing space to ensure that the RV changes lanes safely. The lane-changing behavior causes the internal composition of the agent to change. Return to step (1), step (2) and step (3) in step 1) of the mixed-vehicle fleet ecological collaborative control model of the intersection entrance lane to recalculate and generate the speed trajectory of the single agent; For the RV that is inserted in front of the agent due to lane changing behavior, the calculation formula of the dynamic virtual trajectory described in (3) a is transformed into the following formula: Where: x and y are the longitudinal and lateral displacements of the agent at time t; p i and q j are coefficients that determine the trajectory shape; since the longitudinal and lateral displacements are represented using time-based functions, the velocity v at each time node can be easily obtained by taking the first and second order derivatives x and v y and acceleration a x and a y ; By establishing step 1) of the multi-agent ecological collaborative control model for mixed traffic fleets at intersection entrances, the optimal speed trajectory of each agent is calculated, and the speed trajectory serves as the initial state input of the subsequent multi-agent collaborative control model.

6. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 4 is characterized in that ,Considering the overall traffic efficiency of mixed traffic flow, a multi-agent collaborative control model is established. The specific steps are as follows: (1) In order to reduce the amount of calculation, the single-agent speed trajectory planning model is linked to the multi-agent collaborative control model, and the time it takes for the agent to pass through the intersection is used as the action space of the multi-agent collaborative control model: a j =T d_j The time it takes for each agent to pass the intersection is calculated using the ecological speed trajectory obtained from the single agent ecological speed trajectory planning model in 1). As the initial action of the multi-agent collaborative control model, after the action of the agent is updated, the speed trajectory corresponding to the new action can be output through the smoothness constraint obtained by the trigonometric function in the single-agent speed trajectory planning model; (2) In the present invention, the state space of the agent is defined as the speed and position of the agent under the current action: s j =(v j (t),p j (t)) Where: p j (t) is the position of the agent at the current moment; (3) In order to ensure that the training result of the agent is as close to the optimal solution as possible, the present invention designs the reward function as follows. The specific steps are as follows: a. First, calculate the reward function of a single agent. Considering the safety of the agent during movement, the maximum available deceleration is used as the safety indicator of the agent's operation. The safety reward function calculation formula is as follows: Where: a safe is the maximum available deceleration of the leading vehicle to avoid conflict with the preceding vehicle; safe For penalty items; b. Establishing the multi-agent ecological collaborative control model for mixed-traffic vehicles on the entrance road of an intersection In step 1), the optimal ecological speed trajectory of a single agent is calculated. Therefore, the present invention believes that the closer the action is to the initial action of the agent, the smaller the overall energy consumption of the corresponding speed trajectory, so the calculation formula of the ecological reward function is obtained as follows: c. By combining the safety reward function and the ecological reward function with a certain weight, the reward value of a single agent can be obtained. The calculation formula is as follows: Where: w1 and w2 are weight coefficients; d. The number of vehicles passing through the intersection stop line in each lane of the target entrance road within a cycle is used as the efficiency reward function of the intersection to reflect the traffic efficiency of the entire entrance road. The calculation formula of the entrance road efficiency reward function is as follows: Where: is the lane l in a signal cycle i The number of vehicles passing through the intersection; e. After adding up the single agent reward values ​​of all controllable agents within the control range, it is combined with the entrance efficiency reward function with a certain weight coefficient to obtain the reward value of the entire fleet. The calculation formula is: Where: w3 and w4 are weight coefficients.

7. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 4 is characterized in that The improved Twin Delayed Deep Deterministic policy gradient (TD3) algorithm is used to train the single-agent speed trajectory planning model established in step 1) and the multi-agent collaborative control model established in step 2) of the step of establishing the multi-agent ecological collaborative control model of the mixed-traffic fleet at the entrance of the intersection. The purpose of the training is to simulate the fleet to interact with the environment multiple times based on the optimization results of the model established by the present invention, so as to obtain the overall operation benefit of the fleet corresponding to different decisions of the agents under different environments and vehicle operation information, obtain the optimal benefit through repeated interactions, and store the optimal benefit and the corresponding decision as known experience. In the actual operation of the fleet, if there is an operating condition that is consistent with the known experience, the corresponding optimal solution can be immediately obtained, thereby improving the optimization efficiency; the specific steps are as follows: (1) First, use the DQN solver to solve the single-agent velocity trajectory planning model. The specific steps are as follows: a. Initialize the experience pool Its capacity N single , action-value function and the target network b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M single between; c. Select action And execute, in exploration mode, with ε single The probability of randomly selecting an action In experience mode, select Next The maximum movement is calculated as follows: d. Action Interact with the environment and get rewards And determine the next state Whether the termination condition is triggered; e. Experience Deposit into experience pool f. Repeat steps b to e until the experience pool is full; g. Select a small batch of experience from the experience pool and calculate the target network The objective function is calculated as follows: Where: γ single is the discount factor; h. Calculate the loss function: i. Use the gradient descent method to minimize the loss function of step h, thereby updating the target network Weight j. Every T single Repeat steps g to i at fixed intervals to update the target network parameters; k. Repeat the above steps until the target network converges or the total number of cycles M is reached. single , end the training; After completing the training of the single-agent speed trajectory planning model, the model is solved. The obtained speed trajectory is the optimal speed trajectory of the single agent itself. The time it takes for each agent to pass the intersection based on this trajectory is calculated, and this is used as the initial action when training the multi-agent collaborative control model. (2) The present invention uses the improved TD3 algorithm to train the multi-agent collaborative control model. The specific steps are as follows: a. Initialize the Actor network μ(θ μ ), Critic1 Network Critic2 Network Target Actor Network μ'(θ μ′ ), target Critic1 network Target Critic2 Network Its capacity N single ; b. Get environment status information It contains historical status information, t s The value ranges from 0 to the total number of cycles M multi between; c. Select action To improve the efficiency of the action search of the agent during the training process, the present invention designs the following strategy: As a starting point for the search, that is: If the reward value obtained by the action shifting to the left has an upward trend, update the action to the left, otherwise update it to the right; d. Action Interact with the environment to obtain single-agent rewards After that, calculate the overall reward value of the intersection And determine the next state Whether the termination condition is triggered; e. Experience Deposit into experience pool f. Repeat steps b to e of step (2) in the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at an intersection entrance until the experience pool becomes full; g. Select a small batch of experience from the experience pool Using the target Actor network μ'(θ μ' ) calculated The following actions are taken and noise is added based on the target strategy regularization, in order to limit the complexity of the model and avoid overfitting as much as possible: h. Based on the dual network idea, calculate the target value: Where: γ single is the discount factor. i. Calculate the loss function: j. Use gradient descent to minimize the loss function of step i and update the Critic1 network and Critic1 network Parameters in ; k.Critic1 Network and Critic2 Network Updated critic Then, using the target Actor network μ'(θ μ' )Calculation status Next action: l. Using the Critic1 network and Critic2 Network Calculate the pair Evaluated value: m. Use the gradient ascent method to maximize step l Complete the Actor network μ(θ μ ) updates; n. The new and old target network parameters are weighted averaged through the learning rate κ to complete the update of the target network parameters: i μ′ =κθ μ +(1-κ)θ μ o. Repeat the above steps until the target network converges or the total number of cycles M is reached multi , end the training; Step 3) of the steps of establishing a multi-agent ecological collaborative control model for a mixed traffic fleet at an intersection entrance completes the training of the multi-agent collaborative control model proposed by the present invention.

8. The method for cooperative guidance and control of mixed traffic vehicles at the entrance of an intersection according to claim 1 is characterized in that ,The controllable agent executes the optimized trajectory including the following steps: 1) Send the trained multi-agent collaborative control model to the server in the intersection control area; 2) Perform step 1 to obtain various information parameters of the intersection and upload them to the server. When a vehicle enters the control area, obtain the vehicle composition information of the mixed-traffic fleet and upload it to the fleet operation information storage; 3) Based on the obtained information, the mixed traffic fleet on each lane in the control area is divided into agents of various sizes and types; 4) The fleet operation information storage device uploads the mixed-traffic fleet information in the current control area to the intersection server, and performs step 1) of the steps of establishing the mixed-traffic fleet multi-agent ecological collaborative control model of the intersection entrance road, optimizes the initial ecological speed trajectory of the controllable agent entering the control area through the intersection, and sends it to the controller of the CAV in each controllable agent for execution; 5) Perform step 2) of the steps of establishing a multi-agent ecological collaborative control model for a mixed-traffic fleet at the entrance of an intersection, calculate the time it takes for each agent to pass through the intersection under the current speed trajectory, and the intersection server performs secondary optimization on the speed trajectory of each controllable agent based on the operating conditions of each fleet at the current intersection, taking into account the energy consumption, exhaust emission level, and traffic efficiency of the fleet during operation, and outputs the optimization results to each CAV controller for execution; 6) The fleet operation information storage device uploads the stored information to the historical database of the intersection server at regular intervals. The server uses the updated information to re-acquire road information steps, divide intelligent agents, and establish a multi-agent ecological collaborative control model for mixed-traffic fleets at intersection entrances, thereby ensuring the safety and stability of mixed-traffic fleet operations while ensuring the traffic efficiency of intersection entrances.

Images