Direct and indirect control of a mixed automated vehicle platoon

Abstract

A system for directly and indirectly controlling a mixed automated vehicle receives a traffic state of a group of mixed automated vehicles traveling in the same direction, wherein the group of mixed automated vehicles includes a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group, submits the traffic state to a parameterized function to produce a target headway, the parameterized function trained to convert the traffic state to a target headway for the mixed automated vehicle, and submits the target headway to a headway-based model to produce a target speed, the model configured to map the target headway to a target speed for the mixed automated vehicle. The system determines a control command based on one or a combination of the target headway and the target speed and sends the control command to the controlled vehicle.

Description

Technical Field

[0001] This invention relates generally to traffic control, and more specifically to the control of convoys comprising controlled and uncontrolled vehicles. Background Technology

[0002] Traffic congestion is a major problem in many parts of the world, costing us time, environmental threats, and wasted fuel. In the United States, the cost of traffic congestion can be measured in hundreds of dollars per person per year. Therefore, it is necessary to reduce traffic congestion and / or improve any other transportation goals.

[0003] One potential contribution to solving these problems is allowing vehicles to travel more closely in what is known as a "platoon." The term "platoon" generally refers to a large number of vehicles that are driven as a unit, with short distances between them. Shorter distances allow more vehicles to use the road, and the energy consumption of individual vehicles is also reduced due to decreased drag. Vehicles in a platoon are driven by at least one of automatic control over their speed and direction.

[0004] Today, many vehicles are also equipped with cruise control systems to make driving easier for the driver. In this case, the driver can set the desired speed via, for example, a regulator in the dashboard, and the vehicle's cruise control system then influences the control system to accelerate and brake the vehicle as needed to maintain the desired speed. If the vehicle is equipped with an automatic transmission system, the gear the vehicle is currently in is changed so that the vehicle can maintain the desired speed.

[0005] For example, connected adaptive cruise control (CACC) is a form of autonomous lane keeping where vehicles transmit their positions to a central system to calculate speed controls that ensure stability, robustness, and optimality. Vehicles equipped with CACC can form platoons with other vehicles equipped with CACC. However, traffic may include vehicles with and without CACC. Even if all vehicles are equipped with CACC, some vehicle operators may choose not to use it and drive manually. A significant problem in platoon control is the inefficiency that occurs when manually operated vehicles join and disrupt the platoon. Due to the difficulty in controlling and / or modeling manually operated vehicles, it is nearly impossible to remove vehicles from the platoon.

[0006] For example, one approach divides a fleet into several smaller fleets to ensure homogeneity of fleet control, see, for example, US.6356820. However, this decomposition duplicates the control method and increases the distance between vehicles in different fleets, which in turn increases traffic congestion. Another approach controls the vehicles in a fleet to ensure homogeneity within each fleet, see, for example, US.2012 / 0123658. However, these methods are difficult to apply to manually operated vehicles. Summary of the Invention

[0007] Some implementations aim to provide a system and method for controlling a hybrid convoy. As used herein, such a hybrid convoy includes controlled and uncontrolled vehicles. Examples of controlled vehicles include autonomous and semi-autonomous vehicles willing to form a convoy. For example, controlled vehicles may use connected adaptive cruise control (CACC), which is configured to determine motion commands to the actuators of the controlled vehicles to form and / or maintain a convoy. Examples of uncontrolled vehicles include autonomous and semi-autonomous vehicles unwilling to form a convoy, as well as manually operated vehicles (i.e., vehicles operated by a human driver).

[0008] Another objective of some implementations is to provide indirect control over uncontrolled vehicles forming a convoy. Some implementations are based on the understanding that the movement of controlled vehicles forming a convoy can indirectly affect the movement of uncontrolled vehicles within the convoy. For example, decelerating a controlled vehicle forces a following uncontrolled vehicle to also decelerate. Similarly, increasing the speed of a controlled vehicle can encourage the drivers of following uncontrolled vehicles to accelerate in open space. In this way, direct control of controlled vehicles can be used to encourage self-imposed control over uncontrolled vehicles, thereby allowing convoys to form from a mixture of controlled and uncontrolled vehicles. This encouragement of self-imposed control over uncontrolled vehicles is referred to herein as indirect control.

[0009] On the one hand, such indirect control is unreliable. In reality, the acceleration of a controlled vehicle may not force the acceleration of an uncontrolled vehicle. On the contrary, such acceleration may unintentionally force other drivers to perceive danger and slow down their vehicles. However, some implementations are based on the understanding supported by experimental evidence that direct control of controlled vehicles can be efficiently used for indirect control in mixed convoys if at least two of the following conditions are met.

[0010] The first of these two conditions concerns the means of implementing direct control. Some implementations are based on the understanding that, for efficient control of a mixed convoy, the means of direct control of controlled vehicles in the convoy should be equivalent to the means of self-imposed indirect control by uncontrolled vehicles. Some implementations are based on the understanding that the means of self-imposed indirect control is the distance between vehicles. Indeed, to maintain safety, drivers will maintain what they perceive as a safe distance based on the current traffic speed.

[0011] To this end, in order to unify direct and indirect control, some implementations provide direct control of the controlled vehicle by imposing constraints on the motion of the controlling vehicle. Direct control includes one or a combination of the maximum headway between two vehicles in the platoon and the maximum speed of each vehicle in the platoon. For example, this direct control can be compared to direct commands to the actuators of the vehicle, which is fundamentally different from self-imposed indirect control. Furthermore, constraints on the headway and / or speed of the vehicle allow the controlled vehicle to use its conventional systems for motion control. For example, in some implementations, the controlled vehicle is an autonomous vehicle configured to determine motion commands to the actuators of the constrained autonomous vehicle.

[0012] The second of the two conditions involves the calculation of the means of performing direct control. Some implementations are based on the understanding that, for efficient indirect control, it is necessary to determine constraints on the headway and / or speed of vehicles in the platoon using performance metrics that are likely common or at least relevant to all vehicles in the mixed platoon. Furthermore, such performance metrics should be determined under the assumption that all mixed vehicles are willing and able to participate in platoon formation in a controlled manner.

[0013] However, designing direct and indirect controllers for hybrid autonomous vehicles used to form fleets presents challenges. There are two main approaches to designing such controllers: learning-based controllers or learners, and solution-based controllers or solvers. Both learners and solvers map inputs to outputs. However, learners obtain mappings from data or experience, while solvers obtain mappings for each given input from a model. Neither approach is optimal in this case because the behavioral model of heteronomously driven hybrid autonomous vehicles is unknown, and learners may fail to converge to stable control due to sporadic behaviors of uncontrolled vehicles.

[0014] Some implementations are based on the understanding that reinforcement learning (such as deep reinforcement learning (DRL)) can be modified according to the principles of some implementations to directly and indirectly control platoons of hybrid autonomous vehicles. Specifically, some implementations do not generate actions that change the environment, but instead train parameterized functions (such as DRL controllers) to generate a target headway that forces platoon formation as a reward. In this way, the learner can adapt to the unknown dynamics of hybrid autonomous vehicles.

[0015] For example, some implementations use a headway-based model configured to map a target headway to a target speed of the hybrid autonomous vehicle, ultimately mapping it to the vehicle's movement. The headway-based model allows the use of headway as a control parameter and associates this control parameter with the movements of vehicles available for platooning. In other words, the headway-based model allows learning the unknown behavior of non-autonomous and, more generally, uncontrolled vehicles. An exemplary headway-based vehicle behavior model is the Optimal Speed ​​Model (OVM). The OVM associates the vehicle's headway with a safe driving speed. Other similar models exist and can be used similarly by different implementations.

[0016] Therefore, one embodiment discloses a system for directly and indirectly controlling hybrid autonomous vehicles. The system includes: a receiver configured to receive traffic states of a group of hybrid autonomous vehicles traveling in the same direction, wherein the group includes controlled vehicles willing to participate in platooning and at least one uncontrolled vehicle, and wherein the traffic states indicate the state of each vehicle in the group, the states including the current headway, current speed, and current acceleration of the hybrid autonomous vehicles; a memory configured to store a parameterized function trained to convert traffic states into target headway times for the hybrid autonomous vehicles; and a headway-based model configured to map target headway times to target speeds of the hybrid autonomous vehicles; and a processor configured to submit traffic states to the parameterized function to generate target headway times, wherein the parameterized function is a deep reinforcement learning (DRL) controller. The DRL controller is trained to convert the traffic state of the group of hybrid autonomous vehicles into a target headway, which improves the actions of the hybrid autonomous vehicles defined by the target headway based on a headway-based model. The target headway is determined as a value function of the movement costs of the convoy that improves the actions. This value function is the negative of the sum of the movement costs of the hybrid autonomous vehicles within the planned timeframe, and the movement costs include a component related to the convoy's string stability, a component related to the convoy's equipment stability, and a penalty term for the target headway approaching a stopping headway. The controller submits the target headway to the headway-based model to generate a target speed. It then determines control commands for the controlled vehicles based on one or a combination of the target headway and the target speed. Finally, it includes a transmitter configured to send control commands to the controlled vehicles in the group of hybrid autonomous vehicles, wherein the controlled vehicles are arranged to be controlled by these commands.

[0017] Another embodiment discloses a method for directly and indirectly controlling hybrid autonomous vehicles, wherein the method uses a processor coupled to stored instructions implementing the method, wherein the instructions, when executed by the processor, perform steps of the method, the steps including: receiving traffic states of a group of hybrid autonomous vehicles traveling in the same direction, wherein the group of hybrid autonomous vehicles includes controlled vehicles willing to participate in platooning and at least one uncontrolled vehicle, and wherein the traffic states indicate the state of each vehicle in the group, the states including the current headway, current speed, and current acceleration of the hybrid autonomous vehicles; submitting the traffic states to a parameterization function to generate target headway, the parameterization function being trained to convert the traffic states into target headway for the hybrid autonomous vehicles, wherein the parameterization function is a deep reinforcement learning (DRL) controller, the DRL controller being trained to convert the traffic states of the group of hybrid autonomous vehicles into... The target headway improves the actions of the hybrid autonomous vehicles defined by the target headway based on a headway-based model, wherein the target headway is determined as a value function of the movement costs of the action-improving fleet, wherein the value function is the negative of the sum of the movement costs of the hybrid autonomous vehicles within the planned time frame, and wherein the movement costs include a component related to the chain stability of the fleet, a component related to the equipment stability of the fleet, and a penalty term for the target headway approaching the stop headway; the target headway is submitted to the headway-based model to generate a target speed, the headway-based model being configured to map the target headway to a target speed of the hybrid autonomous vehicles; control commands for the controlled vehicles are determined based on one or a combination of the target headway and the target speed; and control commands are sent to the controlled vehicles in the group of hybrid autonomous vehicles, wherein the controlled vehicles are arranged to be controlled by these commands.

[0018] Another embodiment discloses a non-transitory computer-readable storage medium on which a program executable by a processor for performing a method is implemented. The method includes: receiving traffic states of a group of hybrid autonomous vehicles traveling in the same direction, wherein the group of hybrid autonomous vehicles includes controlled vehicles willing to participate in platooning and at least one uncontrolled vehicle, and wherein the traffic states indicate the state of each vehicle in the group, the states including the current headway, current speed, and current acceleration of the hybrid autonomous vehicles; submitting the traffic states to a parameterized function to generate target headway, the parameterized function being trained to convert the traffic states into target headway for the hybrid autonomous vehicles, wherein the parameterized function is a deep reinforcement learning (DRL) controller, the DRL controller being trained to convert the traffic states of the group of hybrid autonomous vehicles into target headway, the target headway improving the hybrid autonomous vehicles' headway based on headway... The action is defined by a model of distance, wherein the target headway is determined as a value function of the movement cost of the convoy that improves the action, wherein the value function is the negative of the sum of the movement costs of the hybrid autonomous vehicles within the planned time frame, and wherein the movement cost includes a component related to the convoy's string stability, a component related to the convoy's equipment stability, and a penalty term for penalizing the target headway when it approaches the stop headway; the target headway is submitted to a headway-based model to generate a target speed, the headway-based model being configured to map the target headway to the target speed of the hybrid autonomous vehicles; control commands for the controlled vehicles are determined based on one or a combination of the target headway and the target speed; and control commands are sent to the controlled vehicles in the group of hybrid autonomous vehicles, wherein the controlled vehicles are arranged to be controlled by these commands. Attached Figure Description

[0019] [ Figure 1A ]

[0020] Figure 1A A schematic diagram of the control of a fleet of hybrid autonomous vehicles according to some implementation methods is shown.

[0021] [ Figure 1B ]

[0022] Figure 1B The general workflow of a learning-based controller enhanced with a headway-based model, according to some implementations, is shown.

[0023] [ Figure 2 ]

[0024] Figure 2 A schematic diagram is shown, providing an overall overview of the direct and indirect control of convoy formation according to some implementation methods.

[0025] [ Figure 3A ]

[0026] Figure 3A A schematic diagram of a training-enhanced reinforcement learning controller according to some implementations is shown.

[0027] [ Figure 3B ]

[0028] Figure 3B An embodiment is shown. Figure 3A The input and output of learning.

[0029] [ Figure 4 ]

[0030] Figure 4 A graph showing the convergence of model-free and augmented reinforcement learning according to some implementations is presented.

[0031] [ Figure 5 ]

[0032] Figure 5 A block diagram of a system for directly and indirectly controlling a hybrid autonomous vehicle, according to some embodiments, is shown.

[0033] [ Figure 6A ]

[0034] Figure 6A A schematic diagram of a vehicle being directly or indirectly controlled according to some embodiments is shown.

[0035] [ Figure 6B ]

[0036] Figure 6B A schematic diagram illustrates the interaction between a controller that receives control commands determined according to some implementations and other controllers of the vehicle. Detailed Implementation

[0037] System Overview

[0038] This disclosure relates to a group of hybrid autonomous vehicles intended to form a convoy, which is a group of vehicles traveling together in a single lane. The autonomous driving characteristics of this group of vehicles are hybrid, as some vehicles operate autonomously and are willing to form a convoy, while others are unwilling to form a convoy and / or are operated by a human operator. For example, one implementation involves a subgroup of autonomous vehicles controlled by a shared controller to achieve better operation of the convoy as a whole.

[0039] Figure 1AA schematic diagram of the control of a hybrid autonomous vehicle fleet forming a convoy is shown according to some embodiments. According to some embodiments, a group of hybrid autonomous vehicles includes controlled vehicles willing to participate in convoy formation and at least one uncontrolled vehicle. Controller 102 directly controls the controlled autonomous vehicles, such as vehicles 103 and 104, by sending control commands to the controlling vehicle. Controller 102 also indirectly controls the uncontrolled vehicles, such as autonomous vehicle 105 unwilling to participate in the convoy and human-operated vehicle 106. Specifically, controller 102 controls the controlled vehicles based on the traffic conditions of the hybrid autonomous vehicles, such that direct control of the controlled vehicles also provides indirect control of the uncontrolled vehicles.

[0040] To control different types of autonomous vehicles, some implementations use control parameters shared among all vehicles and directly control the controlled vehicle based on these parameters, while simultaneously tracking these parameters for other vehicles. To this end, some implementations achieve this by having vehicles (whether autonomous or human-operated) ultimately control their behavior in a single lane on long road segments based on desired speed and headway (i.e., distance to the vehicle in front). More specifically, the fleet shares desired speed among all vehicles, while headway can vary based on vehicle type, onboard controller, and autonomous driving characteristics.

[0041] For example, some implementations are based on experimental evidence that direct control of controlled vehicles can be efficiently used for indirect control in mixed convoys if at least two of the following conditions are met. The first of these conditions relates to the means of performing direct control. Some implementations are based on the understanding that, for efficient control of mixed convoys, the means of direct control of controlled vehicles in the convoy should be comparable to the means of self-imposed indirect control by uncontrolled vehicles. Some implementations are based on the understanding that the means of self-imposed indirect control is the distance between vehicles. In fact, to maintain safety, drivers maintain what they consider a safe distance based on current traffic speed.

[0042] The second of the two conditions relates to the calculation of the means of performing direct control. Some implementations are based on the understanding that, for efficient indirect control, it is necessary to determine the target headway of the vehicles in the platoon using optimization of performance metrics that are likely common or at least relevant to all vehicles in the mixed platoon. Furthermore, such performance metrics should be determined under the assumption that all mixed vehicles are willing and able to participate in platoon formation in a controlled manner.

[0043] However, designing such direct and indirect controllers for hybrid autonomous vehicles forming a fleet is challenging. There are two approaches to designing such controllers: learning-based controllers or learners, and solution-based controllers or solvers. Both learners and solvers map inputs to outputs. However, learners obtain mappings from data or experience, while solvers obtain mappings from a model for each given input. However, neither approach is optimal in this case because the behavioral model of heteronomous hybrid autonomous vehicles is unknown, and learners may fail to converge to stable control due to sporadic behaviors of uncontrolled vehicles.

[0044] For example, learning-based control can come from two types of learners: deep learners and deep reinforcement learners. In deep learning (DL) and deep reinforcement learning (DRL), training results in a parameterized function with a fixed structure (given by the deep neural network) and many adjustable parameters. The difference between DL and DRL lies in how the function is learned during training. Deep learning is a supervised method that learns parameters by minimizing an error function that depends on the inputs and target outputs in the training set. Deep reinforcement learning, on the other hand, is an unsupervised method that learns from experience, where the error function depends on the values ​​of the states and their successors. In DL and DRL, training can be performed by minimizing the error function via stochastic gradient descent, where the parameter vector is incrementally modified by taking multiple steps in the gradient direction. Similar optimization algorithms are used in policy-based DRL, where the parameterized function represents the policy.

[0045] Some implementations are based on the understanding that reinforcement learning (such as DRL) can be modified according to the principles of some implementations to directly and indirectly control a fleet of hybrid autonomous vehicles. Specifically, instead of generating actions that change the environment, some implementations train parameterized functions (such as a DRL controller) to generate a target headway for platoon formation as a reward. In this way, the learner can be adapted to the unknown dynamics of hybrid autonomous vehicles.

[0046] For example, some implementations use a headway-based model configured to map a target headway to a target speed of the hybrid autonomous vehicle, and ultimately to the vehicle's actions. The headway-based model allows the use of headway as a control parameter and associates this control parameter with the actions of vehicles available for platooning. In other words, the headway-based model allows learning the unknown behavior of non-autonomous and, more generally, uncontrolled vehicles. An exemplary headway-based vehicle behavior model is the Optimal Speed ​​Model (OVM). The OVM associates the vehicle's headway with a safe driving speed. Other similar models exist and can be used similarly by different implementations. For example, some implementations use variations of the OVM, such as the Optimal Speed ​​Function (OVF), the Full Speed ​​Difference Model (FVDM), the Intelligent Drive Model (IDM), and variations thereof.

[0047] For example, in one implementation, the specific learning-based control scheme used is reinforced deep reinforcement learning (DRL). DRL solves the online optimization problem by simultaneously learning the behavior of the system and learning to control the same system. Typically, DRL can be model-free or reinforced using a model. Without a model, the behavior of the DRL controller can become unstable because there are more local optima due to the need to optimize over a larger set of parameters, making convergence difficult. In fact, our own experiments comparing model-free DRL and reinforced DRL show that while model-free DRL typically fails to converge to stable control, the model-based version has almost no difficulty in this regard.

[0048] Figure 1B The general workflow of a learning-based controller enhanced with a headway-based model, according to some embodiments, is illustrated. A headway-based model 150 is used to represent vehicle behavior as a necessary part of the design of a learning-based controller 154, which is trained to control the behavior of vehicles 158 in a fleet of 156. Model 150 is informed 160 as the design choice for the actual vehicle behavior 158, such that connected vehicles are directly controlled and unconnected vehicles are indirectly controlled.

[0049] Generally, the headway of vehicle i in a convoy evolves according to the differential equation:

[0050]

[0051] Among them, h i It is the vehicle's headway, v i It is its speed, v i-1 That is the speed of the vehicle in front. The speed changes according to the following:

[0052]

[0053] Among them, u i This refers to vehicle acceleration. Acceleration is proportional to the force applied to the vehicle, which is transmitted to the vehicle body through the contact between the wheels and the ground. The force itself is determined to be the output of some advanced controller, which may be the human operator in the case of a non-autonomous vehicle, or the speed controller in the case of an autonomous vehicle. For modeling purposes, some implementations represent acceleration as a weighted sum of two parameters, the weights of which vary from vehicle to vehicle.

[0054]

[0055] First parameter It is the difference between the actual speed and the desired speed. The second parameter (v) i-1 -v i The distance to the front of the train is the rate of change. The desired speed is modeled using the Optical Vehicle Frequency (OVF), with the frontage as the input:

[0056]

[0057] Parameter v max h s and h g The symbols differ between vehicles. max (i.e., the speed of the vehicle when there are no obstacles), and the stopping headway h. s (That is, the headway when the vehicle comes to a complete stop, continues moving forward, or moves at full speed), headway h g (That is, the vehicle will travel at its maximum speed v) max (Distance between the front and rear of the moving vehicle).

[0058] The control system includes a communication system that monitors the headway h, speed v, and acceleration u of the uncontrolled vehicles. In some embodiments, the controller may include a communication system in which the uncontrolled vehicles transmit these parameters to a central system; in other embodiments, the controlled vehicles measure these parameters and transmit them to the central system. Generally, there exists a range within which all vehicles should report their parameters or measure their parameters. For this reason, in some embodiments, the hybrid autonomous driving vehicle includes all uncontrolled vehicles within a predetermined range on the flanks of the controlled vehicles in the fleet. For example, in one embodiment, the range D of all vehicle parameters is at least approximately the full-speed headway h of the average controlled vehicle traveling at the maximum permissible speed. g Twice, that is, D≥2h g .

[0059] Figure 2A schematic diagram illustrating a general overview of direct and indirect control of convoy formation according to some implementations is shown. Controlled vehicles connect to a controller and transmit their status 203 to a central controller 201. The traffic status collected by controller 201 may include the current headway, current speed, and current acceleration of the hybrid autonomous vehicles. In some implementations, the hybrid autonomous vehicles include all uncontrolled vehicles within a predetermined flank range 207 of the controlled vehicles in the convoy. In addition to the controlled vehicles, some uncontrolled vehicles also connect 205 to controller 201 to submit their status.

[0060] Controlling vehicles in a convoy is commonly referred to as Cooperative Adaptive Cruise Control (CACC). CACC can be designed for a group of vehicles, all of which are controlled. Because conventional control methods do not account for situations where some vehicles in the convoy may not cooperate, some implementations have developed new methods, but still aim to achieve the performance of a typical convoy of autonomous vehicles. CACC has two objectives: plant stability and string stability. The convoy is plant stable if all vehicles maintain approximately the same, constant speed; the convoy is string stable if speed disturbances across the entire convoy are reduced.

[0061] Some implementations extend the principles of CACC (Controlled Acceleration and Adaptive Cruise Control) designed for controlled vehicles to hybrid autonomous vehicles. These two objectives of CACC inform the reinforcement learning-based controllers of some implementations. Specifically, the learning-based approach uses a cost function designed based on these two objectives. In some implementations, the cost function comprises the sum of two components:

[0062] c i =c i，s +a p c i，p

[0063] Where c i It is the cost generated by each individual observed by the vehicle, c i，s It is a component of the cost function related to string stability, c i，p It is a component of the cost function related to equipment stability; a p These are design parameters that are weighted in series to device stability. The components themselves are given by the following formula:

[0064] c i，s =(h i -h * ) 2 +a v (v i -v * ) 2

[0065] and

[0066]

[0067] Among them, h * and v * It is the team's expected headway and speed, a v It is a design parameter that weights the headway of the vehicle to speed tracking.

[0068] In some implementations, the cost function is further modified because stability is not the only consideration in controlling the vehicle fleet. In particular, an important consideration is preventing vehicles from colliding with each other. For this reason, in some implementations, we modify the cost function to include a penalty term for the cost:

[0069]

[0070] The penalty applies to the distance between the front of the vehicle and the point of approaching the point of parking, because falling below this distance increases the risk of a collision; a c These are design parameters used to weight the constraint satisfaction of fleet control.

[0071] Typically, reinforcement learning algorithms attempt to maximize some value function. Here, the value function V is the negative of the sum of the cost functions of all observed vehicles and the planned range of time T:

[0072]

[0073] Here, A is the set of observed vehicles.

[0074] Typically, the output of a control algorithm is the control itself, which in this case is the acceleration u. i However, as experiments have shown, it is difficult to directly learn appropriate acceleration in complex systems including non-autonomous vehicles. Therefore, some implementations instead set the control output based on the following:

[0075]

[0076] parameter and It is a design parameter that weights the optimal tracking speed to the speed of the vehicle ahead.

[0077] In some implementations, the target headway determined by the controller is the forward headway. As mentioned above, the optimal speed will reduce the headway h between the car and the vehicle. i Parking headway Distance between the front of the car and the front of the car With speed Related. In a preferred embodiment, the time distance to the front of the vehicle when parking. All controlled vehicles are kept stationary. This is done for safety reasons, as it would otherwise be possible to set the stopping headway to a small, unsafe value or a large value that would have unpredictable consequences for traffic; the latter is primarily to protect control of the convoy during training.

[0078] Figure 3A A schematic diagram of training an enhanced reinforcement learning controller according to some implementations is shown. In enhanced reinforcement learning (RL), such as enhanced DRL, the enhanced RL controller 350 interacts with its environment 310 in discrete time steps. At each time t, the RL controller receives observations 320 and rewards 340 regarding traffic state 330 in the environment 310. Ultimately, the enhanced RL controller 350 is used to select an action 360 from the available action set, which is then sent as a control command to the environment to change the traffic state in the environment. Actions are selected to collect as many rewards as possible, and rewards are determined to encourage platoon formation.

[0079] However, unlike model-free RL, the enhanced RL controller is not trained to output actions, but rather to output target headway based on the direct and indirect control principles for the hybrid autonomous vehicles forming a platoon. Therefore, the target headway generated by the enhanced RL is further submitted to a headway-based model 370 to generate control commands specifying actions for controlling the vehicles. To further encourage hybrid autonomous vehicle platoon formation, states 330 and rewards 340 are determined for all hybrid autonomous vehicles. In this way, the enhanced RL controller is trained to translate the traffic state of the hybrid autonomous vehicle group into a target headway that improves the actions of the hybrid autonomous vehicles in the group, with the actions defined by the target headway based on the headway-based model. In effect, the enhanced RL controller allows for the control of hybrid autonomous vehicles during platoon formation.

[0080] Different implementations use different methods to train the parameterized function that forms the RL controller. For example, in some implementations, the parameterized function is trained using one of the following methods: deep deterministic policy gradient method, advantage-actor-critic method, proximal policy optimization method, deep Q-network method, or Monte Carlo policy gradient method.

[0081] Figure 3B An embodiment of the present invention is shown. Figure 3A The inputs and outputs of the learning algorithm 307 are as follows: The inputs to the learning algorithm 307, fed through the reward function, include the current headway h of all observed vehicles (i.e., all vehicles whose parameters are transmitted to the controller). i 301. Current speed vi 303 and current acceleration u i 305. The output is the target acceleration of the connected autonomous vehicle, 309.

[0082] In some implementations, the input is measured and reported directly. Typically, the measurement should be processed by an estimation algorithm to filter out noise in the signal. In other implementations, model-based methods using models such as optimal speed models can be used to determine acceleration given the headway and speed.

[0083] In one implementation, the control algorithm used is based on deep reinforcement learning. Unlike ordinary deep learning, deep reinforcement learning uses deep neural networks to determine the control that minimizes the value function V.

[0084] Figure 4 A graph showing the convergence of model-free and augmented reinforcement learning according to some implementations is presented. Figure 4 The experimental results show that simple deep neural networks without extended tuning exhibit satisfactory performance, strongly suggesting that reinforcement learning is well-suited for this application. The learning of reinforcement optimal control converges quickly 401, while model-free methods using the same learning approach do not converge 402.

[0085] When a controller is applied to a system, it is implemented digitally using an embedded computer. For this reason, the implementation of the controller is typically done using a discrete-time implementation. The process of converting a continuous-time design into a discrete-time design is standard, and various processes exist to do this. In particular, we did this experimentally using a zero-order hold method.

[0086] More importantly, the action space (i.e., the action) The set they belong to is continuous because It is granular, that is, The possible choices are theoretically uncountable. For this reason, we found it necessary to implement a deep reinforcement learning algorithm suitable for use with a continuous action space. This approach is the deep deterministic policy gradient method. This is the method used in one implementation, but the above methods and systems are by no means limited to using one type of reinforcement learning algorithm. Leveraging experimental success similar to other implementations, we have also implemented other methods using the advantage-actor commentator method.

[0087] Exemplary Implementation

[0088] Figure 5A block diagram of a system 500 for directly and indirectly controlling hybrid autonomous vehicles, according to some embodiments, is shown. System 500 may have multiple interfaces for connecting system 500 to other machines and devices. A network interface controller (NIC) 550 includes a receiver adapted to connect system 500 to a network 590 via a bus 506, which connects system 500 to the hybrid autonomous vehicles to receive traffic conditions of a group of hybrid autonomous vehicles traveling in the same direction, wherein the group of hybrid autonomous vehicles includes controlled vehicles willing to participate in platoon formation and at least one uncontrolled vehicle, and wherein the traffic conditions indicate the state of each vehicle and controlled vehicle in the group. For example, in one embodiment, the traffic conditions include the current headway, current speed, and current acceleration of the hybrid autonomous vehicles. In some embodiments, the hybrid autonomous vehicles include all uncontrolled vehicles within a predetermined range on the flanks of the controlled vehicles in the platoon.

[0089] The NIC 550 also includes a transmitter adapted to send control commands to the controlled vehicles via network 590. For this purpose, system 500 includes an output interface, such as control interface 570, configured to submit control commands 575 to the controlled vehicles in the group of hybrid autonomous vehicles via network 590. In this way, system 500 can be deployed on a remote server that communicates directly or indirectly with the hybrid autonomous vehicles.

[0090] System 500 may also include other types of input and output interfaces. For example, system 500 may include a human-machine interface 510. The human-machine interface 510 can connect the controller 500 to a keyboard 511 and a pointing device 512, wherein the pointing device 512 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touch screen, etc.

[0091] System 500 includes a processor 520 configured to execute stored instructions and a memory 540 storing instructions executable by the processor. Processor 520 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. Memory 540 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable storage device. Processor 520 may be connected to one or more input and output devices via bus 506.

[0092] Processor 520 is operatively connected to storage device 530, which stores instructions and processes data used by the instructions. Storage device 530 may form part of memory 540 or be operatively connected to memory 540. For example, memory may be configured to store parameterized function 531 trained to convert traffic states into target headway for hybrid autonomous vehicles; and to store headway-based model 533 configured to map target headway to target speed of hybrid autonomous vehicles.

[0093] Processor 520 is configured to determine control commands for controlled vehicles, which also indirectly control uncontrolled vehicles. To this end, processor is configured to execute control generator 532 to submit traffic states to a parameterization function to generate a target headway, submit the target headway to a headway-based model to generate a target speed, and determine control commands for the controlled vehicles based on one or a combination of the target headway and the target speed.

[0094] In some implementations, parameterization function 531 is a deep reinforcement learning (DRL) controller trained to translate the traffic states of the group of hybrid autonomous vehicles into a target headway, rather than improving the actions of the hybrid autonomous vehicles in the group, where actions are defined by the target headway based on a headway-based model. For example, as described above, the target headway is determined as a value function of the movement costs that improve fleet formation. Examples of value functions include the negative of the sum of the movement costs of the hybrid autonomous vehicles within the planned timeframe.

[0095] In some implementations, the cost is a cost function of multiple terms that incentivize fleet formation. For example, in some implementations, the cost of movement includes a component related to the convoy stability, a component related to the equipment stability of the fleet, and a penalty term for penalizing the target headway when it approaches the stop headway.

[0096] In various implementations, the DRL controller is a model-based DRL enhanced with a headway-based model that solves online optimization, to simultaneously learn the behavior of a traffic system with traffic conditions and learn to control the traffic system to form a convoy, such that the behavior of the traffic system is defined by the target headway generated by the DRL controller, and the control of the traffic system is defined by the acceleration of the hybrid automated vehicles determined from the target headway based on the headway-based model.

[0097] In some implementations, the headway-based model is an optimal speed model (OVM) that maps the target headway to the vehicle speed, such that the DRL controller is an OVM-enhanced DRL. The OVM maps the target headway to the vehicle speed. In some implementations, system 500 is configured to determine the target speed of the controlled vehicle from the target headway based on the OVM, and to formulate control commands for the controlled vehicle to include the corresponding target speed of the controlled vehicle.

[0098] For example, a headway-based model (such as OVM) associates the current headway, stationary headway, and forward headway of the hybrid autonomous vehicle with the target speed of the hybrid autonomous vehicle. The target headway, determined by a parameterized function, is the forward headway, and system 500 uses the determined forward headway and a predetermined stationary headway to determine the target speed of the control command for the hybrid autonomous vehicle according to the headway-based model.

[0099] Figure 6A A schematic diagram of a vehicle 601 directly or indirectly controlled according to some embodiments is shown. As used herein, vehicle 601 can be any type of wheeled vehicle, such as a bus, coach, or off-road vehicle. Furthermore, vehicle 601 can be an autonomous or semi-autonomous vehicle. For example, some embodiments control the movement of vehicle 601. Examples of movement include lateral movement of the vehicle controlled by a steering system 603 of vehicle 601. In one embodiment, the steering system 603 is controlled by a controller 602 communicating with system 500. Alternatively or additionally, the steering system 603 may be controlled by the driver of vehicle 601.

[0100] The vehicle may also include an engine 606, which may be controlled by the controller 602 or other components of the vehicle 601. The vehicle may also include one or more sensors 604 to sense the surrounding environment. Examples of sensors 604 include rangefinders, radar, lidar, and cameras. The vehicle 601 may also include one or more sensors 605 to sense its current motion and internal state. Examples of sensors 605 include Global Positioning System (GPS), accelerometers, inertial measurement units, gyroscopes, shaft rotation sensors, torque sensors, deflection sensors, pressure sensors, and flow sensors. The sensors provide information to the controller 602. The vehicle may be equipped with a transceiver 606, enabling the controller 602 to communicate via wired or wireless communication channels.

[0101] Figure 6BA schematic diagram illustrates the interaction between a controller 602 receiving control commands from system 500 and a controller 600 of vehicle 601, according to some embodiments. For example, in some embodiments, the controller 600 of vehicle 601 is a steering 610 and a brake / throttle controller 620 that control the rotation and acceleration of vehicle 600. In this case, controller 602 outputs control inputs to controllers 610 and 620 to control the state of the vehicle. Controller 600 may also include higher-level controllers, such as a lane-keeping assist controller 630 that further processes the control inputs of predictive controller 602. In both cases, controller 600 maps the use of the output of predictive controller 602 to control at least one actuator of the vehicle (such as the vehicle's steering wheel and / or brakes) to control the vehicle's motion. The state of the vehicle machine x t This can include position, orientation, and longitudinal / lateral velocity; control input u t This may include lateral / longitudinal acceleration, steering angle, and engine / brake torque. State constraints on the system may include lane-keeping constraints and obstacle avoidance constraints. Control input constraints may include steering angle constraints and acceleration constraints. Collected data may include position, orientation, and velocity profiles, acceleration, torque, and / or steering angle.

[0102] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors, whether provided in a single computer or distributed across multiple computers. Such a processor can be implemented as an integrated circuit having one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0103] Furthermore, the various methods or processes outlined herein can be encoded as software executable on one or more processors employing any of a variety of operating systems or platforms. Moreover, such software can be written using any of a variety of suitable programming languages ​​and / or program or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, the functionality of program modules can be combined or distributed as needed in various implementations.

[0104] Furthermore, embodiments of the present invention can be specifically implemented as a method, examples of which have been provided. The actions performed as part of this method can be ordered in any suitable manner. Therefore, embodiments can be constructed that perform actions in a different order than those shown, which may include performing some actions simultaneously, even if they are shown as sequential in the illustrative embodiments.

[0105] Although the invention has been described by way of example of preferred embodiments, it should be understood that various other modifications and alterations can be made within the spirit and scope of the invention. Therefore, the appended claims are intended to cover all such variations and modifications that fall within the true spirit and scope of the invention.

Claims

1. A system for direct and indirect control of a mixed fleet of autonomous vehicles, the system comprising: a receiver configured to receive a traffic state of a group of mixed autonomous vehicles travelling in the same direction, wherein the group of mixed autonomous vehicles comprises a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group of mixed autonomous vehicles, the state comprising a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; a memory configured to store a parameterized function trained to convert the traffic state into a target headway of the mixed autonomous vehicle; and store a headway-based model configured to map the target headway to a target speed of the mixed autonomous vehicle; a processor configured to submit the traffic state into the parameterized function to produce the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the group of mixed autonomous vehicles into a target headway that improves an action of the mixed autonomous vehicle defined from the target headway according to the headway-based model, wherein the target headway is determined such that the action improves a value function of a mobility cost of the platoon formation, wherein the value function is a negative of a sum of mobility costs of the mixed autonomous vehicle over a planning time horizon, and wherein the mobility cost comprises a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term penalizing the target headway from approaching a stop headway; submit the target headway to the headway-based model to produce the target speed; and determine a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and a transmitter configured to transmit the control command to the controlled vehicle in the group of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the command, wherein the headway-based model is an optimal velocity model (OVM) that maps the target headway to a speed of the vehicle such that the DRL controller is an OVM-enhanced DRL.

2. The system of claim 1, wherein, the DRL controller is a model-based DRL enhanced with a headway-based model that solves online optimization to simultaneously learn a behavior of a traffic system with the traffic state and learn a control of the traffic system to form the platoon such that the behavior of the traffic system is defined by the target headway produced by the DRL controller and the control of the traffic system is defined by an acceleration of the mixed autonomous vehicle determined from the target headway according to the headway-based model.

3. The system of claim 1, wherein, The parameterized function is trained using a deep deterministic policy gradient method, an advantage actor critic method, a proximal policy optimization method, a deep Q network method, or a Monte Carlo policy gradient method.

4. The system of claim 1, wherein, The system is arranged on a remote server in direct or indirect wireless communication with the mixed automated driving vehicle.

5. A system for direct and indirect control of a mixed automated driving vehicle, the system comprising: a receiver configured to receive a traffic state of a group of mixed automated driving vehicles travelling in the same direction, wherein the group of mixed automated driving vehicles comprises a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group of mixed automated driving vehicles, the state comprising a current headway, a current speed, and a current acceleration of the mixed automated driving vehicle; a memory configured to store a parameterized function trained to convert the traffic state into a target headway of the mixed automated driving vehicle; and store a headway-based model configured to map the target headway to a target speed of the mixed automated driving vehicle; a processor configured to submit the traffic state into the parameterized function to produce the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the group of mixed automated driving vehicles into a target headway that improves an action of the mixed automated driving vehicle defined from the target headway according to a headway-based model, wherein the target headway is determined such that the action improves a value function of a movement cost of the platoon formation, wherein the value function is a negative of a sum of movement costs of the mixed automated driving vehicle over a planning time horizon, and wherein the movement cost comprises a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term penalizing the target headway from approaching a parking headway; submit the target headway to the headway-based model to produce the target speed; and determine a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and a transmitter configured to transmit the control command to the controlled vehicle in the group of mixed automated driving vehicles, wherein the controlled vehicle is arranged to be controlled by the command, wherein the headway-based model is an optimal velocity model (OVM) mapping the target headway to a target speed of the vehicle, wherein the processor is configured to determine the target speed of the controlled vehicle from the target headway according to the OVM and form a control command to the controlled vehicle comprising the respective target speed of the controlled vehicle.

6. The system of claim 5, wherein, The mixed autonomous vehicle comprises all uncontrolled vehicles within a predetermined range flanking the controlled vehicle in the platoon.

7. A system for direct and indirect control of a mixed autonomous vehicle, the system comprising: a receiver configured to receive a traffic state of a group of mixed autonomous vehicles travelling in the same direction, wherein the group of mixed autonomous vehicles comprises a controlled vehicle willing to participate in platooning and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group of mixed autonomous vehicles, the state comprising a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; a memory configured to store a parameterized function trained to convert the traffic state into a target headway of the mixed autonomous vehicle; and store a headway-based model configured to map the target headway to a target speed of the mixed autonomous vehicle; a processor configured to submit the traffic state into the parameterized function to produce the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the group of mixed autonomous vehicles into a target headway that improves an action of the mixed autonomous vehicle defined by the target headway according to the headway-based model, wherein the target headway is determined such that the action improves a value function of a movement cost of the platooning, wherein the value function is a negative of a sum of movement costs of the mixed autonomous vehicle over a planning time horizon, and wherein the movement cost comprises a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term penalizing the target headway from approaching a stop headway; submit the target headway to the headway-based model to produce the target speed; and determine a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and a transmitter configured to transmit the control command to the controlled vehicle in the group of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the command, wherein the headway-based model relates a current headway, a stop headway, and an advance headway of the mixed autonomous vehicle to a target speed of the mixed autonomous vehicle, wherein the target headway determined by the parameterized function is the advance headway, and wherein the processor determines the target speed of the control command for the mixed autonomous vehicle according to the headway-based model with the determined advance headway and a predetermined fixed stop headway.

8. A method for directly and indirectly controlling a hybrid automated vehicle, wherein, The method uses a processor coupled with storage instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the steps of: receiving traffic states of a group of mixed autonomous vehicles travelling in the same direction, wherein the group of mixed autonomous vehicles comprises controlled vehicles willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic states indicate a state of each vehicle in the group of mixed autonomous vehicles, the state comprising a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic states to a parameterized function to produce a target headway for the mixed autonomous vehicle, the parameterized function trained to convert the traffic states to the target headway, wherein the parameterized function is a deep reinforcement learning, DRL, controller trained to convert traffic states of the group of mixed autonomous vehicles to a target headway that improves an action of the mixed autonomous vehicle defined by a model based on the headway from the target headway, wherein the target headway is determined such that the action improves a value function of a movement cost of the platoon formation, wherein the value function is a negative of a sum of movement costs of the mixed autonomous vehicle over a planning time horizon, and wherein the movement cost comprises a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term penalizing the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed for the mixed autonomous vehicle, the headway-based model configured to map the target headway to the target speed; determining control commands to the controlled vehicles based on one or a combination of the target headway and the target speed; and sending the control commands to the controlled vehicles in the group of mixed autonomous vehicles, wherein the controlled vehicles are arranged to be controlled by the commands, wherein the headway-based model is an optimal velocity model, OVM, that maps the target headway to a speed of the vehicle such that the DRL controller is an OVM-enhanced DRL.

9. A method for directly and indirectly controlling a hybrid automated vehicle, wherein, The method uses a processor coupled with storage instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the steps of: receiving traffic states of a group of mixed autonomous vehicles travelling in the same direction, wherein the group of mixed autonomous vehicles comprises controlled vehicles willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic states indicate a state of each vehicle in the group of mixed autonomous vehicles, the state comprising a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic state to a parametric function to produce a target headway for the platoon of mixed autonomous vehicles, the parametric function trained to convert the traffic state to the target headway, wherein the parametric function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the platoon of mixed autonomous vehicles to a target headway that improves an action of the mixed autonomous vehicles defined by the target headway according to a model based on the headway, wherein the target headway is determined such that the action improves a value function of a mobility cost of the platoon formation, wherein the value function is a negative of a sum of mobility costs of the mixed autonomous vehicles over a planning time horizon, and wherein the mobility cost includes a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term that penalizes the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed for the mixed autonomous vehicles, the headway-based model configured to map the target headway to the target speed; determining a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and sending the control command to the controlled vehicle in the platoon of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the commands, wherein the headway-based model is an optimal velocity model (OVM) that maps the target headway to a target speed of the vehicle, wherein the processor is configured to determine the target speed of the controlled vehicle from the target headway according to the OVM and form a control command to the controlled vehicle that includes the respective target speed of the controlled vehicle.

10. A method for directly and indirectly controlling a hybrid automated vehicle, wherein, The method uses a processor coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor, perform the steps of the method, the method comprising the steps of: receiving a traffic state of a platoon of mixed autonomous vehicles traveling in the same direction, wherein the platoon of mixed autonomous vehicles includes a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the platoon of mixed autonomous vehicles, the state including a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic state to a parameterized function to produce a target headway for the platoon of mixed autonomous vehicles, the parameterized function trained to convert the traffic state to the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the platoon of mixed autonomous vehicles to a target headway that improves an action of the mixed autonomous vehicle defined by the target headway according to a model based on the headway, wherein the target headway is determined such that the action improves a value function of a mobility cost of the platoon formation, wherein the value function is a negative of a sum of mobility costs of the mixed autonomous vehicle over a planning time horizon, and wherein the mobility cost includes a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term that penalizes the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed for the mixed autonomous vehicle, the headway-based model configured to map the target headway to the target speed; determining a control command for the controlled vehicle based on one or a combination of the target headway and the target speed; and sending the control command to the controlled vehicle in the platoon of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the commands, wherein the headway-based model relates a current headway, a stop headway, and an advance headway of the mixed autonomous vehicle to a target speed of the mixed autonomous vehicle, wherein the target headway determined by the parameterized function is the advance headway, and wherein the processor determines the target speed of the control command for the mixed autonomous vehicle according to the headway-based model with the determined advance headway and a predetermined fixed stop headway.

11. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method comprising: receiving a traffic state of a platoon of mixed autonomous vehicles traveling in the same direction, wherein the platoon of mixed autonomous vehicles includes a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the platoon of mixed autonomous vehicles, the state including a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic state into a parameterized function to produce a target headway for the mixed autonomous vehicle, the parameterized function trained to convert the traffic state into the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the group of mixed autonomous vehicles into a target headway that improves an action of the mixed autonomous vehicle defined by the target headway according to a model based on the headway, wherein the target headway is determined such that the action improves a value function of a movement cost of the platoon formation, wherein the value function is a negative of a sum of movement costs of the mixed autonomous vehicle over a planning time horizon, and wherein the movement cost includes a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term that penalizes the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed for the mixed autonomous vehicle, the headway-based model configured to map the target headway to the target speed; determining a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and sending the control command to the controlled vehicle in the group of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the commands, wherein the headway-based model is an optimal velocity model (OVM) that maps the target headway to a speed of the vehicle, such that the DRL controller is an OVM-enhanced DRL.

12. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method comprising: receiving a traffic state of a group of mixed autonomous vehicles traveling in the same direction, wherein the group of mixed autonomous vehicles includes a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group of mixed autonomous vehicles, the state including a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic state into a parameterized function to produce a target headway of the mixed autonomous vehicle, the parameterized function trained to convert the traffic state into the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the group of mixed autonomous vehicles into a target headway that improves an action of the mixed autonomous vehicle defined by the target headway according to a model based on the headway, wherein the target headway is determined such that the action improves a value function of a movement cost of the platoon formation, wherein the value function is a negative of a sum of movement costs of the mixed autonomous vehicle over a planning time horizon, and wherein the movement cost includes a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term that penalizes the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed of the mixed autonomous vehicle, the headway-based model configured to map the target headway to the target speed; determining a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and sending the control command to the controlled vehicle in the group of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the commands, wherein the headway-based model is an optimal velocity model (OVM) that maps the target headway to a target speed of the vehicle, wherein the processor is configured to determine the target speed of the controlled vehicle from the target headway according to the OVM, and form a control command to the controlled vehicle that includes the respective target speed of the controlled vehicle.

13. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method comprising: receiving a traffic state of a group of mixed autonomous vehicles traveling in the same direction, wherein the group of mixed autonomous vehicles includes a controlled vehicle willing to participate in a platoon formation and at least one uncontrolled vehicle, and wherein the traffic state indicates a state of each vehicle in the group of mixed autonomous vehicles, the state including a current headway, a current speed, and a current acceleration of the mixed autonomous vehicle; submitting the traffic state into a parameterized function to produce a target headway for the mixed autonomous vehicle, the parameterized function trained to convert the traffic state into the target headway, wherein the parameterized function is a deep reinforcement learning (DRL) controller trained to convert a traffic state of the set of mixed autonomous vehicles into a target headway that improves an action of the mixed autonomous vehicle defined by the target headway according to a model based on the headway, wherein the target headway is determined such that the action improves a value function of a movement cost of the platoon formation, wherein the value function is a negative of a sum of movement costs of the mixed autonomous vehicle over a planning time horizon, and wherein the movement cost includes a component related to string stability of the platoon, a component related to device stability of the platoon, and a penalty term that penalizes the target headway from approaching a stop headway; submitting the target headway to a headway-based model to produce a target speed for the mixed autonomous vehicle, the headway-based model configured to map the target headway to the target speed; determining a control command to the controlled vehicle based on one or a combination of the target headway and the target speed; and sending the control command to the controlled vehicle of the set of mixed autonomous vehicles, wherein the controlled vehicle is arranged to be controlled by the commands, wherein the headway-based model relates a current headway, a stop headway, and an advance headway of the mixed autonomous vehicle to a target speed of the mixed autonomous vehicle, wherein the target headway determined by the parameterized function is the advance headway, and wherein the processor determines the target speed for the control command to the mixed autonomous vehicle according to the headway-based model with the determined advance headway and a predetermined fixed stop headway.

Images