Truck automatic driving formation decision control method for road mixed driving scene
By employing a two-layer optimization architecture and a multi-agent reinforcement learning-based platooning decision control method, the flexibility and robustness issues of truck platooning in dynamic traffic environments were addressed. This enabled autonomous trucks to form platoons autonomously, safely, and efficiently in mixed-traffic scenarios, thereby improving transportation efficiency and safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-03
AI Technical Summary
Existing truck platooning control technologies lack flexibility and robustness in dynamic traffic environments, are unable to achieve globally optimal decisions, and have fragmented system frameworks that limit overall performance, making it difficult to adapt to the dynamic adjustment needs of vehicles and easily leading to transportation time-saving losses.
A two-layer optimized architecture is adopted. The upper layer uses multi-agent reinforcement learning for platooning and collaborative decision-making, while the lower layer uses optimization modeling methods for multi-vehicle collaborative planning and control. This constructs a platooning decision-making and control method that adapts to the interference of mixed traffic flow on highways. By combining global system rewards and individual behavior rewards, autonomous trucks can form platoons autonomously, safely, and efficiently in dynamic environments.
It enhances the adaptability and intelligence of autonomous truck platoons in dynamic traffic environments, improves the safety and traffic efficiency of platooning, and ensures driving stability and safety.
Smart Images
Figure CN121789436A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving control technology, and more particularly to a method for decision-making and control of autonomous truck platooning in mixed-traffic scenarios on highways. Background Technology
[0002] Highways serve as the core carriers for cross-regional transportation of bulk commodities, industrial raw materials, and essential goods, and their freight efficiency and safety directly impact the stable operation of the national economic supply chain. Freight trucks handle over 80% of my country's road freight volume, forming the backbone of the highway transportation system. To improve freight efficiency and reduce operating costs, truck platooning technology has become a hot research topic. This involves multiple trucks maintaining coordinated speeds and spacing, theoretically reducing air resistance and increasing road traffic density. However, current truck platooning control technology still faces many bottlenecks that urgently need to be addressed, making it difficult to meet the efficient, safe, and stable transportation demands of highway corridors.
[0003] Truck platooning technology is not only a key path to improving freight efficiency and reducing operating costs, but also plays an irreplaceable role in ensuring transportation safety and practicing green and low-carbon development. From an operational economic perspective, multi-vehicle platooning can significantly reduce air resistance, lower fuel consumption, and substantially reduce long-term operating expenses for freight companies. From a traffic efficiency perspective, standardized and coordinated platooning can increase road throughput per unit time, effectively alleviate congestion during peak freight seasons, and improve the overall utilization rate of the road network. From a safety perspective, autonomous driving platooning can avoid risks such as driver fatigue, reaction delays, and operational errors, significantly reducing the incidence of rear-end collisions and scrapes. From an environmental perspective, the reduction in fuel consumption can simultaneously reduce exhaust emissions and noise pollution, aligning with the green and low-carbon development trend in the transportation sector.
[0004] Traditional truck platooning refers to multiple trucks traveling in a convoy with fixed intervals and speeds. This method relies heavily on the driver's subjective experience and operational skills to maintain distance and speed. However, due to the delay in human reaction, the overall convoy speed is relatively low and the intervals between vehicles are large, making it difficult to achieve the ideal timeliness optimization goal. Furthermore, most current research still directly plans the shortest route based on given traffic information. When facing traffic congestion, static strategies such as stopping and waiting or optimizing departure times are often used. These methods cannot adapt to the dynamic adjustment needs of vehicles in motion and are prone to causing losses in transportation timeliness.
[0005] Based on an in-depth analysis of existing technologies, the following key issues exist in the field of multi-CAV cooperative platooning, especially in high-speed, high-traffic scenarios: First, existing methods largely rely on predefined rules or simple optimization objectives. These methods lack flexibility and robustness in dynamic environments with high vehicle density, large state differences, and complex intentions. The system cannot learn from historical experience and struggles to predict the behavior of other vehicles, leading to decisions that may not be globally optimal or even cause conflicts. Second, fragmented system frameworks limit overall performance. Existing solutions typically focus on a single link in the platooning control process, or only design high-level collaborative decision-making rules, such as platoon formation generation, or only optimize mid-level motion planning paths, such as lane change trajectory design, or only improve low-level vehicle control algorithms, such as throttle and braking precision control. There is a lack of a unified technical framework that can seamlessly connect high-level collaborative decision-making, mid-level motion planning, and high-precision vehicle control. This results in complex interface design between system modules and poor information flow, such as mismatches between decision commands and control parameters, ultimately restricting the overall control accuracy and response speed of autonomous truck platooning. Summary of the Invention
[0006] To address the aforementioned technical problems, this invention provides a platooning decision-making and control method for autonomous trucks in mixed-traffic highway scenarios. This invention utilizes a two-layer optimized architecture design. The upper layer employs multi-agent reinforcement learning for collaborative decision-making in platooning formation, while the lower layer uses optimization modeling methods for multi-vehicle collaborative planning and control. This constructs a platooning decision-making and control method adaptable to the interference of mixed-traffic highways, enabling autonomous trucks to form platoons autonomously, safely, and efficiently in dynamic and heterogeneous traffic environments. This aims to improve highway freight efficiency and enhance driving stability and safety. The key aspects include the dynamic generation, speed planning, path planning, and collaborative control of multiple autonomous vehicles within the platoon, with the core objective of improving the safety and traffic efficiency of platooning.
[0007] The technical means employed in this invention are as follows: A method for decision-making and control of autonomous truck platooning in mixed-traffic highway scenarios includes: S1. Define the autonomous truck platooning system as... It consists of 1 autonomous truck and 1 self-driving truck. Each autonomous truck will be accompanied by a follow-up truck; and each autonomous truck will be assigned a unique, fixed number. Among them, the lead truck is The other accompanying trucks are numbered sequentially as follows: ; S2, autonomous trucks within a platoon It uses its own onboard cameras, LiDAR, millimeter-wave radar, GPS and other sensors to collect its own status information, the status information of other autonomous trucks in the platoon, the status information of vehicles in the surrounding environment and lane line information in real time, to form local status information. ; S3, All local state information detected by the autonomous trucks Integrate into a system state information vector The data is then input into the upper-level formation decision module, which outputs the formation decision to the autonomous truck platooning system. ; S4. Formation Decision Under guidance, the lower-level multi-vehicle collaborative planning and control module outputs a set of optimized action sequences for collaborative motion control in order to form a platoon for the autonomous truck platooning system. , ,in For autonomous trucks The corresponding formation forms a motion control optimized action sequence; S5, Utilizing Action Sequences The first action in the process is for autonomous trucks Perform control execution Time step; S6. Based on the updated autonomous truck platooning system status and the mixed traffic environment status on the highway, determine whether the autonomous trucks are in the same lane and whether the relative longitudinal position error, relative speed error and relative heading angle error of all following trucks and the lead truck are within the preset tolerance range; if so, determine that the autonomous trucks have formed a platoon; otherwise, return to step S2 to continue executing the platooning decision control.
[0008] Furthermore, in step S2: The autonomous truck Its own status information includes its own lateral position coordinates, longitudinal position coordinates, longitudinal driving speed, and heading angle; The status information of other autonomous trucks in the platoon refers to the status of other autonomous trucks in the platoon relative to the autonomous trucks. The status information, namely the relative status information, includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The status information of the external surrounding vehicles refers to the detected environmental vehicles relative to the autonomous truck. The status information, namely the relative status information, also includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The lane information includes autonomous trucks. The perpendicular distance from the road boundaries on both sides, and the perpendicular distance from the lane lines on both sides of the lane in which it is located.
[0009] Furthermore, in step S3: The upper-level formation decision module is composed of It consists of multiple agent neural networks, and a single agent neural network. For an autonomous truck Output formation decision Therefore, the autonomous truck platooning system outputs platooning formation decisions. Among them, autonomous trucks Output formation decision Lane-changing decisions made in order to complete platooning tasks when driving in a multi-lane highway environment include left lane change, lane keeping, and right lane change.
[0010] Furthermore, in step S3: The upper-layer formation decision module uses a multi-agent reinforcement learning method to train the agent neural network. The training process is as follows: An autonomous driving simulation environment with three lanes of mixed traffic was set up using autonomous driving simulation software. By randomly distributing autonomous trucks and manually driven vehicles, dynamic heterogeneous traffic flow was simulated. A multi-agent reinforcement learning-based platooning decision reward function is defined to incentivize the safe, efficient, and stable formation of autonomous truck platoons. The reward function employs a hybrid architecture combining global system rewards and individual rewards for each autonomous truck agent in the platoon. Calculate the total reward value separately The calculation formula is as follows:
[0011] In the above formula, This indicates the global formation reward function. This represents the global traffic flow disturbance penalty function. This represents the reward function for individual formation contributions. Represents the individual safety reward function. Represents the individual efficiency reward function; The multi-agent dominant actor-critic algorithm is used to distribute the training of the multi-agent neural network parameters of the upper-level formation formation decision module, so that the trained upper-level formation formation decision module outputs formation decisions for the autonomous truck formation system.
[0012] Furthermore, the global system reward component is shared by all autonomous truck agents within the platoon and is used to guide the cooperative movement behavior among agents at the system level. This includes global platoon formation rewards and global traffic flow disturbance penalties, specifically designed as follows: The formula for calculating the global formation reward is as follows:
[0013] The calculation of the global formation reward is triggered when all autonomous trucks are in the same lane. It is calculated based on the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, aiming to encourage rapid and stable formation of the overall formation. In the above formula, The same lane trigger condition is defined if and only if all autonomous trucks are in the same lane. ;otherwise, ; The norm of the error vector. This represents the error vector consisting of the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, where the relative longitudinal position error is the difference between the current longitudinal distance and the expected following distance; The global traffic flow disturbance penalty aims to incentivize the minimization of disturbances to mixed traffic flow during the formation process, primarily through changes in the acceleration of surrounding vehicles (SVs) caused by formation behavior. The calculation is as follows:
[0014] In the above formula, This indicates the change in acceleration of vehicles (SVs) in the surrounding environment of the formation due to formation behavior.
[0015] Furthermore, the individual reward is applied to each autonomous truck agent in the platoon. Separate calculations are performed to guide the intelligent agents of each autonomous truck. The specific driving behaviors during the formation process include three components: individual formation contribution reward, individual safety reward, and individual efficiency reward. The design of each component is as follows: The formula for calculating the individual formation contribution reward is as follows:
[0016] The individual formation contribution reward is based on the real-time global position and status of all candidate autonomous trucks, dynamically presets a virtual formation with the lead truck as the benchmark, and then rewards each autonomous truck accordingly. Dynamically allocate virtual formation anchor point coordinates This aims to encourage vehicles to move towards the coordinates of the virtual formation anchor point, actively shortening the distance between them; in the above formula, , For autonomous trucks At the present moment The global position; The formula for calculating the individual safety reward is as follows:
[0017] The individual safety reward not only penalizes vehicle collisions but also incentivizes vehicles to maintain a safe state by measuring driving risks. In the above formula, This represents the collision indication function, when an autonomous truck... When a collision occurs, ,otherwise ; Indicates autonomous trucks The first and the second neighboring The distance between the front and rear of the vehicle; The safety time interval constant represents the minimum safe time interval that the workshop is expected to maintain. The formula for calculating the individual efficiency reward is as follows:
[0018] The individual efficiency reward function is designed to incentivize autonomous trucks. j While completing formation tasks, it can maintain efficient driving. In the above formula, This represents the expected driving speed of the autonomous truck.
[0019] Furthermore, the distributed training of the multi-agent neural network parameters of the upper-layer formation decision module using the multi-agent dominant actor-critic algorithm includes: Initialize the actor network for all autonomous truck agents. parameter and the network of critics parameter and experience replay pool And set hyperparameters, including the network learning rate. and Discount Factor Maximum number of training rounds and maximum time step ; For each training round Perform the following steps: step a 1. Reset the three-lane highway mixed traffic simulation environment and randomly initialize the position and status of the autonomous truck platooning system and the vehicles in the environment; step b When the decision-making time step If the platooning is not completed and no collision occurs, the following steps will be executed repeatedly: stepb 1. Intelligent agents for each autonomous truck Based on their respective detected current states Utilizing actor networks Output and sample the formation to form decision actions ; step b 2. Form a set of decision actions in the formation of all intelligent agents. Under guidance, the lower-level multi-vehicle collaborative planning and control module inputs optimized control action sequences into the autonomous truck platooning system, updates the state of the platooning system, and simultaneously guides other vehicles in the environment to update their states, enabling each autonomous truck intelligent agent to achieve this. Obtain new local state information for the next step. ; step b 3. Each autonomous truck intelligent agent Based on local state information The formation is designed to form a decision reward function, and the obtained reward is calculated. and the obtained experience tuples Stored in the experience replay pool middle; step b 4. When the experience replay pool After the number of empirical samples in the dataset meets the training conditions, each agent... From the experience replay pool Randomly sample a small batch of empirical samples and train the network according to the following formula: The critic network parameters are minimized by the loss function. Update:
[0020] Actor network parameters are maximized by the objective function Update:
[0021] Every The training effect of the formation decision neural network is evaluated in several training rounds. P If the average reward value of each evaluation round tends to stabilize, then the convergence condition is met and training stops. After training is completed, the trained multi-agent neural network parameters are saved and deployed to the upper-level formation decision module of the actual autonomous truck for decision guidance in real vehicle formation.
[0022] Furthermore, in step S4, the control method of the lower-level multi-vehicle cooperative planning and control module is as follows: Based on the lane-changing behavior decision output by the upper-level formation decision module, a desired target state is generated for each autonomous truck; based on the current state and the desired target state, a piecewise linear interpolation method is used to generate... The reference trajectory of duration will be optimized in the time domain. Divided into Each time step is obtained. ( (reference state) ; A multi-vehicle cooperative control optimization model is constructed, namely a finite-time domain optimization model with the objective of optimizing the overall control efficiency of the autonomous truck platooning system. This model is solved within each control cycle through objective function and constraint design, providing a solution for each autonomous truck. Output the future Motion control optimization action sequence at each time step ,in For autonomous trucks At time step The control actions include longitudinal acceleration and front wheel steering angle. ; Taking into account the four objectives of trajectory tracking accuracy, control smoothness, formation coordination, and driving safety, the objective function of the multi-vehicle cooperative control optimization model is designed as follows:
[0023] In the above formula, Self-driving trucks With autonomous trucks The expected relative position vectors between them Environmental vehicles At time step The state vector at that location; The constraints for designing the multi-vehicle cooperative control optimization model include dynamic constraints of the autonomous truck platooning system, vehicle state and control variable constraints, and driving safety constraints. The multi-vehicle cooperative control optimization model is optimized and solved using a distributed model prediction framework, and the optimized control action sequence is obtained. The data is transmitted to the corresponding autonomous driving truck execution system to complete the coordinated motion control of the formation.
[0024] Furthermore, the constraints of the multi-vehicle cooperative control optimization model are as follows: The dynamic constraints of the autonomous truck platooning system aim to ensure that the optimized trajectory conforms to the physical motion laws of the autonomous trucks, as shown in the following formula:
[0025] In the above formula, and All of these are state-space equations describing the dynamic characteristics of autonomous trucks; The vehicle state and control variable constraints are used to limit the feasible range of state and control variables for the autonomous truck, as shown in the following formula:
[0026] In the above formula, This represents the set of feasible states for autonomous trucks. , These are the physical limits and comfort limits of the control rate of change for the actuators of the autonomous truck, respectively. The driving safety constraints are the hard conditions designed to ensure driving safety, including internal safety constraints within the platooning system and safety constraints with the external environment, as shown in the following formula:
[0027] In the above formula, For any two vehicles within an autonomous truck platooning system and The minimum safe distance to be maintained throughout the entire prediction time domain; For autonomous trucks Vehicles in any surrounding environment The minimum safe distance to be maintained between them.
[0028] Compared with the prior art, the present invention has the following advantages: 1. This invention proposes an upper-level platooning formation decision model based on multi-agent reinforcement learning. This model enables each truck to make lane-changing decisions autonomously based on local state information through a distributed trained agent network. Combined with a hybrid reward mechanism that combines global system rewards and individual behavior rewards, it can effectively solve the complex decision-making problem of multi-vehicle collaborative platooning in dynamically changing mixed traffic flows, and improve the adaptability and intelligence level of autonomous truck platooning systems in mixed highway scenarios.
[0029] 2. This invention establishes a two-layer collaborative control architecture that combines upper-layer decision-making and lower-layer optimization. This architecture decouples the strategic behavioral decisions of the upper layer from the precise motion control of the lower layer. The lane-changing decisions output by the upper-layer multi-agent reinforcement learning model guide the optimization objectives of the lower-layer model's predictive control. This can achieve closed-loop optimization from macro-strategy to micro-control, and while ensuring driving safety and comfort, it can significantly improve the overall efficiency and stability of formation.
[0030] 3. This invention constructs a lower-level multi-vehicle distributed model predictive control model that incorporates multi-dimensional safety constraints. This model comprehensively considers trajectory tracking, control smoothness, and formation coordination in the optimization objective, and strictly embeds hard constraints such as system dynamics, actuator limits, and internal and external safety distances within the formation into the constraint conditions. This ensures that the generated control action sequence not only conforms to the physical characteristics of the vehicles, but also absolutely guarantees driving safety, and can provide a solid and reliable safety foundation for the formation process. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic diagram illustrating the formation process of autonomous truck platooning in a mixed-traffic scenario where this invention is applied.
[0033] Figure 2 This is a system architecture diagram of the truck autonomous driving platooning decision control method for mixed-traffic highway scenarios proposed in this invention.
[0034] Figure 3 This is a flowchart of the multi-agent reinforcement learning training method for the upper-level formation decision module in this invention.
[0035] Figure 4 This is a flowchart of the lower-level multi-vehicle collaborative planning and control module in this invention. Detailed Implementation
[0036] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0037] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.
[0038] This invention provides a decision-making and control method for autonomous truck platooning in mixed-traffic scenarios, such as highways. Figure 1 As shown, the system architecture diagram of the method of the present invention is as follows. Figure 2 As shown, it includes an upper-level formation decision module and a lower-level multi-vehicle collaborative planning and control module. The specific operation steps of the two modules are as follows: S1. Define the autonomous truck platooning system as... It consists of 1 autonomous truck and 1 self-driving truck. Each autonomous truck will be accompanied by a follow-up truck; and each autonomous truck will be assigned a unique, fixed number. Among them, the lead truck is The other accompanying trucks are numbered sequentially as follows: ; S2, autonomous trucks within a platoon It uses its own onboard cameras, LiDAR, millimeter-wave radar, GPS and other sensors to collect its own status information, the status information of other autonomous trucks in the platoon, the status information of vehicles in the surrounding environment and lane line information in real time, to form local status information. ; S3, All local state information detected by the autonomous trucks Integrate into a system state information vector The data is then input into the upper-level formation decision module, which outputs the formation decision to the autonomous truck platooning system. ; S4. Formation Decision Under guidance, the lower-level multi-vehicle collaborative planning and control module outputs a set of optimized action sequences for collaborative motion control in order to form a platoon for the autonomous truck platooning system. , ,in For autonomous trucks The corresponding formation forms a motion control optimized action sequence; S5, Utilizing Action Sequences The first action in the process is for autonomous trucks Perform control execution Time step; S6. Based on the updated autonomous truck platooning system status and the mixed traffic environment status on the highway, determine whether the autonomous trucks are in the same lane and whether the relative longitudinal position error, relative speed error and relative heading angle error of all following trucks and the lead truck are within the preset tolerance range; if so, determine that the autonomous trucks have formed a platoon; otherwise, return to step S2 to continue executing the platooning decision control.
[0039] In a specific implementation, as a preferred embodiment of the present invention, in step S2: The autonomous truck Its own status information includes its own lateral position coordinates, longitudinal position coordinates, longitudinal driving speed, and heading angle; The status information of other autonomous trucks in the platoon refers to the status of other autonomous trucks in the platoon relative to the autonomous trucks. The status information, namely the relative status information, includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The status information of the external surrounding vehicles refers to the detected environmental vehicles relative to the autonomous truck. The status information, namely the relative status information, also includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The lane information includes autonomous trucks. The perpendicular distance from the road boundaries on both sides, and the perpendicular distance from the lane lines on both sides of the lane in which it is located.
[0040] In a specific implementation, as a preferred embodiment of the present invention, in step S3: The upper-level formation decision module is composed of It consists of multiple agent neural networks, and a single agent neural network. For an autonomous truck Output formation decision Therefore, the autonomous truck platooning system outputs platooning formation decisions. Among them, autonomous trucks Output formation decision Lane-changing decisions made in order to complete platooning tasks when driving in a multi-lane highway environment include left lane change, lane keeping, and right lane change.
[0041] In a specific implementation, as a preferred embodiment of the present invention, in step S3: The upper-layer formation decision module employs a multi-agent reinforcement learning method to train the agent neural network, such as... Figure 3 As shown, the training process is as follows: An autonomous driving simulation environment with three lanes and mixed traffic was set up using autonomous driving simulation software. Dynamic, heterogeneous traffic flow was simulated by randomly distributing autonomous trucks and manually driven vehicles. The autonomous truck platooning system consisted of… The vehicle consists of 1 lead truck and 2 trucks. The status information and action space of the following truck have been defined by steps S2 and S3; A multi-agent reinforcement learning-based platooning decision reward function is defined to incentivize the safe, efficient, and stable formation of autonomous truck platoons. The reward function employs a hybrid architecture combining global system rewards and individual rewards for each autonomous truck agent in the platoon. Calculate the total reward value separately The calculation formula is as follows:
[0042] In the above formula, This indicates the global formation reward function. This represents the global traffic flow disturbance penalty function. This represents the reward function for individual formation contributions. Represents the individual safety reward function. Represents the individual efficiency reward function; The multi-agent dominant actor-critic algorithm is used to distribute the training of the multi-agent neural network parameters of the upper-level formation formation decision module, so that the trained upper-level formation formation decision module outputs formation decisions for the autonomous truck formation system.
[0043] In a specific implementation, as a preferred embodiment of the present invention, the global system reward portion is shared by all autonomous truck agents within the platoon, and is used to guide the cooperative movement behavior between agents at the system level, including global platoon formation rewards and global traffic flow disturbance penalties, specifically designed as follows: The formula for calculating the global formation reward is as follows:
[0044] The calculation of the global formation reward is triggered when all autonomous trucks are in the same lane. It is calculated based on the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, aiming to encourage rapid and stable formation of the overall formation. In the above formula, The same lane trigger condition is defined if and only if all autonomous trucks are in the same lane. ;otherwise, ; The norm of the error vector. This represents the error vector consisting of the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, where the relative longitudinal position error is the difference between the current longitudinal distance and the expected following distance; The global traffic flow disturbance penalty aims to incentivize the minimization of disturbances to mixed traffic flow during the formation process, primarily through changes in the acceleration of surrounding vehicles (SVs) caused by formation behavior. The calculation is as follows:
[0045] In the above formula, This indicates the change in acceleration of vehicles (SVs) in the surrounding environment of the formation due to formation behavior.
[0046] In a specific implementation, as a preferred embodiment of the present invention, the individual reward is applied to each autonomous truck agent in the platoon. Separate calculations are performed to guide the intelligent agents of each autonomous truck. The specific driving behaviors during the formation process include three components: individual formation contribution reward, individual safety reward, and individual efficiency reward. The design of each component is as follows: The formula for calculating the individual formation contribution reward is as follows:
[0047] The individual formation contribution reward is based on the real-time global position and status of all candidate autonomous trucks, dynamically presets a virtual formation with the lead truck as the benchmark, and then rewards each autonomous truck accordingly. Dynamically allocate virtual formation anchor point coordinates This aims to encourage vehicles to move towards the coordinates of the virtual formation anchor point, actively shortening the distance between them; in the above formula, , For autonomous trucks At the present moment The global position; The formula for calculating the individual safety reward is as follows:
[0048] The individual safety reward not only penalizes vehicle collisions but also incentivizes vehicles to maintain a safe state by measuring driving risks. In the above formula, This represents the collision indication function, when an autonomous truck... When a collision occurs, ,otherwise ; Indicates autonomous trucks The first and the second neighboring The distance between the front and rear of the vehicle; The safety time interval constant represents the minimum safe time interval that the workshop is expected to maintain. The formula for calculating the individual efficiency reward is as follows:
[0049] The individual efficiency reward function is designed to incentivize autonomous trucks. jWhile completing formation tasks, it can maintain efficient driving. In the above formula, This represents the expected driving speed of the autonomous truck.
[0050] In a specific implementation, as a preferred embodiment of the present invention, the distributed training of the multi-agent neural network parameters of the upper-level formation decision module using the multi-agent dominant actor-critic algorithm includes: Initialize the actor network for all autonomous truck agents. parameter and the network of critics parameter and experience replay pool And set hyperparameters, including the network learning rate. and Discount Factor Maximum number of training rounds and maximum time step ; For each training round Perform the following steps: step a 1. Reset the three-lane highway mixed traffic simulation environment and randomly initialize the position and status of the autonomous truck platooning system and the vehicles in the environment; step b When the decision-making time step If the platooning is not completed and no collision occurs, the following steps will be executed repeatedly: step b 1. Intelligent agents for each autonomous truck Based on their respective detected current states Utilizing actor networks Output and sample the formation to form decision actions ; step b 2. Form a set of decision actions in the formation of all intelligent agents. Under guidance, the lower-level multi-vehicle collaborative planning and control module inputs optimized control action sequences into the autonomous truck platooning system, updates the state of the platooning system, and simultaneously guides other vehicles in the environment to update their states, enabling each autonomous truck intelligent agent to achieve this. Obtain new local state information for the next step. ; step b 3. Each autonomous truck intelligent agent Based on local state information The formation is designed to form a decision reward function, and the obtained reward is calculated. and the obtained experience tuples Stored in the experience replay pool middle; step b 4. When the experience replay pool After the number of empirical samples in the dataset meets the training conditions, each agent... From the experience replay pool Randomly sample a small batch of empirical samples and train the network according to the following formula: The critic network parameters are minimized by the loss function. Update:
[0051] Actor network parameters are maximized by the objective function Update:
[0052] Every The training effect of the formation decision neural network is evaluated in several training rounds. P If the average reward value of each evaluation round tends to stabilize, then the convergence condition is met and training stops. After training, the trained multi-agent neural network parameters are stored and deployed to the upper-level platooning decision-making module of the actual autonomous truck for guiding platooning decisions. Figure 4 As shown.
[0053] In a specific implementation, as a preferred embodiment of the present invention, the control method of the lower-level multi-vehicle cooperative planning and control module in step S4 is as follows: Based on the lane-changing behavior decision output by the upper-level formation decision module, a desired target state is generated for each autonomous truck; based on the current state and the desired target state, a piecewise linear interpolation method is used to generate... The reference trajectory of duration will be optimized in the time domain. Divided into Each time step is obtained. ( (reference state) ; A multi-vehicle cooperative control optimization model is constructed, namely a finite-time domain optimization model with the objective of optimizing the overall control efficiency of the autonomous truck platooning system. This model is solved within each control cycle through objective function and constraint design, providing a solution for each autonomous truck. Output the future Motion control optimization action sequence at each time step ,in For autonomous trucks At time step The control actions include longitudinal acceleration and front wheel steering angle. ; Taking into account the four objectives of trajectory tracking accuracy, control smoothness, formation coordination, and driving safety, the objective function of the multi-vehicle cooperative control optimization model is designed as follows:
[0054] In the above formula, Self-driving trucks With autonomous trucks The expected relative position vectors between them Environmental vehicles At time step The state vector at that location; The constraints for designing the multi-vehicle cooperative control optimization model include dynamic constraints of the autonomous truck platooning system, vehicle state and control variable constraints, and driving safety constraints. The multi-vehicle cooperative control optimization model is optimized and solved using a distributed model prediction framework, and the optimized control action sequence is obtained. The data is transmitted to the corresponding autonomous driving truck execution system to complete the coordinated motion control of the formation.
[0055] In a specific implementation, as a preferred embodiment of the present invention, the constraints of the multi-vehicle cooperative control optimization model are as follows: The dynamic constraints of the autonomous truck platooning system aim to ensure that the optimized trajectory conforms to the physical motion laws of the autonomous trucks, as shown in the following formula:
[0056] In the above formula, and All of these are state-space equations describing the dynamic characteristics of autonomous trucks; The vehicle state and control variable constraints are used to limit the feasible range of state and control variables for the autonomous truck, as shown in the following formula:
[0057] In the above formula, This represents the set of feasible states for autonomous trucks. , These are the physical limits and comfort limits of the control rate of change for the actuators of the autonomous truck, respectively. The driving safety constraints are the hard conditions designed to ensure driving safety, including internal safety constraints within the platooning system and safety constraints with the external environment, as shown in the following formula:
[0058] In the above formula, For any two vehicles within an autonomous truck platooning system and The minimum safe distance to be maintained throughout the entire prediction time domain; For autonomous trucks Vehicles in any surrounding environment The minimum safe distance to be maintained between them.
[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for decision-making and control of automated truck platooning in mixed-traffic scenarios on highways, characterized in that, include: S1. Define the autonomous truck platooning system as... It consists of 1 autonomous truck and 1 self-driving truck. Each autonomous truck will be accompanied by a follow-up truck; and each autonomous truck will be assigned a unique, fixed number. Among them, the lead truck is The other accompanying trucks are numbered sequentially as follows: ; S2, autonomous trucks within a platoon By utilizing its own sensors to collect real-time status information of itself, other autonomous trucks in the platoon, vehicles in the surrounding environment, and lane markings, it constructs local status information. ; S3, All local state information detected by the autonomous trucks Integrate into a system state information vector The data is then input into the upper-level formation decision module, which outputs the formation decision to the autonomous truck platooning system. ; S4. Formation Decision Under the guidance of the underlying multi-vehicle collaborative planning and control module, the autonomous truck platooning system outputs a set of optimized action sequences for platooning formation and cooperative motion control. , ,in For autonomous trucks The corresponding formation forms a motion control optimized action sequence; S5, Utilizing Action Sequences The first action in the process is for autonomous trucks Perform control execution Time step; S6. Based on the updated autonomous truck platooning system status and the mixed traffic environment status on the highway, determine whether the autonomous trucks are in the same lane and whether the relative longitudinal position error, relative speed error and relative heading angle error of all following trucks and the lead truck are within the preset tolerance range; if so, determine that the autonomous trucks have formed a platoon; otherwise, return to step S2 to continue executing the platooning decision control.
2. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios on highways according to claim 1, characterized in that, In step S2: The autonomous truck Its own status information includes its own lateral position coordinates, longitudinal position coordinates, longitudinal driving speed, and heading angle; The status information of other autonomous trucks in the platoon refers to the status of other autonomous trucks in the platoon relative to the autonomous trucks. The status information, namely the relative status information, includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The status information of the external surrounding vehicles refers to the detected environmental vehicles relative to the autonomous truck. The status information, namely the relative status information, also includes the relative lateral position coordinates, the relative longitudinal position coordinates, the relative longitudinal driving speed, and the relative heading angle; The lane information includes autonomous trucks. The perpendicular distance from the road boundaries on both sides, and the perpendicular distance from the lane lines on both sides of the lane in which it is located.
3. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios on highways according to claim 1, characterized in that, In step S3: The upper-level formation decision module is composed of It consists of multiple agent neural networks, and a single agent neural network. For an autonomous truck Output formation decision Therefore, the autonomous truck platooning system outputs platooning formation decisions. Among them, autonomous trucks Output formation decision Lane-changing decisions made in order to complete platooning tasks when driving in a multi-lane highway environment include left lane change, lane keeping, and right lane change.
4. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios according to claim 3, characterized in that, In step S3: The upper-layer formation decision module uses a multi-agent reinforcement learning method to train the agent neural network. The training process is as follows: An autonomous driving simulation environment with three lanes of mixed traffic was set up using autonomous driving simulation software. By randomly distributing autonomous trucks and manually driven vehicles, dynamic heterogeneous traffic flow was simulated. A multi-agent reinforcement learning-based platooning decision reward function is defined to incentivize the safe, efficient, and stable formation of autonomous truck platoons. The reward function employs a hybrid architecture combining global system rewards and individual rewards for each autonomous truck agent in the platoon. Calculate the total reward value separately The calculation formula is as follows: In the above formula, This indicates the global formation reward function. This represents the global traffic flow disturbance penalty function. This represents the reward function for individual formation contributions. Represents the individual safety reward function. Represents the individual efficiency reward function; The multi-agent dominant actor-critic algorithm is used to distribute the training of the multi-agent neural network parameters of the upper-level formation formation decision module, so that the trained upper-level formation formation decision module outputs formation decisions for the autonomous truck formation system.
5. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios according to claim 4, characterized in that, The global system reward component is shared by all autonomous truck agents within the platoon and is used to guide the cooperative movement behavior among agents at the system level. It includes global platoon formation rewards and global traffic flow disturbance penalties, and the specific design is as follows: The formula for calculating the global formation reward is as follows: The calculation of the global formation reward is triggered when all autonomous trucks are in the same lane. It is calculated based on the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, aiming to encourage rapid and stable formation of the overall formation. In the above formula, The same lane trigger condition is defined if and only if all autonomous trucks are in the same lane. ;otherwise, ; The norm of the error vector. This represents the error vector consisting of the relative longitudinal position error, relative speed error, and relative heading angle error between the following truck and the lead truck, where the relative longitudinal position error is the difference between the current longitudinal distance and the expected following distance; The global traffic flow disturbance penalty aims to incentivize the minimization of disturbances to mixed traffic flow during platooning, primarily through changes in acceleration of vehicles in the surrounding environment caused by platooning behavior. The calculation is as follows: In the above formula, This indicates the change in acceleration of vehicles in the surrounding environment caused by the formation.
6. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios according to claim 4, characterized in that, The individual reward applies to each autonomous truck agent in the platoon. Separate calculations are performed to guide the intelligent agents of each autonomous truck. The specific driving behaviors during the formation process include three components: individual formation contribution reward, individual safety reward, and individual efficiency reward. The design of each component is as follows: The formula for calculating the individual formation contribution reward is as follows: The individual formation contribution reward is based on the real-time global position and status of all candidate autonomous trucks, dynamically presets a virtual formation with the lead truck as the benchmark, and then rewards each autonomous truck accordingly. Dynamically allocate virtual formation anchor point coordinates This aims to encourage vehicles to move towards the coordinates of the virtual formation anchor point, actively shortening the distance between them; in the above formula, , For autonomous trucks At the present moment The global position; The formula for calculating the individual safety reward is as follows: The individual safety reward not only penalizes vehicle collisions but also incentivizes vehicles to maintain a safe state by measuring driving risks. In the above formula, This represents the collision indication function, when an autonomous truck... When a collision occurs, ,otherwise ; Indicates autonomous trucks The first and the surrounding area The distance between the front and rear of the vehicle; The safety time interval constant represents the minimum safe time interval that the workshop is expected to maintain. The formula for calculating the individual efficiency reward is as follows: The individual efficiency reward function is designed to incentivize autonomous trucks. j While completing formation tasks, it can maintain efficient driving. In the above formula, This represents the expected driving speed of the autonomous truck.
7. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios on highways according to claim 4, characterized in that, The method of employing a multi-agent dominant actor-critic algorithm to perform distributed training on the multi-agent neural network parameters of the upper-level formation decision module includes: Initialize the actor network for all autonomous truck agents. parameter and the network of critics parameter and experience replay pool And set hyperparameters, including the network learning rate. and Discount Factor Maximum number of training rounds and maximum time step ; For each training round Perform the following steps: step a 1. Reset the three-lane highway mixed traffic simulation environment and randomly initialize the position and status of the autonomous truck platooning system and the vehicles in the environment; step b When the decision-making time step If the platooning is not completed and no collision occurs, the following steps will be executed repeatedly: step b 1. Intelligent agents for each autonomous truck Based on their respective detected current states Utilizing actor networks Output and sample the formation to form decision actions ; step b 2. Form a set of decision actions in the formation of all intelligent agents. Under guidance, the lower-level multi-vehicle collaborative planning and control module inputs optimized control action sequences into the autonomous truck platooning system, updates the state of the platooning system, and simultaneously guides other vehicles in the environment to update their states, enabling each autonomous truck intelligent agent to achieve this. Obtain new local state information for the next step. ; step b 3. Each autonomous truck intelligent agent Based on local state information The formation is designed to form a decision reward function, and the obtained reward is calculated. and the obtained experience tuples Stored in the experience replay pool middle; step b 4. When the experience replay pool After the number of empirical samples in the dataset meets the training conditions, each agent... From the experience replay pool Randomly sample a small batch of empirical samples and train the network according to the following formula: The critic network parameters are minimized by the loss function. Update: Actor network parameters are maximized by the objective function Update: Every The training effect of the formation decision neural network is evaluated in several training rounds. P If the average reward value of each evaluation round tends to stabilize, then the convergence condition is met and training stops. After training is completed, the trained multi-agent neural network parameters are saved and deployed to the upper-level formation decision module of the actual autonomous truck for decision guidance in real vehicle formation.
8. The truck platooning decision-making and control method for autonomous driving in mixed-traffic scenarios according to claim 1, characterized in that, In step S4, the control method of the lower-level multi-vehicle cooperative planning and control module is as follows: Based on the lane-changing behavior decision output by the upper-level formation decision module, a desired target state is generated for each autonomous truck; based on the current state and the desired target state, a piecewise linear interpolation method is used to generate... The reference trajectory of duration will be optimized in the time domain. Divided into Each time step is obtained. ( (reference state) ; A multi-vehicle cooperative control optimization model is constructed, namely a finite-time domain optimization model with the objective of optimizing the overall control efficiency of the autonomous truck platooning system. This model is solved within each control cycle through objective function and constraint design, providing a solution for each autonomous truck. Output the future Motion control optimization action sequence at each time step ,in For autonomous trucks At time step The control actions include longitudinal acceleration and front wheel steering angle. ; Taking into account the four objectives of trajectory tracking accuracy, control smoothness, formation coordination, and driving safety, the objective function of the multi-vehicle cooperative control optimization model is designed as follows: In the above formula, Self-driving trucks With autonomous trucks The expected relative position vectors between them Environmental vehicles At time step The state vector at that location; The constraints for designing the multi-vehicle cooperative control optimization model include dynamic constraints of the autonomous truck platooning system, vehicle state and control variable constraints, and driving safety constraints. The multi-vehicle cooperative control optimization model is optimized and solved using a distributed model prediction framework, and the optimized control action sequence is obtained. The data is transmitted to the corresponding autonomous driving truck execution system to complete the coordinated motion control of the formation.
9. A method for decision-making and control of truck platooning in a mixed-traffic highway scenario according to claim 8, characterized in that, The constraints of the multi-vehicle cooperative control optimization model are as follows: The dynamic constraints of the autonomous truck platooning system aim to ensure that the optimized trajectory conforms to the physical motion laws of the autonomous trucks, as shown in the following formula: In the above formula, and All of these are state-space equations describing the dynamic characteristics of autonomous trucks; The vehicle state and control variable constraints are used to limit the feasible range of state and control variables for the autonomous truck, as shown in the following formula: In the above formula, This represents the set of feasible states for autonomous trucks. , These are the physical limits and comfort limits of the control rate of change for the actuators of the autonomous truck, respectively. The driving safety constraints are the hard conditions designed to ensure driving safety, including internal safety constraints within the platooning system and safety constraints with the external environment, as shown in the following formula: In the above formula, For any two vehicles within an autonomous truck platooning system and The minimum safe distance to be maintained throughout the entire prediction time domain; For autonomous trucks Vehicles in any surrounding environment The minimum safe distance to be maintained between them.