Multi-agent reinforcement learning based multi-vehicle cooperative lane-changing method
By employing a collaborative lane-changing method based on multi-agent reinforcement learning and fifth-order polynomial programming, the initiative and real-time issues of multi-vehicle collaborative lane changing in complex traffic environments are resolved, achieving a safe and efficient lane-changing process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-04-14
- Publication Date
- 2026-05-29
AI Technical Summary
Existing multi-vehicle cooperative lane-changing methods are difficult to achieve proactive coordination in complex traffic environments, and lack decoupling and real-time performance, resulting in low lane-changing efficiency and easy traffic disturbance.
A multi-agent reinforcement learning approach is adopted, which realizes the cooperative control between vehicles through a distributed decision-making mechanism and five-order polynomial programming, outputs the safe lane-changing distance and generates the desired lane-changing trajectory, and executes the lane-changing process in combination with the trajectory tracking controller.
It improves the feasibility and safety of lane changing in complex traffic environments, reduces the computational coupling of the system, meets the real-time requirements of autonomous driving systems, and enhances lane changing efficiency and stability.
Smart Images
Figure CN122116686A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent connected vehicles and autonomous driving technology, and relates to a multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning. Background Technology
[0002] With the rapid development of intelligent connected vehicles and autonomous driving technologies, vehicle cooperative decision-making in complex traffic environments has become an important research direction for intelligent transportation systems. Lane changing, as a typical high-frequency driving behavior in urban road traffic, directly affects the stability of the overall traffic flow in terms of safety and efficiency. Traditional single-vehicle autonomous lane-changing strategies are mostly based on independent decision-making using local environmental perception, making it difficult to accurately predict the interactive intentions of surrounding vehicles. Especially in high-density urban traffic scenarios, single-vehicle logic often faces problems such as difficulty in obtaining safe gaps, low lane-changing success rates, and a high risk of inducing traffic disturbances because it cannot achieve spatiotemporal coordination with vehicles in front and behind in the target lane. Therefore, achieving multi-vehicle cooperative lane changing through information interaction and collaborative control between vehicles has become an inevitable requirement for addressing complex traffic interaction problems.
[0003] However, existing multi-vehicle cooperative lane-changing methods still have many technical limitations in practical applications, making it difficult to fully meet the needs of cooperative decision-making in complex urban road sections. First, most existing lane-changing strategies lack an active coordination mechanism, with surrounding vehicles often only passively responding to the lane-changing behavior of the target vehicle. When the initial safety clearance of the target lane is insufficient, the system can usually only passively wait for the traffic flow to naturally evolve to create a usable clearance, making it difficult to drive relevant vehicles to actively adjust their speeds to build lane-changing space that meets safety constraints in advance, resulting in low overall lane-changing efficiency. Second, some existing cooperative methods tightly couple high-level lane-changing decision-making logic with the underlying trajectory planning process, resulting in a lack of hierarchy and decoupling in the system architecture. Once faced with traffic scene changes or changes in control strategy requirements, the entire algorithm often needs to be reconstructed or significantly adjusted, severely restricting the flexibility and cross-scenario scalability of the method. Furthermore, some collaborative lane-changing methods based on centralized optimization require the construction and unified solution of a multi-vehicle joint control optimization problem. As the number of vehicles participating in the collaborative game increases, the computational complexity increases exponentially, which can easily lead to the curse of dimensionality and makes it difficult to meet the technical requirements for high real-time decision-making in urban dynamic traffic environments. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning, which aims to improve the execution efficiency and safety of multi-vehicle cooperative lane-changing in complex interactive scenarios, and to establish a hierarchical decoupled distributed decision-making mechanism to overcome the shortcomings of existing cooperative lane-changing methods in terms of initiative, decoupling and real-time performance.
[0005] To achieve the above objectives, the present invention provides the following technical solution: A multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning includes the following steps: Acquire real-time status information of traffic scenarios and construct a distributed multi-agent model, defining the state space and action space of the multi-agent model; Design a reward function for collaborative control objectives, and perform offline training on the multi-agent reinforcement learning model based on the state space and action space to obtain the trained multi-agent reinforcement learning model. The real-time perceived traffic state features are input into the multi-agent reinforcement learning model for online reasoning, and the cooperative control actions of the vehicles are output to form the safe lane-changing distance of the target vehicle. Combining the target vehicle's safe lane-changing distance with the vehicle's motion boundary constraints, a fifth-order polynomial is used to program the target vehicle's desired lane-changing trajectory. Based on the desired lane-changing trajectory, vehicle control commands are generated, and the vehicle is driven to execute and complete the coordinated lane-changing process through the trajectory tracking controller.
[0006] Furthermore, the defined state space of the multi-agent system... With action space Specifically, it includes: The state space of a multi-agent system includes the observation space of the agents, specifically represented as:
[0007]
[0008] in, Represents intelligent agents The observation space Indicates surrounding intelligent vehicles In intelligent agents Relative position in a coordinate system; This refers to each intelligent vehicle. The absolute velocity; Represents intelligent agents A collection of intelligent vehicles in the surrounding area. This indicates the obstacle relative to the agent. The relative position and velocity; Multi-agent action space Defined as:
[0009] in, m / s 2 The intelligent vehicle can only accelerate and decelerate longitudinally. When the safe lane-changing distance is not met, the intelligent vehicle is restricted to longitudinal movement only.
[0010] Furthermore, the design of the reward function oriented towards the cooperative control objective and its offline training specifically includes: The reward function in multi-agent reinforcement learning is defined as individual reward. With global rewards Weighted sum:
[0011] in, and These represent individual rewards. With global rewards Weighting coefficients; Individual rewards Collision Rewards Spacing rewards Speed Reward Comfort rewards and speed constraint rewards Together they constitute:
[0012] Collision Rewards Used to prevent vehicle collisions, it is defined as follows:
[0013] Spacing reward The method used to encourage vehicles to achieve the desired lane-changing distance is calculated as follows:
[0014] in, Indicates spacing reward The weighting coefficients, This indicates the actual distance between the vehicle and the vehicle in front or behind. The desired lane change spacing is expressed by the following formula:
[0015] in, Indicates the vehicle's desired speed. Indicates the time distance between the front and rear of the vehicle. Indicates the length of the vehicle. Indicates the minimum safe distance; Speed Reward It is used to guide vehicles to maintain the desired speed after reaching the desired lane-changing distance, and is defined as follows:
[0016] in, Indicates speed reward The weighting coefficients, Indicates vehicle speed; Comfort reward To ensure driving comfort, the calculation is as follows:
[0017] in, Indicates comfort reward The weighting coefficients, The rate of change of vehicle acceleration is expressed as follows:
[0018] in, and These represent the vehicle's current acceleration and the acceleration at the previous moment, respectively. Indicates a time interval; Speed Constraint Reward This is used to constrain the vehicle's speed to within a preset speed threshold range, and its definition is as follows:
[0019] in, Indicates speed constraint reward The weighting coefficients, and These represent the preset lower speed limit and upper speed limit, respectively. Global Rewards Balanced rewards based on global spacing and global speed bonus Together they constitute:
[0020] Global Spacing Balance Reward The spacing used to guide vehicles to form a balanced distribution is calculated as follows:
[0021] in, Indicates global spacing balance reward The weighting coefficients, and These represent the distances between the target vehicle and the vehicles in front and behind, respectively. Global speed bonus The method used to guide vehicles to achieve speed synchronization is as follows:
[0022] in, Indicates global speed reward The weighting coefficients, This indicates the deviation between the vehicle's average speed and the desired speed. The variance representing vehicle speed; Based on the state space, action space, and reward function of the multi-agent reinforcement learning, an experience pool is constructed, and the multi-agent reinforcement learning model is trained using a framework of centralized training and distributed execution. Based on the sample data in the empirical replay pool, the network loss function is evaluated as follows. Iteratively update the parameters of the evaluation network :
[0023] Based on the action value gradient information output by the evaluation network, the following policy gradient is applied. Parameters of the execution network Update the system to optimize the agent's action output strategy:
[0024] Repeat the update steps of the evaluation network and the execution network until the multi-agent reinforcement learning model meets the preset convergence condition to obtain the target lane-changing multi-agent reinforcement learning model.
[0025] Furthermore, the coordinated control actions of the output vehicle to form a safe lane-changing distance for the target vehicle specifically include: The trained multi-agent reinforcement learning model is deployed to the vehicle. Based on the vehicle's real-time observation of traffic scene state information, the model outputs action commands online. Through continuous information interaction and action execution, each vehicle drives the traffic scene to transition states until the target vehicle meets the preset safe lane-changing distance conditions. The safe lane-changing distance includes the forward safe distance between the target vehicle and the vehicle in front in the target lane, and the rearward safe distance with the vehicle behind. When both of these conditions simultaneously meet preset threshold conditions, the lane-changing condition is determined to be met.
[0026] Furthermore, the desired lane-changing trajectory of the target vehicle using fifth-order polynomial programming specifically includes: After optimizing the upper-level collaborative spacing to obtain safe lane-changing conditions, the displacement equation of the lane-changing trajectory is characterized by a fifth-order polynomial, as follows:
[0027] Taking the first and second derivatives of the displacement equation respectively, we obtain the corresponding velocity and acceleration equations:
[0028]
[0029] Based on the vehicle's displacement, velocity, and acceleration boundary conditions at the initial moment and the moment of lane change completion:
[0030]
[0031] Substituting these equations into the above equations constructs a system of linear equations, which are used to solve for the coefficient vector. : In time hour:
[0032] In time hour:
[0033] Transform the system of linear equations into matrix form:
[0034] Find the inverse of the coefficient matrix:
[0035] This yields the coefficient vector of the fifth-degree polynomial, generating the desired lane-changing trajectory.
[0036] Furthermore, the process of driving the vehicle to perform and complete the coordinated lane-changing process through the trajectory tracking controller specifically includes: The generated desired lane-changing trajectory is converted into a sequence of target heading angle and target acceleration. Model predictive control is used for underlying trajectory tracking and command execution.
[0037] The beneficial effects of this invention are as follows: (1) By introducing a multi-agent reinforcement learning collaborative mechanism, the present invention enables the target vehicle and surrounding vehicles to actively adjust their longitudinal motion state and collaboratively form a lane-changing distance that meets safety constraints. This overcomes the problem of passive lane changing that relies on natural gaps in traditional methods, thereby improving the feasibility of lane changing in complex traffic environments and reducing the disturbance to traffic flow.
[0038] (2) The present invention adopts a hierarchical control architecture that decouples decision-making and trajectory planning, separates multi-vehicle collaborative decision-making from single-vehicle trajectory generation, reduces the computational coupling of the system, improves the scalability and stability of the method, and is conducive to flexible deployment in different traffic scenarios.
[0039] (3) The present invention is based on a strategy that combines offline training and online distributed execution. While ensuring the collaborative decision-making ability of multiple vehicles, it reduces the online computational complexity, enabling vehicles to respond quickly to dynamic traffic environments, thereby meeting the real-time requirements of autonomous driving systems.
[0040] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a flowchart of a multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the two-layer control framework of the multi-vehicle cooperative lane-changing method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the initial simulation scenario of multi-agent cooperative lane changing according to an embodiment of the present invention; Figure 4 This is a convergence curve of the reward function during training of the multi-agent reinforcement learning model according to an embodiment of the present invention; Figure 5 This is a graph showing the change in vehicle spacing during multi-vehicle cooperative lane changing in an embodiment of the present invention. Figure 6 This is a graph showing the vehicle speed change during the multi-vehicle cooperative lane-changing process according to an embodiment of the present invention. Figure 7 This is a graph showing the vehicle acceleration variation during multi-vehicle cooperative lane changing in an embodiment of the present invention. Figure 8 This is a schematic diagram of the single-vehicle trajectory planning and execution process according to an embodiment of the present invention; Figure 9 This is a graph showing the vehicle speed change during the trajectory execution process according to an embodiment of the present invention; Figure 10 This is a graph showing the change in longitudinal acceleration of the vehicle during the trajectory execution process according to an embodiment of the present invention. Figure 11 This is a graph showing the change in lateral acceleration of a vehicle during trajectory execution in an embodiment of the present invention. Detailed Implementation
[0042] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0043] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0044] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0045] Please see Figures 1 to 11 This is a multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning.
[0046] Example like Figure 1 As shown in the figure, this invention discloses a multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning, characterized by the following steps: S1. Obtain real-time status information of traffic scenarios and construct a distributed multi-agent model, defining the state space and action space of the multi-agent model; S2. Design a reward function for the cooperative control objective, and perform offline training on the multi-agent reinforcement learning model based on the state space and action space to obtain the trained multi-agent reinforcement learning model. S3. Input the real-time perceived traffic state features into the multi-agent reinforcement learning model for online reasoning, and output the cooperative control actions of the vehicles to form the safe lane-changing distance of the target vehicles. S4. Combining the target vehicle's safe lane-changing distance with the vehicle's motion boundary constraints, a fifth-order polynomial is used to program the target vehicle's desired lane-changing trajectory. S5. Generate vehicle control commands based on the desired lane-changing trajectory, and drive the vehicle to execute and complete the coordinated lane-changing process through the trajectory tracking controller.
[0047] The following is a detailed explanation of each of the above steps.
[0048] S1, such as Figure 3 As shown, this embodiment constructs a multi-vehicle cooperative lane-changing traffic scenario. The red vehicle ahead in the right lane is a stationary obstacle, and the green vehicle 2 behind it is a normally moving vehicle. Due to the presence of the obstacle, vehicle 2 needs to change lanes to avoid a collision, and is therefore identified as the target lane-changing vehicle. Yellow vehicles 0 and 1 in the middle lane are adjacent vehicles. Because these two vehicles occupy key positions in the target lane, vehicle 2 cannot directly perform a lane-changing operation. To achieve a safe lane change, vehicle 2 needs to cooperate with vehicles 0 and 1, adjusting their longitudinal positional relationships through multi-vehicle collaboration to create a safe and feasible lane-changing gap for the target vehicle. Therefore, vehicle 2, along with vehicles 0 and 1, constitutes a multi-agent system.
[0049] The traffic scenario is constructed based on the SUMO simulation platform, and the simulated road is a 300-m long, one-way, three-lane urban road. In this traffic scenario, the initial spatial positions and motion states of each vehicle are as follows: Vehicle 0 and Vehicle 1 are both located in the middle lane, with Vehicle 0 at a longitudinal position of x=20 m and Vehicle 1 at a longitudinal position of x=44 m; Vehicle 2 is located in the right lane with an initial longitudinal position of x=32 m; a stationary obstacle is located x=145 m ahead of the right lane. The initial speeds of Vehicle 0, Vehicle 1, and Vehicle 2 are all set to 8 m / s, where the longitudinal position x is defined along the vehicle's direction of travel.
[0050] Other parameter settings for the vehicle are shown in Table 1.
[0051] Table 1 SUMO Parameter Settings
[0052] S2, the defined state space of the multi-agent system With action space Specifically, it includes: The state space of a multi-agent system includes the observation space of the agents, specifically represented as:
[0053]
[0054] in, Represents intelligent agents The observation space Indicates surrounding intelligent vehicles In intelligent agents Relative position in a coordinate system; This refers to each intelligent vehicle. The absolute velocity; Represents intelligent agents A collection of intelligent vehicles in the surrounding area. This indicates the obstacle relative to the agent. The relative position and velocity; Multi-agent action space Defined as:
[0055] in, m / s 2 The intelligent vehicle can only accelerate and decelerate longitudinally. When the safe lane-changing distance is not met, the intelligent vehicle is restricted to longitudinal movement only.
[0056] S3. The design of the reward function oriented towards the cooperative control objective and its offline training specifically includes: S31. The reward function in multi-agent reinforcement learning is defined as individual reward. With global rewards Weighted sum:
[0057] in, and These represent individual rewards. With global rewards Weighting coefficients; S32, Individual Rewards Collision Rewards Spacing rewards Speed Reward Comfort rewards and speed constraint rewards Together they constitute:
[0058] Collision Rewards Used to prevent vehicle collisions, it is defined as follows:
[0059] Spacing reward The method used to encourage vehicles to achieve the desired lane-changing distance is calculated as follows:
[0060] in, Indicates spacing reward The weighting coefficients, This indicates the actual distance between the vehicle and the vehicle in front or behind. The desired lane change spacing is expressed by the following formula:
[0061] in, Indicates the vehicle's desired speed. Indicates the time distance between the front and rear of the vehicle. Indicates the length of the vehicle. Indicates the minimum safe distance; To verify the effectiveness of the method, this embodiment specifically sets the various physical parameters of the desired lane-changing distance. The desired vehicle speed is set based on the characteristics of typical urban road traffic flow. The speed is 8 m / s, and the headway is... The time is 1.5 seconds, and the vehicle length is... Set at 5 m, minimum safety distance Let's assume it's 2 m. According to the formula... The calculated lane-changing distance in this embodiment is 19m. Speed Reward It is used to guide vehicles to maintain the desired speed after reaching the desired lane-changing distance, and is defined as follows:
[0062] in, Indicates speed reward The weighting coefficients, Indicates vehicle speed; Comfort reward To ensure driving comfort, the calculation is as follows:
[0063] in, Indicates comfort reward The weighting coefficients, The rate of change of vehicle acceleration is expressed as follows:
[0064] in, and These represent the vehicle's current acceleration and the acceleration at the previous moment, respectively. Indicates a time interval; Speed Constraint Reward This is used to constrain the vehicle's speed to within a preset speed threshold range, and its definition is as follows:
[0065] in, Indicates speed constraint reward The weighting coefficients, and These represent the preset lower speed limit and upper speed limit, which are set to 6 m / s and 10 m / s respectively in this embodiment; S33, Global Rewards Balanced rewards based on global spacing and global speed bonus Together they constitute:
[0066] Global Spacing Balance Reward The spacing used to guide vehicles to form a balanced distribution is calculated as follows:
[0067] in, Indicates global spacing balance reward The weighting coefficients, and These represent the distances between the target vehicle and the vehicles in front and behind, respectively. Global speed bonus The method used to guide vehicles to achieve speed synchronization is as follows:
[0068] in, Indicates global speed reward The weighting coefficients, This indicates the deviation between the vehicle's average speed and the desired speed. The variance representing vehicle speed; The coefficient settings for the reward items are shown in Table 2.
[0069] Table 2 Reward Item Coefficient Settings
[0070] S34. Based on the state space, action space, and reward function of the multi-agent reinforcement learning, an experience pool is constructed. In this embodiment, the multi-agent reinforcement learning model specifically adopts the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm and uses a centralized training and distributed execution framework to train the multi-agent reinforcement learning model. Based on the sample data in the empirical replay pool, the network loss function is evaluated as follows. Iteratively update the parameters of the evaluation network :
[0071] Based on the action value gradient information output by the evaluation network, the following policy gradient is applied. Parameters of the execution network Update the system to optimize the agent's action output strategy:
[0072] Repeat the update steps of the evaluation network and the execution network until the MADDPG model meets the preset convergence condition to obtain the target lane-changing MADDPG model.
[0073] The parameter coefficients of MADDPG are set as shown in Table 3.
[0074] Table 3 MADDPG parameter settings
[0075] S4. The coordinated control actions of the output vehicle to form a safe lane-changing distance for the target vehicle specifically include: The trained MADDPG model is deployed to the vehicle. Based on the vehicle's real-time observation of traffic scene state information, the model outputs action commands online. Through continuous information interaction and action execution, each vehicle drives the traffic scene to transition states until the target vehicle meets the preset safe lane-changing distance conditions. Specifically, the safe lane-changing distance includes the forward safe distance between the target vehicle and the vehicle in front in the target lane, and the rearward safe distance with the vehicle behind. When both of these conditions simultaneously meet the preset threshold conditions, the system determines that the lane-changing condition is met and triggers the underlying trajectory planning described below.
[0076] S5. The desired lane-changing trajectory of the target vehicle using fifth-order polynomial programming specifically includes: After optimizing the upper-level collaborative spacing to obtain safe lane-changing conditions, the displacement equation of the lane-changing trajectory is characterized by a fifth-order polynomial, as follows:
[0077] Taking the first and second derivatives of the displacement equation respectively, we obtain the corresponding velocity and acceleration equations:
[0078]
[0079] Based on the vehicle's displacement, velocity, and acceleration boundary conditions at the initial moment and the moment of lane change completion:
[0080]
[0081] Substituting these equations into the above equations constructs a system of linear equations, which are used to solve for the coefficient vector. : In time hour:
[0082] In time hour:
[0083] In this embodiment, the lane change completion time is set to 3.5 seconds, the longitudinal displacement span of the trajectory planning is 30 meters, and the initial and post-lane change longitudinal expected vehicle speeds are 8 m / s. Based on these parameters, the boundary condition equations of the fifth-degree polynomial are substituted into: At the start of a lane change Displacement ,speed acceleration ; Lane change termination time Displacement ,speed acceleration .
[0084] Transform the system of linear equations into matrix form:
[0085] Find the inverse of the coefficient matrix:
[0086] This yields the coefficient vector of the fifth-degree polynomial, generating the desired lane-changing trajectory.
[0087] S6. The process of driving the vehicle to perform and complete the coordinated lane-changing process through the trajectory tracking controller specifically includes: The generated desired lane-changing trajectory is converted into a sequence of target heading angle and target acceleration. Model predictive control is used for underlying trajectory tracking and command execution.
[0088] In this embodiment, the lane-changing trajectory is executed in CARLA with a simulation step size of 0.1s. The specific lane-changing scenario is based on the preset safe lane-changing distance condition reached by the target vehicle in step S4. The model predictive control continuously outputs control commands to drive the target vehicle to accurately track the desired lane-changing trajectory, thereby smoothly completing the multi-vehicle cooperative lane-changing process.
[0089] To further verify the effectiveness and feasibility of the multi-agent cooperative lane-changing method based on reinforcement learning proposed in this invention under complex interactive scenarios, this embodiment conducts a system simulation experiment based on the hierarchical control architecture and preset parameters constructed above. The following will, in conjunction with experimental test data and specific figures, elaborate on the actual technical effects of this invention from the aspects of the training process of the upper-layer multi-agent decision-making model and the dynamic evolution of the microscopic motion state during the execution phase of multi-agent cooperative lane-changing: Figure 4 The convergence curve of the reward function during the training process of the MADDPG model is shown. As can be seen from the figure, with the increase of training rounds, the cumulative reward value of each agent shows a steady upward trend, and begins to stabilize and converge after approximately 900 training rounds. This indicates that the MADDPG algorithm can effectively learn multi-vehicle cooperative lane-changing strategies, and the model possesses good convergence and learning efficiency.
[0090] Figures 5 to 7 Together, they reflect the dynamic evolution of the vehicle's longitudinal motion state during the upper-level multi-agent collaborative decision-making process. Specifically, Figure 5 The diagram shows the vehicle spacing variation curves during the multi-vehicle cooperative lane-changing process. It can be observed that, under the control of the cooperative strategy, the vehicles in the target lane actively interact and successfully reach the predetermined safe lane-changing distance (i.e., 19 meters) with the vehicles in front and behind at approximately the 7th second.
[0091] Figure 6 The corresponding vehicle speed change curves are shown. To assist the target vehicle in establishing a safe lane-changing gap, intelligent agent vehicle 0 behind the target lane actively adopts a deceleration strategy within the first 2 seconds, while intelligent agent vehicle 1 in front simultaneously accelerates, working together to quickly increase the spatial gap. Subsequently, as the vehicle spacing gradually approaches the preset safe lane-changing gap, the speed control strategies of vehicle 0 and vehicle 1 smoothly transition, actively approaching and converging towards the predetermined desired speed of 8 m / s, thereby stabilizing the current vehicle spacing and providing a stable traffic flow environment for the target vehicle to perform a lane change.
[0092] Figure 7 The diagram shows the vehicle acceleration variation curves during multi-vehicle cooperative lane changing. Analysis reveals that in the initial stage of cooperation (within the first 3 seconds), due to the significant difference between the initial spacing and the safe lane-changing spacing, each intelligent vehicle outputs large acceleration and deceleration commands to quickly respond to the cooperation needs. However, once the spacing approaches the target value, the acceleration and deceleration amplitudes of each vehicle significantly slow down. This result fully validates the effectiveness of the "comfort reward" design in the aforementioned reward function, which effectively suppresses frequent fluctuations and instantaneous jumps in acceleration while achieving the cooperation goal, ensuring the smoothness of the lane-changing process and thus obtaining a more comfortable driving experience.
[0093] Figures 8 to 11 The verification results of the underlying single-vehicle lane-changing trajectory planning and tracking execution are presented. The relevant experiments were conducted on the CARLA co-simulation platform. Figure 8 A physical space diagram illustrating the planning and execution process of a single vehicle's lane-changing trajectory visually demonstrates the smooth lane-changing path generated based on a fifth-order polynomial.
[0094] Figure 9 The target vehicle's speed change curve during the underlying trajectory tracking process is shown. Simulation data indicates that the target vehicle's speed can be maintained very stably near the desired speed of 8 m / s throughout the entire lane-changing maneuver, demonstrating the excellent steady-state tracking performance of the underlying control system.
[0095] Figure 10 and Figure 11 The curves showing the changes in the vehicle's longitudinal and lateral acceleration during trajectory execution are displayed. Figure 10 It can be seen that during lane changing, the longitudinal acceleration of vehicles remains at a low level and changes relatively smoothly, causing minimal disturbance to the longitudinal movement of traffic. From Figure 11 As can be seen, since lane changing inevitably involves lateral movement, the vehicle's lateral acceleration changes to some extent. However, this change curve is continuous and gentle, with the overall peak value strictly controlled within the range of ±1.5 m / s². This parameter range fully complies with vehicle dynamics boundary constraints and human comfort standards, further demonstrating the smoothness and safety of the proposed method in the actual vehicle control execution phase.
[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning, characterized in that: Includes the following steps: Acquire real-time status information of traffic scenarios and construct a distributed multi-agent model, defining the state space and action space of the multi-agent model; Design a reward function for collaborative control objectives, and perform offline training on the multi-agent reinforcement learning model based on the state space and action space to obtain the trained multi-agent reinforcement learning model. The real-time perceived traffic state features are input into the multi-agent reinforcement learning model for online reasoning, and the cooperative control actions of the vehicles are output to form the safe lane-changing distance of the target vehicle. Combining the target vehicle's safe lane-changing distance with the vehicle's motion boundary constraints, a fifth-order polynomial is used to program the target vehicle's desired lane-changing trajectory. Based on the desired lane-changing trajectory, vehicle control commands are generated, and the vehicle is driven to execute and complete the coordinated lane-changing process through the trajectory tracking controller.
2. The multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to claim 1, characterized in that: The defined state space of the multi-agent system With action space Specifically, it includes: The state space of a multi-agent system includes the observation space of the agents, specifically represented as: in, Represents intelligent agents The observation space Indicates surrounding intelligent vehicles In intelligent agents Relative position in a coordinate system; Represents each intelligent vehicle The absolute velocity; Represents intelligent agents The surrounding collection of intelligent vehicle agents; Indicates the obstacle relative to the agent. The relative position and velocity; Multi-agent action space Defined as: in, m / s 2 Intelligent vehicles can only accelerate and decelerate longitudinally. When the safe lane-changing distance is not met, intelligent vehicles are restricted to longitudinal movement only.
3. The multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to claim 1, characterized in that: The design of the reward function oriented towards the cooperative control objective and its offline training specifically includes: The reward function in multi-agent reinforcement learning is defined as individual reward. With global rewards Weighted sum: in, and These represent individual rewards. With global rewards Weighting coefficients; Individual rewards Collision Rewards Spacing rewards Speed Reward Comfort rewards and speed constraint rewards Together they constitute: Collision Rewards Used to prevent vehicle collisions, it is defined as follows: Spacing reward The method used to encourage vehicles to achieve the desired lane-changing distance is calculated as follows: in, Indicates spacing reward The weighting coefficients, This indicates the actual distance between the vehicle and the vehicle in front or behind. The desired lane change spacing is expressed by the following formula: in, Indicates the vehicle's desired speed. Indicates the time distance between the front and rear of the vehicle. Indicates the length of the vehicle. Indicates the minimum safe distance; Speed Reward It is used to guide vehicles to maintain the desired speed after reaching the desired lane-changing distance, and is defined as follows: in, Indicates speed reward The weighting coefficients, Indicates vehicle speed; Comfort reward To ensure driving comfort, the calculation is as follows: in, Indicates comfort reward The weighting coefficients, The rate of change of vehicle acceleration is expressed as follows: in, and These represent the vehicle's current acceleration and the acceleration at the previous moment, respectively. Indicates a time interval; Speed Constraint Reward This is used to constrain the vehicle's speed to within a preset speed threshold range, and its definition is as follows: in, Indicates speed constraint reward The weighting coefficients, and These represent the preset lower speed limit and upper speed limit, respectively. Global Rewards Balanced rewards based on global spacing and global speed bonus Together they constitute: Global Spacing Balance Reward The spacing used to guide vehicles to form a balanced distribution is calculated as follows: in, Indicates global spacing balance reward The weighting coefficients, and These represent the distances between the target vehicle and the vehicles in front and behind, respectively. Global speed bonus The method used to guide vehicles to achieve speed synchronization is as follows: in, Indicates global speed reward The weighting coefficients, This indicates the deviation between the vehicle's average speed and the desired speed. The variance representing vehicle speed; Based on the state space, action space, and reward function of the multi-agent reinforcement learning, an experience pool is constructed, and the multi-agent reinforcement learning model is trained using a framework of centralized training and distributed execution. Based on the sample data in the empirical replay pool, the network loss function is evaluated as follows. Iteratively update the parameters of the evaluation network : Based on the action value gradient information output by the evaluation network, the following policy gradient is applied. Parameters of the execution network Update the system to optimize the agent's action output strategy: Repeat the update steps of the evaluation network and the execution network until the multi-agent reinforcement learning model meets the preset convergence condition to obtain the target lane-changing multi-agent reinforcement learning model.
4. The multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to claim 1, characterized in that: The coordinated control actions of the output vehicle to form a safe lane-changing distance for the target vehicle specifically include: The trained multi-agent reinforcement learning model is deployed to the vehicle. Based on the vehicle's real-time observation of traffic scene state information, the model outputs action commands online. Through continuous information interaction and action execution, each vehicle drives the traffic scene to undergo state transition until the target vehicle meets the preset safe lane-changing distance condition. The safe lane-changing distance includes the forward safe distance between the target vehicle and the vehicle in front in the target lane, and the rearward safe distance between the target vehicle and the vehicle behind. When both of them meet the preset threshold conditions, it is determined that the lane-changing condition is met.
5. The multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to claim 1, characterized in that: The desired lane-changing trajectory of the target vehicle, obtained using fifth-order polynomial programming, specifically includes: After optimizing the upper-level collaborative spacing to obtain safe lane-changing conditions, the displacement equation of the lane-changing trajectory is characterized by a fifth-order polynomial, as follows: Taking the first and second derivatives of the displacement equation respectively, we obtain the corresponding velocity and acceleration equations: Based on the vehicle's displacement, velocity, and acceleration boundary conditions at the initial moment and the moment of lane change completion: Substituting these equations into the above equations constructs a system of linear equations, which are used to solve for the coefficient vector. : In time hour: In time hour: Transform the system of linear equations into matrix form: Find the inverse of the coefficient matrix: This yields the coefficient vector of the fifth-degree polynomial, generating the desired lane-changing trajectory.
6. The multi-vehicle cooperative lane-changing method based on multi-agent reinforcement learning according to claim 1, characterized in that: The process of driving the vehicle to perform and complete the coordinated lane change through the trajectory tracking controller specifically includes: The generated desired lane-changing trajectory is converted into a sequence of target heading angle and target acceleration. Model predictive control is used for underlying trajectory tracking and command execution.