Multi-vehicle cooperative lane-changing method and system based on reinforcement learning in mixed traffic environment

Abstract

The application relates to the technical field of intelligent driving, in particular to a multi-vehicle cooperative lane-changing method and system based on reinforcement learning in a mixed traffic environment, vehicle information and environment information are acquired in real time; according to the vehicle information and the environment information, a double-layer decision model is constructed to plan a driving track and a driving speed of the vehicle; a high layer of the decision model determines an optimal vehicle behavior according to the vehicle information and the environment information, and a low layer plans the driving track and the speed according to the optimal vehicle behavior and the vehicle information; in the planned driving track and speed, a collision ellipse model is constructed to detect a conflict between an ICV (Intelligent Connected Vehicle) and an HDV (Heavy Duty Vehicle) according to the track and the speed, and the driving speed is adjusted according to the conflict. The scheme can improve the multi-vehicle cooperative scheduling efficiency and the adaptability to a dynamic traffic environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a multi-vehicle cooperative lane-changing method and system based on reinforcement learning in a mixed traffic environment. Background Technology

[0002] With the widespread application of intelligent connected vehicles (ICVs) and manually driven vehicles (HDVs) in mixed traffic environments, traditional traffic management and vehicle scheduling methods face numerous challenges, especially in complex traffic scenarios such as multi-lane highways. Existing technologies mainly rely on static traffic signal control and fixed scheduling mechanisms, which are often unable to effectively address the collaborative needs between autonomous and manually driven vehicles when facing mixed traffic flows.

[0003] Traditional vehicle dispatching methods often focus on the dispatching and management of single vehicles or single lanes, lacking sufficient consideration for multi-vehicle collaborative behavior. This results in the underutilization of the communication and collaboration capabilities of autonomous vehicles, leading to significant problems in vehicle collaboration efficiency and right-of-way allocation in scenarios such as highway ramp merging. Furthermore, traditional dispatching strategies cannot dynamically adjust based on real-time traffic flow, road conditions, and environmental changes, impacting traffic efficiency and safety.

[0004] To address the problems of traditional vehicle scheduling methods, researchers have proposed a reinforcement learning-based approach for vehicle behavior decision-making and trajectory planning. This approach optimizes vehicle lane-changing decisions and behavior control through reinforcement learning algorithms. However, existing reinforcement learning methods primarily focus on the decision-making level of single-vehicle behavior and lack comprehensive solutions for multi-vehicle cooperative lane-changing scenarios.

[0005] The existing methods still have the following shortcomings in practical applications:

[0006] Low efficiency of multi-vehicle collaborative scheduling: Existing methods fail to fully consider the collaborative lane changing and scheduling issues between multiple autonomous vehicles and human-driven vehicles, resulting in low scheduling efficiency in multi-vehicle mixed flow environments; especially in trajectory planning, trajectory conflicts must be avoided, but the trajectories of human-driven vehicles are uncertain and difficult to plan, affecting collaborative scheduling.

[0007] Low adaptability to dynamic traffic environments: Most existing methods rely on pre-set static rules or simple behavioral decision-making models, which are difficult to cope with rapidly changing traffic environments;

[0008] Lack of comprehensive path planning and control strategies: Although some methods utilize reinforcement learning to optimize path planning, multi-vehicle collaborative decision-making, obstacle avoidance, and real-time trajectory planning still face significant challenges, such as the conflict between local optimization and global objectives, the balance between computational efficiency and accuracy in trajectory planning, and the interaction dilemma between CAV and HDV, especially in complex scenarios with multi-lane mixed traffic flow.

[0009] Therefore, there is an urgent need for a reinforcement learning-based multi-vehicle cooperative lane-changing method and system in mixed traffic environments. This system should be able to utilize reinforcement learning and multi-vehicle cooperative decision-making, combined with real-time vehicle and environmental information, to perform multi-vehicle cooperative lane-changing, improve the efficiency of multi-vehicle cooperative scheduling and adaptability to dynamic traffic environments, and ensure the safety of multi-vehicle cooperative lane-changing. Summary of the Invention

[0010] One of the objectives of this invention is to provide a multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment. This method can utilize reinforcement learning and multi-vehicle cooperative decision-making, combined with real-time vehicle information and environmental information, to perform multi-vehicle cooperative lane-changing, thereby improving the efficiency of multi-vehicle cooperative scheduling and adaptability to dynamic traffic environments, and ensuring the safety of multi-vehicle cooperative lane-changing.

[0011] The basic solution provided by this invention is a multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment, comprising:

[0012] Real-time acquisition of vehicle and environmental information;

[0013] Based on vehicle and environmental information, a two-layer decision-making model is constructed to plan the vehicle's trajectory and speed.

[0014] The high-level decision-making model determines the optimal vehicle behavior based on vehicle and environmental information, while the low-level model plans the driving trajectory and speed based on the optimal vehicle behavior and vehicle information.

[0015] In the planned driving trajectory and speed, based on the trajectory and speed, a collision ellipse model is constructed to detect the conflict between the ICV and HDV, and the driving speed is adjusted according to the conflict.

[0016] The second objective of this invention is to provide a multi-vehicle cooperative lane-changing system based on reinforcement learning in a mixed traffic environment. This system can utilize reinforcement learning and multi-vehicle cooperative decision-making, combined with real-time vehicle information and environmental information, to perform multi-vehicle cooperative lane-changing, thereby improving the efficiency of multi-vehicle cooperative scheduling and adaptability to dynamic traffic environments, and ensuring the safety of multi-vehicle cooperative lane-changing.

[0017] This invention provides a second basic solution: a multi-vehicle cooperative lane-changing system based on reinforcement learning in a mixed traffic environment, which adopts the above-mentioned multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment.

[0018] Beneficial Effects: This solution targets intelligent connected mixed traffic environments. Through a two-layer decision-making model, it achieves accurate prediction and dynamic adjustment of vehicle behavior in complex traffic flows. Specifically, it senses vehicle and environmental information in real time. The higher layer makes vehicle behavior decisions based on this information, intelligently selecting the optimal vehicle behavior. Compared with traditional traffic control methods, it can more flexibly respond to the collaborative lane-changing needs of different vehicle types and road conditions, improving the smoothness of traffic flow. The lower layer, based on the optimal vehicle behavior decisions made by the higher layer and combined with vehicle information, performs trajectory and speed planning to generate corresponding trajectories, thereby controlling vehicle movement and ensuring that vehicles can safely and efficiently execute the vehicle behavior decisions made by the higher layer.

[0019] The upper layer makes vehicle behavior decisions, while the lower layer plans the trajectory and speed and controls the vehicle. Both layers make decisions and plans based on reinforcement learning, thereby achieving collaboration, enabling multi-vehicle collaborative lane changing, improving the efficiency of multi-vehicle collaborative scheduling and adaptability to dynamic traffic environments.

[0020] Furthermore, during the low-level trajectory and speed planning process, collision detection is performed on the trajectory and speed, especially for ICV and HDV collision detection. This is not a simple detection based on whether the trajectories overlap, as is the case in existing technologies. Since HDVs are manually driven vehicles, their trajectories are uncertain. Therefore, this solution constructs a collision ellipse model for ICVs and HDVs to reflect the uncertainty of HDV trajectories. If the collision ellipse model is valid, a collision is determined to exist, thereby accurately identifying the collision between ICVs and HDVs, making timely adjustments, reducing the impact on collaborative scheduling, and further improving the efficiency of multi-vehicle collaborative scheduling and adaptability to dynamic traffic environments.

[0021] In summary, this solution can utilize reinforcement learning and multi-vehicle collaborative decision-making, combined with real-time vehicle and environmental information, to perform multi-vehicle collaborative lane changing, thereby improving the efficiency of multi-vehicle collaborative scheduling and adaptability to dynamic traffic environments, and ensuring the safety of multi-vehicle collaborative lane changing. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to the present invention.

[0023] Figure 2 This is a schematic diagram of the decision model structure in an embodiment of the multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to the present invention.

[0024] Figure 3 This is a schematic diagram illustrating lane-changing priority in an embodiment of the reinforcement learning-based multi-vehicle cooperative lane-changing method for mixed traffic environments according to the present invention.

[0025] Figure 4This is a schematic diagram of a multi-vehicle cooperative lane-changing scenario in a three-lane mixed traffic environment, as described in an embodiment of the reinforcement learning-based multi-vehicle cooperative lane-changing method for mixed traffic environments of the present invention. Detailed Implementation

[0026] The following detailed description illustrates the specific implementation method:

[0027] The markings in the accompanying drawings include:

[0028] Example 1

[0029] This embodiment is basically as shown in the appendix. Figure 1 As shown, a reinforcement learning-based multi-vehicle cooperative lane-changing method in a mixed traffic environment includes:

[0030] Real-time acquisition of vehicle and environmental information;

[0031] Based on vehicle and environmental information, a two-layer decision model is constructed to plan the vehicle's trajectory and speed. Specifically, through the constructed reinforcement learning two-layer decision model, the upper layer makes vehicle behavior decisions based on real-time perceived vehicle and environmental information to obtain the optimal vehicle behavior, while the lower layer plans the trajectory and speed based on the optimal vehicle behavior and real-time perceived vehicle information to generate the corresponding trajectory and control the vehicle's movement based on the trajectory.

[0032] The decision-making model is structured into high-level and low-level components, such as... Figure 2 As shown;

[0033] The higher level is used to make vehicle behavior decisions based on vehicle and environmental information to obtain the optimal vehicle behavior.

[0034] Specifically, the higher layer calculates the reward function value of the action combination in the action space formed by the vehicle information and environmental information and the constructed reward function of the higher layer, selects the action combination with the highest reward function value of the higher layer as the optimal vehicle behavior, generates the corresponding global instruction, and sends it down to the lower layer.

[0035] The high-level state space includes vehicle information and environmental information; the environmental information includes map information and traffic flow.

[0036] Map information, including traffic light information and speed limit zones (L1∈(60,80), L2∈(80,100), L3∈(100,120)), enables global path planning to create the most efficient path;

[0037] Vehicle information, including: Lane status:

[0038]

[0039] in The current lane status, i.e., the lane currently occupied by the vehicle (the vehicle performing trajectory planning):

[0040]

[0041] in The coordinates of the current vehicle;

[0042] The current speed of the vehicle;

[0043] The distance between the current vehicle and the vehicle in front in the same lane;

[0044] This is the distance between the current vehicle and the vehicle behind it in the same lane.

[0045] and This shows the status of the surrounding lanes of the current lane, specifically the status of the adjacent left and right lanes:

[0046] ;

[0047] ;

[0048] in The coordinates of the left adjacent vehicle are given; the left adjacent vehicle is the closest vehicle behind the current vehicle in the left lane.

[0049] The speed of the vehicle to the left;

[0050] The coordinates of the right adjacent vehicle are given; where the right adjacent vehicle is the closest vehicle behind the current vehicle in the right lane.

[0051] The speed of the vehicle to the right;

[0052] Obtaining basic driving information of surrounding vehicles is essential to making safer and more effective lane-changing decisions.

[0053] Traffic flow This includes: traffic flow in the current lane. Traffic flow in the left lane Traffic flow in the right lane ;

[0054] If the traffic flow in an adjacent lane is lower than the traffic flow in the current lane, the adjacent lane will be used as the target lane for lane changing to improve efficiency. If the traffic flow on a certain path... If the traffic flow is too high, it may cause congestion. Therefore, choose a lane with lower traffic flow and switch lanes to improve efficiency.

[0055] The high-level action space includes: action sets and ;

[0056] Action set This includes: maintaining the original lane, changing lanes to the left, and changing lanes to the right;

[0057] Action set: A∈(0,1,2);

[0058] Where 0 indicates that the lane-keeping strategy is activated;

[0059] 1, 2 indicates that the lane-changing strategy is activated and the target lane is specified;

[0060] Action set This includes: maintaining a constant speed, accelerating, and decelerating;

[0061] Action set: B∈(0,1,2);

[0062] Where 0 indicates that the constant speed driving strategy is activated;

[0063] 1 indicates that the acceleration strategy is activated;

[0064] 2 indicates that the deceleration strategy is activated.

[0065] The high-level reward functions include: lane-changing reward function and lane-keeping reward function;

[0066] The lane-change reward function is:

[0067]

[0068] in These are the weighting coefficients. As a safety distance bonus during lane changes, As a penalty for collision, Rewards for efficiency This is a reward for traffic differences; in this embodiment, it is set as follows: ;

[0069] Among them, the safety distance bonus during lane changes It encourages intelligent agents (the current vehicle) to maintain a safe distance from vehicles in front and behind;

[0070]

[0071] in Safety distance threshold:

[0072]

[0073] Where τ is the current vehicle braking reaction time, in seconds; The speed of the following vehicle is expressed in m / s. The speed of the vehicle in front is expressed in m / s. The current acceleration of the vehicle. This represents the maximum acceleration of the current vehicle. This represents the maximum acceleration of the vehicle during deceleration. The minimum acceleration required for the current vehicle to decelerate;

[0074] Collision Penalty When an intelligent agent collides with surrounding vehicles, it is given a large negative reward;

[0075]

[0076] Efficiency Rewards This encourages intelligent agents to complete lane-changing actions quickly and efficiently.

[0077]

[0078] in This is a preset lane-changing time threshold;

[0079] Traffic Difference Rewards Encourage vehicles to choose lanes with less traffic.

[0080]

[0081] in The current traffic flow in the lane. The target lane's flow rate is the target lane's flow rate; if the target lane's flow rate is less than the current lane's flow rate, a positive reward is given; otherwise, a negative reward is given.

[0082] The lane-keeping reward function is:

[0083]

[0084] in These are the weighting coefficients. To maintain a safe distance in the current lane, For speed rewards, As an acceleration reward; this embodiment sets ;

[0085] Reward for maintaining a safe distance in the current lane A positive reward is given when the agent maintains its current lane and keeps a safe distance from vehicles in front and behind.

[0086]

[0087] Speed ​​Reward This encourages intelligent agents to maintain a reasonable speed range in order to improve driving efficiency;

[0088]

[0089] in and These are the preset minimum speed and the preset maximum speed, which are 15 m / s and 25 m / s respectively in this embodiment;

[0090] Acceleration bonus It encourages agents to maintain a steady acceleration and avoids sudden acceleration or deceleration.

[0091]

[0092] in and These are the preset minimum acceleration value and the preset maximum acceleration value, which in this embodiment are -3 respectively. and 2 ;

[0093] The lower layer is used to perform trajectory and speed planning based on optimal vehicle behavior and real-time perceived vehicle information, generate corresponding planning results, and control the movement of the vehicle based on the planning results.

[0094] Specifically, the lower layer calculates the reward function value of trajectory and speed planning in the lower layer action space based on the state space formed by the global instructions (i.e. optimal vehicle behavior) and vehicle information, and constructs the underlying reward function. The trajectory and speed with the highest reward function value in the lower layer are selected as the planning result. Based on the planning result, control instructions are generated and sent to the ICV controller to control the current vehicle to execute the optimal vehicle behavior decision according to the planning result.

[0095] The lower-level state space includes: global commands and vehicle information;

[0096] Global instructions are instructions for optimal vehicle behavior generated by the higher layer, including: changing lanes to the left, changing lanes to the right, or maintaining the original lane, as well as accelerating, decelerating, or maintaining a constant speed, which is the action space of the higher layer.

[0097] Vehicle information, including: Lane status:

[0098]

[0099] in Current lane status:

[0100]

[0101] in This represents the current acceleration of the vehicle.

[0102] This is the current yaw angle of the vehicle;

[0103] The acceleration of the vehicle at the previous moment;

[0104] This represents the steering angle of the vehicle at the previous moment.

[0105] These are the coordinates of the currently selected front viewpoint;

[0106] The target lane state refers to the lane to which the vehicle changes lanes in the optimal vehicle behavior. This can be either the adjacent left target lane state or the right target lane state.

[0107] ;

[0108] in The coordinates of the vehicle ahead in the target lane;

[0109] The speed of the vehicle ahead in the target lane;

[0110] The coordinates of the vehicle following in the target lane;

[0111] The speed of the vehicle behind in the target lane.

[0112] The lower-level action space includes: trajectory generation, velocity planning and conflict correction, and control command output;

[0113] Specifically, the trajectory generation uses a third-order Bezier curve, which has good smoothness and computational efficiency, and is suitable for path planning in traffic environments;

[0114] The trajectory is controlled by four control points:

[0115]

[0116] in For the first The coordinates of the control points The Bernstein basis functions for the third-order Bezier curves are as follows:

[0117]

[0118] Control point settings: The first control point is the current vehicle position; the fourth control point is the center of the result area of ​​the optimal vehicle behavior; the middle control points are offset by a fixed ratio to control the shape of the curve.

[0119] The trajectory is discretized into several points, forming a trajectory point set, which in this embodiment consists of 101 points:

[0120] ;

[0121] Speed ​​planning and conflict correction include:

[0122] Generate the desired velocity based on the trajectory length and the desired time window:

[0123]

[0124] in For the trajectory length, For the desired time window, The initial velocity;

[0125] A proportional controller is used to generate longitudinal acceleration:

[0126]

[0127] in For the maximum permissible acceleration, Current speed;

[0128] Collision detection is performed based on trajectory and speed, including: ICV-ICV collision detection and ICV–HDV collision detection;

[0129] The ICV-ICV conflict detection includes: analyzing whether the distance between the trajectory points of two ICV vehicles is lower than a preset trajectory point threshold. If so, a conflict is determined, and a vehicle to be avoided is determined according to a preset priority rule.

[0130] The preset priority rule is that when multiple vehicles intend to enter the same area, coordination is based on priority, with higher-priority vehicles being allocated that area first. Figure 3 HDV has higher priority than ICV; straight-ahead ICV has higher priority than lane-changing ICV; forced lane-changing ICV has higher priority than free lane-changing ICV; if all are free lane-changing ICVs, then the priority objective function value of the free lane-changing ICV is calculated according to the priority objective function. The larger the priority objective function value, the higher the priority. The priority objective function can be set according to lane-changing efficiency. The higher the lane-changing efficiency, the higher the priority. In this embodiment, the priority objective function is equal to the planned speed of the free lane-changing ICV, that is, the faster the speed, the higher the priority.

[0131] ICV–HDV Conflict Detection: Considering the trajectory uncertainty of HDV vehicles, a collision ellipse model is used to determine potential conflicts. If the collision ellipse model holds true, a conflict is determined to exist. The collision ellipse model is as follows:

[0132]

[0133] in These are the trajectory coordinates of the HDV. The trajectory coordinates of the ICV. For the major semi-axis of the collision ellipse:

[0134] ;

[0135] in The current length of the vehicle. For time window, To the minor semi-axis of the collision ellipse, The current width of the vehicle;

[0136] If a conflict is determined, and the ICV vehicle is of low priority, reduce the ICV's acceleration to avoid a collision:

[0137] ;

[0138] in For the maximum permissible deceleration, This is the current speed of the ICV. This is the current speed of HDV;

[0139] Control command output includes: after completing trajectory and speed planning, converting the optimal vehicle behavior generated by the upper layer into control commands that can be executed by the ICV controller;

[0140] Specifically, after completing trajectory generation, speed planning and conflict correction, the lower layer transforms the optimal vehicle behavior generated by the higher layer into control commands that can be executed by the ICV controller, including: acceleration (longitudinal control) and steering angle (lateral control).

[0141] Among them, acceleration This has been achieved through speed planning and conflict correction;

[0142] The steering angle changes according to the direction of the trajectory;

[0143] To achieve directional control, a forward-looking point is selected in the trajectory, and a preview vector is constructed. Connect the current position of the car's front end with the forward viewpoint (distance is...) ):

[0144] Preview vector (2D):

[0145]

[0146] Forward sight distance (module length):

[0147]

[0148] Lateral error angle (angle between preview vector and vehicle body):

[0149]

[0150] in The current heading angle of the vehicle;

[0151] bicycle model steering angle (including) ):

[0152]

[0153] Longitudinal acceleration (and) Uncoupled):

[0154]

[0155] in The x-coordinate of the foreground viewpoint is... The ordinate of the foreground viewpoint is y. Let x be the x-coordinate of the current vehicle. This represents the ordinate of the current vehicle;

[0156] Based on the kinematic bicycle model, the angle between the preview vector and the current orientation of the vehicle is converted into a steering angle:

[0157]

[0158] in This is the distance from the rear wheel of the current vehicle to the preview point; This is the wheelbase of the current vehicle;

[0159] The control command output format is:

[0160] ,

[0161] ;

[0162] Control commands are sent to the actuators of the ICV vehicle to realize the implementation of the high-level behavioral intentions.

[0163] Low-level reward function :

[0164]

[0165] in As a penalty for failure, In this embodiment, a success reward is set as follows: , ;

[0166] As a security reward, Rewards for completing the task. As a stability reward, As a comfort reward;

[0167] Security rewards :

[0168]

[0169] in To adjust the negative parameter of safety importance; in this embodiment ;

[0170] Task completion reward :

[0171]

[0172] in It is a negative weight in the task completion reward; in this embodiment ; For trajectory tracking error, For speed tracking error; The target trajectory point set is generated by a third-order Bezier curve;

[0173] Stability Rewards :

[0174]

[0175] in and It is a positive weight in the stability reward; in this embodiment , ; It is the change in acceleration. It is the change in steering angle;

[0176] Comfort reward :

[0177]

[0178] in It is the current acceleration of the vehicle. It is the rate of change of steering angle. It is a negative comfort weight; in this embodiment .

[0179] In the specific implementation process, such as Figure 4 The application scenario shown is based on a multi-vehicle cooperative lane-changing scenario in a three-lane mixed traffic situation. In this intelligent connected mixed traffic environment scenario, the basic road section is a one-way three-lane road. The short white lane lines are dashed lines, allowing unrestricted lane changing between vehicles. The long white lane lines are solid lines, prohibiting lane changing. The solid yellow lines are the lane dividers. Vehicle types are divided into intelligent connected vehicles (ICVs, yellow vehicles) and manually driven vehicles (HDVs, blue vehicles). The roadside is equipped with a roadside cooperative control unit (RCCU). Intelligent connected vehicles can report their own operating status information (vehicle position, speed, heading angle, etc.) in real time and receive guidance and control information or other information (such as obstacles, pedestrians, etc.) from the RCCU to assist vehicles in making lateral lane-changing decisions.

[0180] This solution employs a two-layer multi-vehicle cooperative lane-changing method based on reinforcement learning, significantly improving the efficiency and safety of multi-vehicle cooperative lane-changing in intelligent connected mixed traffic environments. Through an efficient two-layer control architecture, this method can accurately predict and dynamically adjust vehicle behavior in complex traffic flows, effectively solving the bottleneck problems of existing technologies that cannot handle complex traffic situations and lack coordinated scheduling.

[0181] Highly efficient collaborative decision-making capability: Through a multi-agent decision-making framework based on reinforcement learning, this invention can intelligently select the optimal lane-changing strategy based on real-time traffic flow data, road conditions, and vehicle dynamics. Compared with traditional traffic control methods, it can more flexibly respond to collaborative lane-changing needs under different vehicle types and road conditions, thereby improving traffic flow smoothness.

[0182] Dynamically adapting to changing traffic environments: The dual-layer structure of this invention (vehicle behavior decision layer and trajectory planning control layer) enables the system to adapt to complex and constantly changing traffic environments in real time. Whether facing densely trafficked urban expressways or high-traffic highways, this invention can adjust the vehicle's travel path and speed in real time based on dynamic traffic information, avoiding traffic congestion and potential traffic accidents.

[0183] Significantly Enhanced Safety: Through precise vehicle status monitoring and collaborative decision-making, this invention effectively reduces traffic conflicts caused by uncoordinated driving behaviors. Reinforcement learning algorithms enable vehicles to anticipate potential hazards during lane changes and take appropriate evasive action, thereby reducing the probability of accidents and improving driving safety.

[0184] Improving overall traffic efficiency: This method achieves efficient coordinated lane changing and scheduling among multiple vehicles, reducing traffic delays caused by inconsistent vehicle behavior. Particularly in the complex traffic flow of multi-lane highways, it can significantly improve the efficiency of merging and lane changing, optimizing the overall capacity of the traffic flow.

[0185] Promoting the widespread application of intelligent connected vehicles: This solution provides technical support for the application of intelligent connected vehicles in mixed traffic environments, promoting their popularization and application. By achieving efficient collaboration and intelligent scheduling among vehicles, it lays the foundation for the construction of future intelligent transportation systems, further driving the development of modern transportation systems towards greater intelligence and efficiency.

[0186] In summary, this invention addresses the challenges of multi-vehicle cooperative lane changing in intelligent connected mixed traffic environments by combining reinforcement learning with multi-vehicle cooperative lane changing strategies. It provides a safer, more efficient, and intelligent traffic management solution, and strongly supports the widespread application of intelligent transportation technologies.

[0187] This embodiment also provides a multi-vehicle cooperative lane-changing system based on reinforcement learning in a mixed traffic environment, which adopts the above-mentioned multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment.

[0188] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment, characterized in that, include: Real-time acquisition of vehicle and environmental information; Based on vehicle and environmental information, a two-layer decision-making model is constructed to plan the vehicle's trajectory and speed. The decision-making model's higher-level components determine the optimal vehicle behavior based on vehicle and environmental information, while the lower-level components plan the driving trajectory and speed based on the optimal vehicle behavior and vehicle information. In the planned driving trajectory and speed, based on the trajectory and speed, a collision ellipse model is constructed to detect the conflict between the ICV and HDV, and the driving speed is adjusted according to the conflict. The determination of optimal vehicle behavior based on vehicle information and environmental information includes: Based on the state space formed by vehicle information and environmental information and the constructed high-level reward function, calculate the reward function value of action combination in the high-level action space; The action combination with the highest reward function value at higher levels is selected as the optimal vehicle behavior; The process of planning the driving trajectory and speed based on optimal vehicle behavior and vehicle information includes: Based on the state space formed by the optimal vehicle behavior and vehicle information and the constructed underlying reward function, the reward function value of the trajectory and speed planning performed in the underlying action space is calculated. The trajectory and velocity with the highest reward function value at the lower level are selected as the planning result; The reward functions for the higher layers include: lane-changing reward function and lane-keeping reward function; The lane-change reward function is: in These are the weighting coefficients. As a safety distance bonus during lane changes, As a penalty for collision, Rewards for efficiency Rewards based on traffic differences; The lane-keeping reward function is: in These are the weighting coefficients. To maintain a safe distance in the current lane, For speed rewards, This is an acceleration bonus.

2. The multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to claim 1, characterized in that, The collision ellipse model is as follows: in These are the trajectory coordinates of the HDV. The trajectory coordinates of the ICV. For the major semi-axis of the collision ellipse: ； in The current length of the vehicle. For time window, To the minor semi-axis of the collision ellipse, The current width of the vehicle. This represents the current speed.

3. The multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to claim 1, characterized in that, The collision detection based on trajectory and velocity also includes: For ICV-ICV collision detection, analyze whether the distance between the trajectory points of the two ICV vehicles is lower than the preset trajectory point threshold. If so, a collision is determined, and the vehicle to be avoided is determined according to the preset priority rules and the collision is avoided.

4. The multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to claim 1, characterized in that, If a conflict is determined, and the ICV vehicle is of low priority, reduce the ICV's acceleration to avoid a collision: ； in For the maximum permissible deceleration, This is the current speed of the ICV. This is the current speed of HDV.

5. The multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to claim 1, characterized in that, The lower-level action space includes: trajectory generation, velocity planning and conflict correction, and control command output; Trajectory generation uses a third-order Bezier curve, and the trajectory is discretized into several points to form a trajectory point set. ; Velocity planning and conflict correction include: generating the desired velocity based on the trajectory length and the desired time window, and generating longitudinal acceleration using a proportional controller. Collision detection is performed based on trajectory and speed; Control command output includes: after completing trajectory and speed planning, converting it into control commands that can be executed by the ICV controller.

6. The multi-vehicle cooperative lane-changing method based on reinforcement learning in a mixed traffic environment according to claim 5, characterized in that, The lower-level reward function : in As a penalty for failure, A reward for success; As a security reward, Rewards for completing the task. As a stability reward, As a comfort bonus.

7. A multi-vehicle cooperative lane-changing system based on reinforcement learning in a mixed traffic environment, characterized in that, The method for multi-vehicle cooperative lane changing in a mixed traffic environment based on reinforcement learning, as described in any one of claims 1-6, is adopted.

Images