Dynamic speed limit control method for CAV vehicle platoon based on multi-agent reinforcement learning

Abstract

The application provides a kind of dynamic speed limit control method for CAV vehicle leading vehicle group based on multi-agent reinforcement learning: a bottleneck area simulation environment is constructed, a following and lane changing model is established;According to the maximum capacity of bottleneck, the optimal vehicle group size is calculated and the vehicle group coefficient is constructed;The vehicle group characteristics are used as the state space, the merging, segmentation and migration are used as the action space, and the Mask-DQN model is trained to maximize the vehicle group coefficient, and the optimal vehicle group leading by the CAV vehicle is output;For local communication scene, IPPO algorithm is used to train the first speed limiting strategy, and for global communication scene, MAPPO algorithm is used to train the second speed limiting strategy.In the online control stage, real-time traffic flow data is collected, the optimal vehicle group is obtained through the Mask-DQN model, and the corresponding strategy model is selected according to whether the CAV vehicle can obtain global information to output real-time acceleration adjustment instruction.The application realizes the safe, efficient and collaborative control of highway mixed traffic flow.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a dynamic speed limit control method for a CAV (Carrier Aerial Vehicle) navigator group based on multi-agent reinforcement learning. Background Technology

[0002] With the development of autonomous driving and connected communication technologies, intelligent connected vehicles are gradually acquiring characteristics such as precise control, information exchange, and rapid response. Dynamic speed limit control based on intelligent connected vehicles (CAVs) has become a research hotspot in the field of speed limit management. Existing technologies mainly calculate dynamic speed limit values ​​through optimization algorithms and reinforcement learning, and issue deceleration commands to CAVs in mixed traffic flow, enabling them to guide neighboring manually driven vehicles (HDVs) to simultaneously implement deceleration behavior, thereby achieving dynamic speed limit control of the overall traffic flow.

[0003] However, existing technologies have the following shortcomings. First, there is a lack of vehicle group segmentation mechanisms in multi-lane scenarios: Existing technologies mostly use fixed thresholds or rules to mechanically divide traffic flow into multiple discrete vehicle groups in single-lane scenarios. This is difficult to adapt to the complex topological structure formed by the heterogeneous spatial distribution of vehicles in multi-lane scenarios, and there is a lack of segmentation mechanisms that use cross-lane vehicle groups as CAV control units. At the same time, segmentation based on fixed thresholds ignores the capacity constraints of downstream bottleneck areas, making it difficult for vehicle group flow to match road capacity, thereby reducing the controllability of vehicle groups. Second, there is insufficient modeling of collaborative control between CAV control units: Existing technologies obtain traffic flow information around CAV vehicles and in bottleneck areas through real-time information transmission and independently calculate their deceleration values ​​within the control cycle. However, they do not explicitly model collaborative control methods between CAV control units, making it difficult to adapt to the disturbances caused by the complex interactive behavior of heterogeneous traffic flows in real-world scenarios, and there is a lack of a global collaborative speed optimization control mechanism for multiple control units. Third, in CAV-based control technologies, there is a lack of classification strategy modeling for different communication state scenarios: the speed limit system is paralyzed when communication is missing, and it is difficult to fully utilize data for collaborative management when there is global communication.

[0004] Therefore, there is an urgent need for a method that can optimize vehicle group segmentation based on bottleneck area capacity constraints in multi-lane mixed traffic flow scenarios and implement multi-CAV collaborative speed limit control for different communication scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic speed limit control method for CAV (Carrier Availability) vehicle navigator groups based on multi-agent reinforcement learning. This method aims to address the technical problems in existing technologies, such as the lack of vehicle group segmentation mechanisms, constraints related to multi-lane scenarios and bottleneck traffic capacity, insufficient CAV cooperative control modeling, and the lack of classification strategies for different communication scenarios. The specific technical solution is as follows: A dynamic speed limit control method for a CAV (Carrier Automobile) vehicle navigator group based on multi-agent reinforcement learning, comprising an offline training phase and an online control phase; The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. When the agent can only obtain local observation information, the first velocity limiting policy model is trained by using the local observation information as input and the agent's acceleration adjustment command as output, and the independent proximal policy optimization algorithm is used. When the agent can obtain global observation information, the second speed limiting policy model is trained by using the global observation information as input and the agent's acceleration adjustment command as output, and the multi-agent proximal policy optimization algorithm is used. The online management phase includes: Collect real-time traffic flow data within the upstream control area of ​​the bottleneck area, input the real-time traffic flow data into the trained Mask-DQN model, and obtain several optimal vehicle groups led by CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. Based on the type of information that the intelligent agent can currently obtain, either the first speed limit strategy model or the second speed limit strategy model is adopted to output acceleration adjustment commands in real time.

[0006] Preferably, the lane-changing behavior models of the CAV vehicle and the HDV vehicle are the same, wherein the lane-changing motivation is expressed as: Lane change safety verification is represented as follows: in, For lane changing motives in the first traffic zone, The lane-changing motive for the second passage area; 0 indicates no lane-changing motive, 1 indicates free lane-changing motive, and 2 indicates forced lane-changing motive. The speed difference between the vehicle in front and the vehicle in front; The speed difference threshold; For the exit lane, To reduce the number of lanes; This marks the boundary between the upstream and downstream sections of the second passage area. This is the current location of this vehicle. This vehicle, The position of vehicles behind the target lane; Position of the vehicle ahead in the target lane; This is the minimum acceptable clearance for lane changes.

[0007] Preferably, the optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, specifically: in, To achieve the optimal vehicle group size; This represents the maximum traffic capacity of the bottleneck area. The expected speed of the vehicle group in the bottleneck area; The length of the road occupied by the group of vehicles; The desired spacing between the vehicles; The critical density of the bottleneck region; The critical velocity for the bottleneck region; The vehicle lane-changing rate for the downstream section of the second traffic zone; This is the capacity loss coefficient caused by lane-changing conflicts.

[0008] Preferably, the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle, specifically: in, For vehicle group coefficient, Matching item to vehicle group size; For CAV vehicle navigation gain; To achieve the optimal vehicle group size; For the first The size of a vehicle group; The number of vehicles in the group; The penetration rate of CAV vehicles within the upstream control area of ​​the bottleneck region; For the first CAV vehicle navigation gain coefficient for a vehicle group; The standard deviation is Gaussian.

[0009] Preferably, the training of the first speed limiting policy model is specifically as follows: A1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and use the local observation information of the intelligent agent as the state space; A2. Set the action space of each agent to continuous acceleration adjustment commands; A3. Constructing the policy network for each agent. and value network The policy network It is used to output the mean and standard deviation of the probability distribution of actions based on local observation information, and to construct a Gaussian distribution strategy; A4. Each agent, based on its current state space, accesses the policy network... Output the action, and after the action is executed, the state transition probability will be transferred to the next state space; A5. Calculate the immediate reward obtained by each agent after performing an action, and store the state space before the action, the current action, the immediate reward after the action, and the state space after the action as experience into the experience pool. A6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. value network The mean squared error between the estimated state value and the target value is used as the loss function to update the value network parameters. ; A7. Repeat steps A4 to A6 until the first rate limiting strategy model converges.

[0010] Preferably, the training of the second speed limiting strategy model is as follows: B1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and use global observation information as the state space; B2. Set the action space of each agent to continuous acceleration adjustment commands; B3. Constructing the policy network for each agent. and value network The policy network It is used to output the mean and standard deviation of the action probability distribution based on the local observation information of the agent, and to construct a Gaussian distribution strategy; B4. Each agent, based on current local observation information, communicates through the policy network. Output the action, and after the action is executed, the state transition probability will be transferred to the next state space; B5. Calculate the immediate reward obtained by each agent after performing an action, and store the state space before the action, the currently executed action, the immediate reward after the action, and the state space after the action as experience into the experience pool. B6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. value network The mean squared error between the estimated state value and the target value is used as the loss function to update the value network parameters. ; B7. Repeat steps B4 to B6 until the second rate limiting strategy model converges.

[0011] Preferably, the immediate rewards during the training of the first and second speed-limiting policy models are both represented as: in, for Time-based intelligent agent Instant rewards; for Time-based intelligent agent The discrete form of the distance-collision time; The target speed for the vehicle group to pass through; for Time-based intelligent agent The vehicle speed; and All are weighting coefficients.

[0012] Preferably, the state transition probabilities during the training of the first speed limiting policy model Represented as: State transition probabilities during training of the second speed limiting policy model Represented as: in, express Local observation information of all agents at any given moment; express Local observation information of all agents at any given time, ; express Actions performed by all agents at any given moment; Indicates the total number of agents; Indicates the first A smart agent Local observation information at any given time; express Global observation information at any given time. express Global observation information at any given moment.

[0013] Preferably, the local observation information of the agent is represented as follows: in, For intelligent agents Local observation information; For intelligent agents Its own speed; Represents intelligent agents The speed difference between the last car in the previous optimal car group; For intelligent agents The distance between the vehicle and the last vehicle in the previous optimal vehicle group; For intelligent agents Collision time index; For intelligent agents The size of the vehicle group it belongs to; For intelligent agents The average speed of vehicles within the observation range.

[0014] Preferably, the global observation information is represented as follows: in, Represents global observation information; It is a set of local observation information of each intelligent agent; The average speed of vehicles in the bottleneck area; The total number of intelligent agents within the control area; The average velocity of each agent; This represents the average risk level of the road.

[0015] The application of the technical solution of the present invention has the following beneficial effects: This invention calculates the optimal vehicle group size by introducing the maximum capacity of the bottleneck area. Based on the optimal vehicle group size and the CAV (Car Access Vehicle) leadership principle, it constructs a vehicle group coefficient that includes a vehicle group size matching term and a CAV leadership gain term. The rationality of the vehicle group segmentation structure is evaluated from two dimensions: the degree of deviation between the vehicle group size and the optimal size, and whether the vehicle group is led by CAVs. A larger vehicle group coefficient value indicates that the vehicle group size is close to the optimal size and the leading vehicle is a CAV, thus ensuring that the segmented vehicle group matches the bottleneck capacity and has the ability to be effectively controlled by the control system. This invention solves the problems of existing technologies where vehicle group segmentation relies on fixed rules and does not consider the heterogeneous spatial distribution of multi-lane traffic and the constraints of downstream bottleneck capacity.

[0016] This invention models the vehicle grouping problem as an objective optimization problem. Through a mask mechanism, it dynamically identifies and masks invalid merging, splitting, and migration actions in the current state, focusing the agent's attention on a subset of effective actions. This overcomes the inefficiency of standard DQN in exploring a large action space, significantly improving the training efficiency and policy quality of the vehicle grouping strategy. Furthermore, in the Mask-DQN model training, this invention uses the single-step improvement of the vehicle group coefficient as an immediate reward, ensuring the agent receives immediate feedback after each partitioning operation. This directly quantifies the contribution of actions to the vehicle group structure optimization, guiding the agent to quickly converge towards maximizing the vehicle group coefficient.

[0017] This invention constructs first and second speed-limiting strategy models for two scenarios: local communication and global communication. These models enable agents to make independent decisions based solely on local observations when communication is limited, and to achieve multi-agent collaborative optimization using global information when communication is complete, thus balancing control effectiveness under different communication conditions. Furthermore, the reward functions of the first and second speed-limiting strategy models are designed as multi-objective weighted forms that simultaneously consider driving safety and traffic efficiency. The risk level is measured by the distance to collision time, and traffic efficiency is measured by the degree to which the vehicle speed approaches the target speed. By balancing these two objectives through weighting coefficients, the agents are guided to optimize both safety and efficiency dimensions simultaneously during the learning process, avoiding problems such as excessively low speeds or excessively high risks caused by optimizing a single objective.

[0018] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart of the dynamic speed limit control method for the CAV vehicle navigator group based on multi-agent reinforcement learning in Example 1; Figure 2 This is a schematic diagram of the optimal vehicle group division in Example 1; Figure 3 This is a schematic diagram of the intelligent agents in the optimal vehicle group in Example 1; Figure 4 This is a schematic diagram illustrating the change in the cumulative TET value in the simulation case of Example 1; Figure 5 This is a schematic diagram illustrating the change in the cumulative TIT value in the simulation case of Example 1; Figure 6 This is a schematic diagram of travel time in the simulation case of Example 1. Detailed Implementation

[0020] To facilitate understanding of the present invention, a more complete description is provided below, along with preferred embodiments. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the present invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0022] Example 1: See Figure 1 This embodiment provides a dynamic speed limit control method for a CAV vehicle navigator group based on multi-agent reinforcement learning, including an offline training phase and an online control phase; The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. When the agent can only obtain local observation information, the first velocity limiting policy model is trained by using the local observation information as input and the agent's acceleration adjustment command as output, and the independent proximal policy optimization algorithm is used. When the agent can obtain global observation information, the second speed limiting policy model is trained by using the global observation information as input and the agent's acceleration adjustment command as output, and the multi-agent proximal policy optimization algorithm is used. The online management and control phase includes: Collect real-time traffic flow data within the upstream control area of ​​the bottleneck area, input the real-time traffic flow data into the trained Mask-DQN model, and obtain several optimal vehicle groups led by CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. Based on the type of information that the intelligent agent can currently obtain, either the first speed limit strategy model or the second speed limit strategy model is adopted to output acceleration adjustment commands in real time.

[0023] The dynamic speed limiting control method of this embodiment will be described in detail below: In this embodiment, CAV vehicles refer to connected and automated vehicles (CAVs), HDV vehicles refer to human-driven vehicles (HDVs), and the Mask-DQN model is an improved model that combines the action mask mechanism with the classic reinforcement learning algorithm DQN (Deep Q-Network).

[0024] In this embodiment, the bottleneck area refers to the bottleneck area of ​​a highway. A bottleneck area is a critical road segment where traffic capacity is significantly reduced and traffic flow easily transitions from free flow to congested or even blocked flow due to factors such as road geometry, traffic demand, traffic composition, and driving behavior. Based on the characteristics of the bottleneck and whether its spatial location changes over time, it can generally be divided into two main categories: fixed bottlenecks and moving bottlenecks.

[0025] In this embodiment, the road segment within the upstream control area of ​​the bottleneck area is divided into a first traffic area and a second traffic area along the vehicle's direction of travel. Vehicles traveling in the same direction will sequentially pass through the first traffic area, the second traffic area, and the bottleneck area. The control area can be set according to actual conditions. For example, the road segment 0-3 kilometers upstream of the bottleneck area can be set as the control area. Once a vehicle enters the control area, its speed will be adjusted to ensure it reaches the target speed when it reaches the bottleneck area.

[0026] The congestion and disturbances in the bottleneck area mainly stem from three types of behaviors: free lane changing, slowing down and waiting, and forced lane changing. Free lane changing often occurs in the first traffic zone upstream of the bottleneck, where vehicles actively change lanes to achieve higher speeds. This behavior prematurely increases the traffic density in the bottleneck area, exacerbating congestion. Slowing down and forced lane changing are concentrated in the second traffic zone. In this zone, limited by the bottleneck's capacity, vehicles must slow down and wait for an acceptable lane-changing gap to merge. This process directly leads to a decrease in traffic speed, gradually forming congestion and queues. Simultaneously, due to road conditions, vehicles often force lane changes within small gaps, causing significant speed fluctuations for following vehicles and further increasing the risk of rear-end collisions.

[0027] Therefore, in this embodiment, CAV vehicles and HDV vehicles are mixed within the upstream control range of the bottleneck area to construct a simulation environment upstream of the bottleneck area. The first passage area is located at the very upstream, where vehicles mainly engage in free lane changing and straight-line travel. This area is also where vehicle control begins to significantly reduce traffic pressure on subsequent road sections. The second passage area is located in the middle, where forced lane changing behavior increases significantly, and vehicles will also be controlled in this area. The bottleneck area is located downstream, where traffic capacity is limited. Vehicles will reach the target speed before reaching this area, thus passing through the bottleneck area more smoothly.

[0028] Furthermore, in order to achieve accurate simulation of mixed traffic flow, it is necessary to model the car-following behavior and lane-changing behavior of CAV vehicles and HDV vehicles respectively.

[0029] Specifically, in this implementation, an Intelligent Driver Model (IDM) is used to model the following behavior of HDV vehicles. The Intelligent Driver Model (IDM) is represented as follows: in, The acceleration to be used by the following vehicle in the next moment; The maximum acceleration that can be used by following vehicles; To follow the current speed of the vehicle, To match the desired speed of the vehicle, The speed difference between the following vehicle and the vehicle in front; For acceleration exponent, in this embodiment we take... =4; The expected safe headway for following vehicles; The current actual headway between the following vehicle and the vehicle in front; This is the minimum safe distance when stationary; For safe headway; To reduce the speed of vehicles following the car.

[0030] The Intelligent Driver Model (IDM) takes the current vehicle speed, relative speed to the vehicle in front, actual vehicle distance, and expected safe distance as inputs, and outputs the specific acceleration value of the following vehicle at each step, thereby quantitatively describing the dynamic change process of the following vehicle's acceleration. It can reproduce the phase transition phenomenon of traffic flow from free flow and synchronous flow to congested flow, as well as real driving characteristics such as bias towards the expected driving speed and sudden braking.

[0031] Specifically, the following behavior of CAV vehicles is divided into two types depending on the type of vehicle in front. When a CAV vehicle follows an HDV vehicle, an adaptive cruise control (ACC) model is used, which is represented as follows: in, This is the current position of the vehicle in front. The current location of the vehicle being followed; The length of the vehicle in front; The current speed of the vehicle in front. The current speed of the vehicle following it; For cruise control of the vehicle's headway; This is the gain coefficient for the positioning error. This is the gain coefficient for the speed error; The acceleration of the following vehicle output by the adaptive cruise control model; Indicates the vehicle in front. Indicates a vehicle following another vehicle.

[0032] The adaptive cruise control model (ACC) takes safe distance constraints and desired vehicle speed as its core control objectives, and outputs smooth acceleration commands in real time. It reflects the performance characteristics of adaptive cruise control such as vehicle speed maintenance and distance adjustment, and can accurately characterize the micro-behavioral features of CAV vehicles when acquiring external information and using it for longitudinal control.

[0033] Furthermore, when CAV vehicles form a platoon with other CAV vehicles, a cooperative adaptive cruise control model (CACC) is used, represented as: in, This refers to the following distance error; The first derivative of the spacing error is expressed as the relative velocity deviation. The real-time acceleration of the vehicle in front; For proportional gain, For differential gain, This is the feedforward gain.

[0034] As can be seen from the first derivative of the error term, the error variation is closely related to the speed and acceleration of the vehicle in front, as well as the speed and acceleration of the vehicle itself. The key to CACC's superiority over ACC lies in the acceleration of the vehicle in front obtained through communication. As a feedforward term, it affects the acceleration value of the final output.

[0035] Since the first passage area, the second passage area, and the bottleneck area have different vehicle driving characteristics, after constructing a car-following model for the vehicle, different parameters can be selected to simulate the driving style of the corresponding road segment. The specific parameter values ​​can be calibrated with reference to common knowledge in the field or actual test data.

[0036] Preferably, this embodiment proposes an improved lane-changing model for the upstream of the bottleneck region based on the LC2013 lane-changing model and the lane-changing behavior characteristics of vehicles upstream of the bottleneck region. The LC2013 lane-changing model is a hierarchical microscopic vehicle lane-changing model developed by the German Aerospace Center for the open-source microscopic traffic simulation platform SUMO.

[0037] The improved lane-changing model in this embodiment divides vehicle lane-changing behavior into two stages: lane-changing motivation generation and lane-changing safety verification and execution. Since the upstream area of ​​the highway bottleneck region in this embodiment is not subject to traffic rule constraints and ignores vehicles' proactive yielding behavior, in order to recreate the road congestion and high-risk driving conditions caused by lane-changing behavior upstream of the bottleneck region, this embodiment only considers tactical and strategic lane-changing motivations, corresponding to free lane-changing and forced lane-changing scenarios, respectively. Specifically, free lane-changing is used in the first traffic area, free lane-changing is used in the upstream section of the second traffic area, and forced lane-changing is used in the downstream section.

[0038] Specifically, in the first traffic zone, HDV vehicles apply tactical lane-changing mechanisms, while CAV vehicles do not perform lane-change control. In the second traffic zone, based on the vehicle's position, free lane-changing or forced lane-changing mechanisms are applied to each CAV and HDV vehicle, forcing them to change lanes to a lane where they can continue driving. The expression for the lane-changing mechanism is: in, For lane changing motives in the first traffic zone, The lane-changing motive for the second passage area; 0 indicates no lane-changing motive, 1 indicates free lane-changing motive, and 2 indicates forced lane-changing motive. The speed difference between the vehicle in front and the vehicle in front; The speed difference threshold; For the exit lane, To reduce the number of lanes; This marks the boundary between the upstream and downstream sections of the second passage area. This is the current location of this vehicle. This refers to this vehicle.

[0039] Furthermore, lane-change safety verification is based on the minimum acceptable distance in the target lane, ensuring that the distance between the vehicle in front and the vehicle behind in the target lane is large enough to avoid a collision. Its expression is: in, The position of vehicles behind the target lane; Position of the vehicle ahead in the target lane; This is the minimum acceptable clearance for lane changes.

[0040] In this embodiment, HDV vehicles' free lane changing and forced lane changing correspond to Take 30m and 18m respectively; CAV vehicles' free lane changing and forced lane changing correspond to The distances were set to 25m and 15m respectively. Because CAV vehicles have a rapid following response, their minimum acceptable clearance is relatively small; while vehicles with mandatory lane-changing motivation have more lenient requirements on following distance in lane-changing behavior. Furthermore, to simulate the deceleration and waiting behavior of vehicles with mandatory lane-changing motivation but lacking safe lane-changing conditions, the car-following model further reduces the target speed, increases the target following distance, and adjusts the target minimum distance based on road segment parameters.

[0041] After the simulation environment is built, the vehicle grouping strategy can be trained within it. Preferably, in this embodiment, the optimal vehicle group size is calculated based on the maximum capacity of the bottleneck area as follows: in, To achieve the optimal vehicle group size; This represents the maximum traffic capacity of the bottleneck area. The expected speed of the vehicle group in the bottleneck area; The length of the road occupied by the group of vehicles; The desired spacing between the vehicles; The critical density of the bottleneck area (i.e., the traffic density at which the traffic flow in the bottleneck area reaches its maximum value). The critical speed of the bottleneck area (the vehicle speed corresponding to the traffic density of the bottleneck area equaling the critical density); The vehicle lane change rate is the vehicle lane change rate in the downstream section of the second traffic area, which is included in the control area. The vehicle lane change rate is expressed as the number of lane-changing vehicles / the number of vehicles passing through. The capacity loss coefficient caused by lane-changing conflicts is set to 0.8 in this embodiment. The minimum safe distance between vehicles; The expected safe distance for the vehicle group; The desired distance between vehicles within the vehicle group; This refers to the length of the vehicle body; For the first The speed of the vehicle; For the first The maximum acceleration of the vehicle.

[0042] Considering the frequent lane-changing behavior upstream of the bottleneck area, which causes additional disturbances, speed fluctuations, and effective clearance loss, the actual throughput capacity will decrease to some extent compared to the theoretical capacity. Therefore, this embodiment introduces lane-changing rate and lane-changing conflict loss coefficient to reduce the bottleneck capacity. At the micro level, the longitudinal organization of the vehicle group should ideally ensure safety during driving. Therefore, this embodiment uses a spacing strategy based on safe time distance to calculate the expected spacing of the vehicle group, which is jointly determined by the minimum static safe clearance and the dynamic safe clearance based on the expected time distance, i.e., the larger of the two values ​​is taken. Similarly, adjacent vehicles within the vehicle group should also maintain a reasonable and safe time distance to ensure the stability of the vehicle group when the vehicle group speed is limited. The expected spacing within the vehicle group consists of the safe time distance and the safe braking distance, reflecting the collision avoidance requirements under the expected time distance and the maximum deceleration constraint, respectively. Thus, the total length of the vehicle group can be expressed as a linear combination of the vehicle body length and the spacing between vehicles. Among them, the minimum safe distance, safe time distance, and braking distance are all calculated with reference to the parameters in the car-following model.

[0043] Preferably, the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle, specifically as follows: in, For vehicle group coefficient, Matching item to vehicle group size; For CAV vehicle navigation gain; For the first The size of a vehicle group (i.e., the total number of vehicles). The number of vehicles in the group; The penetration rate of CAV vehicles within the upstream control area of ​​the bottleneck region; For the first CAV vehicle navigation gain coefficient for a vehicle group; The standard deviation is Gaussian.

[0044] In this embodiment, the vehicle group coefficient evaluates vehicle group segmentation from two dimensions: vehicle group size matching degree and CAV vehicle leadership. A Gaussian kernel is used to establish a vehicle group size matching term to characterize the impact of offset on the vehicle group structure. The vehicle group size matching term reaches its maximum value when the vehicle group size equals the optimal size; as the degree of deviation increases, its contribution decreases exponentially, reflecting the losses caused by the vehicle group structure deviating from the optimal operating state. The CAV vehicle leadership gain term is used to evaluate whether the vehicle group has the ability to be effectively controlled by the control system. In vehicle group control, the lead vehicle usually determines the dynamic behavior of the entire vehicle group. When the lead vehicle is a CAV vehicle, the system can implement variable speed limit or cooperative speed limit control better. Therefore, when all lanes in the vehicle group are led by CAV vehicles, the highest gain value will be obtained; if only some lanes are led by CAV vehicles, the gain value drops to 1; when the entire vehicle group is led by HDV vehicles, the system will exhibit stronger randomness and uncertainty, and the gain will be 0.

[0045] Specifically, the larger the vehicle group coefficient value, the closer the vehicle group division result is to the expected distribution, that is, the number of vehicles in the group approaches the optimal size and the leading vehicle is a CAV vehicle; conversely, the smaller the vehicle group coefficient value, the worse the vehicle group control performance under the current division strategy is, and a second division is needed to further optimize the vehicle group division strategy.

[0046] Preferably, the initial vehicle group division is performed based on simulated traffic flow data within the upstream control area of ​​the bottleneck region, specifically: Vehicles within the upstream control area of ​​the bottleneck zone will be numbered sequentially from downstream to upstream according to their spatial location. The vehicle closest to the bottleneck area is defined as the first vehicle.

[0047] Will The initial vehicle grouping is performed based on the headway of consecutive vehicles: vehicle number 1 is taken as the first vehicle of the first vehicle group, and the vehicle groups are divided sequentially. Within all lanes, if the headway between the target vehicle and the vehicle in front is less than a set threshold (100m in this embodiment), the target vehicle is divided into the same vehicle group as the vehicle in front; otherwise, it is divided into a new vehicle group.

[0048] After the initial vehicle grouping, traffic flow can be defined as an initial set of vehicle groups consisting of multiple vehicle groups. , among which, the Individual car group The size (i.e., the total number of vehicles) is denoted as After vehicle grouping is performed, the current vehicle grouping situation can be evaluated based on the vehicle grouping coefficient.

[0049] Furthermore, this embodiment models the vehicle grouping problem as an objective optimization problem and uses the Mask-DQN model to solve for the optimal vehicle grouping strategy. The essence of vehicle grouping is to adjust the vehicle group structure through three types of operations—merging, splitting, and migrating—based on the initial partitioning, so that the overall partitioning result reaches the optimal level, thus obtaining the vehicle group coefficient. maximize.

[0050] Specifically, in this embodiment, the merging operation refers to merging adjacent vehicle groups. With a group of cars Merging; splitting operations refer to dividing a group of vehicles segmentation Vehicles form a sub-vehicle group and The original vehicle group numbers are sequentially incremented; the migration operation refers to transferring the vehicle group... Upstream (or downstream) Vehicles moved to adjacent vehicle groups (or The total number of vehicles remains unchanged. The maximum value is determined by the number of vehicles in the vehicle group, and the specific segmentation is obtained by the learning strategy, such as: The value can be 1-3. Reinforcement learning can use model training to determine whether to segment 1, 2 or 3 vehicles, or choose other two types of actions.

[0051] Preferably, the feature vector of the vehicle group includes five types of feature vectors: total number of vehicles in the group, size of each group, type of lead vehicle in each group, CAV vehicle penetration rate within the control range, and distance between vehicle groups. The state space of the Mask-DQN model is constructed based on these five types of feature vectors. It can comprehensively reflect the rationality of the vehicle group division structure, and the dimensions are moderate, which is conducive to the convergence of reinforcement learning and training.

[0052] The action space of the Mask-DQN model consists of the three types of actions for the aforementioned vehicle group partitioning decision variables, represented as follows: However, under certain conditions, not all actions are effective or feasible. For example, when the number of vehicles in a group... The segmentation action cannot be performed when the vehicle group size is large. Less than the number to be migrated Transfer actions cannot be executed at this time. Directly exploring and learning across the entire action space leads to a large number of invalid attempts, significantly reducing learning efficiency and policy quality. To address this, this embodiment introduces a Mask mechanism. The core idea is to dynamically identify and mask invalid actions in the current action space state during the two key stages of action selection and Q-value update, thus focusing the agent's attention on a subset of valid actions.

[0053] Specifically, the optimal action selection (i.e., selecting the action with the largest Q value) and the calculation of the target Q value are expressed as follows: in, for t The execution of actions at any given moment for t The state space at any given moment; For state space The effective action space below; In the state space The main evaluation network parameters are: Take action below The cumulative expected return that can be obtained afterward; Represents the current state space Take action below The cumulative expected return of the target at that time; express Instant reward refers to the immediate reward after taking an action; This represents the discount factor, indicating the importance of future rewards. These represent the parameters of the target evaluation network; Represents the state space at the next time step. Take action below Estimated cumulative expected return at that time; This is an indicator function; if a valid action exists in the state, the function value is 1; otherwise, it is 0.

[0054] The Mask mechanism used in this embodiment is a common technique in reinforcement learning for handling invalid actions. Its basic idea is to mark the validity of each action in the current state by using a binary mask vector, and to mask the Q value corresponding to invalid actions, so that the agent can only select and update in the subset of valid actions.

[0055] Furthermore, to guide the agent in learning the optimal vehicle grouping strategy, a suitable reward signal needs to be designed. This embodiment uses the single-step improvement in the vehicle group coefficient as the immediate reward, and constructs a loss function based on the mean squared error (MSE) between the current Q-value and the target Q-value. Guide the agent to learn the optimal vehicle grouping strategy: in, for The vehicle group coefficient at any given time This represents the single-step improvement amount of the vehicle group coefficient. Instant rewards for moments The reward design directly quantifies the actions performed. The real-time changes in the optimization effect of the rear vehicle group structure encourage the agent to take actions that can improve the overall vehicle group coefficient. S Value partitioning strategy.

[0056] like Figure 2 As shown, in this embodiment, the Mask-DQN model is trained to output several optimal vehicle groups led by CAV vehicles. The specific method is as follows: S1. Obtain simulated traffic flow data within the upstream control area of ​​the bottleneck area, number the vehicles according to their spatial location and divide them into an initial vehicle group set based on the headway. Among them, consecutive vehicles with a headway less than a set threshold are divided into the same vehicle group, otherwise they are divided into a new vehicle group. S2. Use the feature vectors of the current vehicle group set as the state space of the Mask-DQN model. The merging, splitting, and migrating of vehicles within a group are treated as the action space, and a mask mechanism is used to identify the current state space. The effective action space below; S3. Calculate the Q-value (cumulative expected return) of each effective action through the main evaluation network, select the action with the largest Q-value to execute, and update the vehicle group set to obtain the next state space. ; Calculate the change in the vehicle group coefficient before and after the action is executed as the immediate reward. The state space before the action Current action being performed Instant rewards after an action and the state space after the action Stored as experience in the experience pool; S4. Calculate the target Q value (target cumulative expected return) using the target evaluation network, and use the mean square error between the Q value output by the main evaluation network and the target Q value as the loss function to backpropagate and update the parameters of the main evaluation network. S5. Repeat steps S2 through S4, each loop... The parameters of the main evaluation network are then synchronized to the target evaluation network. When the Mask-DQN model converges, the optimal vehicle grouping strategy that maximizes the vehicle group coefficient is output, resulting in several optimal vehicle groups led by CAV vehicles.

[0057] Specifically, the convergence of the Mask-DQN model refers to the following: when the increase in the vehicle group coefficient between any two adjacent iterations is less than a preset threshold in a series of consecutive iterations, or when the maximum number of training iterations is reached. In this embodiment... The value range is set to 5-10.

[0058] Furthermore, after obtaining the optimal vehicle group, speed limit strategy models can be trained for different communication scenarios.

[0059] Specifically, based on the Independent Proximal Policy Optimization (IPPO) algorithm, this embodiment models the lead CAV vehicles within the upstream control range of the bottleneck area as a multi-agent system, such as... Figure 3 As shown, all lead CAVs in each vehicle group are considered as one agent (i.e., the lead CAVs in each lane of a single vehicle group are collectively considered as one agent), possessing their own policy network and value network. They can interact and learn from the traffic flow environment to optimize speed limit policies and guide the speed control of the vehicle group. The remaining vehicles in the vehicle group all use a car-following model to simulate micro-behavior.

[0060] Specifically, the optimal multi-agent system for vehicle corps navigation using CAV (Car Aid) It can be represented as: in, This refers to the number of vehicles currently on the road led by CAV vehicles, i.e., the number of intelligent agents. Global observation information for all CAV vehicles; This refers to the set of local observation information from each intelligent agent (i.e., the lead CAV vehicle). ; The action space for each intelligent agent; The evolution mechanism of traffic flow is jointly determined by the real-time behavior of vehicles, the road physical environment, and vehicle dynamics. The reward function for each agent; This is the discount factor.

[0061] When CAV vehicles cannot obtain global observation information (i.e., CAV vehicles can only obtain the state information of vehicles within a limited range around them through onboard sensors (such as cameras and radar) and obtain the state information of neighboring CAV vehicles through vehicle-to-vehicle communication), the local observation information of each agent is defined as: in, For intelligent agents Local observation information (i.e., the state space of the first speed limiting strategy model); For intelligent agents Its own speed; Represents intelligent agents The speed difference between the last car in the previous optimal car group; For intelligent agents The distance between the vehicle and the last vehicle in the previous optimal vehicle group; For intelligent agents Collision time index; For intelligent agents The size of the vehicle group is sent to each lead CAV vehicle by the roadside unit after the optimal vehicle group is divided. For intelligent agents The average speed of vehicles within the observable range; in this embodiment, the intelligent agent can acquire the speed information of four vehicles in front and behind in real time.

[0062] Preferably, the collision time index is expressed as: in, Represents intelligent agents exist The collision time index at any given moment. express t Time-based intelligent agent The net distance between the last car in the previous optimal car group and the car in the previous optimal car group. Represents intelligent agents exist The speed of time express Time-based intelligent agent The speed of the last vehicle in the previous optimal vehicle group. Furthermore, when the collision time metric is below a set critical threshold... At that time, the vehicle is considered to be in a high-risk state.

[0063] Preferably, when an optimal vehicle group includes two or more lead CAV vehicles, , , , and The lead CAV vehicle at the very front is selected for data calculation and acquisition. In this embodiment, the local observation information, considering spatiotemporal and safety information, is scaled to a relatively small dimension to enable rapid training convergence and avoid the curse of dimensionality and information redundancy interference.

[0064] Furthermore, the agent's action space is a continuous acceleration adjustment command, and the output range is... The unit of acceleration adjustment command is The primary function is deceleration, while retaining a small range of acceleration capability to ensure the vehicle's speed recovery ability.

[0065] Furthermore, in traffic flow scenarios, the state transition probability (i.e., the evolution mechanism of traffic flow) can be expressed as the probability of changes in vehicle position and speed in the overall system after a vehicle performs an action; therefore, the state transition probability in the first speed limit strategy model... Represented as: in, express Local observation information of all agents at any given time, express Local observation information of all agents at any given moment; express Actions performed by all agents at any given moment; , Indicates the first A smart agent Local observation information at any given time.

[0066] Furthermore, based on the two key optimization objectives of driving safety and traffic efficiency under variable speed limit control, the reward function in this embodiment adopts a multi-objective weighted approach to comprehensively consider the two types of evaluation indicators, as shown in the following expression: in, for Time-based intelligent agent Instant rewards; for Time-based intelligent agent The discrete form of the distance-collision time is used to measure the degree of risk in the current state; The target speed for the traffic group. express Time-based intelligent agent The vehicle speed, the reward function through Encourage CAV pilot speeds to approach the target speed; and All are weighting coefficients.

[0067] Furthermore, intelligent agents The discrete form of the distance-collision time is expressed as: in, Indicates the critical threshold. Indicates the time step.

[0068] Furthermore, in this embodiment, the agent adopts the Actor-Critic framework, where both the policy network (Actor) and the value network (Critic) are implemented based on two fully connected neural networks, with 128 neurons in each layer, and the activation function is... , is represented as: in, This represents the first layer of a fully connected neural network. This represents the second fully connected neural network layer; This represents the weight parameters of the first fully connected neural network layer. This represents the weight parameters of the second fully connected neural network layer; This represents the bias parameters of the first fully connected neural network layer. This represents the bias parameters of the second fully connected neural network layer; For the first Local observation information of each agent is used as input to the policy network and value network; Activation function is used to improve the network's ability to approximate complex functions, thereby improving training stability and convergence speed.

[0069] The policy network models continuous actions by outputting the mean and standard deviation of the action distribution, as follows: in, Let be the mean of the probability distribution of the actions. The standard deviation of the probability distribution of actions; This is a scaling factor used to map the output motion to the motion space. For the weight parameters of the action output layer, These are the bias parameters for the action output layer.

[0070] Based on the output of the policy network, the final policy can be modeled as a Gaussian distribution: in, Represents intelligent agents In state space Select action The probability distribution; Represents intelligent agents Possible actions; Indicates a Gaussian distribution; These are the policy network parameters.

[0071] The value network is used to estimate state value, which is expressed as the cumulative reporting expectation calculated by each agent based on the current local observation information, as follows: in, Represents intelligent agents Based on the current state space The expected cumulative return The weights of the current value output layer. The bias parameters for the current value output layer. These are the parameters of the value network.

[0072] Preferably, in this embodiment, training the first speed limiting strategy model specifically involves: A1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and integrate the local observation information of the intelligent agent. As a state space; A2. Set the action space of each agent to continuous acceleration adjustment commands, and the range of the acceleration adjustment commands is: The unit of acceleration adjustment command is ; A3. Constructing the policy network for each agent. and value network The policy network is used to output the mean of the action probability distribution based on local observation information. with standard deviation And construct a Gaussian distribution strategy The value network is used to estimate state value (i.e., cumulative reporting expectation). ); A4. Each agent outputs an action based on its current state space through the policy network. The probability of state transition after the action is performed. Transition to the next state; where, Represents intelligent agents exist The action output at any time; A5. Calculate the immediate rewards obtained by each agent after performing an action. The state space before the action Current action being performed Instant rewards after an action and the state space after the action This experience is stored in the experience pool; among which... Represents intelligent agents Execute action The immediate reward afterwards Represents intelligent agents exist The state space at any given moment; A6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. The mean squared error between the estimated state value and the target value of the value network is used as the loss function to update the value network parameters. ; A7. Repeat steps A4 to A6 until the first rate limiting strategy model converges.

[0073] Preferably, the advantage function is used to measure the merits of taking a certain action relative to the average action in the current state space, and is usually calculated using the Generalized Advantage Estimation (GAE) method.

[0074] Those skilled in the art will understand that the shearing loss function constrains the update magnitude within a certain range, and can transform the objective function into the expected form based on samples sampled using the old strategy. Its formula can be expressed as: in, This represents the cut-and-substitute objective function; The advantage function is used to evaluate the current action. The relative merits and demerits of the average movement; For shear hyperparameters; This represents the probability ratio between the old and new strategies. This represents the mathematical expectation of samples at each time step in the sampling experience. When the ratio... When the value goes out of bounds, the clip function can clip it to the boundary value, thereby constraining the difference between the old and new strategies and preventing training divergence.

[0075] Preferably, the target value is expressed as: in, express The value of a goal at any given moment; Represents the old value network in relation to the state space. Estimated state value.

[0076] Preferably, the convergence criteria for the first rate-limiting strategy model are: the immediate reward tends to stabilize within a number of consecutive training cycles (i.e., the difference between two adjacent training cycles is less than the threshold), or the loss function tends to stabilize and no longer decreases significantly, or the training cycle reaches the preset maximum number of training cycles.

[0077] Furthermore, in an ideal environment where traffic scenarios are covered by full-domain network communication and global information exchange (i.e., CAV vehicles can obtain global observation information), IPPO modeling can be optimized into a multi-agent proximal policy optimization algorithm (MAPPO) to obtain a second speed limit policy model. This utilizes global observation information to improve the rationality and coordination of the speed limit policy of the lead CAV vehicle, thereby optimizing the overall traffic control objectives.

[0078] At this point, the agent can obtain local observation information from each agent and overall traffic flow information (i.e., the position, speed, acceleration, and lane of all vehicles). Therefore, the global observation information is represented as: in, Represents global observation information; This refers to the set of local observation information from each intelligent agent (i.e., the lead CAV vehicle). ; The average speed of vehicles in the bottleneck area; The total number of intelligent agents within the control area; The average velocity of each agent; This represents the average risk level of the road (i.e., the average collision time index of each intelligent agent within the control area).

[0079] In this embodiment, the MAPPO modeling maintains consistency with the IPPO framework in the design of the action space, state transition probabilities, and reward function. However, the agent network modeling is optimized based on the CTDE training paradigm and the global state space. The policy network (Actor) retains the framework of a two-layer fully connected neural network and obtains the action distribution from local observations. The value network (Critic) incorporates global observation information. As output, it is extended to a three-layer fully connected neural network to adapt to high-dimensional global observation information and improve feature extraction capabilities. Finally, the output layer also uses global observation information as input to establish a global state value estimate, the expression of which is as follows: in, This represents the first fully connected layer of a value network. This represents the second fully connected neural network layer of the value network. This represents the third fully connected neural network layer of the value network. This represents the weight parameters of the first fully connected neural network layer. This represents the weight parameters of the second fully connected neural network layer. This represents the weight parameters of the third fully connected neural network layer; This represents the bias parameters of the first fully connected neural network layer. This represents the bias parameters of the second fully connected neural network layer. This represents the bias parameters of the third fully connected neural network layer; This indicates that the agent is based on global observation information. The expected cumulative return; The weights of the current value output layer. The bias parameters for the current value output layer. This represents the parameters of the value network in the second rate-limiting strategy model.

[0080] Furthermore, in the second speed-limiting strategy model, the state transition probability... Represented as: in, express Global observation information at any given time. express Global observation information at any given moment; This represents the actions performed by all agents at the current moment.

[0081] Preferably, in this embodiment, training the second speed limiting strategy model specifically involves: B1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and integrate global observation information. As a state space; B2. Set the action space of each agent to continuous acceleration adjustment commands, with the range of acceleration adjustment commands being: The unit of acceleration adjustment command is ; B3. Constructing the policy network for each agent. and value network The policy network is used to base its decisions on the agent's local observation information. Mean of the probability distribution of output actions with standard deviation And construct a Gaussian distribution strategy The value network is used to base its data on global observation information. Estimate the state value (i.e., the expected cumulative return) The strategy network structure is a two-layer fully connected neural network, and the value network structure is a three-layer fully connected neural network; wherein, This represents the parameters of the value network in the second rate-limiting strategy model. This represents the parameters of the policy network in the second speed limiting policy model; B4. Each agent, based on the current local observation information... Output actions through the policy network The probability of state transition after the action is performed. Transition to the next state; where, Represents intelligent agents exist Local observation information at a given time, Represents intelligent agents exist The action output at any time; B5. Calculate the immediate rewards obtained by each agent after performing an action. The state space before the action Current action being performed Instant rewards after an action and the state space after the action This experience is stored in the experience pool; among which... Represents intelligent agents Execute action The immediate reward afterwards; B6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. The mean squared error between the estimated state value and the target value of the value network is used as the loss function to update the value network parameters. ; B7. Repeat steps B4 to B6 until the second rate limiting strategy model converges.

[0082] Preferably, the training of the second speed-limiting policy model differs from that of the first speed-limiting policy model in that the value network input of the second speed-limiting policy model is global observation information. Furthermore, the value network is expanded into a three-layer fully connected neural network; training adopts a centralized training and distributed execution paradigm, meaning that during training, the value network can access global observation information to guide the policy network update, while during execution, each agent only relies on its own policy network and local observation information to output actions. Moreover, the shearing loss function and the calculation of the target value in the second rate-limiting policy model are the same as in the first rate-limiting policy model.

[0083] Preferably, the convergence criteria for the second rate limiting strategy model are: the immediate reward tends to stabilize within a number of consecutive training cycles (i.e., the difference between two adjacent training cycles is less than the threshold), or the loss function tends to stabilize and no longer decreases significantly, or the training cycle reaches the preset maximum number of training cycles.

[0084] The offline training phase is now complete. In the online control phase, the trained Mask-DQN model, the first speed limit strategy model, and the second speed limit strategy model are deployed on roadside equipment. Specifically, the roadside sensing equipment collects real-time traffic flow data (including vehicle position, speed, acceleration, and lane information) within the upstream control area of ​​the highway bottleneck region. This real-time traffic flow data is input into the trained Mask-DQN model, which then infers and outputs several optimal vehicle groups led by CAVs online. Subsequently, based on the type of information the leading CAVs can currently acquire, either the first or second speed limit strategy model is loaded: if the CAVs can only acquire local observation information, the first speed limit strategy model is loaded, and each agent outputs acceleration adjustment commands based on the local observation information; if the CAVs can acquire global observation information, the second speed limit strategy model is loaded, and each agent outputs acceleration adjustment commands based on the global observation information. After the lead CAV vehicle executes the acceleration adjustment command to adjust its speed, its speed change is transmitted to other vehicles in the group through following behavior, guiding the entire group to decelerate synchronously, thereby smoothing traffic flow fluctuations, reducing driving risks, and improving traffic efficiency in bottleneck areas.

[0085] Simulation Case: This embodiment constructs a simulation environment to compare and analyze three strategies: no control, variable speed limiting control based on the first speed limiting strategy model, and variable speed limiting control based on the second speed limiting strategy model. The selected analysis indicators include the cumulative TET value, the cumulative TIT value, and the travel time.

[0086] The cumulative TET value indicates that the agent is at a critical threshold throughout the entire experimental period. The cumulative duration of the following states, i.e., the total duration for which the agent is exposed to dangerous states, is expressed as: The cumulative TIT value indicates that TTC is at a critical threshold. The combined effect of severity and duration is expressed as follows: in, This represents the total number of steps in the simulation experiment; For characteristic functions, i.e. when The value is 1 if the condition is true, and 0 otherwise. It was a short time; For the first An intelligent agent in The collision time index at any given moment.

[0087] Depend on Figure 4 and Figure 5As can be seen from the changes in the cumulative TET and TIT values, the variable speed limit control in this embodiment effectively reduces the driving risk in the bottleneck area of ​​the highway. Compared with the scenario without control, the total cumulative risk value has decreased significantly.

[0088] Specifically, in the initial stage of the simulation, the cumulative risk values ​​corresponding to both the first and second speed-limiting strategy models increased to some extent compared to the uncontrolled scenario. However, in the later stages of the simulation, the risk value in the uncontrolled scenario continued to increase approximately linearly, while under the control of the method in this embodiment, the growth trend of the cumulative TET and TIT values ​​gradually slowed down and stabilized. This indicates that in the initial stage of the simulation, vehicles detected the formation of congestion in the bottleneck area through local or global observation and implemented speed-limiting strategies in advance to reduce the speed of the vehicle group. This process may have generated small speed disturbances within the vehicle group, leading to an early increase in the short-term risk index. Subsequently, by reducing the arrival flow in the bottleneck area, the large-scale generation of driving risks in the bottleneck area was effectively controlled. Furthermore, regarding the cumulative TET value, the first and second speed-limiting strategy models showed similar control effects, but in terms of TIT, the cumulative value corresponding to the second speed-limiting strategy model was significantly smaller than that of the first speed-limiting strategy model, demonstrating better performance in reducing the occurrence of high-risk events. This indicates that the second speed limit strategy model, based on global observation information, can help vehicle groups make more rational speed coordination and control decisions, resulting in a safer headway, reduced severity of potential conflicts, and a lower risk level. Furthermore, this result demonstrates that, compared to the first speed limit strategy model, the second model has stronger forward-looking perception and real-time adjustment capabilities, enabling it to identify traffic condition evolution trends earlier and implement speed limit control in a timely manner, thereby avoiding more collision risks.

[0089] Figure 6 This demonstrates the travel time distribution under three scenarios: no control, variable speed limiting control based on a first speed limiting strategy model, and variable speed limiting control based on a second speed limiting strategy model. Figure 6 It can be seen that the average vehicle travel time without control is around 220 seconds, and the median position in the box plot also indicates that the time required for more vehicles to pass through the bottleneck is more than 220 seconds. This shows that under no intervention conditions, congestion development in the bottleneck area significantly prolongs vehicle travel time. In contrast, the first and second speed limit strategy models significantly reduce the overall travel time, with the average travel time for both models dropping to around 180 seconds. This demonstrates that the method in this embodiment has a significant effect on improving traffic efficiency in the bottleneck area.

[0090] Meanwhile, in the uncontrolled scenario, the travel time distribution is relatively wide, roughly ranging from 180 to 260 seconds, reflecting significant fluctuations in traffic flow. Under speed control using the first and second speed limit strategy models, the travel time is concentrated in the 170-200 second range, exhibiting a more stable traffic flow. This phenomenon indicates that in the uncontrolled scenario, vehicles experience a gradual increase in travel time due to the spread of congestion in bottleneck areas as traffic flow evolves, with significant differences in travel time between different vehicles. Under the control of the first and second speed limit strategy models, the vehicle group can achieve a more regular and gradual process of reaching the bottleneck through coordinated speed limits, thereby effectively improving the overall traffic efficiency in the bottleneck area.

[0091] Example 2: This embodiment provides a dynamic speed limit control method for a CAV (Carrier Automobile) navigator group based on multi-agent reinforcement learning, including an offline training phase and an online control phase. Unlike Embodiment 1, this embodiment only trains and deploys the first speed limit policy model, specifically: The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. When the intelligent agent can only obtain local observation information, the first speed limit strategy model is trained by using the local observation information as input and the acceleration adjustment command of the intelligent agent as output, and the independent proximal strategy optimization algorithm is used to train the first speed limit strategy model. The online management and control phase includes: Collect real-time traffic flow data within the upstream control area of ​​the bottleneck area, input the real-time traffic flow data into the trained Mask-DQN model, and obtain several optimal vehicle groups led by CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent, and the first speed limit strategy model is used to output acceleration adjustment commands in real time.

[0092] The optimal vehicle group division and the training of the first speed limit strategy model in this embodiment are the same as in Embodiment 1. Therefore, this embodiment will not provide a detailed explanation of the specific technical details.

[0093] Example 3: This embodiment provides a dynamic speed limit control method for a CAV (Carrier Automobile) navigator group based on multi-agent reinforcement learning, including an offline training phase and an online control phase. Unlike Embodiment 1, this embodiment only trains and deploys a second speed limit strategy model. Specifically: The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The leading CAV vehicles in each lane of a single optimal vehicle group are treated as a single agent. With the agent able to obtain global observation information, the second speed limit strategy model is trained using the global observation information as input and the agent's acceleration adjustment command as output, and the multi-agent proximal strategy optimization algorithm is used to train the second speed limit strategy model. The online management and control phase includes: Collect real-time traffic flow data within the upstream control area of ​​the bottleneck area, input the real-time traffic flow data into the trained Mask-DQN model, and obtain several optimal vehicle groups led by CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent, and the second speed limit strategy model is used to output acceleration adjustment commands in real time.

[0094] The optimal vehicle group division and the training of the second speed limit strategy model in this embodiment are the same as in Embodiment 1. Therefore, this embodiment will not provide a detailed explanation of the specific technical details.

[0095] Example 4: This embodiment provides an optimal vehicle group segmentation method for dynamic speed limit control of vehicle groups, including an offline training phase and an online control phase; The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The online management and control phase includes: Real-time traffic flow data is collected from the upstream control area of ​​the bottleneck region. The real-time traffic flow data is then input into the trained Mask-DQN model to obtain several optimal vehicle groups led by CAV vehicles.

[0096] The specific details of the optimal vehicle group division in this embodiment are the same as those in Embodiment 1. Therefore, this embodiment will not provide a detailed explanation of the specific technical details.

[0097] Example 5: This embodiment provides a dynamic speed limit control system for a CAV (Cross-Agent Vehicle) navigator group based on multi-agent reinforcement learning, including a memory and a processor. The memory stores a computer program, and when the processor runs the computer program, it executes the dynamic speed limit control method for a CAV navigator group based on multi-agent reinforcement learning in Embodiment 1.

[0098] Example 6: This embodiment provides a storage medium storing a computer program, which, when executed by a processor, performs the dynamic speed limit control method for CAV vehicle navigator groups based on multi-agent reinforcement learning in Embodiment 1.

[0099] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning, characterized in that, It includes an offline training phase and an online management phase; The offline training phase includes: The road segment is divided into a first traffic area, a second traffic area, and a bottleneck area along the direction of vehicle travel. Car-following behavior models and lane-changing behavior models are constructed for CAV vehicles and HDV vehicles, respectively. The optimal vehicle group size is calculated based on the maximum traffic capacity of the bottleneck area, and the vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV vehicle navigation principle. The initial vehicle group set is obtained by first dividing the simulated traffic flow data within the upstream control range of the bottleneck area. The feature vector of the vehicle group set is used as the state space, the merging, splitting and migration between vehicle groups are used as the action space, the single-step improvement of the vehicle group coefficient is used as the immediate reward, and the Mask-DQN model is trained with the goal of maximizing the vehicle group coefficient. The Mask-DQN model is used to output the optimal vehicle group led by several CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. When the agent can only obtain local observation information, the first velocity limiting policy model is trained by using the local observation information as input and the agent's acceleration adjustment command as output, and the independent proximal policy optimization algorithm is used. When the agent can obtain global observation information, the second speed limiting policy model is trained by using the global observation information as input and the agent's acceleration adjustment command as output, and the multi-agent proximal policy optimization algorithm is used. The online management phase includes: Collect real-time traffic flow data within the upstream control area of ​​the bottleneck area, input the real-time traffic flow data into the trained Mask-DQN model, and obtain several optimal vehicle groups led by CAV vehicles. The lead CAV vehicles in each lane of a single optimal vehicle group are treated as a single intelligent agent. Based on the type of information that the intelligent agent can currently obtain, either the first speed limit strategy model or the second speed limit strategy model is adopted to output acceleration adjustment commands in real time.

2. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 1, characterized in that, The lane-changing behavior models for CAV vehicles and HDV vehicles are the same, where the lane-changing motivation is expressed as: Lane change safety verification is represented as follows: in, For lane changing motives in the first traffic zone, The lane-changing motive for the second passage area; 0 indicates no lane-changing motive, 1 indicates free lane-changing motive, and 2 indicates forced lane-changing motive. The speed difference between the vehicle in front and the vehicle in front; The speed difference threshold; For the exit lane, To reduce the number of lanes; This marks the boundary between the upstream and downstream sections of the second passage area. This is the current location of this vehicle. This vehicle, The position of vehicles behind the target lane; Position of the vehicle ahead in the target lane; This is the minimum acceptable clearance for lane changes.

3. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 1, characterized in that, The optimal vehicle group size is calculated based on the maximum capacity of the bottleneck area, specifically: in, To achieve the optimal vehicle group size; This represents the maximum traffic capacity of the bottleneck area. The expected speed of the vehicle group in the bottleneck area; The length of the road occupied by the group of vehicles; The desired spacing between the vehicles; The critical density of the bottleneck region; The critical velocity for the bottleneck region; The vehicle lane-changing rate for the downstream section of the second traffic zone; This is the capacity loss coefficient caused by lane-changing conflicts.

4. The dynamic speed limit control method for CAV vehicle navigator groups based on multi-agent reinforcement learning according to claim 1, characterized in that, The vehicle group coefficient is constructed based on the optimal vehicle group size and the CAV (Caravan Leader) vehicle navigation principle. Specifically: in, For vehicle group coefficient, Matching item to vehicle group size; For CAV vehicle navigation gain; To achieve the optimal vehicle group size; For the first The size of a vehicle group; The number of vehicles in the group; The penetration rate of CAV vehicles within the upstream control area of ​​the bottleneck region; For the first CAV vehicle navigation gain coefficient for a vehicle group; The standard deviation is Gaussian.

5. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 1, characterized in that, The specific steps for training the first speed limiting policy model are: A1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and use the local observation information of the intelligent agent as the state space; A2. Set the action space of each agent to continuous acceleration adjustment commands; A3. Constructing the policy network for each agent. and value network The policy network It is used to output the mean and standard deviation of the probability distribution of actions based on local observation information, and to construct a Gaussian distribution strategy; A4. Each agent, based on its current state space, accesses the policy network... Output the action, and after the action is executed, the state transition probability will be transferred to the next state space; A5. Calculate the immediate reward obtained by each agent after performing an action, and store the state space before the action, the current action, the immediate reward after the action, and the state space after the action as experience into the experience pool. A6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. value network The mean squared error between the estimated state value and the target value is used as the loss function to update the value network parameters. ; A7. Repeat steps A4 to A6 until the first rate limiting strategy model converges.

6. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 5, characterized in that, The specific steps for training the second speed limiting policy model are as follows: B1. Treat the lead CAV vehicles in each lane of a single optimal vehicle group as a single intelligent agent, and use global observation information as the state space; B2. Set the action space of each agent to continuous acceleration adjustment commands; B3. Constructing the policy network for each agent. and value network The policy network It is used to output the mean and standard deviation of the action probability distribution based on the local observation information of the agent, and to construct a Gaussian distribution strategy; B4. Each agent, based on current local observation information, communicates through the policy network. Output the action, and after the action is executed, the state transition probability will be transferred to the next state space; B5. Calculate the immediate reward obtained by each agent after performing an action, and store the state space before the action, the currently executed action, the immediate reward after the action, and the state space after the action as experience into the experience pool. B6. Sample experience from the experience pool to calculate the advantage function, and update the policy network parameters using the shearing loss function optimized by the near-end policy. value network The mean squared error between the estimated state value and the target value is used as the loss function to update the value network parameters. ; B7. Repeat steps B4 to B6 until the second rate limiting strategy model converges.

7. The dynamic speed limit control method for a CAV (Cardboard Automation Vehicle) navigator group based on multi-agent reinforcement learning according to claim 6, characterized in that, The immediate rewards during the training of the first and second speed-limiting policy models are both represented as: in, for Time-based intelligent agent Instant rewards; for Time-based intelligent agent The discrete form of the distance-collision time; The target speed for the vehicle group to pass through; for Time-based intelligent agent The vehicle speed; and All are weighting coefficients.

8. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 6, characterized in that, State transition probabilities during training of the first speed limiting policy model Represented as: State transition probabilities during training of the second speed limiting policy model Represented as: in, express Local observation information of all agents at any given moment; express Local observation information of all agents at any given time, ; express Actions performed by all agents at any given moment; Indicates the total number of agents; Indicates the first A smart agent Local observation information at any given time; express Global observation information at any given time. express Global observation information at any given moment.

9. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 1, characterized in that, The local observation information of the agent is represented as: in, For intelligent agents Local observation information; For intelligent agents Its own speed; Represents intelligent agents The speed difference between the last car in the previous optimal car group; For intelligent agents The distance between the vehicle and the last vehicle in the previous optimal vehicle group; For intelligent agents Collision time index; For intelligent agents The size of the vehicle group it belongs to; For intelligent agents The average speed of vehicles within the observation range.

10. The dynamic speed limit control method for a CAV (Continuous Aerial Vehicle) navigator group based on multi-agent reinforcement learning according to claim 1, characterized in that, The global observation information is represented as follows: in, Represents global observation information; It is a set of local observation information of each intelligent agent; The average speed of vehicles in the bottleneck area; The total number of intelligent agents within the control area; The average velocity of each agent; This represents the average risk level of the road.

Images