A dynamic group enhanced deep reinforcement learning ramp-merging cooperative decision and control method

Abstract

This invention belongs to the field of intelligent traffic control technology, specifically relating to a dynamic grouping-enhanced deep reinforcement learning-based collaborative decision-making and control method for ramp merging, applicable to highway ramp merging scenarios for intelligent connected autonomous vehicles. First, a virtual queue is constructed based on the predicted arrival times of vehicles, and a dynamic grouping threshold is adaptively calculated to group adjacent vehicles. Second, a grouping collaborative controller assigns a target passage time to each group and generates a target acceleration guidance control signal. Finally, the state information and grouping features are input into a deep reinforcement learning module, trained using a reward function containing a grouping guidance penalty term, and the resulting acceleration control command is output. This invention provides merging gaps for ramp vehicles through dynamic grouping, and the grouping collaborative controller generates a target acceleration guidance signal, improving training stability and decision interpretability. By incorporating the grouping guidance penalty term into the reward function and embedding collaborative scheduling knowledge into the training process, the consistency of collaborative decision-making is improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent traffic control technology, specifically to a dynamic grouping-enhanced deep reinforcement learning method for collaborative decision-making and control of ramp merging. Background Technology

[0002] The merging area of ​​highway ramps is a critical node where mainline traffic and ramp traffic converge. Due to the road geometry and the interweaving of traffic flows, this area has long been a high-incidence zone for traffic conflicts and congestion. With the continuous increase in highway traffic volume, the traffic pressure in the merging area of ​​ramps is becoming increasingly prominent. Traditional ramp control methods mainly rely on traffic light timing control and variable speed limit signs, which depend on preset timing rules or fixed thresholds to intervene in traffic flow. While these methods are applicable to low-flow or stable traffic scenarios, they are difficult to dynamically adapt to real-time traffic changes under high-flow and complex road conditions. In recent years, the rapid development of intelligent connected and autonomous driving vehicle technology has provided a new technical path for ramp merging collaborative control. With the help of vehicle-to-everything (V2X) communication technology, real-time status information sharing and collaborative decision-making can be achieved between mainline and ramp vehicles. Deep reinforcement learning has been introduced into this field due to its adaptive decision-making capabilities in complex dynamic environments. Intelligent agents autonomously learn control strategies through continuous interaction with the simulation environment, which has become an important direction in current research on ramp merging collaborative control.

[0003] However, existing technologies still have significant shortcomings in the following aspects: First, existing ramp merging collaborative control methods generally lack a structured grouping management mechanism for vehicles within the control area, treating all vehicles as discrete individuals and failing to identify and utilize the inherent correlation of highly similar behaviors between adjacent vehicles with close time intervals. This leads to a sharp expansion of the decision state space dimension with the number of vehicles, resulting in low system scheduling efficiency and an inability to provide structured merging gaps for ramp vehicles. Second, existing deep reinforcement learning-based control methods rely solely on data-driven methods to output acceleration control commands, lacking a group collaborative controller based on vehicle kinematic constraints to provide structured guidance for the agent's decision-making process. In the early stages of training, the agent is prone to policy oscillations and slow convergence speed due to the lack of prior physical constraints, resulting in insufficient physical interpretability of the obtained control strategy. Third, the reward function design of existing deep reinforcement learning methods is mainly guided by traffic efficiency, failing to integrate group collaborative scheduling knowledge into the reward mechanism. The agent's learning process lacks guidance and constraints from the group collaborative control level, making it difficult to organically combine the physical laws of vehicle group scheduling with the strategy optimization process of reinforcement learning, resulting in insufficient consistency between the output control strategy and the group collaborative scheduling results.

[0004] Therefore, this invention proposes a dynamic grouping-enhanced deep reinforcement learning method for collaborative decision-making and control of ramp inflow. Summary of the Invention

[0005] To address the technical problems mentioned in the background section, such as the lack of a structured vehicle grouping management mechanism in existing technologies, the lack of structured guidance from the grouping collaborative controller during the deep reinforcement learning training process leading to policy oscillations and insufficient interpretability, and the lack of integration of grouping collaborative scheduling knowledge into the reward function resulting in insufficient consistency between the control strategy and the grouping scheduling results, the present invention aims to provide a dynamic grouping-enhanced deep reinforcement learning-based ramp merging collaborative decision-making and control method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A dynamic grouping-enhanced deep reinforcement learning-based collaborative decision-making and control method for ramp inflow includes: S1: Obtain the real-time status information of all networked autonomous vehicles in the target area, calculate the predicted arrival time of each vehicle at the ramp merging point, and sort all vehicles according to the predicted arrival time to form a virtual queue. S2: Based on the current average vehicle speed, adaptively calculate the dynamic grouping threshold, and sequentially check the time interval between adjacent vehicles in the virtual queue. When the time interval is less than the dynamic grouping threshold, the corresponding adjacent vehicles are assigned to the same vehicle group, thus dynamically forming several vehicle groups. S3: Treat each vehicle group as a whole unit through the group coordination controller, assign a target passage time to each vehicle group, and generate a target acceleration for each vehicle based on the target passage time as a group-level guidance control signal; S4: Input the real-time status information and the vehicle group features into the deep reinforcement learning decision module, and train it using a reward function that includes a group guidance penalty term. The group guidance penalty term constrains the acceleration control command output by the deep reinforcement learning decision module to be consistent with the target acceleration, and outputs the final acceleration control command for each vehicle.

[0007] Furthermore, the target area includes the main line of the highway and the corresponding ramp merging area, and all vehicles in the target area are connected and automated vehicles with communication and automatic control capabilities. By deploying roadside units within the merging area of ​​the ramp or sharing information via vehicle-mounted terminals, the position, speed, acceleration, and distance between vehicles are periodically collected to form the real-time status information.

[0008] Furthermore, step S1 specifically includes: The predicted arrival time is calculated based on the current speed of each vehicle and the physical distance of each vehicle from the ramp merging point, and the denominator of the calculation is protected by a preset minimum speed threshold. Ramp vehicles are projected onto the mainline queue according to their predicted arrival times, and then sorted together with the mainline vehicles according to their predicted arrival times to form the virtual queue that satisfies the non-decreasing constraint of predicted arrival times. The virtual queue serves as the input sequence for the dynamic grouping execution in step S2.

[0009] Furthermore, in step S2, the dynamic grouping threshold is based on the basic headway threshold, and is superimposed with the product of the speed sensitivity coefficient and the difference between the preset target speed and the average speed of vehicles in the current control area. The calculation result is constrained between the preset minimum grouping threshold and the maximum grouping threshold by a limiting function. When the average speed of vehicles in the control area is lower than the target speed, the dynamic grouping threshold is increased to accommodate more vehicles into the same group. When the average speed is close to the target speed, the dynamic grouping threshold approaches the larger value between the basic headway threshold and the preset minimum grouping threshold.

[0010] Furthermore, after completing the initial grouping in step S2, the method also includes constraint processing on the number of vehicle groups, specifically: The total number of current vehicle groups is counted. If it exceeds the preset maximum number of groups, the dynamic grouping threshold is gradually increased by a preset step size, and the grouping determination is repeated until the total number of vehicle groups is not greater than the maximum number of groups. If the constraint is still not met when the dynamic grouping threshold is increased to the upper limit of the system's absolute safety time interval, the current grouping result is forcibly retained as the input for the scheduling in step S3.

[0011] Furthermore, in step S3, the process by which the group cooperative controller allocates target transit time includes: Extract the predicted arrival times of all vehicles in each vehicle group and calculate the group average predicted arrival time. Using the group's average predicted arrival time, the number of vehicles in the group, and the target speed as inputs, an optimal group-level target passage time that satisfies the inter-group safety interval constraint is assigned to each vehicle group through a timetable lookup strategy based on vehicle kinematic constraints. Starting from the time when the group-level target passes through, a preset safe following time and sight distance are superimposed sequentially according to the driving order of vehicles in the group, which is mapped to the final target passing time of each vehicle in the group. The final target passing time of each vehicle serves as the basis for the calculation of target acceleration generated by the group cooperative controller.

[0012] Furthermore, after completing the target passage time allocation, the group cooperative controller further generates the target acceleration for each vehicle, which is composed of the superposition of a speed matching term and a time synchronization term: The speed matching term is calculated based on the difference between the vehicle's current speed and the preset target speed, and is used to guide the vehicle to converge toward the preset target speed; The time synchronization term is calculated based on the difference between the vehicle's own predicted arrival time and the average predicted arrival time of the vehicle group, and is used to guide the vehicle to keep synchronized with the average predicted arrival time of the vehicle group. The target acceleration is output after normalization to ensure that the target acceleration is within the acceleration range allowed by the vehicle actuator, and serves as a reference benchmark for the grouped guidance penalty item in step S4.

[0013] Furthermore, step S4 specifically includes: In constructing the state space and action space, the state vector is formed by the longitudinal position, speed and lane type of up to a preset number of vehicles in the control area. When the number of vehicles is insufficient, the missing positions are filled with zero or masked. The action vector is formed by the longitudinal acceleration control commands of each vehicle, and the acceleration is constrained within a preset physical range. The reward function is constructed by weighting three parts: total driving time reward, time allocation penalty, and group guidance penalty. The group guidance penalty is quantified by the root mean square error between the acceleration control command output by the deep reinforcement learning decision module and the target acceleration generated by the group cooperative controller in step S3. The deviation is included in the policy optimization objective with negative weights, so as to guide the reinforcement learning training to a control policy space consistent with the target acceleration of the group cooperative controller. In training execution, the state, action, reward, and state of the next moment at each control moment are stored in the experience pool as experience samples. The proximal policy optimization algorithm is used to iteratively update the policy network parameters by editing the objective function until the policy converges and the final longitudinal acceleration control command of each vehicle is output.

[0014] A dynamic grouping-enhanced deep reinforcement learning ramp merging collaborative decision-making and control device, comprising: The information acquisition module is used to acquire real-time status information of all networked autonomous vehicles within the target area; The group planning module is used to calculate the predicted arrival time of each vehicle at the ramp merging point based on the real-time status information and form a virtual queue, perform adaptive grouping on the virtual queue based on a dynamic grouping threshold to obtain several vehicle groups, and assign target passage time to each vehicle group and generate corresponding target acceleration through the group coordination controller. The control decision module is used to input the real-time state information and the vehicle group features into the deep reinforcement learning decision module, train the deep reinforcement learning decision module based on the reward function containing the group guidance penalty term, and output the final acceleration control command for each vehicle.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 8.

[0016] Compared with the prior art, the advantages of the present invention are as follows: 1. This invention constructs a virtual queue based on predicted arrival time and combines a speed-adaptive dynamic grouping threshold and a maximum group number constraint processing mechanism to identify and divide adjacent vehicles with similar behaviors within the control area into several vehicle groups. This enables vehicles within a group to coordinate closely and maintain sufficient safe time slots between groups, solving the problems of existing methods lacking structured grouping management of vehicles and the expansion of the scheduling state space with the number of vehicles. Thus, it realizes structured grouping management of vehicles within the control area and provides clear and usable merging gaps for ramp vehicles.

[0017] 2. This invention assigns target passage times to each vehicle group based on vehicle kinematic constraints through a grouped collaborative controller, and generates target accelerations with clear physical meaning for each vehicle by combining speed matching terms and time synchronization terms. These accelerations serve as group-level guided control signals input to the deep reinforcement learning decision module. This solves the problems of policy oscillation, slow convergence, and insufficient interpretability of control strategies in the early stages of training caused by the lack of structured prior constraints in existing deep reinforcement learning methods. It guides the agent's decision-making to a control strategy space that conforms to vehicle dynamics constraints, thereby improving training stability and the physical interpretability of control actions.

[0018] 3. This invention introduces a group-guided penalty term into the reward function to quantify the deviation between the acceleration control command output by the deep reinforcement learning model and the target acceleration generated by the group-coordinated controller. The deviation is then incorporated into the policy optimization objective with negative weights. This explicitly integrates group-coordinated scheduling knowledge into the deep reinforcement learning training process, solving the problems of existing methods where the reward function does not incorporate group-coordinated scheduling knowledge and the consistency between the control strategy and the group scheduling results is insufficient. This guides the agent to learn a control strategy consistent with the group-coordinated control objective, thereby improving the overall consistency of collaborative decision-making and the system coordination efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the workflow of the method of the present invention; Figure 2This is a schematic diagram illustrating the dynamic grouping and merging of vehicles according to the present invention; Figure 3 This is a comparison of the average reward value convergence curves during training of the method of this invention and the benchmark deep reinforcement learning method. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] To achieve the above objectives, this invention provides a dynamic grouping-enhanced deep reinforcement learning-based collaborative decision-making and control method for ramp inflow, such as... Figures 1-3 As shown, the method includes: S1: Obtain the real-time status information of all networked autonomous vehicles in the target area, calculate the predicted arrival time of each vehicle at the ramp merging point, and sort all vehicles according to the predicted arrival time to form a virtual queue.

[0023] S101: This method is applied to the ramp merging scenario in a fully connected autonomous vehicle environment. It assumes that all vehicles entering the control area on the ramp and mainline are connected autonomous vehicles with communication and automatic control capabilities. Roadside units are deployed in the ramp merging control area. Alternatively, information can be shared through vehicle-mounted terminals to periodically obtain real-time status information of all relevant vehicles within the control area; The real-time status information specifically includes the position, speed, acceleration of each vehicle, and the distance between vehicles. At the same time, the system predetermines the absolute position of the merging point and calculates the physical distance of each vehicle in the control area from the merging point based on the acquired position information. All of the above status data will serve as the basis for subsequent collaborative decision input. S102: For each vehicle within the control area, based on the acquired real-time status information, calculate the predicted arrival time at the merging point for each vehicle using its current speed and physical distance from the ramp merging point; for any tested vehicle within the control area... The calculation formula is: in, Indicates the vehicle being tested Predict the time to reach the ramp merging point; Indicates the vehicle being tested Physical distance from the merging point; Indicates the vehicle being tested The current speed; This represents the preset minimum speed threshold, used to avoid abnormal predicted values ​​caused by a denominator that is zero or too small. In this embodiment, based on the well-known low-speed protection and numerical stability processing principles in existing intelligent connected vehicle trajectory prediction, vehicle motion state estimation, and longitudinal control of autonomous driving, the minimum speed threshold is... The preferred value range is defined as 0.1-1.0 m / s. Those skilled in the art can determine the specific value within this range based on the ramp queue length, vehicle start-up response characteristics, and system sampling period. In this embodiment, The preferred value is 0.5 m / s; S103: After calculating the predicted arrival times of all vehicles within the control area, the ramp vehicles are projected onto the mainline queue according to their predicted arrival times, and then sorted together with the mainline vehicles according to their predicted arrival times to form a virtual queue arranged in order of predicted arrival times. The mathematical expression is ;in, to This represents the vehicles after they have been sorted in order. This represents the total number of vehicles currently participating in scheduling in the virtual queue, and all vehicles in the queue satisfy the time constraint condition: .

[0024] S2: Based on the current average vehicle speed, adaptively calculate the dynamic grouping threshold, and sequentially check the time interval between adjacent vehicles in the virtual queue. When the time interval is less than the dynamic grouping threshold, the corresponding adjacent vehicles are assigned to the same vehicle group, thus dynamically forming several vehicle groups.

[0025] S201: Obtain the virtual queue The vehicles in the queue have arrived at the merging point according to the predicted time. A global sorting was completed to serve as the input sequence for subsequent adaptive dynamic grouping; S202: Obtain the speed status information of vehicles within the current observation window to calculate the dynamic grouping threshold. The calculation formula is: in, This is a dynamic grouping threshold; The basic headway threshold; The velocity sensitivity coefficient; To preset the target vehicle speed; To control the average speed of all relevant vehicles within the area; and These are the lower and upper limits of the dynamic grouping threshold, respectively; This represents the limiting function, used to constrain the calculation results within a preset range; In this embodiment, based on existing connected autonomous driving cooperative adaptive cruise control... The well-known safe headway standard, the basic headway threshold The preferred value range is consistently defined between 0.6 and 1.5 seconds. Those skilled in the art can determine the specific value within this range based on whether the system is in a high-efficiency traffic scenario approaching the lower limit or a high-safety-margin scenario approaching the upper limit; this is based on the well-known intelligent driver car-following model in classical traffic flow theory. Dynamic compensation mechanism, the speed sensitivity coefficient The preferred value range is defined as between 0.01 and 0.1. The speed sensitivity coefficient described in this embodiment... The dimensions are The preferred value is 0.05; In this embodiment, based on the known safe time distance and acceptable merging gap design principles in existing highway ramp merging collaborative control, platoon following control, and traffic flow gap acceptance theory, the lower limit of the dynamic grouping threshold is... The time between the front and rear of the vehicle should not be less than the minimum acceptable safe headway of the system, preferably within the range of 0.8-1.2 seconds, and in this embodiment, a value of 1.0 second is preferred; the upper limit of the dynamic grouping threshold. The time interval should not exceed the maximum effective inter-group discrimination time interval set to ensure traffic efficiency in the scenario of ramp merging. The preferred value range is between 2.0 and 3.0 seconds, and the preferred value in this embodiment is 2.5 seconds. S203: Check the virtual queues sequentially. The time interval between adjacent vehicles is defined as the time interval between the vehicles in front. With the car behind The predicted arrival time interval between two adjacent workshops is The expression is: ,and ; When the time interval between two adjacent vehicles Less than the dynamic grouping threshold At that time, that is The two adjacent vehicles are assigned to the same vehicle group; when the time interval between the two adjacent vehicles... Not less than the dynamic grouping threshold At that time, that is Then in the vehicle and The division is made by cutting off the vehicle. As the last vehicle in the current vehicle group, and by Start a new trainset; The virtual queues are traversed sequentially. All vehicles in the dataset are subjected to adaptive grouping based on the above conditions, resulting in several groups of vehicles with similar behaviors. S204: Finally, count the total number of vehicle groups obtained from the current division. If the number of vehicle groups after grouping exceeds the preset maximum number of groups, then introduce a preset step size and gradually increase the dynamic grouping threshold by the preset step size. And require the enlarged version The absolute safe time interval limit of the system must not be exceeded; subsequently, the updated time interval will be used. Repeat the grouping determination process in step S203 above until the final number of vehicle groups is not greater than the maximum number of groups. If the absolute upper limit is reached and the grouping constraint still cannot be met, the last grouping result will be forcibly taken. In this embodiment, based on the physical space constraints of the ramp merging control area and the communication scheduling window capacity, the preset maximum number of groups is preferably defined within a constant range of 3-5 groups. Those skilled in the art can determine the value within this range based on the actual traffic flow density of the main line. In this embodiment, the preferred value is 4 groups. Based on the maximum critical condition for the stability of connected vehicle platooning in classical traffic flow theory, the upper limit of the absolute safe time distance of the system is preferably defined within a range of 2.0-3.0 seconds. In this embodiment, the preferred value is 2.5 seconds. Based on the underlying standard control cycle of autonomous vehicles and the conventional communication frequency of vehicle-to-everything (V2X) networks, to ensure the smoothness of algorithm iteration, the balance between grouping accuracy and system computational load, the preferred value of the preset step size is defined within a range of 0.05-0.15 seconds. In this embodiment, the preferred value is 0.1 seconds.

[0026] S3: Treat each vehicle group as a whole unit through the group coordination controller, assign a target passage time to each vehicle group, and generate a target acceleration for each vehicle based on the target passage time as a group-level guidance control signal.

[0027] S301: Treat each vehicle group obtained in step S2 as a whole vehicle unit and perform unified scheduling. For any vehicle group obtained in step S2... ,in For the vehicle group number, , The maximum number of vehicle groups currently defined; extract the vehicle groups. The predicted arrival time of all vehicles at the merge point calculated in step S1. And calculate the vehicle group Predicted average time to reach the inlet point The calculation formula is: in, For vehicle group The total number of vehicles included; S302: A timetable lookup based on vehicle motion models is used to assign target times for each vehicle group to pass through the ramp merging point. Specifically, a group-level passing timetable is pre-constructed based on vehicle kinematic constraints, control area length, target speed, and safe headway. This timetable records candidate group-level passing times for different vehicle groups under different average predicted arrival times, group sizes, and operating conditions. During the lookup, the average predicted arrival time of the vehicle group, the number of vehicles within the group, and the target speed are used as input parameters to select the group-level target passing time that satisfies the safety interval constraint while also considering traffic efficiency; this is taken as the optimal group-level passing time for that vehicle group. ; Assuming that vehicles within the same vehicle group are not disturbed by external vehicles during merging, the system prioritizes the average time of the vehicle group. Determine the optimal passing order and group-level passing time for each vehicle group as a whole; then map the obtained global vehicle group passing order to the final target passing time of each specific vehicle within the corresponding vehicle group, using the following mapping formula: in, For vehicles The final assigned target passes through time; For the system for vehicle groups The optimal group-level throughput time for overall allocation; For vehicles In vehicle group Internal driving sequence number, and ; The preset safe following time and sight distance for vehicles within the group; In this embodiment, based on the timetable scheduling strategy based on the vehicle motion model and the typical communication and sensing physical range of the roadside equipment for the ramp merging area, the optimal group-level passing time allocated by the system for the entire vehicle group is preferably defined between 10.0 and 40.0 seconds. Those skilled in the art can perform specific calculations within this time boundary range based on the actual physical distance of the vehicle group from the merging point and its real-time speed. The basic calculation logic is the ratio of the physical distance of the vehicle group from the merging point to its current speed. In this embodiment, taking a typical vehicle group cruising smoothly at a preset target speed of 20.0 m / s, approximately 400 meters from the ramp merging point, as an example, the system directly calculates based on the quotient of the distance and speed, and the optimal group-level passing time allocated to this vehicle group is preferably 20.0 seconds. Based on the following stability theory of the cooperative adaptive cruise control system and the low-latency communication characteristics of connected vehicles, the preset safe following time line of sight for vehicles within the group is... The preferred value range is defined as 0.5-1.5 seconds. Those skilled in the art can make a specific value within this range based on the vehicle's braking performance and communication delay level. In this embodiment, the preferred value is 1.0 seconds. S303: The group coordination controller generates group-level guidance control signals for each vehicle in the vehicle group, wherein the guidance control signals are target accelerations; for vehicles belonging to the vehicle group... any vehicle Combined with the vehicle group The predicted average time to reach the merging point and the vehicle Its own real-time status, vehicle The formula for calculating the target acceleration is: in, This indicates that the group cooperative controller is responsible for the vehicle. Output target acceleration; Preset the target vehicle speed for the system; For vehicles The current speed; Indicates vehicle The vehicle group Predict the average time to reach the inflow point; Indicates vehicle It predicts the time it will take to reach the inflow point. This represents the preset speed matching gain coefficient, used to adjust the response weight for the vehicle to converge toward the target speed; This represents the preset vehicle spacing synchronization gain coefficient, which is used to adjust the response weight to keep the average arrival time of this vehicle synchronized with that of the vehicle group. It is a normalization function used to limit the calculated acceleration value within the safe physical boundaries allowed by the vehicle actuators and system; In this embodiment, based on the relevant design specifications for highway ramps and merging areas, such as the typical speed limit for ramps being set at 40-60 km / h, and combined with the actual average speed of traffic flow on the outer lane of the mainline at the merging point to ensure smooth merging, the system presets a target speed. The preferred value range is defined as 15.0-22.0 m / s, approximately 54-80 km / h, and the preferred value in this embodiment is 20.0 m / s; In this embodiment, based on the critical damping condition in linear feedback control theory that ensures system state convergence and avoids overshoot, the preset velocity matching gain coefficient... The preferred value range is defined as between 0.5 and 2.0. Those skilled in the art can determine the value within this range based on the response sensitivity of the vehicle's underlying actuator. The preset speed matching gain coefficient mentioned in this embodiment... The dimensions are The preferred value is 1.2. Based on the requirements for position / time error dissipation rate in multi-agent consensus cooperative control theory, to avoid excessive acceleration abrupt changes affecting ride comfort, the preferred value range for the preset vehicle distance synchronization gain coefficient is defined as 0.1-0.8. In this embodiment, the preset vehicle distance synchronization gain coefficient... The dimensions are The preferred value is 0.5; it should be noted that the preset vehicle distance synchronization gain coefficient The design values ​​should ensure that the contribution of the time synchronization term to acceleration does not exceed that of the speed matching term, so as to ensure that the response priority of speed adjustment is higher than that of time synchronization adjustment, and to avoid excessive acceleration abrupt changes that may affect ride comfort.

[0028] S4: Input the real-time status information and the vehicle group features into the deep reinforcement learning decision module, and train it using a reward function that includes a group guidance penalty term. The group guidance penalty term constrains the acceleration control command output by the deep reinforcement learning decision module to be consistent with the target acceleration, and outputs the final acceleration control command for each vehicle.

[0029] S401: At each control moment The system extracts the operational status information of connected autonomous vehicles within the control area and constructs the state space of a deep reinforcement learning model. To facilitate standardized input, the maximum number of vehicles participating in decision-making within the control area is pre-set. ,in, Design the corresponding hyperparameters for the state vector dimension; when the number of vehicles actually participating in decision-making within the control area is less than When missing vehicle positions, zero-value filling, preset invalid identifier filling, or masking are used to ensure that the input dimension of the deep reinforcement learning model remains consistent; the vehicle states are preferably sorted and input according to the longitudinal distance of the vehicle from the conflict point within the control area from near to far. During control time , No. The status of a vehicle is represented as follows: in, Indicates the first Vehicle under control The state vector; Indicates the first Vehicle under control The longitudinal position; Indicates the first Vehicle under control The speed of travel; Indicates the first Vehicle under control The discrete lane category identifier variable is used to characterize whether the vehicle is located in the mainline lane or the ramp lane; The states of each vehicle within the control area are combined in a preset order to form the control time. Environment status input: in, This indicates the deep reinforcement learning model at the control time. Received status input; The action space of a deep reinforcement learning model is defined as the set of longitudinal acceleration control variables for each vehicle; at control time... The output action is represented as: in, Indicates control time The action vector; This indicates the deep reinforcement learning model at the control time. The output of the first The longitudinal acceleration of the vehicle; To meet the physical constraints of the vehicle actuators, Limited to a preset acceleration range: in, This indicates the minimum allowed longitudinal acceleration for output; This indicates the maximum allowed longitudinal acceleration for output; Finally, the obtained state input and action output This is used for subsequent reward value calculation and policy network training; To ensure consistency between training and reward calculation of deep reinforcement learning models under fixed-dimensional input conditions, for invalid vehicle positions, it is preferable to use a masking mechanism to prevent them from participating in subsequent reward calculations and policy updates; or to simultaneously set the target acceleration and policy output acceleration at the corresponding positions to zero to avoid interference from empty positions on the reward function value. S402: Construct the overall reward function to ensure the deep reinforcement learning model improves traffic efficiency while maintaining consistency with the group collaborative control results in step S3. The total reward function is a weighted sum of the total driving time reward, the through time allocation penalty, and the group guidance penalty, expressed as: in, Indicates control time Total reward value; Indicates total driving time bonus items The corresponding weighting coefficients are used to adjust the degree of attention the model pays to the traffic efficiency optimization objective; Indicates the penalty item allocated through time. The corresponding weighting coefficients are used to adjust the model's attention to the timing coordination constraints of vehicle passage; Indicates group-guided penalty items The corresponding weighting coefficients are used to adjust the strength of the tracking constraint on the target acceleration by the model's output action; Indicates control time Total driving time bonus items; Indicates control time Penalty items for passing time allocation; Indicates control time Grouped guidance penalty items; In this embodiment, based on the existing technical principle of weighted design of multi-objective reward functions in deep reinforcement learning, the... , , The values ​​of each reward are typically set based on its dimensions, range, and the importance of the control objectives, with non-negative real numbers being preferred, and preferably satisfying the following conditions: Based on existing experience in tuning conventional parameters for reward shaping in continuous control tasks, the aforementioned The preferred value range is 0.4-0.7, the The preferred value range is 0.1-0.3, the The preferred value range is 0.2-0.4. In this embodiment, the preferred value is [value missing]. This aims to improve traffic efficiency while ensuring reasonable allocation of passage time and consistency in group guidance. Total driving time bonus This is used to characterize the overall traffic efficiency of the controlled area. The optimal value is constructed based on the cumulative running time of all valid vehicles within the controlled area. A larger reward value indicates a smaller total system delay and higher overall traffic efficiency. The expression is: in, Indicates control time The number of valid vehicles within the controlled area; The time allocation penalty item This is used to characterize the deviation between the predicted vehicle passage time and the target passage time allocation result. The larger the deviation, the more significant the deviation between the vehicle passage time sequence and the cooperative scheduling result, and the greater the penalty imposed. The expression is: in, Indicates vehicle The predicted arrival time at the ramp merging point; This indicates that step S3 is for vehicles. The assigned target passes through time; The group guidance penalty item The deviation between the control action output by the deep reinforcement learning model and the target acceleration calculated in step S3 is expressed as: in, The first step calculated in step S3 Vehicle under control The target acceleration; This represents the squared deviation between the output acceleration of deep reinforcement learning and the target acceleration; Based on the above expression, when and The closer they get, the more likely they are to be grouped and receive penalties. The value is small; when the deviation between the two increases, the group-guided penalty term is small. By increasing the value of the grouping guidance penalty term in the total reward function, the deep reinforcement learning model can be guided to learn action outputs consistent with the dynamic grouping guidance results. The final total reward value This is used to construct training samples and update policy network parameters; S403: During the control moment Deep reinforcement learning models are based on state input Output action vector Traffic environment execution action vector Subsequently, the vehicle status within the controlled area is updated to the status of the next moment. The control time is calculated according to the total reward function. Corresponding reward value ; Construct an empirical sample from a single state transition process: in, Indicates control time A generated empirical sample; Indicates control time The environmental conditions; Indicates control time The action output; Indicates control time Total reward value; Indicates control time The environmental conditions; continuously generated experience samples The data are sequentially stored in the experience pool, serving as the data source for subsequent training of the near-end policy optimization algorithm; S404: Sample a batch of experience samples from the experience pool, and iteratively update the policy network using the near-end policy optimization algorithm; to limit the variation between the old and new policies, a clipping objective function is used as the optimization objective, expressed as: in, This represents the clipping objective function of the policy network; Indicates the policy network parameters; This represents the expected value calculation for empirical samples within a sampling batch; Indicates control time Applying new and old strategies to actions The probability ratio; Indicates control time The estimated advantage; Indicates the editing factor; This represents the probability ratio. Limited to the range Inside; The probability ratio Represented as: in, Indicates the current state of the policy network. Down Output Action The probability distribution; This indicates that the policy network was in state before the update. Down Output Action The probability distribution; In this embodiment, based on existing techniques for parameterized representation of policy networks in deep reinforcement learning, the policy network parameters... The parameter set consists of connection weights and bias terms in a neural network. Its values ​​are typically determined through random initialization. To avoid excessively large initial network outputs leading to training instability, and considering general experience in weight initialization for deep neural networks, the policy network parameters are... The initial value range is preferably defined within Within this range, those skilled in the art can conventionally select based on state dimension, number of network layers, and activation function type. In this embodiment, a zero-mean small-range random initialization method is preferred, wherein the weight parameters are preferably initialized within a certain range. Between these parameters, the bias term is preferably initialized to 0. The above parameter initialization method is a conventional technique in existing neural network training, which can effectively avoid the problem of excessive gradient or output saturation in the early stage of training and improve the stability of policy iteration. In this embodiment, based on the existing technical principles of near-end strategy optimization algorithms, the clipping coefficients... Typically, a smaller positive number is chosen to control the magnitude of change between the old and new strategies, combined with existing... Based on the algorithm's experience with conventional parameter settings in continuous control tasks, the clipping coefficients... The preferred value range for the coefficient is defined as between 0.1 and 0.3. Those skilled in the art can select the coefficient within this range based on training stability, convergence speed, and control effect. The clipping coefficient described in this embodiment... The preferred value is 0.2, at which point the probability ratio is... The editing interval corresponds to ; By maximizing the clipping target function The update range of the policy network is limited to a preset range to avoid policy oscillations or training instability caused by excessive parameter changes during training. After multiple rounds of iterative training, when the policy network converges, it outputs the final longitudinal acceleration control command for each vehicle. It also controls the mainline vehicles and ramp vehicles to complete the coordinated merging according to the dynamic grouping guidance results.

[0030] This invention also provides a dynamic grouping-enhanced deep reinforcement learning ramp merging collaborative decision-making and control device, including an information acquisition module, a grouping planning module, and a control decision module. The information acquisition module is used to acquire real-time status information of all networked autonomous vehicles within a target area. The grouping planning module is used to calculate the predicted arrival time of each vehicle at the ramp merging point based on the real-time status information and form a virtual queue. Based on a dynamic grouping threshold, the virtual queue is adaptively grouped to obtain several vehicle groups. A grouping collaborative controller is used to assign target passage times to each vehicle group and generate corresponding target accelerations. The control decision module is used to input the real-time status information and the vehicle group features into the deep reinforcement learning decision module, train the deep reinforcement learning decision module based on a reward function containing a grouping guidance penalty term, and output the final acceleration control command for each vehicle.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0032] Figure 3 This diagram illustrates the comparison of the average reward convergence curves during training between the method of this invention and a benchmark deep reinforcement learning method that does not employ a dynamic grouping strategy. The horizontal axis represents the number of training epochs, and the vertical axis represents the normalized average reward value. Figure 3 It can be seen that the method using the dynamic grouping enhancement strategy of this invention exhibits a significantly faster convergence speed than the benchmark method in the early stages of training, and its average reward value continues to rise and tends to stabilize with the increase of training rounds. In contrast, the pure deep reinforcement learning benchmark method without the dynamic grouping strategy has a slower convergence speed, and its final convergence level is also lower than that of the method of this invention. The above results show that the dynamic grouping structured guidance mechanism and the introduction of the grouping guidance penalty term proposed in this invention can effectively improve the training efficiency of the deep reinforcement learning decision module, accelerate the convergence process of the policy network, and obtain a control strategy with higher passage efficiency and cooperative consistency.

[0033] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0034] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic group-enhanced deep reinforcement learning ramp-merging cooperative decision and control method, characterized in that, include: S1: Obtain the real-time status information of all networked autonomous vehicles in the target area, calculate the predicted arrival time of each vehicle at the ramp merging point, and sort all vehicles according to the predicted arrival time to form a virtual queue. S2: Based on the current average vehicle speed, adaptively calculate the dynamic grouping threshold, and sequentially check the time interval between adjacent vehicles in the virtual queue. When the time interval is less than the dynamic grouping threshold, the corresponding adjacent vehicles are assigned to the same vehicle group, thus dynamically forming several vehicle groups. S3: Treat each vehicle group as a whole unit through the group coordination controller, assign a target passage time to each vehicle group, and generate a target acceleration for each vehicle based on the target passage time as a group-level guidance control signal; S4: Input the real-time status information and the vehicle group features into the deep reinforcement learning decision module, and train it using a reward function that includes a group guidance penalty term. The group guidance penalty term constrains the acceleration control command output by the deep reinforcement learning decision module to be consistent with the target acceleration, and outputs the final acceleration control command for each vehicle.

2. The dynamic group-enhanced deep reinforcement learning ramp-merging cooperative decision and control method according to claim 1, characterized in that, The target area includes the main line of the highway and the corresponding ramp merging area. All vehicles in the target area are connected and automated vehicles with communication and automatic control capabilities. By deploying roadside units within the merging area of ​​the ramp or sharing information via vehicle-mounted terminals, the position, speed, acceleration, and distance between vehicles are periodically collected to form the real-time status information. 3.The dynamic group-enhanced deep reinforcement learning ramp-merging cooperative decision and control method according to claim 1, wherein, The S1 step specifically includes: The predicted arrival time is calculated based on the current speed of each vehicle and the physical distance of each vehicle from the ramp merging point, and the denominator of the calculation is protected by a preset minimum speed threshold. Ramp vehicles are projected onto the mainline queue according to their predicted arrival times, and then sorted together with the mainline vehicles according to their predicted arrival times to form the virtual queue that satisfies the non-decreasing constraint of predicted arrival times. The virtual queue serves as the input sequence for the dynamic grouping execution in step S2.

4. The method for collaborative decision-making and control of ramp merging using dynamic grouping enhancement deep reinforcement learning as described in claim 1, characterized in that, In step S2, the dynamic grouping threshold is based on the basic headway threshold, and is superimposed with the product of the speed sensitivity coefficient and the difference between the preset target speed and the average speed of vehicles in the current control area. The calculation result is constrained between the preset minimum grouping threshold and the maximum grouping threshold by a limiting function. When the average speed of vehicles in the control area is lower than the target speed, the dynamic grouping threshold is increased to accommodate more vehicles into the same group. When the average speed is close to the target speed, the dynamic grouping threshold approaches the larger value between the basic headway threshold and the preset minimum grouping threshold.

5. The method for collaborative decision-making and control of ramp merging using dynamic grouping enhancement deep reinforcement learning as described in claim 4, characterized in that, After completing the initial grouping in step S2, the process also includes constraint processing on the number of vehicle groups, specifically: The total number of current vehicle groups is counted. If it exceeds the preset maximum number of groups, the dynamic grouping threshold is gradually increased by a preset step size and the grouping determination is repeated until the total number of vehicle groups is not greater than the maximum number of groups. If the constraint is still not met when the dynamic grouping threshold is increased to the upper limit of the system's absolute safety time interval, the current grouping result is forcibly retained as the input for the S3 step scheduling.

6. The method for collaborative decision-making and control of ramp merging using dynamic grouping enhancement deep reinforcement learning as described in claim 1, characterized in that, In step S3, the process of the group cooperative controller allocating the target transit time includes: Extract the predicted arrival times of all vehicles in each vehicle group and calculate the group average predicted arrival time. Using the group's average predicted arrival time, the number of vehicles in the group, and the target speed as inputs, an optimal group-level target passage time that satisfies the inter-group safety interval constraint is assigned to each vehicle group through a timetable lookup strategy based on vehicle kinematic constraints. Starting from the time when the group-level target passes through, a preset safe following time and sight distance are superimposed sequentially according to the driving order of vehicles in the group, which is mapped to the final target passing time of each vehicle in the group. The final target passing time of each vehicle serves as the basis for the calculation of target acceleration generated by the group cooperative controller.

7. The method for collaborative decision-making and control of ramp merging using dynamic grouping enhancement deep reinforcement learning as described in claim 6, characterized in that, After completing the target passage time allocation, the group cooperative controller further generates the target acceleration for each vehicle, which is composed of the superposition of a speed matching term and a time synchronization term: The speed matching term is calculated based on the difference between the vehicle's current speed and the preset target speed, and is used to guide the vehicle to converge toward the preset target speed; The time synchronization term is calculated based on the difference between the vehicle's own predicted arrival time and the average predicted arrival time of the vehicle group, and is used to guide the vehicle to keep synchronized with the average predicted arrival time of the vehicle group. The target acceleration is output after normalization to ensure that the target acceleration is within the acceleration range allowed by the vehicle actuator, and serves as a reference benchmark for the grouped guidance penalty item in step S4.

8. The method for collaborative decision-making and control of ramp merging with dynamic grouping enhancement according to claim 1, characterized in that, The S4 step specifically includes: In constructing the state space and action space, the state vector is formed by the longitudinal position, speed and lane type of up to a preset number of vehicles in the control area. When the number of vehicles is insufficient, the missing positions are filled with zero or masked. The action vector is formed by the longitudinal acceleration control commands of each vehicle, and the acceleration is constrained within a preset physical range. The reward function is constructed by weighting three parts: total driving time reward, time allocation penalty, and group guidance penalty. The group guidance penalty is quantified by the root mean square error between the acceleration control command output by the deep reinforcement learning decision module and the target acceleration generated by the group cooperative controller in step S3. The deviation is included in the policy optimization objective with negative weights, so as to guide the reinforcement learning training to a control policy space consistent with the target acceleration of the group cooperative controller. In training execution, the state, action, reward, and state of the next moment at each control moment are stored in the experience pool as experience samples. The proximal policy optimization algorithm is used to iteratively update the policy network parameters by editing the objective function until the policy converges and the final longitudinal acceleration control command of each vehicle is output.

9. A dynamic grouping-enhanced deep reinforcement learning ramp merging collaborative decision-making and control device, characterized in that, include: The information acquisition module is used to acquire real-time status information of all networked autonomous vehicles within the target area; The group planning module is used to calculate the predicted arrival time of each vehicle at the ramp merging point based on the real-time status information and form a virtual queue, perform adaptive grouping on the virtual queue based on a dynamic grouping threshold to obtain several vehicle groups, and assign target passage time to each vehicle group and generate corresponding target acceleration through the group coordination controller. The control decision module is used to input the real-time state information and the vehicle group features into the deep reinforcement learning decision module, train the deep reinforcement learning decision module based on the reward function containing the group guidance penalty term, and output the final acceleration control command for each vehicle.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

Images