Road ramp behavior planning method and system, and computer readable storage medium

Abstract

This invention belongs to the field of intelligent driving technology, specifically disclosing a method, system, and computer-readable storage medium for behavior planning at road ramps. Addressing the challenge of balancing traffic efficiency and safety in existing mixed traffic ramp scenarios for connected autonomous vehicles, the invention first establishes a communication model and a multi-view safety prediction model, generating a multi-view graph structure and quantified values ​​of driving behavior safety risks. Then, the behavior planning problem is constructed as an optimization problem with the objective of minimizing total travel time, constrained by collision-free operation and communication quality, and transformed into a graph Markov decision process. Finally, a graph reinforcement learning network is used to solve the problem, outputting the optimal lane-changing operation. This invention effectively balances traffic efficiency and driving safety while ensuring zero collisions and reliable communication, significantly improving decision convergence speed and driving stability.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method, system, and computer-readable storage medium for planning behavior at road ramps. Background Technology

[0002] With the acceleration of urbanization, the urban population and the number of motor vehicles continue to grow, putting severe pressure on the transportation network. Especially at the merging sections of road ramps, due to the diverse driving purposes, complex driving behaviors, and the common presence of mixed traffic of manually driven vehicles (HV) and connected autonomous vehicles (CAV), the frequency of vehicle interaction has increased significantly, traffic conflicts occur frequently, and these areas have become typical areas of traffic congestion and accident-proneness.

[0003] Existing research has made some progress in vehicle behavior planning for exit ramp divergence sections. Early studies mainly relied on optimization motion models, using mathematical programming or control theory methods to establish and solve vehicle trajectory and behavior models. These methods have some applicability in highly structured traffic networks, ensuring the feasibility and smoothness of the planning results. However, optimization models typically assume precise knowledge of the environmental state and ignore complex vehicle interaction rules, making it difficult to handle mixed and heterogeneous traffic environment information. In scenarios where HV and CAV traffic coexist, their adaptability and robustness are significantly limited.

[0004] To overcome the aforementioned problems, researchers have recently begun to employ reinforcement learning (RL)-based methods to solve vehicle behavior planning problems in mixed traffic environments. RL, with its sequential decision-making capabilities, can learn optimal strategies to adapt to dynamic and uncertain traffic environments through continuous interaction with the environment. However, existing RL-based mixed traffic behavior planning methods still face many difficulties in solving multi-vehicle cooperative decision-making problems. On the one hand, the complex interaction relationships between vehicles are difficult to effectively model using traditional RL frameworks; on the other hand, urban traffic environments are highly dynamic and uncertain, and RL models are less effective at policy transfer and real-time adaptation, easily leading to low-cooperation behavior and affecting planning efficiency and overall system performance.

[0005] Graph Deep Reinforcement Learning (GDRL) methods, by introducing graph neural networks to model the spatial and interactive relationships between Carrier Abilities (CAVs), can enhance the collaborative decision-making capabilities of multi-agents based on structured representations, making them suitable for behavior planning tasks in urban mixed traffic networks. However, existing GDRL-based research still pays relatively little attention to ensuring the safe driving behavior of CAVs during the planning process. In densely congested urban mixed traffic networks, neglecting safety constraints may lead to frequent collisions or dangerous approaches by CAVs, not only reducing the safety of behavior planning but also potentially exacerbating local traffic congestion, creating a vicious cycle. Summary of the Invention

[0006] This invention provides a method, system, and computer-readable storage medium for road ramp behavior planning. The technical problem it solves is: how to effectively improve the safety behavior planning capability of CAVs at ramps in densely congested urban mixed traffic networks while ensuring traffic efficiency.

[0007] To address the above technical problems, this invention provides a road ramp behavior planning method, comprising:

[0008] A communication model and a multi-view safety prediction model are established for connected autonomous vehicles in road ramp traffic networks. The multi-view safety prediction model includes a multi-view dynamic interaction model and a driving behavior safety prediction model. The multi-view safety prediction model generates a multi-view graph structure based on vehicle speed, driving intention, lane changing behavior and communication status. The driving behavior safety prediction model calculates and superimposes the driving behavior safety risk quantification value based on the interactions between vehicles, between vehicles and road boundaries, and between vehicles and destinations.

[0009] Based on the aforementioned communication model, the mixed traffic driving behavior planning problem at ramp entrances is constructed as an optimization problem with the objective of minimizing the total time cost for all connected autonomous vehicles to complete their departure from their designated ramps, and with collision-free constraints and communication quality constraints as conditions.

[0010] The optimization problem is transformed into a Graph Markov Decision Process, where the state space is composed of the vehicle feature matrix and the adjacency matrix of the multi-view graph structure, the action space is the set of lane change operations, and the reward function is obtained by weighted summation of speed reward, collision penalty, frequent lane change penalty, reward based on the quantified safety risk value, and acceleration penalty.

[0011] The vehicle feature matrix and multi-view adjacency matrix are input into a graph reinforcement learning network. The graph reinforcement learning network outputs the Q-value of each action in the action space. The network parameters are updated according to the reward function, and the lane change operation corresponding to the largest Q-value is selected as the current driving strategy.

[0012] The present invention also provides a road ramp behavior planning system, which is characterized by comprising: a communication model and multi-view safety prediction model establishment module, an optimization problem construction module, a graph Markov decision process definition module, and a graph reinforcement learning solution module, which are respectively used to execute the corresponding steps in the above-mentioned road ramp behavior planning method.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the key feature of which is that the computer program, when executed by a processor, implements the above-mentioned road ramp behavior planning method.

[0014] This invention provides a road ramp behavior planning method, system, and computer-readable storage medium. It generates a multi-view graph structure and quantitative values ​​of driving behavior safety risks by establishing a communication model and a multi-view safety prediction model. This leads to the construction of an optimization problem with the objective of minimizing total travel time and constraints of collision-free operation and communication quality. The problem is then transformed into a graph Markov decision process, and finally, a graph reinforcement learning network is used to solve for the optimal lane change operation. This invention can effectively balance traffic efficiency and driving safety in densely congested mixed traffic environments, significantly reducing the collision rate and ensuring zero-collision cooperative driving with good communication quality. Simultaneously, through joint optimization of multi-view safety prediction and reward function, it effectively suppresses frequent lane changes and abrupt acceleration and deceleration, improving driving comfort. Furthermore, this invention has faster convergence speed and higher training stability, exhibits strong robustness to congested conditions, and can comprehensively improve traffic flow and system operating efficiency in ramp areas. Attached Figure Description

[0015] Figure 1 This is a flowchart of the road ramp behavior planning method provided in an embodiment of the present invention;

[0016] Figure 2 This is an example diagram of the communication model provided in the embodiments of the present invention;

[0017] Figure 3 This is an example diagram of the driving behavior safety prediction model provided in the embodiments of the present invention;

[0018] Figure 4 This is an example diagram of the state space provided in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of the driving intention reward mechanism provided in an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of the graph reinforcement learning network provided in an embodiment of the present invention;

[0021] Figure 7 This is a comparison chart of the cumulative reward trends of PPO algorithms based on the MPDI model and the MPSP model, respectively.

[0022] Figure 8 This is a comparison chart of the cumulative reward trends of DDQN algorithms based on the MPDI model and the MPSP model, respectively.

[0023] Figure 9 This is a comparison chart of the cumulative reward trends of A2C algorithms based on the MPDI model and the MPSP model, respectively.

[0024] Figure 10 This is a comparison chart of the cumulative step count trends of PPO algorithms based on the MPDI model and the MPSP model, respectively.

[0025] Figure 11 This is a comparison chart of the cumulative step count trends of the DDQN algorithm based on the MPDI model and the MPSP model, respectively.

[0026] Figure 12 This is a comparison chart of the cumulative step count trends for A2C algorithms based on the MPDI model and the MPSP model, respectively.

[0027] Figure 13 This is a comparison chart of the speed trends of PPO algorithms based on the MPDI model and the MPSP model, respectively.

[0028] Figure 14 These are speed trend comparison charts for DDQN algorithms based on the MPDI and MPSP models, respectively.

[0029] Figure 15 These are speed trend comparison charts for A2C algorithms based on the MPDI and MPSP models, respectively.

[0030] Figure 16 This is a comparison chart of the cumulative reward value trends between the MPSP-JMDC framework provided by this invention and a benchmark algorithm based on the MPDI model;

[0031] Figure 17 This is a comparison chart of the speed change trends between the MPSP-JMDC framework provided by this invention and a benchmark algorithm based on the MPDI model;

[0032] Figure 18 This is a comparison chart showing the trend of step count changes between the MPSP-JMDC framework provided by this invention and a benchmark algorithm based on the MPDI model. Detailed Implementation

[0033] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. The embodiments are given for illustrative purposes only and should not be construed as limiting the present invention. The accompanying drawings are for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.

[0034] The road ramp behavior planning method provided in this embodiment of the invention, such as... Figure 1 The flowchart shown includes the following steps:

[0035] S1. Establish a communication model and a multi-view safety prediction model: Establish a communication model and a multi-view safety prediction model for connected autonomous vehicles in the traffic network at road ramps; the multi-view safety prediction model includes a multi-view dynamic interaction model and a driving behavior safety prediction model. The multi-view safety prediction model generates a multi-view graph structure based on vehicle speed, driving intention, lane changing behavior and communication status. The driving behavior safety prediction model calculates and superimposes the driving behavior safety risk quantification value based on the interactions between vehicles, between vehicles and road boundaries, and between vehicles and destinations.

[0036] S2. Constructing the behavior planning optimization problem: Based on the communication model, the mixed traffic driving behavior planning problem at the ramp is constructed as an optimization problem with the objective of minimizing the total time cost for all connected autonomous vehicles to complete their departure from their designated ramps, and with collision-free constraints and communication quality constraints as conditions.

[0037] S3. Define a graph Markov decision process: Transform the optimization problem into a graph Markov decision process, wherein the state space is composed of the vehicle feature matrix and the adjacency matrix of the multi-view graph structure, the action space is the set of lane change operations, and the reward function is obtained by weighted summation of speed reward, collision penalty, frequent lane change penalty, reward based on the quantified safety risk value, and acceleration penalty.

[0038] S4. Solve the graph Markov decision process using a graph reinforcement learning network: Input the vehicle feature matrix and multi-view adjacency matrix into the graph reinforcement learning network. The graph reinforcement learning network outputs the Q value of each action in the action space. Update the network parameters according to the reward function and select the lane change operation corresponding to the maximum Q value as the current driving strategy.

[0039] Step S1 sets the signal-to-noise ratio threshold and communication range through the communication model, and generates a multi-view graph structure and quantified safety risk value through the multi-view safety prediction model, realizing a unified quantitative representation of the complex spatiotemporal interaction and multi-dimensional safety risks between vehicles in mixed traffic at ramp entrances. Step S2 constructs the above output into a mathematical optimization problem with the goal of minimizing total travel time and constraints of collision-free operation and communication quality, realizing a formal definition and constraint expression of the behavior planning problem. Step S3 transforms the optimization problem into a graph Markov decision process and designs a reward function with safety risk penalties, realizing a solvable mapping of the optimization objective and constraints in the reinforcement learning framework. Step S4 solves the process using a graph reinforcement learning network composed of parallel graph convolution and fully connected layers, realizing state-action value estimation and optimal lane-changing strategy output based on multi-view graph features. Overall, through the progressive coupling of the above four steps, this method effectively balances vehicle traffic efficiency and driving safety while ensuring reliable zero-collision communication, improving decision convergence speed and driving stability.

[0040] The following is a detailed explanation of each step.

[0041] (1) S1: Establish communication model and multi-perspective security prediction model

[0042] refer to Figure 2 The example diagram of the communication model shown illustrates how, in a mixed traffic scenario at a road ramp, the communication model supports information exchange between connected autonomous vehicles (CAVs), roadside units (RSUs), and base stations (BSs), thereby ensuring communication quality and collaborative decision-making capabilities during vehicle operation. The communication link consists of three types of physical links and one type of logical line. The channel characteristics of each type of link are explained below. In a real mixed traffic scenario at a road ramp, besides connected autonomous vehicles (CAVs) (the number of which is...),... The total number of vehicles in the environment, including manually driven vehicles (HV), is: Human-driven vehicles (HVs) do not participate in communication, while connected autonomous vehicles (CAVs) only detect the distance to human-driven vehicles (HVs) to plan lane-changing behavior, calculate potential forces, etc.

[0043] 1) V2V link (vehicle-to-vehicle link)

[0044] V2V links are used for direct communication between CAVs, employing Dedicated Short Range Communication (DSRC) or Cellular Vehicle-to-Everything (C-V2X) direct communication modes. The effective communication range of a V2V link is affected by transmit power, path loss, and shadowing fading. To ensure communication reliability, the signal-to-noise ratio (SNR) at the receiver must not be lower than a preset threshold (e.g., 5 dB).

[0045] 2) V2I Link (Vehicle-to-Infrastructure Link)

[0046] V2I links are used for communication between CAVs and Roadside Units (RSUs). RSUs are typically deployed on roadside poles or gantries, and communication with vehicles also uses DSRC or C-V2X direct connections. Channel frequency bands, bandwidth, and transmit power are the same as for V2V links. Because RSUs are fixed in location and usually higher than vehicles, their line-of-sight transmission conditions are better than V2V, with slightly lower path loss, but they are still affected by non-line-of-sight factors such as building and vehicle obstruction. The signal-to-noise ratio threshold requirement is the same as for V2V.

[0047] 3) B2I link (link between base station and RSU)

[0048] The link between the base station (BS) and the RSU is a wired or wireless backhaul link, and its channel characteristics differ fundamentally from V2V / V2I. In practical deployments, fiber optic wired connections are preferred, as the channel is considered ideal, with bandwidth exceeding 1 Gbps, extremely low bit error rate, latency less than 1 millisecond, and no signal-to-noise ratio (SNR) attenuation. When wiring is not feasible, microwave point-to-point wireless backhaul can be used, operating at frequencies above 6 GHz or in millimeter-wave bands (such as 24 GHz or 60 GHz), with bandwidth exceeding 100 MHz, high transmit power (100 mW to 1 W), and communication distances ranging from hundreds of meters to several kilometers. For simplification, this model treats the link between the base station and the RSU as an ideal channel, requiring only its reliable transmission of aggregated vehicle data and global decision-making, without imposing SNR threshold constraints.

[0049] 4) Intended Path (Logical Path)

[0050] Intent-based connections are not physical channels and therefore do not involve physical layer parameters such as carrier frequency, bandwidth, and signal-to-noise ratio. They are logical connections based on driving intent and established through the existence of V2V physical links. Specifically, the system divides vehicles into different intent groups according to the CAV's predetermined exit ramp (channel 1 or channel 2). If a valid V2V link exists between vehicles in the same group, the intent-based connection is automatically activated to share path planning preferences and phased decision-making information.

[0051] In the example of the ramp diversion scenario provided in this embodiment, two exit ramps are configured, referred to as Channel 1 and Channel 2, respectively. The system consists of one CAV (Carrier Available Vehicle) and multiple roadside units (RSUs). Each CAV acts as a mobile terminal, equipped with positioning, sensing, and communication modules, continuously generating its own status information and interacting with the surrounding environment via V2V and V2I links. The traffic environment comprises CAVs and HVs (human-driven vehicles), with different color codes for CAVs with different intentions to visually indicate their driving intent. In this example, the CAVs are further divided into Ramp1 and Ramp2, representing CAVs planning to exit from lane 1 and lane 2, respectively. RSUs deployed along the road act as fixed nodes, collecting vehicle status information through short-range communication, performing local aggregation, and uploading the results to the base station to support collaborative perception and decision-making.

[0052] The Multi-Perspectives Safety Prediction (MPSP) model serves as both the primary channel for transmitting state information to the decision-making model and a crucial source of environmental feedback for state transitions. First, based on the different relationships between CAVs (Autonomous Vehicles), this module constructs a multi-perspective spatiotemporal graph, modeling dynamic communication across multiple perspectives to form a comprehensive graph representation of the autonomous driving scenario. Then, it calculates the driving behavior risks between elements within each IoV, constructing a driving behavior safety prediction graph. Finally, a graph reinforcement learning network constructs a vehicle interaction graph, acquiring structural information through neighborhood feature aggregation, inferring local and global graph context information, and then training the optimal driving strategy based on this feature information. This ultimately generates a cooperative strategy that guides the CAV to safely and efficiently complete ramp exit operations while maintaining overall traffic flow stability and smoothness.

[0053] The MPSP model comprises a Multi-Perspectives Dynamic Interaction (MPDI) model and a Driving Behavior Safety Prediction (DBSP) model. On one hand, it effectively improves the coordination of behaviors among CAVs (Carrier Aerial Vehicles); on the other hand, it enhances the safety of driving behavior planning from a mechanistic perspective. From the perspectives of operational efficiency and safety, it optimizes the overall performance of the system and effectively alleviates traffic congestion in ramp areas.

[0054] The MPDI model generates a multi-view graph structure based on vehicle speed, driving intent, lane change behavior, and communication status. The multi-view graph structure is a combination of a speed graph structure, an intent graph structure, and a frequent lane change graph structure.

[0055] 1) Speed ​​Graph Structure: The dynamic interactions between vehicles are closely related to their speeds. Vehicles with similar speeds typically exhibit similar traffic conditions at the same time. Logical relationships based on speed can be represented by a graph. It means that among them Indicates time The set of vehicles (vertices). This represents the set of speed-based edges between vehicles (characterizing the speed correlation between vehicles). CAV and CAV The edge between Determined by their speed, expressed as:

[0056] (1)

[0057] in, and Representing vehicles and At any moment speed, This represents the speed difference threshold.

[0058] Adjacency matrix based on velocity Depend on The edge relationships are constructed as follows:

[0059] (2)

[0060] 2) Intent Graph Structure: In a traffic environment, vehicles do not make decisions in a uniform way. Different driving intentions lead to different behavioral patterns. Vehicles with similar decision intentions often exhibit consistent motion characteristics and similar dynamic states. CAVs (Carrier Aerial Vehicles) that intend to exit at the same ramp are grouped together based on the same intention. Logical relationships based on intention can be represented by a graph. It means that among them This represents the set of intention-based edges between vehicles (representing the intention correlation between vehicles). CAV and CAV The edge between Determined by their driving intentions, namely:

[0061] (3)

[0062] in, and They represent CAV respectively and CAV Between Intent at any given moment. Adjacency matrix based on intent. Depend on The edge relationships are constructed as follows:

[0063] (4)

[0064] 3) Frequent Lane Change Graph Structure: In CAV ramp diversion scenarios, lane change behavior typically reflects the phased changes in vehicle state. Vehicles with similar lane change trends within a short period are often in similar driving phases and tend to adopt similar strategies. Based on historical lane change behavior, a frequent lane change graph can be constructed. To represent the logical relationship between actions between vehicles, where This represents the edges between vehicles based on lane-changing behavior (representing the logical relationship between vehicles' lane-changing actions). CAV and CAV The edge between Determined by their lane-changing behavior, specifically expressed as:

[0065] (5)

[0066] in, and They represent in CAV Time and CAV The time difference between the last lane change and the last lane change. This represents the lane change time difference threshold. Therefore, the adjacency matrix is ​​based on lane change behavior. Depend on The edge relation construction is as follows:

[0067] (6)

[0068] Communication distance directly affects the ability of a CAV to exchange information with surrounding vehicles. Other CAVs beyond this distance will disconnect from the current CAV, causing a communication interruption. To ensure the quality and efficiency of communication between CAVs, and considering practical realities (obstacles on urban roads, dynamic vehicle movement, etc.), a minimum SNR threshold is defined for the signal-to-noise ratio (SNR) between CAVs. Based on this, an adjacency matrix with a multi-view graph structure of the MPDI model considering communication quality is established. :

[0069] (7)

[0070] (8)

[0071] Among them, symbols This represents element-wise matrix multiplication, which calculates the product of corresponding elements of two matrices of the same dimension. , , Indicates corresponding to respectively , , The adjacency matrix takes into account communication quality; Indicates in The communication state matrix at time t, Indicates a predefined communication range Adjacent CAVs within and CAV The communication status between them, specifically, takes the following values:

[0072] (9)

[0073] It indicates that it is within a predefined communication range. Adjacent CAVs within and CAV The corresponding element is set to 1, while adjacent nodes CAV outside this range and CAV The corresponding element is set to 0.

[0074] refer to Figure 3 The example diagram of the DBSP model shown is used to calculate and superimpose the quantitative value of driving behavior safety risk based on the interaction between vehicles, between vehicles and road boundaries, and between vehicles and destinations. It is composed of the superposition of three types of driving behavior safety prediction potential fields: vehicle-to-vehicle, vehicle-to-road, and vehicle-to-destination.

[0075] 1) Vehicle-to-Vehicle Driving Behavior Safety Prediction Potential Field: The vehicle-to-vehicle driving behavior safety prediction potential field is a force field formed by other vehicles in the surrounding traffic environment. Its strength is related to the vehicle's type, mass, speed, acceleration, and other dynamic states. To uniformly quantify the safety risks caused by the mass and speed of obstructing vehicles, an equivalent mass is introduced, defined as:

[0076] (10)

[0077] in, Indicates vehicles with obstacles Equivalent quality (including CAV and HV), This indicates its actual quality. Furthermore, when CAV... Located in the obstacle vehicle When a vehicle is within the potential field of the generated driving risk, the strength of the potential field it experiences depends on its relationship with the obstacle vehicle. The relative positions of the vehicles. Meanwhile, the obstacle vehicles. The motion state of an object affects the strength of its potential field, among which velocity... and acceleration It is the main influencing factor. Therefore, CAV Obstacle vehicles The potential field model can be defined as:

[0078] (11)

[0079] in, and For undetermined coefficients, It is an undetermined speed-related parameter. Indicates vehicles with obstacles At any moment acceleration, Indicates pseudo-distance. The modulus is:

[0080] (12)

[0081] in, and These represent the critical thresholds for longitudinal and lateral safety distances, respectively. CAV The coordinates of the centroid; Indicates vehicles with obstacles The original coordinates, Indicates vehicles with obstacles The offset coordinates after the turning operation satisfy:

[0082] (13)

[0083] in, Indicates vehicles with obstacles The clockwise turning angle.

[0084] When CAV With obstacle vehicles When the distance is far enough, it is no longer affected by the potential field of the obstacle vehicle. In this case, the obstacle vehicle... The potential field function must be changed accordingly, that is:

[0085] (14)

[0086] in, This represents the extended boundary of the potential field, the maximum value of which is limited by the safe interaction distance between the CAV and the obstacle vehicle.

[0087] 2) Vehicle-Road Driving Behavior Safety Prediction Potential Field: The vehicle-road driving behavior safety prediction potential field generally consists of lane line potential field and boundary potential field. To avoid the lane line potential field easily causing the optimization strategy to converge to a local optimum, only the boundary potential field is modeled. By introducing control constraints to regulate the CAV's driving behavior, this method can effectively prevent vehicles from crossing boundaries or colliding with roadside obstacles. At that moment, CAV The boundary potential field at the current location is defined as:

[0088] (15)

[0089] in, It is the coefficient of the potential field at the road boundary. It is CAV The number of lanes from the right edge of the road. It is the width of a single lane. CAV The number of lanes leading to the right edge of the road. CAV The distance to the right lane line of its lane.

[0090] 3) Vehicle-Destination Driving Behavior Safety Prediction Potential Field: Besides surrounding obstacles, vehicles, and the road environment, destination information plays a crucial role in CAV motion planning. To ensure the vehicle can reach the target area while avoiding potential risks, the gravitational potential field generated by the destination is added to the potential field model. Based on the distance between the target point and the CAV, the corresponding vehicle-destination driving behavior safety prediction potential field can be constructed, defined as:

[0091] (16)

[0092] in, Represents the coordinates of the target point. and It is a determining factor used to unify the magnitude of the gravitational potential field experienced by the CAV in the longitudinal and transverse directions.

[0093] 4) Superimposed Potential Field: By superimposing all the above potential fields, a unified driving safety potential field model can be constructed, i.e., at any given time... CAV The overall potential field is:

[0094] (17)

[0095] It indicates that, at any time CAV The total potential field is equal to the sum of the potential fields of all obstacle vehicles, the potential field of the road boundary, and the potential field of the destination.

[0096] Again, time CAV The total potential field is normalized by applying minimum-maximum normalization to obtain the normalized total potential field value. As CAV The quantitative value of driving behavior safety risk.

[0097] This step enables a structured representation of the complex interactions between vehicles in mixed traffic scenarios at ramp entrances and a unified quantification of multidimensional safety risks. This lays the foundation for the formal definition of subsequent behavior planning problems and reinforcement learning solutions, providing a structured state input and risk perception capability.

[0098] (1) S2: Constructing a behavioral planning optimization problem

[0099] This invention aims to alleviate traffic congestion at ramp entrances in densely populated urban mixed traffic networks through Mixed Traffic Driving Behavior Planning (MTDBP). Specifically, the planning objective is to enable each agent to quickly exit its designated ramp while ensuring collision-free driving, thereby effectively solving traffic congestion and ensuring the efficient, safe, and orderly operation of densely populated urban traffic networks. Therefore, the MTDBP problem is modeled as a mathematical problem in urban mixed traffic networks, namely:

[0100] (18)

[0101] in, and Representing vehicles in the environment and vehicles At any moment Location, and They represent the times at time 1 and 2 respectively. The number of CAVs and the total number of vehicles (including CAVs and HVs) in the environment. CAV and CAV The signal-to-noise ratio between them Defined as the minimum signal-to-noise ratio threshold required to ensure communication between CAVs.

[0102] In MTDBP, it is necessary to minimize the total time cost of each agent (CAV) to improve the efficiency of the urban hybrid transportation network; therefore, the objective function is... It minimizes the total time for all CAVs to complete the ramp departure task. As a destination constraint, it indicates that the plan starts from the first... CAVs exiting from each ramp should depart from the designated ramp exit. Exit means that each CAV must exit at its pre-designated ramp exit (it cannot take the wrong exit). Indicates the plan to start from the The collection of CAVs exiting from each ramp This indicates the total number of ramps. As a communication quality constraint, it means that the signal-to-noise ratio between any two CAVs must not be lower than the minimum signal-to-noise ratio threshold in order to ensure reliable information exchange required for V2V collaborative decision-making. This indicates no collision constraints, meaning that no two vehicles can occupy the same position at the same time.

[0103] Step S2 transforms the mixed traffic driving behavior planning problem at ramp entrances into a constrained multi-objective optimization problem. By minimizing the total time for all connected autonomous vehicles to complete their ramp exit tasks, and introducing destination constraints (vehicles must exit from the designated ramp), communication quality constraints (the signal-to-noise ratio between any two vehicles must not be lower than a threshold), and collision-free constraints (the positions of any two vehicles must not overlap), a complete mathematical programming model is constructed. This model clarifies the trade-off between efficiency and safety and abstracts actual traffic rules and communication limitations into quantifiable constraints, providing a clear problem boundary and optimization direction for subsequent transformation into a Tumarkov decision process.

[0104] (3) S3, Define the Graph Markov Decision Process

[0105] The MTDBP problem under ramp congestion in urban mixed transportation networks is a non-convex optimization problem. Due to network characteristics such as the mobility and time-varying service demands of CAVs, future time slot availability cannot be accurately predicted or analytically modeled. Therefore, traditional modeling methods are insufficient to solve this problem. To address this challenge, step S3 models the system as a Graphical Markov Decision Process (GMDP), defining the corresponding state space, action space, and reward function.

[0106] 1) State Space

[0107] refer to Figure 4 The example state-space diagram shown illustrates the state at the ramp entrance in a mixed urban transportation network. At any given time, the state space of CAV is used matrix This is determined by the driving characteristics of the vehicle and the mutual influence and interaction between vehicles, and can be expressed as:

[0108] (19)

[0109] in, Indicates time Vehicle driving feature matrix, Indicates time The adjacency matrix represents the mutual influence and interaction between vehicles.

[0110] Vehicle driving feature matrix Included in time At time t, the eigenvectors of all CAVs. CAV The driving characteristics include its normalized position. Normalized speed ,Lane and intention When there is in the scene When there are multiple CAVs, the feature matrix is... Defined as:

[0111] (20)

[0112] time CAV Normalized position Normalized speed They are defined as follows:

[0113] ,(twenty one)

[0114] ,(twenty two)

[0115] in, and Representing time respectively CAV Position and velocity, The total length of the road. This indicates the maximum speed limit.

[0116] Lane Index One-hot encoding is used to indicate the current lane of a vehicle on a multi-lane road. Taking a three-lane road as an example, it employs... , and These correspond to the rightmost, middle, and leftmost lanes, respectively. Similarly, driving intention... Represented using one-hot encoding, where , and These respectively indicate exiting from Channel 1, exiting from Channel 2, and continuing straight along the main line.

[0117] The mutual influence and interaction between vehicles are achieved through the adjacency matrix. This indicates that the dimensions of the matrix are fixed. It can adapt to changes in the number of vehicles in the scene. When the number of vehicles is less than... When some vehicles leave the scene, the corresponding elements in the matrix will be set to 0, thus maintaining the integrity of the structure and ensuring the consistency of graph neural network calculations.

[0118] 2) Action Space

[0119] CAV uses a discrete action space to represent its lateral decision-making behavior. This action set corresponds to a series of lane-changing operations that can change the vehicle's lateral position on the road, thus satisfying the MTDBP process required for urban road ramp exit tasks. Action Space Designed as follows:

[0120] ,(twenty three)

[0121] In this context, "left" means changing lanes to the left lane, "stay" means staying in the current lane, and "right" means changing lanes to the right lane.

[0122] 3) Reward function

[0123] To alleviate congestion at ramps in urban mixed-use transportation networks, the primary objective is to improve the efficiency of behavioral planning at ramps across the entire network and enhance driving safety. The goal is to encourage vehicles to navigate ramps efficiently and safely, reducing rapid acceleration and deceleration during lane changes, and minimizing emissions while ensuring driving safety and comfort. Rewards are divided into average speed rewards and penalties for frequent lane changes and collisions, i.e., rewards based on speed. CAV The total reward is:

[0124] ,(twenty four)

[0125] in, , , , , , Representing time respectively The rewards include driving intention rewards, speed rewards, frequent lane change rewards, collision rewards, prediction rewards, and acceleration rewards.

[0126] This invention designs a driver intent reward mechanism with guidance function for ramp scenarios in urban mixed traffic networks. This mechanism penalizes vehicles that occupy the leftmost lane before entering the ramp, prompting them to change lanes to the right in a timely manner during their journey, thus ensuring smooth exit from the designated ramp. The reward setting can improve overall traffic efficiency. (Reference) Figure 5 The diagram shown illustrates the intention reward, specifically the driving intention reward value. The specific calculation formula is as follows:

[0127] (25)

[0128] in, This is a category indicator variable used to identify the first... Does each CAV (Continuous Awareness and Activation) fall into a stage that requires reward or punishment? This represents the total number of CAVs in the current scene, where and The values ​​are all (A value of 1 corresponds to) The value of 2 corresponds to ).in Indicates the first The destination of the CAV is When driving in Rewards for specific road sections Then it represents the first The destination of the CAV is When driving in Penalties for violations on designated road sections Indicates departure from the destination The A CAV in The distance traveled within the designated area. and They represent and The length of the road segment in the area. The positive or negative value is determined jointly by the road segment and the lane where the CAV is currently located. Furthermore, by... The absolute value is modeled as the longitudinal position of the vehicle. The function can effectively incentivize CAVs to make lane-changing decisions earlier while driving, thus improving the timeliness and collaborative efficiency of path planning.

[0129] Speed ​​Reward It is measured by the sum of the relative speeds of all CAVs, and represents the operational efficiency of traffic flow. This mechanism encourages CAVs to maintain high speeds as much as possible within the legal speed range, thereby improving the overall traffic efficiency and operational quality of the system, specifically expressed as:

[0130] (26)

[0131] in, and Defined respectively as the current time of the first The speed of the CAV and the maximum speed of all CAVs.

[0132] The collision reward mechanism uses penalties to reinforce the system's constraint on driving safety. Once a collision occurs that is associated with CAV (Car Access Vehicle) or HV (Hard Vehicle) in the environment, negative feedback is generated, making the model more inclined to avoid collisions during training, thus improving the safety and reliability of the entire driving process. Represented as:

[0133] (27)

[0134] in, This represents the collision reward factor.

[0135] To reduce the potential safety risks associated with frequent lane changes, CAVs are penalized for excessive consecutive lane changes. This mechanism provides a negative reward for each lane change, encouraging the vehicle to maintain a stable driving path and thus improving overall safety. Therefore, the reward for frequent lane changes manifests as follows:

[0136] (28)

[0137] in, This indicates a reward factor for frequent lane changes.

[0138] The predictive reward aims to enhance the CAV's ability to predict safe driving behavior, effectively improve MTDBP efficiency, ensure orderly operation at ramps in urban mixed traffic networks, and thus significantly alleviate environmental congestion. This is achieved by converting driving behavior risk values ​​into predictive reward values ​​and feeding them back to the CAV, specifically as follows:

[0139] (29)

[0140] in, This represents the predicted reward factor.

[0141] To ensure vehicle safety during operation, the acceleration penalty function is designed as follows:

[0142] (30)

[0143] in, This represents the acceleration penalty function factor. The acceleration indicator variable is expressed as:

[0144] (31)

[0145] in, Indicates the absolute acceleration threshold. Defined in the current The arithmetic mean of the accelerations of all CAVs in a densely congested urban mixed traffic network at any given time.

[0146] Step S3 transforms the non-convex, dynamic, and analytically difficult-to-solve hybrid traffic behavior planning and optimization problem into a Graph Markov Decision Process, thus adapting it to the reinforcement learning solution framework. In the state space, a combination of vehicle feature matrices and multi-view adjacency matrices is used to represent the state, preserving the vehicle's own motion attributes and the spatiotemporal dependencies and interactions between vehicles. The action space is designed with three discrete lateral operations: left lane change, lane keeping, and right lane change, closely aligning with actual driving behavior at ramp entrances. The reward function integrates speed rewards, collision penalties, frequent lane change penalties, prediction rewards based on quantified safety risks, and acceleration penalties, achieving multi-objective guidance for traffic efficiency, safety, and comfort. This step enables the reinforcement learning agent to learn the optimal driving strategy that balances efficiency and safety through interaction with the environment.

[0147] (4) S4. Use graph reinforcement learning networks to solve graph Markov decision processes.

[0148] Each CAV is considered as an intelligent agent, and this intelligent agent adopts the following... Figure 6 The graph reinforcement learning network shown solves the graph Markov decision process. The graph reinforcement learning network (GRL) consists of a policy network and a value network based on a graph convolutional model and deep reinforcement learning. The policy network and value network use the same network structure. The policy network acts as the "decision-maker": it encodes the current environmental state as graph structure features and then directly outputs the lateral action the vehicle should take—change lanes left, stay in its lane, or change lanes right. Specifically, the graph convolutional model extracts graph embedding feature vectors from multiple perspectives through multiple aggregators (aggregator 1, aggregator 2), fusing the node feature matrix with adjacency matrices such as speed, intent, and lane-changing behavior to obtain graph structure features. These features are then fed into three parallel hidden layers (hidden layer 1, hidden layer 2, hidden layer 3) in the value network of the deep reinforcement learning module to output state values. These values ​​are then fed into three parallel hidden layers (hidden layer 1, hidden layer 2, hidden layer 3) in the policy network to calculate the Q-value of each candidate action and select the action accordingly. The parameters of the policy network are updated through policy gradients. The core of the policy network is to maximize the expected cumulative reward and to update the parameters using the policy gradient algorithm.

[0149] The value network acts as an "evaluator." Instead of directly outputting actions, it estimates the state value of the current state—the expected cumulative reward obtainable from that state according to the current policy. During training, the value network helps calculate the advantage function—the difference between the actual cumulative reward and the estimated state value. This advantage value reflects how good or bad the chosen action is relative to the average level, and the policy network uses this information to stably update its parameters. The value network itself continuously refines its estimation by minimizing the mean squared error loss, making it closer to the true reward.

[0150] In congested ramp scenarios, each CAV (Carrier Ability Vehicle) possesses its own independent policy network and value network. The policy network is responsible for making lane-changing decisions based on multi-view graph information perceived by the vehicle (speed, intent, lane-changing history, and communication quality of surrounding vehicles); the value network evaluates the safety, efficiency, and progress towards the ramp in the current local traffic state, providing a low-variance baseline for policy updates. The two work together: the value network significantly reduces the variance of the policy gradient, avoiding the oscillations and instability common in pure policy gradient methods, while the policy network utilizes the advantageous signals provided by the value network to learn a balanced driving strategy between ensuring safety (predicted rewards, collision penalties) and pursuing efficiency (speed rewards, intent rewards), ultimately achieving safe and efficient multi-vehicle cooperative ramp exit behavior planning in dense urban mixed traffic.

[0151] To systematically verify the efficiency and stability of the proposed MPSP-based road ramp behavior planning method (referred to as the MPSP-JMDC algorithm), comparative simulation experiments were conducted between the MPSP-JMDC algorithm and three benchmark algorithms: A2C (Actor-Critic) algorithm, DDQN (Dual Deep Q Network) algorithm, and PPO (Proximal Policy Optimization) algorithm.

[0152] The experiment was conducted on a computing platform equipped with a 14th-generation Intel® Core™ i9-14900K 32-core processor, 128 GB of memory, and an NVIDIA GeForce RTX 4090 graphics card (24 GB of VRAM). Using this computing system, a simulation platform for simulating densely congested urban mixed traffic networks was developed using Flow. Flow is a high-level programming interface based on the Simulation of Urban Mobility (SUMO) platform, supporting the proposed MTDBP framework. The constructed traffic simulation platform was configured with a three-lane road segment with two right-hand ramps along the driving direction. To ensure reliable communication quality between CAVs, the transmit power and communication bandwidth were set to 10 mW and 2 MHz, respectively. Detailed hyperparameter configurations for the MPSP algorithm are shown in Table 1.

[0153] Table 1 Simulation Experiment Parameters

[0154]

[0155] To systematically evaluate the efficiency and reliability of the proposed algorithm in MTDBP, the proposed algorithm and the benchmark algorithm were compared and analyzed from four key performance indicators: average driving speed, average time consumption, collision rate, and normalized reward. The details are as follows.

[0156] Average driving speed ( This evaluation metric calculates the average speed of CAV during the training phase and is designed to assess the efficiency of the algorithm in MTDBP.

[0157] Average time consumption: To evaluate the task execution efficiency of the proposed algorithm, this metric is measured by the number of time steps required to complete the task.

[0158] Collision rate ( ): Record the number of collisions that occur during CAV training and calculate the average collision rate to evaluate the safety and reliability of the algorithm in MTDBP, i.e. , and These represent the total number of collisions and the total number of training rounds, respectively.

[0159] Normalized reward: This evaluation metric aims to serve as a standard for evaluating the performance of algorithms. Its specific expression is as follows: , This represents the cumulative reward value during the training process.

[0160] To evaluate the efficiency and generality of the proposed MPSP model, benchmark algorithms implemented using the MPDI and MPSP models were compared. The MTDBP capability of CAVs to safely and efficiently reach their destination was analyzed and compared. The cumulative reward trend comparison charts for PPO algorithms based on the MPDI and MPSP models are shown below. Figure 7 As shown in the figure, the cumulative reward trend comparison charts for the DDQN algorithm classes based on the MPDI model and the MPSP model are as follows: Figure 8 As shown in the figure, the cumulative reward trend comparison charts for the A2C algorithm classes based on the MPDI model and the MPSP model are as follows: Figure 9 As shown.

[0161] Among them, such as Figure 7 and Figure 9 As shown, PPO and A2C exhibit significant oscillations in the early stages of training, and the exploration phase is relatively long. Although the reward curve eventually converges, there are still significant fluctuations in the later stages. In contrast, the MPDI-based PPO and A2C algorithms converge faster and maintain more stable curves in the later training stages. The PPO and A2C algorithms implemented based on the model proposed in this chapter have the fastest convergence speed in the early stages and maintain stable and highest reward values ​​in the later stages, demonstrating significant advantages in learning efficiency and stability. Figure 8As shown, unlike the PPO and A2C series algorithms, the reward curves of the DDQN series algorithms exhibit a more consistent trend. Compared to DDQN, the MPDI-based DDQN algorithm converges faster. However, its final reward value does not show a significant improvement. In contrast, the MPSP-based DDQN algorithm achieves the highest reward with the shortest exploration time, demonstrating the superior efficiency of the proposed model.

[0162] Next, the cumulative step count trends of the benchmark algorithm under the MPDI and MPSP models are compared. The cumulative step count trends of the PPO algorithm class based on the MPDI and MPSP models are shown in the following graphs. Figure 10 As shown in the figure, the cumulative step count trends of the DDQN algorithm based on the MPDI model and the MPSP model are compared. Figure 11 As shown in the figure, the cumulative step count trends of the A2C algorithm based on the MPDI model and the MPSP model are compared. Figure 12 As shown. Comparison Figure 10 , Figure 11 and Figure 12 It is evident that the baseline algorithm involves extensive exploration in the early stages, resulting in significant fluctuations and the slowest convergence speed. The MPDI model requires numerous exploration steps to converge, also exhibiting considerable volatility. In contrast, the MPSP implementation requires fewer exploration steps to converge and demonstrates the highest stability. Although the baseline algorithms differ, the MPSP-based implementation consistently achieves better convergence performance, indicating the proposed model's strong robustness and versatility.

[0163] Next, we compare the speed trend graphs of the benchmark algorithm under the MPDI and MPSP models. The speed trend comparison graphs of the PPO algorithm class based on the MPDI and MPSP models are shown below. Figure 13 As shown in the figure, the speed trend comparison charts of the DDQN algorithm based on the MPDI model and the MPSP model are as follows: Figure 14 As shown in the figure, the speed trend comparison charts for A2C algorithms based on the MPDI model and the MPSP model are as follows: Figure 15 As shown. Comparison Figure 13 , Figure 14 and Figure 15 As can be seen, the speed curve stabilizes only after a prolonged period of fluctuation and instability. To improve stability, the MPDI model is used. When combined with a baseline algorithm, the MPDI-based method can reach a stable driving state more quickly with less speed fluctuation. This allows for speed adjustment to improve driving comfort and ensure safety. Meanwhile, the baseline algorithm based on the MPSP model can achieve a stable and comfortable driving speed in the shortest time, offering better versatility, efficiency, and enhanced safety and comfort.

[0164] The experimental results show that the proposed MPSP model significantly improves driving safety and comfort, ensures efficient behavioral decision-making, and demonstrates strong applicability to congested urban mixed traffic networks.

[0165] After verifying the generality and effectiveness of the MPSP model, the following experiments were conducted to evaluate the performance of the MPSP-JMDC algorithm based on this model. A comparison of the cumulative reward value trends of the MPSP-JMDC framework and the benchmark algorithm based on the MPDI model is shown below. Figure 16 As shown. From Figure 16 It can be seen that the DDQN-MPDI algorithm has a long exploration period in the early stage of training, with large fluctuations, and the lowest reward after convergence. The PPO-MPDI algorithm converges faster than DDQN-MPDI, has smaller fluctuations in the exploration phase, and is more stable than DDQN-MPDI in the later stage of training. The MPSP-JMDC framework proposed in this embodiment achieves the smoothest training curve, the fastest speed, and the best reward, indicating that it has higher efficiency and better overall performance.

[0166] The speed and step count trends of the MPSP-JMDC framework and the MPDI-based benchmark algorithm are compared as follows: Figure 17 and Figure 18 As shown. From Figure 17 and Figure 18 It can be seen that all three algorithms based on the MPDI model achieved speed convergence. The PPO-MPDI algorithm has stronger behavioral planning capabilities than the DDQN-MPDI algorithm, requiring fewer steps to converge and exhibiting less fluctuation. The MPSP-JMDC framework demonstrates stable and efficient solution capabilities from the outset. Its step count curve shows a smooth downward trend with minimal fluctuations until convergence, exhibiting excellent stability and convergence. In terms of step count, the DDQN-MPDI algorithm experiences significant fluctuations in its speed curve during the initial training phase due to continuous environment exploration and experience accumulation, and still exhibits noticeable fluctuations after convergence. The PPO-MPDI algorithm shows slight fluctuations in the early stages of training, followed by a rapid decrease in speed and a tendency to stabilize and converge. The MPSP-JMDC framework proposed in this chapter has higher learning efficiency and stronger global exploration capabilities than the benchmark algorithms. Its speed curve smoothly decreases and converges without significant fluctuations, indicating that it is safer in behavioral planning and can improve driving comfort.

[0167] Table 2 shows the key performance indicators of the three algorithms based on the MPDI and MPSP models, respectively. As can be seen from Table 2, the DDQN-MPSP algorithm has the slowest average speed, requires the most steps to reach the destination, maintains a collision rate of 0, and has a normalized reward of 1.85. Although this algorithm can guarantee safe navigation, its average step efficiency is relatively low. The PPO-MPSP algorithm has a faster average speed than DDQN-MPSP, requires fewer than 510 steps, and also maintains a 0 collision rate. However, even with a normalized reward of 1.83, it is still lower than DDQN-MPSP, indicating that its overall driving performance is unstable and inefficient. The proposed MPSP-JMDC framework performs best among the three algorithms, achieving the fastest speed, the fewest steps, zero collisions, and the highest normalized reward of 1.86. The results show that MPSP-JMDC has a faster convergence speed than the benchmark algorithm and better safety performance.

[0168] Table 2 Key Performance Indicators for Mixed Traffic Driving Behavior Planning

[0169]

[0170] The above experimental results show that the road ramp behavior planning method provided by the embodiments of the present invention has achieved significant improvements in key indicators such as average driving speed, number of steps to complete the task, collision rate and normalized reward value. At the same time, the training curve is smoother and the convergence speed is faster, which effectively ensures driving safety and traffic efficiency in high-density mixed traffic flow at urban road ramps.

[0171] This invention also provides a road ramp entrance behavior planning system for executing the steps in the above method embodiments. Specifically, the system includes:

[0172] The communication model and multi-view security prediction model establishment module, optimization problem construction module, graph Markov decision process definition module, and graph reinforcement learning solution module all operate in the same way as their corresponding method steps S1 to S4, and will not be described again here. Those skilled in the art should understand that this system can be implemented in software, hardware, or a combination of both. For example, each module can be configured as a computer program module stored in memory and executed by a processor, or it can be implemented in hardware forms such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0173] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements all or part of the steps of the aforementioned road ramp behavior planning method. The computer-readable storage medium can be any tangible medium containing or storing a program, such as: hard disk, optical disk, DVD, CD-ROM, flash memory, solid-state drive (SSD), read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), or magnetic cards, optical cards, etc. When the computer program is executed by one or more processors, the processors perform the following operations: establishing a communication model and a multi-view safety prediction model; constructing a behavior planning optimization problem; defining a graph Markov decision process; using a graph reinforcement learning network to solve the graph Markov decision process and outputting the optimal lane change operation. Specific implementation processes can be found in the descriptions in the method embodiments and will not be repeated here.

[0174] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for behavior planning at road ramp entrances, characterized in that, include: Establish a communication model and a multi-view safety prediction model for connected autonomous vehicles in road ramp traffic networks; The multi-view safety prediction model includes a multi-view dynamic interaction model and a driving behavior safety prediction model. The multi-view safety prediction model generates a multi-view graph structure based on vehicle speed, driving intention, lane changing behavior and communication status. The driving behavior safety prediction model calculates and superimposes the driving behavior safety risk quantification value based on the interactions between vehicles, between vehicles and road boundaries, and between vehicles and destinations. Based on the aforementioned communication model, the mixed traffic driving behavior planning problem at ramp entrances is constructed as an optimization problem with the objective of minimizing the total time cost for all connected autonomous vehicles to complete their departure from their designated ramps, and with collision-free constraints and communication quality constraints as conditions. The optimization problem is transformed into a Graph Markov Decision Process, where the state space is composed of the vehicle feature matrix and the adjacency matrix of the multi-view graph structure, the action space is the set of lane change operations, and the reward function is obtained by weighted summation of speed reward, collision penalty, frequent lane change penalty, reward based on the quantified safety risk value, and acceleration penalty. The vehicle feature matrix and multi-view adjacency matrix are input into a graph reinforcement learning network. The graph reinforcement learning network outputs the Q-value of each action in the action space. The network parameters are updated according to the reward function, and the lane change operation corresponding to the largest Q-value is selected as the current driving strategy.

2. The road ramp behavior planning method according to claim 1, characterized in that: The communication model includes vehicle-to-vehicle links, vehicle-to-infrastructure links, links between base stations and roadside units, and logical routes based on driving intentions. The multi-view graph structure generated by the multi-view dynamic interaction model includes a speed graph structure, an intention graph structure, and a frequent lane change graph structure. In the speed graph structure, the edges between vehicles are determined based on whether the absolute value of the speed difference is less than or equal to a speed difference threshold. In the intention graph structure, the edges between vehicles are determined based on whether the vehicles have the same predetermined intention to exit the ramp. In the frequent lane change graph structure, the edges between vehicles are determined based on whether the absolute value of the time difference from the last lane change is less than or equal to a lane change time difference threshold. The multi-view dynamic interaction model also constructs a communication state matrix based on a preset signal-to-noise ratio threshold and communication range, and performs element-wise multiplication of the adjacency matrices of the speed graph structure, intention graph structure, and frequent lane change graph structure with the communication state matrix, and then adds the multiplication results to obtain the adjacency matrix of the final multi-view graph structure.

3. The road ramp behavior planning method according to claim 2, characterized in that: The driving behavior safety prediction model includes a vehicle-to-vehicle driving behavior safety prediction potential field, a vehicle-to-road driving behavior safety prediction potential field, and a vehicle-to-destination driving behavior safety prediction potential field. The vehicle-to-vehicle driving behavior safety prediction potential field calculates the equivalent mass based on the actual mass and speed of the obstacle vehicle, and constructs a potential field model based on the pseudo-distance between the connected autonomous vehicle and the obstacle vehicle, the obstacle vehicle's acceleration, and the turning angle. The potential field value is set to zero when the pseudo-distance exceeds a preset extension boundary. The vehicle-to-road driving behavior safety prediction potential field only models the road boundary, calculated based on the number of lanes from the connected autonomous vehicle to the right boundary of the road, the distance to the right lane line of the current lane, the lane width, and the boundary potential field coefficient. The vehicle-to-destination driving behavior safety prediction potential field is calculated based on the longitudinal and lateral distances between the current position of the connected autonomous vehicle and the target point, and their corresponding determination coefficients. The three potential fields are added together to obtain the total potential field, and then min-max normalization is applied to obtain the quantitative value of the driving behavior safety risk.

4. The road ramp behavior planning method according to claim 2, characterized in that: The objective function of the optimization problem is to minimize the total time for all connected autonomous vehicles to complete the exit task of the ramp. The constraints of the optimization problem include destination constraints, communication quality constraints, and collision-free constraints. The destination constraint requires that connected autonomous vehicles planning to exit from any ramp must exit from the designated ramp exit. The communication quality constraint requires that the signal-to-noise ratio between any two connected autonomous vehicles is not lower than a preset minimum signal-to-noise ratio threshold. The collision-free constraint requires that the positions of any two vehicles do not overlap at the same time.

5. The road ramp intersection behavior planning method according to claim 2, characterized in that: In the state space of the Graph Markov Decision Process, each row of the vehicle feature matrix corresponds to the normalized position, normalized speed, one-hot encoded representation of the lane, and one-hot encoded representation of the driving intention of a connected autonomous vehicle; the adjacency matrix of the multi-view graph structure is a binary matrix, the dimension of which is fixed to the maximum number of connected autonomous vehicles in the scene, and the corresponding elements are set to zero when the actual number of vehicles is less than this dimension; the action space includes three discrete actions: changing lanes to the left, keeping the lane, and changing lanes to the right.

6. The road ramp behavior planning method according to claim 1, characterized in that: The reward function is obtained by summing driving intention reward, speed reward, collision penalty, frequent lane change penalty, prediction reward based on safety risk quantification value, and acceleration penalty; The driving intention reward is calculated based on the vehicle's current road segment, current lane, and longitudinal position; the speed reward is the arithmetic mean of the ratio of the current speed to the maximum speed of all connected autonomous vehicles; the collision penalty is a negative constant when a collision occurs, otherwise it is zero; the frequent lane change penalty is a negative constant when a sharp lane change is detected, otherwise it is zero; the prediction reward is the normalized safety risk quantification value multiplied by a preset prediction reward factor; the acceleration penalty is determined based on the comparison between the average acceleration of all connected autonomous vehicles and a preset absolute acceleration threshold.

7. The road ramp behavior planning method according to claim 1, characterized in that: The graph reinforcement learning network includes a policy network and a value network. The policy network takes the vehicle feature matrix and multi-view adjacency matrix in the state space as input, extracts multi-view graph embedding features, and then processes them through a fully connected layer and a deep reinforcement learning module to output the expected cumulative reward value corresponding to each action in the action space. The lane change operation corresponding to the maximum expected cumulative reward value is selected as the current driving strategy. The value network is used to estimate the state value of the current state and calculate the advantage function to update the parameters of the policy network, thereby optimizing the policy network in the direction of maximizing cumulative rewards.

8. The road ramp entrance behavior planning method according to claim 1, characterized in that: Each connected autonomous vehicle has its own independent policy network and value network; the parameters of the policy network are updated using the policy gradient method, and the parameters of the value network are updated by minimizing the mean squared error loss.

9. A road ramp behavior planning system, characterized in that, include: The communication model and multi-view safety prediction model establishment module, the optimization problem construction module, the graph Markov decision process definition module, and the graph reinforcement learning solution module are respectively used to execute the corresponding steps in the road ramp behavior planning method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the road ramp behavior planning method according to any one of claims 1 to 8.

Images