Highway merging decision-making method for intelligent connected vehicles considering action asynchrony

Abstract

This invention belongs to the field of intelligent vehicle technology and discloses a method for highway merging decision-making for intelligent connected vehicles (ICVs) that considers motion asynchrony. The method comprises the following steps: S1. Functionally defining an intelligent connected vehicle (ICV) and a human-driven vehicle (HDV); S2. Representing the highway merging decision-making for an ICV in mixed traffic as a multi-agent reinforcement learning problem based on the Stackelberg game, and designing a spatiotemporal Markov decision process that considers the asynchrony of vehicle motion; S3. Constructing a Stackelberg game model; and S4. Using an Actor-Critic multi-agent reinforcement learning framework for training and evaluation. By addressing the asynchrony of vehicle motion coordination in highway merging areas, the present invention can effectively improve the merging decision-making capabilities of ICVs in mixed traffic highway merging areas.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent vehicles, and in particular relates to a highway merging decision method for intelligent connected vehicles taking into account action asynchrony. Background Art

[0002] Freeway ramp merging is one of the most difficult tasks in autonomous driving, and related research has always been a hot topic in the field of autonomous driving. Enabling intelligent connected vehicles to make reasonable decisions in highway merging areas has important application value for promoting the commercial application of autonomous driving.

[0003] Early research primarily employed rule-based approaches, which are unsuitable for complex merging scenarios. Optimization-based approaches have shown promising results, but they rely on precise models, are computationally complex, and have limited adaptability to mixed traffic. With the advancement of artificial intelligence (AI), multi-agent reinforcement learning (MRL) can leverage the function approximation capabilities of neural networks to understand state information in the environment and learn the mapping between states and actions to create optimal policies, demonstrating promising performance. However, existing MRL research primarily relies on fully cooperative multi-agent systems, using Markov decision processes as the underlying model and assuming that all agents make decisions simultaneously, ultimately converging to a Nash equilibrium. These approaches fail to explicitly model and analyze inter-vehicle conflicts, ignoring the asynchronous coordination of vehicle actions in reality. Their ability to handle complex interactions between agents is limited, leading to uncertain and suboptimal convergence results and overly conservative merging decisions. Therefore, how to account for the asynchrony of vehicle actions in decision-making models is a key issue in improving the safety and reliability of highway merging for intelligent connected vehicles. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method for intelligent connected vehicles to make merging decisions on highways that takes into account the asynchrony of vehicle movements. This method can address the asynchrony of vehicle movement coordination in highway merging areas. By constructing a Stackelberg game model, the merging decision-making process of intelligent connected vehicles on highways under mixed traffic conditions is characterized as a multi-agent reinforcement learning problem based on the Stackelberg game. A spatiotemporal sequence Markov decision process that takes into account the asynchrony of vehicle movements is designed to improve the asynchronous collaborative processing capability of the intelligent connected vehicle merging decision-making model.

[0005] To achieve the above objectives, the present invention provides a method for making decisions about merging on highways for intelligent connected vehicles taking into account asynchrony of movement, comprising the following steps:

[0006] S1. Select a highway merging area with mixed traffic and define the functions of the intelligent connected vehicle (ICV) and the human-driven vehicle (HDV).

[0007] S2. Represent the merging decision-making of intelligent connected vehicles on highways under mixed traffic as a multi-agent reinforcement learning problem based on the Stackelberg game, and design a spatiotemporal Markov decision process that takes into account the asynchrony of vehicle actions.

[0008] S3. Construct Stackelberg game model;

[0009] S4. Use the Actor-Critic multi-agent reinforcement learning framework for training and evaluation.

[0010] Furthermore, in step S1, the functions of the intelligent connected vehicle (ICV) and the human-driven vehicle (HDV) are defined as follows:

[0011] ICV can obtain the operating status information of surrounding vehicles, such as speed and position, through sensors;

[0012] ICVs can obtain information from other ICVs and traffic communication facilities through on-board communication modules, such as real-time traffic information and road conditions, to achieve complete information sharing. HDVs do not have communication functions.

[0013] ICV's decision-making is based on a hierarchical decision-making control model. High-level decisions are made by a multi-agent reinforcement learning algorithm, and throttle and steering operations are implemented with the help of an underlying PID controller. HDV's decision-making is based on instructions output by the following model and lane-changing model.

[0014] Furthermore, the high-level decisions include changing lanes to the left, changing lanes to the right, accelerating, decelerating, and maintaining the current state.

[0015] Furthermore, in step S2, a spatiotemporal Markov decision process is designed to address the asynchrony of vehicle coordination in the merging area of ​​the highway, which can be expressed as:

[0016] STMDP={S,A,P,R,γ,O}

[0017] Where S represents the state space; A represents the action space; P represents the state transfer matrix, which represents the dynamic characteristics of the entire system; R represents the reward function; γ represents the discount factor; o i represents the action sequence of agent i, O=(o 1 , o 2 ,...,o m ) represents the action sequence of all agents, m represents the number of ICVs in the scene, and the order is based on the safety priority index p of each ICV i to decide, indicating the importance of different ICVs in the decision-making stage;

[0018] The state space of the multi-agent system as a whole is the Cartesian product of all ICV state spaces, expressed as:

[0019] S=S1×S2×…×S m

[0020]

[0021] Where S t is the state space of an ICV at time t; represents the longitudinal position of the vehicle, represents the lateral position of the vehicle, The specific value of is calculated with the intersection of the vertical line of the vehicle's leftmost edge relative to the driving direction on the main road of the expressway merging area and the horizontal line of the vehicle's uppermost longitudinal edge relative to the road section entrance as the origin of the Cartesian coordinate system; Indicates the longitudinal speed of the vehicle; represents the lateral speed of the vehicle; n represents the number of all ICVs and HDVs. Assume that the environment contains nm HDVs and m ICVs;

[0022] Among them, the action space of the entire system is the joint action of each ICV, which is expressed as:

[0023] A=A1×A2×…A m

[0024] A i ={a left , a right , a up , a down , a cruising}

[0025] Where a left , a right , a up , a down , a cruising They are changing lanes to the left, changing lanes to the right, accelerating, decelerating and maintaining the status quo; A i is a set of discrete high-level control decisions, is the action space of the i-th ICV; when an acceleration or deceleration operation is performed, the corresponding acceleration change of the vehicle is 2m / s 2 .

[0026] Furthermore, the process of constructing the Stackelberg game model in step S3 is as follows:

[0027] S3.1 describes the vehicle merging game scenario using multi-agent reinforcement learning and identifies the game theory elements involved in the merging process on highway ramps.

[0028] The elements include: participants, strategy sets, payoff functions, information sets and equilibrium;

[0029] Participants are all ICVs that enter the confluence area; the strategy set is the action space of the ICVs; the payoff function is the reward function of the ICVs; the information set is the state space of the ICVs; and the equilibrium is a stable state reached by multiple ICVs when they respond to each other optimally.

[0030] S3.2 uses multi-agent reinforcement learning to construct a Stackelberg game model, which can be expressed as:

[0031]

[0032] Where, Indicates that the leader makes a decision π l and the follower makes a decision π f The leader’s strategy function when represents the strategy function of the follower under the same circumstances; BR(π l ) indicates that the follower observes the leader's decision π f The best response strategy after

[0033] S3.3 further clarifies the equilibrium goal of the Stackelberg game, which is expressed as:

[0034]

[0035] Where, Indicates that the leader makes a decision π lSE and the follower makes a decision π fSE The state value function of the leader in state s; Indicates that the leader makes a decision π l and the follower makes a decision π f The state value function of the leader in state s; BR(π lSE ) indicates that the follower observes the leader's decision π lSE The best response strategy after

[0036] S3.4 Design an ICV security priority index. Determine the leader and follower levels based on the ICV security priority index. The larger the security priority index, the closer the ICV's position is to the top layer in the multi-layer leader-follower structure.

[0037] Define the safety priority index of the i-th ICV as p i , expressed as:

[0038] p i =λ1p l +λ2p e +λ3p h +η i

[0039] Where λ1, λ2, and λ3 represent the merge priority metrics p l , merge distance priority metric p e And the headway priority metric p h The weight coefficient, η i is subject to η i The purpose is to prevent the occurrence of two ICV safety priority indicators p i Same situation; p l , p e , p h Respectively expressed as:

[0040]

[0041] Where x and L are the distance traveled by the i-th ICV in the merging lane and the length of the merging lane respectively; d headway is the headway; t h is the predefined headway threshold; v t is the vehicle speed.

[0042] Furthermore, step S4 includes the following sub-steps:

[0043] S4.1 At time t, the i-th ICV changes its current state Enter the Actor network and take actions that have a higher security priority than its own ICV in the Stackelberg game model As a prerequisite, combine the action probability output by the Actor network to make a response decision

[0044] S4.2Critic will be the current state and response decisions As input, it outputs the current state function value to evaluate the utility of the action taken in the current state;

[0045] S4.3 Each time a decision is made After that, the central controller assigns immediate rewards to the actions taken according to the reward function

[0046] S4.4 After a spatiotemporal Markov decision process is completed, if the merging goal has been achieved, the entire decision process ends; if not, the above operations are repeated;

[0047] The entire state transition process will be recorded by the experience replay buffer, and the Actor-Critic neural network will sample and train these stored experience data to continuously update the optimal strategy. The overall loss function J(θ i ) is expressed as:

[0048]

[0049] Where, represents the objective function of the Actor network; β1 and β2 are weight coefficients; the Actor network uses θ to learn the strategy function π θ ; Critic network uses φ to learn the state value function v φ ; Represents the loss function of the Critic network; represents the entropy regularization term, which can be expressed as:

[0050]

[0051] Where, Denote the advantage function as follows:

[0052]

[0053] Where, Indicates that the i-th ICV follows the policy function π θ The expected return that can be obtained when Indicates that the i-th ICV is replaying the buffer according to the experience The expected return from the data distribution in R i,t It represents the immediate reward obtained by the i-th ICV at time t.

[0054] Beneficial effects:

[0055] The present invention provides a method for intelligent connected vehicles to make merging decisions on highways that takes into account the asynchrony of vehicle movements. This method can effectively improve the merging decision-making ability of intelligent connected vehicles in highway merging areas under mixed traffic conditions by targeting the asynchrony of vehicle movement coordination in highway merging areas.

[0056] The present invention proposes a highway merging decision-making method for intelligent connected vehicles that takes into account the asynchrony of vehicle movements. This method, combined with the intelligent and information-based application requirements of my country's highway traffic management, supports the innovative development of intelligent connected vehicles and autonomous driving technologies, enabling intelligent connected vehicles to make reasonable decisions in complex scenarios such as mixed-traffic highway merging areas. This method has certain practical significance for promoting the commercial application of autonomous driving and improving traffic conditions in highway merging areas.

[0057] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a highway merging decision-making method for an intelligent connected vehicle considering action asynchrony of the present invention;

[0059] Figure 2 Schematic diagram of the execution mechanism of ICV under the spatiotemporal sequence Markov decision framework;

[0060] Figure 3 Schematic diagram of the Stackelberg game model in the highway merging area;

[0061] Figure 4 Flowchart for training highway merging decisions for intelligent connected vehicles;

[0062] Figure 5 The figure is a schematic diagram of the experimental scene setting of the highway merging area in this embodiment. DETAILED DESCRIPTION

[0063] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0064] like Figure 1 and Figure 4 As shown, the present invention provides a method for making decisions on highway merging for intelligent connected vehicles considering action asynchrony, comprising the following steps:

[0065] S1. Select a highway merging area with mixed traffic and define the functions of the intelligent connected vehicle (ICV) and the human-driven vehicle (HDV).

[0066] In this embodiment, the decision-making of ICVs in the merging area of ​​a highway is simulated based on the Highway-env simulation platform. In this environment, the ICV interacts with the environment through a reinforcement learning algorithm, and an experimental scenario is designed, which consists of a two-lane main road and a single-lane ramp. The total length of the road is 520m, of which the AB section is 220m, the BC section is 100m, the merging lane CD section is 100m, and the DE section is 100m. 18 vehicle generation points are set on the main road and ramp of the AB section. In order to simulate the traffic scene in the real world, these vehicle generation points are set according to the law of uniform distribution, such as Figure 5 shown.

[0067] In order to ensure the effectiveness and robustness of the model under diverse traffic conditions, three traffic density patterns with different initial numbers of vehicles are set, expressed as:

[0068] Easy mode: 1-3 ICVs and 1-3 HDVs;

[0069] Medium mode: 2-4 ICVs and 2-4 HDVs;

[0070] Hard Mode: 4-6 ICVs and 3-5 HDVs.

[0071] ICV can obtain the operating status information of surrounding vehicles, including speed and position, through sensors;

[0072] ICV can obtain information from other ICVs and traffic communication facilities through the on-board communication module, including real-time traffic information and road conditions, to achieve complete information sharing. HDV does not have communication functions.

[0073] ICV decisions are based on a hierarchical decision-making control model. High-level decisions (changing lanes left, changing lanes right, accelerating, decelerating, and maintaining the status quo) are made by a multi-agent reinforcement learning algorithm, with the help of an underlying PID controller to implement throttle and steering operations. HDV decisions are based on instructions output by the following model and the lane-changing model.

[0074] S2. The highway merging decision of intelligent connected vehicles under mixed traffic is represented as a multi-agent reinforcement learning problem based on Stackelberg game. A spatiotemporal Markov decision process is designed considering the asynchrony of vehicle actions, such as Figure 2 As shown;

[0075] S2.1 Aiming at the asynchronous nature of vehicle coordination in merging areas on highways, a spatiotemporal Markov decision process is designed, which can be expressed as:

[0076] STMDP={S,A,P,R,γ,O}

[0077] Where S represents the state space; A represents the action space; P represents the state transfer matrix, which represents the dynamic characteristics of the entire system; R represents the reward function; γ represents the discount factor; o i represents the action sequence of agent i, O=(o 1 , o 2 ,...,o m ) represents the action sequence of all agents, m represents the number of ICVs in the scene, and the order is based on the safety priority index p of each ICV i to decide, indicating the importance of different ICVs in the decision-making stage;

[0078] S2.2State space definition;

[0079] The state space S of an ICV at time t t It can be defined as:

[0080]

[0081] Where, represents the longitudinal position of the vehicle, represents the lateral position of the vehicle, The specific value of is calculated by taking the intersection of the vertical line of the vehicle on the main road of the expressway merging area relative to the leftmost edge in the driving direction and the horizontal line of the vehicle relative to the longitudinal uppermost edge of the road section entrance as the origin of the Cartesian coordinate system. represents the longitudinal velocity of the vehicle, represents the lateral velocity of the vehicle. The vehicle here refers to any vehicle in the environment, which can be an ICV or an HDV. n represents the number of all ICVs and HDVs. Assume that the environment contains (nm) HDVs and m ICVs. The state space of the multi-agent system is the Cartesian product of the state spaces of all ICVs, which can be expressed as:

[0082] S=S1×S2×…×S m

[0083] S2.3 Action space definition;

[0084] The action space of the i-th ICV is defined as:

[0085] A i ={a left , a right , a up , a down , a cruising}

[0086] Where A i is a set of discrete high-level control decisions, a left , a right , aup , a down , a cruising They are changing lanes to the left, changing lanes to the right, accelerating, decelerating, and maintaining the status quo. Executing an acceleration or deceleration operation will result in a vehicle acceleration change of 2 m / s. 2 The underlying controller generates steering and throttle control signals to steer the ICV based on the actions in the action space. At each moment, the ICV calculates the action to be performed based on the currently observed state. The action space of the entire system is the combined actions of each ICV, expressed as:

[0087] A=A1×A2×…A m

[0088] S2.4 Reward function definition;

[0089] S2.41 The reward function of the i-th ICV is defined as:

[0090] r i =w c r c +w s r s +w h r h +w m r m

[0091] Where w c , w s , w h and w m The collision reward r is c , speed reward r s , headway reward r h and the confluence cost reward r m The corresponding weight coefficient. The four reward indicators are defined as follows:

[0092]

[0093]

[0094] Where, according to the speed range observed in the NGSIM dataset, the minimum vehicle speed is set to v min =10m / s, the maximum speed is set to v max =30m / s;d headway is the headway, t h is the predefined headway threshold, set to 1.2s; the merging cost reward r m The design is to prevent vehicles from waiting too long in the acceleration lane and causing collisions. x is the distance the vehicle travels on the ramp, and L is the length of the ramp.

[0095] S2.42 The ultimate goal of asynchronous collaboration of multiple ICVs is to maximize the global reward, which is expressed as:

[0096]

[0097] Where m represents the number of ICVs, and each ICV is assigned the same global average reward, expressed as:

[0098]

[0099] S3. Construct a Stackelberg game model, such as Figure 3 As shown;

[0100] S3.1 describes the vehicle merging game scenario using multi-agent reinforcement learning and identifies the game theory elements involved in the merging process on highway ramps.

[0101] The elements include: participants, strategy sets, payoff functions, information sets and equilibrium;

[0102] Participants are all ICVs that enter the confluence area; the strategy set is the action space of the ICVs; the payoff function is the reward function of the ICVs; the information set is the state space of the ICVs; and the equilibrium is a stable state reached by multiple ICVs when they respond to each other optimally.

[0103] S3.2 uses multi-agent reinforcement learning to construct a Stackelberg game model, which can be expressed as:

[0104]

[0105] Where, Indicates that the leader makes a decision π l and the follower makes a decision π f The leader’s strategy function when represents the strategy function of the follower under the same circumstances; BR(π l ) indicates that the follower observes the leader's decision π f The best response strategy after

[0106] S3.3 further clarifies the equilibrium goal of the Stackelberg game, which is expressed as:

[0107]

[0108] Where, Indicates that the leader makes a decision π lSE and the follower makes a decision π fSE The state value function of the leader in state s; Indicates that the leader makes a decision πl and the follower makes a decision π f The state value function of the leader in state s; BR(π lSE ) indicates that the follower observes the leader's decision π lSE In Stackelberg equilibrium, no participant can improve their state-value function by changing their strategy.

[0109] S3.4 Design an ICV security priority indicator. Based on the size of the ICV security priority indicator, determine the leader and follower levels. The larger the value of the security priority indicator, the closer the ICV is to the top level in the multi-layer leader-follower structure. The definition of the security priority indicator follows the following rules:

[0110] I. Vehicles in the acceleration lane have higher priority than vehicles on the ramp and main road. Because vehicles in the acceleration lane face more urgent merging tasks in terms of time and space, they should be given higher priority than vehicles on the ramp and main road.

[0111] II. Vehicles closer to the end of the acceleration lane should have higher priority because they are more prone to collisions and congestion.

[0112] III. Vehicles with smaller headway should have higher priority because they are more likely to collide with the vehicle in front.

[0113] Define the safety priority index of the i-th ICV as p i , expressed as:

[0114] p i =λ1p l +λ2p e +λ3p h +η i

[0115] Where λ1, λ2, and λ3 represent the merge priority metrics p l , merge distance priority metric p e And the headway priority metric p h The weight coefficient, η i is subject to η i The purpose is to prevent the occurrence of two ICV safety priority indicators p i Same situation; p l , p e , p h Respectively expressed as:

[0116]

[0117] Where x and L are the distance traveled by the i-th ICV in the merging lane and the length of the merging lane respectively; d headway is the headway; t h is the predefined headway threshold; v t is the vehicle speed.

[0118] S4. Use the Actor-Critic multi-agent reinforcement learning framework for training and evaluation;

[0119] S4.1 At time t, the i-th ICV changes its current state Enter the Actor network and take actions that have a higher security priority than its own ICV in the Stackelberg game model As a prerequisite, combine the action probability output by the Actor network to make a response decision

[0120] S4.2Critic will be the current state and response decisions As input, it outputs the current state function value to evaluate the utility of the action taken in the current state;

[0121] S4.3 Each time a decision is made After that, the central controller assigns immediate rewards to the actions taken according to the reward function

[0122] S4.4 After a spatiotemporal Markov decision process is completed, if the merging goal has been achieved, the entire decision process ends; if not, the above operation is repeated; the entire state transition process is recorded by the experience replay buffer, and the Actor-Critic neural network samples and trains these stored experience data to continuously update the optimal strategy. The overall loss function J(θ i ) is expressed as:

[0123]

[0124] Where, represents the objective function of the Actor network; β1 and β2 are weight coefficients; the Actor network uses θ to learn the strategy function π θ ; Critic network uses φ to learn the state value function v φ ; Represents the loss function of the Critic network; represents the entropy regularization term, which can be expressed as:

[0125]

[0126] Where, Denote the advantage function as follows:

[0127]

[0128] Where, Indicates that the i-th ICV follows the policy function π θ The expected return that can be obtained when Indicates that the i-th ICV is replaying the buffer according to the experience The expected return from the data distribution in R i,t represents the immediate reward obtained by the i-th ICV at time t;

[0129] The overall loss function J(θ i ) integrates the Stackelberg game model and considers the asynchronous coordination of vehicles in the merging area of ​​the highway. The Actor network outputs the probability value of each action, which is used to learn the policy function of ICV The value of the Critic network output state value function Used to guide the policy updates of the Actor network.

[0130] In this embodiment, the parameters are determined as follows:

[0131] The Actor Network structure consists of five layers: one input layer, three hidden layers, and one output layer. The input layer is the state space, and the hidden layers consist of 128, 64, and 32 neurons, respectively. The activation function is set to the ReLU function. The output layer is the probability of each action. The Critic Network structure is similar to the Actor Network structure, except that the input layer consists of the state space and the encoding of the actions generated by the Actor Network. Because the input layer contains additional action encoding information and has a larger dimension, the number of hidden layer neurons is larger than that of the Actor Network: 256, 128, and 64, respectively. The discount factor γ is 0.99, the experience replay pool size is set to 30,000, and the batch size is set to 128.

[0132] In each training round, the initial speed of the vehicle is randomly selected between 25 and 27 m / s. The ICV simulation frequency, that is, the control sampling frequency, is 5 Hz, which means that the ICV performs 5 simulation steps per second and takes an action every 0.2 seconds. The strategy frequency is 1 Hz, which means that the ICV strategy is evaluated and updated once in each simulation step. The experiment selected 3 different random seeds for a total of 2 million steps of training, and the ICVs shared the same random seeds. Each complete training process consists of 20,000 independent rounds, each with 100 time steps. At each time step, the ICV makes a decision based on its currently learned strategy. In each round, the ICV continuously interacts with the environment and updates its strategy from the initial state until it completes the merge and reaches the terminal state or a collision occurs, which is considered a complete training round.

[0133] Experimental results

[0134] To verify the effectiveness and robustness of the method, the model was trained under three traffic density modes with different initial vehicle numbers. After training, all algorithms were further tested 30 times with random seeds under the three traffic densities and compared with the baseline models MAACKTR and MAPPO. The average collision rates and vehicle speeds are shown in Table 1.

[0135] Table 1 Comparison of collision rate and average driving speed under three traffic density levels

[0136]

[0137] The results show that compared with other benchmark algorithms, the method in this embodiment achieves higher safety and efficiency, indicating that considering the asynchrony of vehicle actions in the decision-making model can improve the safety, reliability and traffic efficiency of highway merging for intelligent connected vehicles.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of protection of the present invention.

Claims

1. A highway merging decision-making method for intelligent connected vehicles considering asynchrony of movement, characterized by: The following steps are involved: S1. Select a highway merging area with mixed traffic and define the functions of the intelligent connected vehicle (ICV) and the human-driven vehicle (HDV). S2. Represent the merging decision-making of intelligent connected vehicles on highways under mixed traffic as a multi-agent reinforcement learning problem based on the Stackelberg game, and design a spatiotemporal Markov decision process that takes into account the asynchrony of vehicle actions. In step S2, a spatiotemporal Markov decision process is designed to address the asynchrony of vehicle coordination in the merging area of ​​the highway, which can be expressed as: STMDP={S,A,P,R,γ,O} Where S represents the state space; A represents the action space; P represents the state transfer matrix, which represents the dynamic characteristics of the entire system; R represents the reward function; γ represents the discount factor; o i represents the action sequence of agent i, O=(o 1 ,o 2 ,…,o m ) represents the action sequence of all agents, m represents the number of ICVs in the scene, and the order is based on the safety priority index p of each ICV i to decide, indicating the importance of different ICVs in the decision-making stage; The state space of the multi-agent system as a whole is the Cartesian product of all ICV state spaces, expressed as: S=S1×S2×…×S m Where S t is the state space of an ICV at time t; represents the longitudinal position of the vehicle, represents the lateral position of the vehicle, The specific value of is calculated with the intersection of the vertical line of the vehicle's leftmost edge relative to the driving direction on the main road of the expressway merging area and the horizontal line of the vehicle's uppermost longitudinal edge relative to the road section entrance as the origin of the Cartesian coordinate system; Indicates the longitudinal speed of the vehicle; represents the lateral speed of the vehicle; n represents the number of all ICVs and HDVs. Assume that the environment contains nm HDVs and m ICVs; Among them, the action space of the entire system is the joint action of each ICV, which is expressed as: A=A1×A2×…A m A i ={a left ,a right ,a up ,a down ,a cruising } Where a left ,a right ,a up ,a down ,a cruising They are changing lanes to the left, changing lanes to the right, accelerating, decelerating and maintaining the status quo; A i is a set of discrete high-level control decisions, is the action space of the i-th ICV; when an acceleration or deceleration operation is performed, the corresponding acceleration change of the vehicle is 2m / s 2 ; S3. Construct Stackelberg game model; The process of constructing the Stackelberg game model in step S3 is as follows: S3.1 describes the vehicle merging game scenario using multi-agent reinforcement learning and identifies the game theory elements involved in the merging process on highway ramps. The elements include: participants, strategy sets, payoff functions, information sets and equilibrium; Participants are all ICVs that enter the confluence area; the strategy set is the action space of the ICVs; the payoff function is the reward function of the ICVs; the information set is the state space of the ICVs; and the equilibrium is a stable state reached by multiple ICVs when they respond to each other optimally. S3.2 uses multi-agent reinforcement learning to construct a Stackelberg game model, which can be expressed as: Where, Indicates that the leader makes a decision π l and the follower makes a decision π f The leader’s strategy function when represents the strategy function of the follower under the same circumstances; BR(π l ) indicates that the follower observes the leader's decision π l The best response strategy after S3.3 further clarifies the equilibrium goal of the Stackelberg game, which is expressed as: Where, Indicates that the leader makes a decision π lSE and the follower makes a decision π fSE The state value function of the leader in state s; Indicates that the leader makes a decision π l and the follower makes a decision π f The state value function of the leader in state s; BR(π lSE ) indicates that the follower observes the leader's decision π lSE The best response strategy after S3.4 Design an ICV security priority index. Based on the value of the ICV security priority index, determine the leader and follower levels. The larger the value of the security priority index, the closer the ICV's position is to the top level in the multi-layer leader-follower structure. Define the safety priority index of the i-th ICV as p i , expressed as: p i =λ1p l +λ2p e +λ3p h +n i Where λ1, λ2, and λ3 represent the merge priority metrics p l , merge distance priority metric p e And the headway priority metric p h The weight coefficient, η i is subject to η i The purpose is to prevent the occurrence of two ICV safety priority indicators p i Same situation; p l ,p e ,p h Respectively expressed as: Where x and L are the distance traveled by the i-th ICV in the merging lane and the length of the merging lane respectively; d headway is the headway; t h is the predefined headway threshold; v t is the vehicle speed; S4. Use the Actor-Critic multi-agent reinforcement learning framework for training and evaluation.

2. The method for making decisions on highway merging for intelligent connected vehicles considering asynchrony of movement according to claim 1, characterized in that: In step S1, the functions of the intelligent connected vehicle (ICV) and the human-driven vehicle (HDV) are defined as follows: ICV can obtain the operating status information of surrounding vehicles, including speed and position, through sensors; ICVs can obtain information from other ICVs and traffic communication facilities through on-board communication modules, including real-time traffic information and road conditions, to achieve complete information sharing. HDVs do not have communication functions. ICV decisions are based on a hierarchical decision-making control model. High-level control decisions are made by a multi-agent reinforcement learning algorithm, and throttle and steering operations are implemented with the help of an underlying PID controller. HDV decisions are based on instructions output by the following model and lane-changing model.

3. The method for making decisions on highway merging for intelligent connected vehicles considering asynchrony of movement according to claim 1, characterized in that: The step S4 includes the following sub-steps: S4.1 At time t, the i-th ICV changes its current state Enter the Actor network and take actions that have a higher security priority than its own ICV in the Stackelberg game model As a prerequisite, combine the action probability output by the Actor network to make a response decision S4.2Critic will be the current state and response decisions As input, it outputs the current state function value to evaluate the utility of the action taken in the current state; S4.3 Each time a decision is made After that, the central controller assigns immediate rewards to the actions taken according to the reward function S4.4 After a spatiotemporal sequence Markov decision process is completed, if the merging goal has been achieved, the entire decision process ends; if not, the above operations are repeated; The entire state transition process will be recorded by the experience replay buffer, and the Actor-Critic neural network will sample and train these stored experience data to continuously update the optimal strategy. The overall loss function J(θ i ) is expressed as: Where, represents the objective function of the Actor network; β1 and β2 are weight coefficients; the Actor network uses θ to learn the strategy function π θ ; Critic network uses φ to learn the state value function v φ ; Represents the loss function of the Critic network; represents the entropy regularization term, which can be expressed as: Where, Denote the advantage function as follows: Where, Indicates that the i-th ICV follows the policy function π θ The expected return that can be obtained when Indicates that the i-th ICV is replaying the buffer according to the experience The expected return from the data distribution in R i,t It represents the immediate reward obtained by the i-th ICV at time t.

Images