An intelligent vehicle scheduling method for ramp merging scene

Abstract

The application discloses a kind of intelligent vehicle scheduling methods for ramp merging scene, comprising: receiving the local state information sent by each intelligent connected vehicle in current ramp merging scene, and combined into joint state space with the global state information obtained;Joint state space is input into the depth reinforcement learning model trained, and the joint action space of intelligent connected vehicle is obtained;According to the joint action space, each intelligent connected vehicle is scheduled;Wherein the depth reinforcement learning model includes current policy network, target policy network and value network;The acquisition method of the depth reinforcement learning model includes: iteratively performing total cycle steps until reaching preset condition, and obtaining the depth reinforcement learning model trained.The interaction information of each step in the interaction process of historical intelligent connected vehicle and environment is used to train and update the parameters of current policy network, target policy network and value network in depth reinforcement learning model.The application improves the vehicle throughput in ramp merging scene.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, and specifically relates to an intelligent vehicle scheduling method for ramp merging scenarios. Background Technology

[0002] Intelligent transportation technology is a product of the deep integration of the automotive industry with new-generation information technologies such as the Internet of Things, artificial intelligence, and high-performance computing. It is the main direction of the intelligent and connected development of the traditional transportation sector and has great significance in ensuring traffic efficiency, improving traffic safety, and enhancing the user's driving experience.

[0003] The Vehicle-Infrastructure Cooperative System (VICS) utilizes advanced wireless communication and next-generation Internet technologies to achieve comprehensive real-time information interaction between vehicles and the road network. Based on the collection and fusion of dynamic traffic information across all times and spaces, it enables vehicle safety control and collaborative road management, ensuring traffic safety, improving traffic efficiency, and achieving a safe, efficient, and environmentally friendly road traffic system. The VICS is mainly divided into roadside units, on-board units, and cloud systems. The Roadside Unit (RSU) is the core infrastructure of the VICS, serving as a hub for sensing road network characteristics and exchanging information about road participants. Its main coverage scenarios include intersections, highways, and ramps. The RSU not only interfaces with various signals and monitoring and management systems but also needs to interact with edge computing and the traffic management system. The On-Board Unit (OBU) is a hardware unit installed in a vehicle that supports vehicle-to-everything (V2X) wireless communication technology. The OBU can acquire raw information from various sensors and on-board networks, calculate the underlying information required for VICS applications, and transmit it to the Roadside Unit.

[0004] Despite significant progress in autonomous driving over the past few years, the number of traffic accidents involving Connected Autonomous Vehicles (CAVs) has continued to rise. These accidents are often caused by CAVs' inability to react promptly to dynamic driving environments, particularly in traffic where CAVs and human-driven vehicles (HDVs) share the road. CAVs not only need to react to objects on the road but also need to be aware of the behavior of HDVs. Among the many challenging driving scenarios, ramp merging is one of the most difficult tasks for CAVs. Summary of the Invention

[0005] Because both commercially active vehicles (CAVs) and high-level traffic vehicles (HDVs) coexist in mixed traffic flows, and HDVs tend to maximize individual interests rather than the overall benefit, coupled with the inherent randomness of human driving, these are all significant causes of traffic congestion and collisions. Autonomous driving systems, controlled by predefined policies, can act selflessly and drive strategically to influence human driver behavior, thereby reducing congestion and improving overall traffic efficiency.

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent vehicle scheduling method for ramp merging scenarios. This method, targeting ramp merging scenarios with mixed traffic flows, employs an intelligent scheduling strategy based on deep reinforcement learning algorithms to schedule vehicles more safely and efficiently, thereby improving the overall traffic efficiency of ramp entrances and reducing congestion.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an intelligent vehicle scheduling method for ramp merging scenarios, the method comprising:

[0009] It receives local state information from each intelligent connected vehicle in the current ramp merging scenario and combines it with the acquired global state information to form a joint state space.

[0010] Joint state space By inputting the data into a pre-trained deep reinforcement learning model, the joint action space of the intelligent connected vehicle is obtained.

[0011] According to the joint action space Dispatch and manage all intelligent connected vehicles;

[0012] The deep reinforcement learning model includes a current policy network, a target policy network, and a value network; the method for obtaining the deep reinforcement learning model includes iteratively executing the overall loop steps until a preset condition is met to obtain a trained deep reinforcement learning model.

[0013] In some embodiments, the total loop step includes:

[0014] S1. Execute the first loop step to obtain the interaction information at each step in the interaction process between the intelligent connected vehicle and the environment. The first loop step includes:

[0015] S101. Receive local state information sent by each intelligent connected vehicle in the ramp merging scenario, and combine it with the acquired global state information to form a joint state space.

[0016] S102, Combine the state space Inputting this into the current deep reinforcement learning model yields the joint action space of the intelligent connected vehicle. Calculate the reward value r t This forms the interactive information for the corresponding steps. and joint action space Send to the corresponding intelligent connected vehicle;

[0017] S103, responding to intelligent connected vehicles based on joint action space Perform the corresponding action to transition the environment to the next joint state.

[0018] S2, record the interaction information at each step. All values ​​are input into the value network to obtain the state value V = [V1, V2, ..., V] at each step. T ], and calculate the discount reward R = [R1, R2, ..., R] for each step based on the state value. T ];

[0019] S3. Calculate the loss function L(ω) of the value network based on the state value and the discounted reward, and update the parameters of the value network using gradient descent and backpropagation.

[0020] S4. Execute the second loop step, updating the parameters of the current policy network M times, where M≤T, to obtain the updated parameters of the current policy network; wherein the second loop step includes:

[0021] The state s in the interaction information t and action a t The inputs are fed into the current policy network and the target policy network respectively, and the importance weights I are calculated using the importance sampling method. t (θ); Calculate the dominance function based on state value and discounted reward. And combined with importance weight I t (θ) Find the objective function L of the current policy network. CLIP (θ), using backpropagation to update the parameters of the current policy network;

[0022] S5. Update the parameters of the target policy network using the updated parameters of the current policy network.

[0023] The joint state space for:

[0024]

[0025] s i =(v i ,h f ,h r ,h lf,h lr ,h rf ,h rr ,v f ,v r ,v lf ,v lr ,v rf ,v rr ,q i ,q l ,q r )

[0026] Among them, s i Let v represent the state space of the i-th intelligent connected vehicle, n represent the maximum number of intelligent connected vehicles in the joint state space, and v i Let h represent the longitudinal velocity of the i-th intelligent connected vehicle. f ,h r ,h lf ,h lr ,h rf ,h rr Let v represent the relative distances between the i-th intelligent connected vehicle and the vehicle in front, the vehicle in behind, the vehicle in front in the adjacent lane to the left, the vehicle in front in the adjacent lane to the left, the vehicle in front in the adjacent lane to the right, and the vehicle in front in the adjacent lane to the right, respectively; f ,v r ,v lf ,v lr ,v rf ,v rr Let q represent the relative speeds between the i-th intelligent connected vehicle and the vehicles in front, behind, to the left of the vehicle in the adjacent lane, to the right of the vehicle in the adjacent lane, and to the right of the vehicle in the adjacent lane. i ,q l ,q r Let q represent the equivalent average traffic flow in the lane where the i-th connected vehicle is located, the lane adjacent to its left, and the lane adjacent to its right, respectively; where the equivalent average traffic flow q in lane l is... l for:

[0027] q l =ρ l ·v l

[0028]

[0029]

[0030] Where v l Let ρ be the average longitudinal velocity of all vehicles in lane l. l For the density of lane l, S l N is the length of lane l.l The number of vehicles in lane l;

[0031] In some embodiments, the joint action space for:

[0032]

[0033] a i =(a,c) a∈[-4.5,2.5]m / s 2 c∈{0,0.5,1}

[0034] Where a i Let represent the action space of the i-th intelligent connected vehicle, n represent the maximum number of intelligent connected vehicles in the joint action space, a represent the acceleration of the i-th vehicle, and c represent the lane-changing action of the i-th vehicle, where 0, 0.5, and 1 represent going straight, changing lanes to the left, and changing lanes to the right, respectively.

[0035] In some embodiments, the calculation of the reward value r t ,include:

[0036] r t =β1r1+β2r2+β3r3

[0037]

[0038] r2=δ(t)-δ(t+1)

[0039] r3∈{-1,0}

[0040] Where: r1 is the average speed bonus, N is the number of vehicles in the scene, and V is the average speed bonus. max r1 is the maximum set vehicle speed; r2 is the traffic flow unevenness bonus; r3 is the lane change penalty, r3 = -1 when performing a lane change, otherwise r3 = 0; v i Let β1, β2, and β3 represent the longitudinal speed of the i-th intelligent connected vehicle, and β3 be weighting factors; δ(t) represents the equivalent average traffic flow q of each lane at step t. l The standard deviation.

[0041] In some embodiments, the state value V at step t t =V(s) t ;ω), where V(s) t ;ω) represents the value function, ω represents the parameters of the value network, and the state s at step t is... t The input is fed into the value network to obtain the state value V. t ;

[0042] The formula for calculating discount rewards is:

[0043] Rt =r t +γV(s t+1 ;ω)

[0044] Therefore, using the state value V at step T T Derive the discount reward R at step t forward. t for:

[0045]

[0046] Where, γ k-t γ T-t r is the discount factor for the corresponding step. t Let be the reward value at step t.

[0047] In some embodiments, the method for calculating the loss function L(ω) of the value network includes:

[0048] L(ω)=E t [(R t -V t ) 2 ]

[0049] Where E t Let R be the expectation function. t For the discount reward at step t, V t Let t be the state value at step t;

[0050] Update the parameter ω of the value network using gradient descent based on L(ω):

[0051]

[0052] Where α is the learning rate of the gradient method, The gradient operator uses gradient descent and backpropagation to update the parameters of the value network, making the output state value of the value network closer to the true state value.

[0053] In some embodiments, the importance weight I t The methods for calculating (θ) include:

[0054]

[0055] Where π(·|s) t ;θ),π(·|s t ;θ old ) indicates that the state s t The policy distributions obtained by inputting into the current policy network and the target policy network, where θ represents the parameters of the current policy network. old π(a) represents the parameters of the target policy network. t |s t;θ),π(a t |s t ;θ old This indicates the actual action to be taken (a). t The value of the action obtained by inputting it into the corresponding policy distribution is the value obtained under the corresponding policy distribution.

[0056] In some embodiments, the advantage function is calculated based on state value and discount reward. And combined with importance weight I t (θ) Find the objective function L of the current policy network. CLIP (θ), including:

[0057] Based on the state value V t and discount rewards R t Calculate the advantage function

[0058]

[0059] Based on the advantage function And importance weight I t (θ), find the objective function L of the current policy network. CLIP (θ);

[0060]

[0061] Where ε is the weight of the constraint importance I t The hyperparameters of (θ), clip(I) t (θ), 1-ε, 1+ε) represent the expression for I t The value of (θ) is restricted to the range (1-ε, 1+ε), when I t When (θ) < 1-ε, let I t (θ)=1-ε, similarly when I t When (θ)>1+ε, let I t (θ) = 1 + ε;

[0062] Based on the objective function L of the current policy network CLIP (θ), which uses backpropagation to update the parameters θ of the current policy network.

[0063] In some embodiments, the parameters θ of the target policy network are updated using the updated parameters θ of the current policy network. old ,include:

[0064] θ old ←θ

[0065] The current policy network serves to collect a certain amount of interaction information between intelligent connected vehicles and the environment, which is used to update the value network and the target policy network, thus completing one iteration.

[0066] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0067] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0068] This invention introduces reinforcement learning for multiple intelligent connected vehicles, combining the local action spaces observed by multiple intelligent connected vehicles into a joint state space. The traffic flow density of each lane observed by the RSU is added to the local state space, enabling the deep reinforcement learning strategy to output not only the longitudinal acceleration action of the vehicle, but also the lane-changing action, thus enabling more flexible scheduling of intelligent connected vehicles, thereby improving the overall throughput of the ramp and reducing congestion. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of a ramp merging scenario model according to an embodiment of the present invention.

[0070] Figure 2 This is a flowchart illustrating an intelligent vehicle scheduling method for ramp merging scenarios according to an embodiment of the present invention. Detailed Implementation

[0071] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0072] Before detailing the embodiments of the present invention, a specific scenario example is given to illustrate the scenarios in which the technical solution of the present invention can be used. For example... Figure 1 The diagram shown is a model of a ramp merging scenario in this embodiment. Some relevant settings for the scenario model are as follows:

[0073] (1) The intelligent connected vehicles (CAVs) in the scenario are equipped with on-board units (OBU) and high-precision sensors, GPS, LiDAR and other equipment, which can obtain their own speed, position and perceive the relative position and speed of surrounding vehicles.

[0074] (2) The ramp merging scenario includes a main road and a merging lane, where the main road is further divided into three lanes: L1, L2, and L3.

[0075] (3) Roadside units (RSUs) are deployed on the roadside at road intersections and equipped with LiDAR and MEC servers, covering a circular area with a radius of 400 meters. RSUs can assist CAVs in reinforcement learning training and LiDAR can detect the vehicle density of each lane within the coverage area.

[0076] (4) The traffic flow in the ramp merging scenario is a mixed traffic flow, which includes both intelligent connected vehicles (CAVs) and human-driven vehicles (HDVs). Among them, a% (penetration rate) of vehicles are intelligent connected vehicles (CAVs), and the rest are human-driven vehicles (HDVs).

[0077] (5) Intelligent connected vehicles (CAVs) can communicate with roadside units (RSUs) and other intelligent connected vehicles (CAVs) within the communication range. The delay and packet loss of vehicle-to-vehicle and vehicle-to-infrastructure communication are negligible.

[0078] Example 1

[0079] Firstly, this embodiment provides an intelligent vehicle scheduling method for ramp merging scenarios, wherein the ramp merging scenario includes intelligent connected vehicles (CAVs), human-driven vehicles (HDVs), and roadside units (RSUs); the executing entity of the method can be a roadside unit (RSU), and the method includes:

[0080] It receives local state information from each intelligent connected vehicle in the current ramp merging scenario and combines it with the acquired global state information to form a joint state space.

[0081] Joint state space By inputting the data into a pre-trained deep reinforcement learning model, the joint action space of the intelligent connected vehicle is obtained.

[0082] According to the joint action space Dispatch and manage all intelligent connected vehicles;

[0083] The deep reinforcement learning model includes a current policy network, a target policy network, and a value network; the method for obtaining the deep reinforcement learning model includes iteratively executing the total loop steps S1-S5 until the preset conditions are met, thereby obtaining a trained deep reinforcement learning model.

[0084] S1. Execute the first loop step to obtain the interaction information at each step in the interaction process between the intelligent connected vehicle and the environment. The first loop step includes:

[0085] S101. Receive local state information sent by each intelligent connected vehicle in the ramp merging scenario, and combine it with the acquired global state information to form a joint state space.

[0086] S102, Combine the state space Inputting this into the current deep reinforcement learning model yields the joint action space of the intelligent connected vehicle. Calculate the reward value r t This generates interactive information for the corresponding steps. and joint action space Send to the corresponding intelligent connected vehicle;

[0087] S103, responding to intelligent connected vehicles based on joint action space Perform the corresponding action to transition the environment to the next joint state.

[0088] S2, record the interaction information at each step. All values ​​are input into the value network to obtain the state value V = [V1, V2, ..., V] at each step. T ], and calculate the discount reward R = [R1, R2, ..., R] for each step based on the state value. T ];

[0089] S3. Calculate the loss function L(ω) of the value network based on the state value and the discounted reward, and update the parameters of the value network using gradient descent and backpropagation.

[0090] S4. Execute the second loop step, updating the parameters of the current policy network M times, where M≤T, to obtain the updated parameters of the current policy network; wherein the second loop step includes:

[0091] The state s in the interaction information t and action a t The inputs are fed into the current policy network and the target policy network respectively, and the importance weights I are calculated using the importance sampling method. t (θ); Calculate the dominance function based on state value and discounted reward. And combined with importance weight I t (θ) Find the objective function L of the current policy network. CLIP (θ), using backpropagation to update the parameters of the current policy network;

[0092] S5. Update the parameters of the target policy network using the updated parameters of the current policy network.

[0093] In some specific embodiments, such as Figure 2 As shown, the intelligent vehicle scheduling method for ramp merging scenarios in this embodiment includes the following steps:

[0094] Step 1: Initialize the environmental parameters and deep reinforcement learning model parameters for the ramp merging scenario used for training. Environmental parameters include lane length, width, lane distribution, and the initial positions, speeds, and numbers of vehicles in the ramp merging scenario. This scheduling algorithm uses a PPO-based deep reinforcement learning algorithm; therefore, the deep reinforcement learning model includes a current policy network, a target policy network, and a value network. The parameters θ, θ' ... old ω is initialized. The policy network is used to select the appropriate action based on the current state, and the value network is used to score the current state.

[0095] Step 2: During the training process in step t, the intelligent connected vehicle observes local state information through its own sensors and sends it to the Roadside Unit (RSU). This information is then combined with the global state information observed by the RSU to form a joint state space.

[0096] The joint state space during step t for:

[0097]

[0098] s i =(v i ,h f ,h r ,h lf ,h lr ,h rf ,h rr ,v f ,v r ,v lf ,v lr ,v rf ,v rr ,q i ,q l ,q r )

[0099] Among them, s i Let v represent the state space of the i-th intelligent connected vehicle, n represent the maximum number of intelligent connected vehicles in the joint state space, and v i Let h represent the longitudinal velocity of the i-th intelligent connected vehicle. f ,h r ,h lf ,h lr ,h rf ,h rr Let v represent the relative distances between the i-th intelligent connected vehicle and the vehicle in front, the vehicle in behind, the vehicle in front in the adjacent lane to the left, the vehicle in front in the adjacent lane to the left, the vehicle in front in the adjacent lane to the right, and the vehicle in front in the adjacent lane to the right, respectively; f,v r ,v lf ,v lr ,v rf ,v rr Let q represent the relative speeds between the i-th intelligent connected vehicle and the vehicles in front, behind, to the left of the vehicle in the adjacent lane, to the right of the vehicle in the adjacent lane, and to the right of the vehicle in the adjacent lane. i ,q l ,q r Let q represent the equivalent average traffic flow in the lane where the i-th connected vehicle is located, the lane adjacent to its left, and the lane adjacent to its right, respectively; where the equivalent average traffic flow q in lane l is... l for:

[0100] q l =ρ l ·v l

[0101]

[0102]

[0103] Where v l Let ρ be the average longitudinal velocity of all vehicles in lane l. l For the density of lane l, S l N is the length of lane l. l The number of vehicles in lane l;

[0104] Step 3, during the t-th training process, the roadside unit (RSU) will... The input is fed into the current deep reinforcement learning model to obtain the joint action space of the intelligent connected vehicle. And calculate the reward value r t Then The message is sent to the corresponding vehicle, which then performs the appropriate action to transition the environment to the next state.

[0105] The joint action space during step t for:

[0106]

[0107] a i =(a,c) a∈[-4.5,2.5]m / s 2 c∈{0,0.5,1}

[0108] Where a iLet represent the action space of the i-th intelligent connected vehicle, n represent the maximum number of intelligent connected vehicles in the joint action space, a represent the acceleration of the i-th vehicle, and c represent the lane-changing action of the i-th vehicle, where 0, 0.5, and 1 represent going straight, changing lanes to the left, and changing lanes to the right, respectively.

[0109] In this embodiment, the reward value r is calculated. t ,include:

[0110] r t =β1r1+β2r2+β3r3

[0111]

[0112] r2=δ(t)-δ(t+1)

[0113] r3∈{-1,0}

[0114] Where: r1 is the average speed bonus, N is the number of vehicles in the scene, and V is the average speed bonus. max r1 is the maximum set vehicle speed; r2 is the traffic flow unevenness bonus; r3 is the lane change penalty, r3 = -1 when performing a lane change, otherwise r3 = 0; v i Let β1, β2, and β3 represent the longitudinal speed of the i-th intelligent connected vehicle, and β3 be weighting factors; δ(t) represents the equivalent average traffic flow q of each lane at step t. l The standard deviation.

[0115] Step 4: Repeat the interaction process between the intelligent connected vehicle and the environment in steps 2 and 3 T times, and record the interaction information at each step. All are stored in the RSU's local cache;

[0116] Step 5: Store the interaction information from each step. All values ​​are input into the value network to obtain the state value V = [V1, V2, ..., V] at each step. T Then, the discount reward R = [R1, R2, ..., R] is calculated for each step. T ];

[0117] The state value V at step t t =V(s) t ;ω), where V(s) t ;ω) represents the value function, ω represents the parameters of the value network, and the state s at step t is... t The input is fed into the value network to obtain the state value V. t ;

[0118] The formula for calculating discount rewards is:

[0119] R t =rt +γV(s t+1 ;ω)

[0120] Therefore, using the state value V at step T T Derive the discount reward R at step t forward. t for:

[0121]

[0122] Where, γ k-t γ T-t r is the discount factor for the corresponding step. t Let be the reward value at step t.

[0123] Step 6: Calculate the loss function L(ω) of the value network, and update the parameters of the value network using gradient descent and backpropagation;

[0124] In this embodiment, the loss function L(ω) of the value network is calculated as follows:

[0125] L(ω)=E t [(R t -V t ) 2 ]

[0126] Update the parameter ω of the value network using gradient descent based on L(ω):

[0127]

[0128] Where α is the learning rate of the gradient method, As a gradient operator, using gradient descent and backpropagation to update the parameters of the value network can make the output state value of the value network closer to the true state value.

[0129] Step 7: Calculate the importance weights I of the current policy network and the target policy network using the importance sampling method. t (θ);

[0130] In this embodiment, the importance weight I t The formula for calculating (θ) is:

[0131]

[0132] Where π(·|s) t ;θ),π(·|s t ;θ old ) indicates that the state s t The policy distributions obtained by inputting into the current policy network and the target policy network, where θ represents the parameters of the current policy network. old π(a) represents the parameters of the target policy network.t |s t ;θ),π(a t |s t ;θ old This indicates the actual action to be taken (a). t The value of the action obtained by inputting it into the corresponding policy distribution is the value obtained under the corresponding policy distribution.

[0133] Step 8: Calculate the dominance function And combined with the importance weight I calculated in step 7 t (θ) Find the objective function L of the current policy network. CLIP (θ), using backpropagation to update the parameters of the current policy network;

[0134] In this embodiment, the advantage function The calculation formula is:

[0135]

[0136] Based on the advantage function And importance weight I t (θ), find the objective function L of the current policy network. CLIP The formula for calculating (θ) is:

[0137]

[0138] Where ε is the weight of the constraint importance I t The hyperparameters of (θ), clip(I) t (θ), 1-ε, 1+ε) represent the expression for I t The value of (θ) is restricted to the range (1-ε, 1+ε), when I t When (θ) < 1-ε, let I t (θ)=1-ε, similarly when I t When (θ)>1+ε, let I t (θ) = 1 + ε;

[0139] Based on the objective function L of the current policy network CLIP (θ), using backpropagation to update the parameters θ of the current policy network.

[0140] Step 9: Repeat steps 7 and 8 to update the parameters of the current policy network M times, where M ≤ T;

[0141] Step 10: Update the parameters θ of the target policy network using the updated parameters θ of the current policy network. old ,include:

[0142] θ old ←θ

[0143] The current policy network is used to collect a certain amount of interaction information between intelligent connected vehicles and the environment, which is used to update the value network and the target policy network, thus completing one iteration.

[0144] Step 11: Repeat steps 1-10 for E iterations to obtain the trained deep reinforcement learning model.

[0145] Step 12: In real-time applications, obtain the current joint state space. By inputting the data into a pre-trained deep reinforcement learning model, the joint action space of the intelligent connected vehicle is obtained. According to the joint action space Dispatch and manage all intelligent connected vehicles.

[0146] Example 2

[0147] Secondly, based on Embodiment 1, this embodiment provides an intelligent vehicle dispatching device for ramp merging scenarios, including a processor and a storage medium;

[0148] The storage medium is used to store instructions;

[0149] The processor is configured to operate according to the instructions to execute the method according to Embodiment 1.

[0150] Example 3

[0151] Thirdly, based on Embodiment 1, this embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Embodiment 1.

[0152] Example 4

[0153] Fourthly, based on Embodiment 1, this embodiment provides a device, including,

[0154] Memory;

[0155] processor;

[0156] as well as

[0157] Computer programs;

[0158] The computer program is stored in the memory and configured to be executed by the processor to implement the method described in Embodiment 1.

[0159] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0163] The above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and are not intended to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An intelligent vehicle scheduling method for ramp merging scenarios, characterized in that, The method includes: Receive local state information sent by each intelligent connected vehicle in the current ramp merging scenario, and combine it with the acquired global state information to form a joint state space; By inputting the joint state space into the trained deep reinforcement learning model, the joint action space of the intelligent connected vehicle is obtained. Based on the aforementioned joint action space, each intelligent connected vehicle is scheduled; The deep reinforcement learning model includes a current policy network, a target policy network, and a value network; the method for obtaining the deep reinforcement learning model includes: iteratively executing the overall loop steps until a preset condition is met to obtain a trained deep reinforcement learning model; the overall loop steps include: S1. Execute the first loop step to obtain the interaction information at each step in the interaction process between the intelligent connected vehicle and the environment. The first cyclic step includes: S101. Receive local state information sent by each intelligent connected vehicle in the ramp merging scenario, and combine it with the acquired global state information to form a joint state space. ; S102, Combine the state space Inputting this into the current deep reinforcement learning model yields the joint action space of the intelligent connected vehicle. Calculate the reward value This forms the interactive information for the corresponding steps. ; and will combine action space Send to the corresponding intelligent connected vehicle; S103, responding to intelligent connected vehicles based on joint action space Perform the corresponding action to transition the environment to the next joint state. ; S2, record the interaction information at each step. All data is input into the value network to obtain the state value at each step. And calculate the discount reward for each step based on the state value. ; S3. Calculate the loss function of the value network based on state value and discounted reward. The parameters of the value network are updated using gradient descent and backpropagation. S4. Execute the second loop step, and adjust the parameters of the current policy network. Next update The updated parameters of the current policy network are obtained; wherein the second loop step includes: The state in the interaction information and actions The inputs are fed into the current policy network and the target policy network respectively, and importance weights are calculated using the importance sampling method. Calculate the advantage function based on state value and discount reward. And combined with importance weights Find the objective function of the current policy network. The parameters of the current policy network are updated using backpropagation; S5. Update the parameters of the target policy network using the updated parameters of the current policy network; Wherein, the joint state space for: ， ， in, Indicates the first The state space of a smart connected vehicle. This represents the maximum number of intelligent connected vehicles in the joint state space. Indicates the first The longitudinal speed of a smart connected vehicle They represent the first The relative distance between a smart connected vehicle and the vehicle in front in the same lane, the vehicle behind in the same lane, the vehicle in front in the adjacent lane on the left, the vehicle behind in the adjacent lane on the left, the vehicle in front in the adjacent lane on the right, and the vehicle behind in the adjacent lane on the right. They represent the first The relative speeds between a smart connected vehicle and the vehicle in front in the same lane, the vehicle behind in the same lane, the vehicle in front in the left adjacent lane, the vehicle behind in the left adjacent lane, the vehicle in front in the right adjacent lane, and the vehicle behind in the right adjacent lane. They represent the first The equivalent average traffic flow in the lane where the intelligent connected vehicle is located, the left adjacent lane, and the right adjacent lane; where the lane Equivalent average traffic flow for: ， ， ， in For lane The average longitudinal speed of all vehicles For lane density, For lane Length, For lane The number of vehicles on board; The joint action space for: ， ， ， ， in Indicates the first The movement space of an intelligent connected vehicle This indicates the maximum number of intelligent connected vehicles in the joint action space. Indicates the first The acceleration of the vehicle, Indicates the first The lane-changing actions of a vehicle are indicated by 0, 0.5, and 1, which represent going straight, changing lanes to the left, and changing lanes to the right, respectively. The calculation of reward value ,include: ， ， ， ， in: As an average speed bonus, This represents the number of vehicles in the scene. The maximum value set for the vehicle speed; Incentives are awarded for traffic flow unevenness. As a penalty for changing lanes, when performing a lane-changing maneuver. ,otherwise ; Indicates the first The longitudinal speed of a smart connected vehicle As a weighting factor; Indicates the first Equivalent average traffic flow for each lane The standard deviation.

2. The intelligent vehicle scheduling method according to claim 1, characterized in that, No. The value of a step's state ,in Represents the value function. The parameters representing the value network will be the first... Step state The state value is obtained by inputting into the value network. ; The formula for calculating discount rewards is: ， Therefore, using the first The value of a step's state Derive forward to the first Step discount rewards for: ， in, , The discount factor for the corresponding step. For the first The reward value for each step.

3. The intelligent vehicle scheduling method according to claim 1, characterized in that, Loss function of value network The calculation methods include: ， in Let be the expected function. For the first Step discount rewards, For the first The state value of a step; based on Update the parameters of the value network using gradient descent. : ， in The learning rate for the gradient method is... The gradient operator uses gradient descent and backpropagation to update the parameters of the value network, making the output state value of the value network closer to the true state value.

4. The intelligent vehicle scheduling method according to claim 1, characterized in that, Importance weight The calculation methods include: ， in , Indicates the state The policy distributions obtained by inputting into the current policy network and the target policy network This represents the parameters of the current policy network. The parameters represent the target policy network. , This indicates the actual action to be taken. The value of the action obtained by inputting it into the corresponding policy distribution is the value obtained under the corresponding policy distribution.

5. The intelligent vehicle scheduling method according to claim 1, characterized in that, Calculate the advantage function based on state value and discount reward. And combined with importance weights Find the objective function of the current policy network. ,include: Based on state value and discount rewards Calculate the advantage function : ， Based on the advantage function And importance weight Find the objective function of the current policy network. ; ， in, To limit importance weights hyperparameters, Indicates will The value is limited to a range In the middle, when season Similarly, season ; Based on the objective function of the current policy network Update the parameters of the current policy network using backpropagation. .

6. The intelligent vehicle scheduling method according to claim 1, characterized in that, Utilize the updated parameters of the current policy network Update the parameters of the target policy network ,include: ， The current policy network serves to collect a certain amount of interaction information between intelligent connected vehicles and the environment, which is used to update the value network and the target policy network, thus completing one iteration.

7. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images