Intelligent vehicle on-ramp merging decision method based on reinforcement learning

Abstract

The present application relates to a kind of intelligent vehicle highway ramp merging decision method based on reinforcement learning, it is related to the field of intelligent vehicle, for improving intelligent vehicle merging success rate, reduce traffic accident.The method sets the intersection of highway ramp and main road as merging point, sets ramp and the main road located S1 meter after merging point and S2 meter before merging point as control area;In control area, the target lane of intelligent vehicle is projected to main road, and the distance of projected intelligent vehicle to merging point is equal to the distance of intelligent vehicle on ramp to merging point;Determine the vehicle in front of projected intelligent vehicle and the vehicle behind, obtain their speed and position as environmental vehicle information;Intelligent vehicle information is obtained, such as the distance of intelligent vehicle to merging point, intelligent vehicle speed and acceleration;According to intelligent vehicle information, environmental vehicle information, acceleration and front wheel rotation angle are adjusted using reinforcement learning DDPG model, and gradually realize successful merging.

Description

Technical Field

[0001] This invention relates to intelligent vehicles, and more particularly to a reinforcement learning-based method for intelligent vehicles to make decisions on highway ramp merging. Background Technology

[0002] Highway entrance ramp merging is a crucial scenario for autonomous driving applications, characterized by complex traffic environments and high accident rates, making it a "bottleneck" section for vehicles traveling on highways. Existing rule-based decision-making algorithms suffer from overly conservative decisions and an inability to handle unpredictable emergencies, while reinforcement-based algorithms suffer from low sample utilization and poor training stability. Furthermore, there is still room for optimization in the setting of reward functions. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, the present invention aims to propose a reinforcement learning-based intelligent vehicle highway ramp merging decision method, which assists intelligent vehicles within the control area in making effective decisions within a set distance range at the merging point under given speed limits, thereby improving the safety of intelligent vehicles.

[0004] To achieve the above objectives, the technical solution in this case is as follows.

[0005] Firstly, this invention proposes a reinforcement learning-based intelligent vehicle merging decision method for highway ramps. The method defines the intersection of the highway ramp and the main road as the merging point, and the ramp and the main road located S1 meters after and S2 meters before the merging point as control areas, where S1 and S2 are set values. Within the control area, the intelligent vehicle is projected onto the target lane of the main road, with the distance from the projected intelligent vehicle to the merging point equal to the distance from the intelligent vehicle on the ramp to the merging point. The vehicles in front of and behind the projected intelligent vehicle are determined, and their speeds and positions are acquired as environmental vehicle information. The distance between the intelligent vehicle and the merging point, the intelligent vehicle's speed, and acceleration are acquired as intelligent vehicle information. Based on the intelligent vehicle information and the environmental vehicle information, the intelligent vehicle uses a reinforcement learning DDPG model to adjust its acceleration and front wheel angle, gradually achieving successful merging.

[0006] In the above technical solution, the reward obtained by the agent in the reinforcement learning DDPG model includes at least a first reward, the calculation steps of which include:

[0007] The import process is divided into several stages, and an impact factor is set for each stage.

[0008] Based on the distance of the intelligent vehicle to the merging point, determine its stage and calculate the reward using the influence factor of that stage.

[0009] In the above technical solution, the formula for calculating the first reward is as follows:

[0010]

[0011] In the formula: k i Let w be the influence factor for stage i, where i = 1, 2, ..., n, and n is the total number of stages. m It is the weight of the incoming position reward, Δv max v is the maximum permissible speed difference between the speed of the intelligent vehicle and the average speed of the two vehicles in front and behind it. p1 v is the speed of the first vehicle in front of the intelligent vehicle. f1 v is the speed of the first vehicle behind the intelligent vehicle. m For intelligent vehicle speed, d m Let w be the distance from the center point of the intelligent vehicle to the merging point. The value of w ranges from [0, 1], where 0 indicates that the distance between the intelligent vehicle and the first vehicle in front and the intelligent vehicle and the first vehicle behind are the same, and 1 indicates that the distance between the intelligent vehicle and the first vehicle in front or the first vehicle behind is zero. The definition of w is as follows:

[0012]

[0013] In the formula: d p1 d represents the distance from the vehicle in front of the intelligent vehicle to the merging point. f1 l is the distance from the vehicle behind the intelligent vehicle to the merging point. p1 l m These are the lengths of the first vehicle preceding the intelligent vehicle and the intelligent vehicle itself.

[0014] In the above technical solution, the reward obtained by the agent in the reinforcement learning DDPG model is the sum of the first reward and the second reward, where the second reward is the value of any one of the following, or the sum of any two, or the sum of any multiple:

[0015] Collision penalties and non-collision rewards, safe driving speed rewards, stop penalties for failing to reach the designated destination, rewards for reaching the destination, and passenger comfort rewards; among which:

[0016] The collision penalty and non-collision reward are calculated as follows:

[0017]

[0018] In the formula: r col Indicates vehicle collision penalty, r collision The penalty value set for a vehicle collision;

[0019] The safe driving speed bonus is calculated as follows:

[0020]

[0021] In the formula: r v (v) represents the high-speed bonus for the intelligent vehicle at its current speed v. For intelligent vehicles, r h_s This is the reward value for vehicles traveling at high speeds;

[0022] Penalty for failure to reach the designated destination:

[0023]

[0024] In the formula: r s The reward indicates that the vehicle has successfully arrived. stop The reward value for a vehicle's successful arrival;

[0025] The reward for reaching the destination is calculated as follows:

[0026]

[0027] In the formula: r a The reward indicates that the vehicle has successfully arrived. arrival The reward value set for a vehicle successfully reaching its destination;

[0028] Passenger comfort is calculated as follows:

[0029]

[0030] In the formula: w j For the weighting of comfort penalty reward, j max The maximum impact force allowed for passenger comfort. Let j be the acceleration derivative of the intelligent vehicle. m The degree of impact on passengers.

[0031] In the above technical solution, the agent in the reinforcement learning DDPG model selects actions through an Actor network and evaluates the Q-value of the selected actions through a Critic network. Both the Actor network and the Critic network consist of temporal neural network layers and fully connected layers; wherein:

[0032] Both the Actor network and the Critic network take the historical state information of intelligent vehicle information and environmental vehicle information as input. The action output by the Actor network is concatenated with the output of the temporal neural network layer in the Critic network and then passed to the fully connected layer in the Critic network. This fully connected layer calculates and generates the evaluation Q value of the action.

[0033] In the above technical solution, both the Actor network and the Critic network have network parameters and target network parameters, which are the same when initialized.

[0034] During the update, N historical state-action pairs are randomly sampled from the experience replay pool D, where N is a set value. The network parameters θ of the Critic network are first updated by minimizing the loss function. Q Then, by maximizing the Q-value estimated from the Critic network, the network parameters θ of the Actor network are updated. μ ;

[0035] Obtain the target network parameters θ of the Actor network and Critic network. μ′ , using τθ μ +(1-τ)θ μ′ Update the target network parameters θ of the Actor network μ’ τ is the approximation coefficient, expressed as τθ. Q +(1-τ)θ Q′ Update the target network parameters θ of the Critic network Q’ ;

[0036] After the reinforcement learning DDPG model is trained, the Actor network uses the target network parameters to make decisions about the arrival of intelligent vehicles.

[0037] In the above technical solution, the experience replay pool is divided into two experience replay pools: a positive sample experience replay pool and a negative sample experience replay pool.

[0038] When the agent samples in the experience pool, it averages the data from the two experience replay pools according to the set number of samples, and then combines the sampled data for training.

[0039] Positive samples are those where intelligent vehicles have successfully joined, while negative samples are those where intelligent vehicles have failed to join.

[0040] In the above technical solutions, the temporal neural network is any one of LSTM, GRU, Bi-LSTM, and RNN.

[0041] Secondly, the present invention proposes an intelligent vehicle having a system implemented by any of the above methods, which controls the intelligent vehicle to successfully merge from a highway ramp into the main highway.

[0042] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above.

[0043] The technical solution of the present invention has the following technical effects:

[0044] (1) Project the intelligent vehicle onto the target lane of the main road. Combine the perception range of the intelligent vehicle and take the vehicles in front of and behind the projected intelligent vehicle as environmental vehicles. Obtain environmental vehicle information and combine it with the intelligent vehicle's own information. Use reinforcement learning to help the intelligent vehicle merge from the ramp into the environmental vehicles on the main road based on the traffic density of the main road, avoid collisions and improve the merging success rate.

[0045] (2) By taking advantage of the different distances of intelligent vehicles from the merging point after being projected onto the main road, the merging process is divided into several stages, and an influencing factor is set for each stage to measure the merging success rate of intelligent vehicles merging from the ramp into the highway and reduce traffic accidents.

[0046] (3) By adding a temporal neural network to DDPG, the merging failure behaviors such as collision and driving out of the road boundary can be effectively avoided.

[0047] (4) During training, the positive and negative sample experience replay pool is designed to improve sample utilization and training efficiency, so that the trained decision model has strong robustness and strong stability. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A schematic diagram of a highway ramp merging in one specific embodiment, wherein the intelligent vehicle is in the AB merging phase;

[0050] Figure 2 A schematic diagram of a highway ramp merging when an intelligent vehicle is in the BC merging phase in one specific embodiment;

[0051] Figure 3 A schematic diagram of a highway ramp merging when an intelligent vehicle is in the CD merging stage in one specific embodiment;

[0052] Figure 4 A schematic diagram of a DDPG structure incorporating LSTM in one specific embodiment;

[0053] Figure 5 A schematic diagram of the positive and negative sample experience replay pool structure in one specific embodiment;

[0054] Figure 6 A schematic diagram of a typical reinforcement learning model. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0057] A highway entrance ramp consists of one main highway lane and one merging lane. The merging environment includes the merging lane with its geometric shape and structure, and contains merging vehicles and surrounding vehicles. The length of the acceleration lane of the merging lane is determined by the maximum design speed of the main line, according to the "Highway Route Design Specifications." According to the "Regulations for the Implementation of the Road Traffic Safety Law," the minimum speed limit on highways is 60 km / h, meaning vehicles must be at least 60 km / h to enter the main highway, which has a speed limit of 120 km / h.

[0058] The merging vehicles are intelligent vehicles equipped with perception sensors. Merging behavior on highway entrance ramps is a mandatory merging. They need to enter within a standard acceleration lane of fixed length, at the prescribed speed limit, and in combination with the traffic density of the main line. They need to comprehensively consider time and space boundary factors to formulate a safe, reasonable, and generalizable merging decision strategy, select the optimal merging time, and complete the merging into the main line.

[0059] To improve the success rate and safety of intelligent vehicles merging from ramps onto highway main roads, this invention proposes a technical solution for intelligent vehicle highway ramp merging decision-making based on reinforcement learning. The intersection of the highway ramp and the main road is defined as the merging point. The ramp and the main road located S1 meters after and S2 meters before the merging point are defined as control areas, where S1 and S2 are set values. Within the control area, the intelligent vehicle is projected onto the target lane of the main road, with the distance from the projected intelligent vehicle to the merging point equal to the distance from the intelligent vehicle on the ramp to the merging point. The vehicles in front of and behind the projected intelligent vehicle are determined, and their speeds and positions are acquired as environmental vehicle information. The distance between the intelligent vehicle and the merging point, the intelligent vehicle's speed, and acceleration are acquired as intelligent vehicle information. Based on the intelligent vehicle information and the environmental vehicle information, the intelligent vehicle uses a reinforcement learning DDPG model to adjust its angular velocity and front wheel steering angle, gradually achieving successful merging.

[0060] In one implementation of a simulation of an intelligent vehicle merging from a ramp onto a highway main road, it is assumed that no other vehicles besides the intelligent vehicle are merging on the entrance ramp. The refresh frequency of the merging environment is 15Hz, meaning the environment is refreshed 15 times per second to ensure real-time updates of vehicle positions, directions of travel, and other statuses. The intelligent vehicle's perception range is a circle with a radius of 200m, which is the typical perception range of a general sensor. The intelligent vehicle is located at the intersection of the entrance ramp and the control area, heading towards the merging point, which is the intersection of the highway ramp and the main road. The initial speed of the intelligent vehicle is randomly selected within the range of 16.7-27.8 m / s. A merging simulation ends when the vehicle stops, collides, or successfully merges. One simulation can be considered as one round of training for the next simulation. After a merging simulation ends, the intelligent vehicle is subsequently deleted. If a simulation is repeated, the intelligent vehicle is regenerated with new initial conditions.

[0061] The simulated environment vehicle conforms to the randomness of merging into the environment from the highway entrance ramp. The starting position is randomly selected within the range of 0-150 meters from the starting point, and the speed of the environment vehicle is randomly selected within the range of 25-33 m / s. The longitudinal driving strategy of the environment vehicle is based on the Intelligent Driver Model (IDM), including vehicle following and collision avoidance. Since the main road is a single lane, the environment vehicle does not have lane changing behavior.

[0062] Based on the intelligent vehicle's perception range of 200m, the control area of ​​the intelligent vehicle is exemplarily defined as 230m after the merging point (exemplary value of S1) and 100m before the merging point (exemplary value of S2). The reinforcement learning-based merging decision is only used to control the intelligent vehicle within the control area. It is assumed that the intelligent vehicle makes a valid decision within 230m after the merging point because if the merging is unsuccessful, the intelligent vehicle can come to a complete stop within 230m under a given speed limit. Furthermore, the initial section of the entrance ramp is designed for merging vehicles to accelerate from low speed, and the intelligent vehicle will not make a merging decision there. The 100m before the merging point is used to evaluate whether the merging is successful because if the intelligent vehicle does not choose a suitable merging timing and vehicle gap, a collision may occur after merging.

[0063] The inflow process of an intelligent vehicle can be viewed as a Markov Decision Process (MDP). Therefore, an MDP model can be performed on the intelligent vehicle, represented by (S, A, R), where S is the state space, A is the action space, and R is the reward function. The details are as follows.

[0064] (1) State space

[0065] In the highway entrance ramp merging environment built on highway-env, intelligent vehicles can obtain the environmental state within the detectable range based on their own sensors. Within the detectable range, they can obtain the position and speed information of other vehicles in the environment, as well as the position information of obstacles in the road, and the speed, acceleration, and front wheel angle of the intelligent vehicle.

[0066] For example, the environmental state within the perception range includes the state of 5 vehicles: the merging vehicle (m) and its two preceding vehicles (p1, p2) and two following vehicles (f1, f2). In this case, the order of the vehicles is with reference to the intelligent vehicle, and the intelligent vehicle is traveling in the forward direction. The first preceding vehicle and the first following vehicle are vehicles adjacent to the intelligent vehicle.

[0067] When an intelligent vehicle on the entrance ramp is projected onto the target lane on the main line, the distance from the projected intelligent vehicle to the merging point is the same as the distance from the intelligent vehicle on the ramp to the merging point. The distance from the center point of the intelligent vehicle to the merging point is taken as the total distance from the intelligent vehicle to the merging point. The state of the intelligent vehicle includes the distance d from its center point to the merging point. m Speed ​​v m and acceleration a m These are used as intelligent vehicle information; the states of the two vehicles in front and the two vehicles behind include the distance to the merging point (d). p1 d p2 d f1 d f2 ) and velocity (v) p1 v p2 v f1 v f2 These are considered as environmental vehicle information; the state space consists of intelligent vehicle information and environmental vehicle information, and can be represented as s = [d p2 v p2 d p1 v p1 d m v m a m d f1 v f1 d f2 v f2 The order of status information content does not limit the application of the method.

[0068] Within the perception range of the intelligent vehicle, considering the average vehicle length (5m in this example), it is reasonable to assume approximately four vehicles in front of and behind it. During simulation, the speed and distance of the intelligent vehicle and surrounding vehicles change in real time. When fewer than two vehicles are detected in front of or behind the intelligent vehicle, virtual vehicles can be assumed at the intersection of the perception range and the main road to construct a state vector of five vehicles. If any vehicle is just outside the perception range, the virtual vehicle has a speed value within the speed limit of the main road. Note that the virtual vehicle is determined relative to the intelligent vehicle or its projection in front of or behind it. If the intelligent vehicle decelerates on the ramp and its projection moves behind the vehicles on the main road, then the following vehicle becomes the preceding vehicle. In reality, when the intelligent vehicle is traveling on the ramp, the vehicles in front of and behind its projected vehicle on the main road may change, with the following vehicle becoming the preceding vehicle. By acquiring environmental vehicle information in real time, the intelligent vehicle's merging decision actions can be adjusted in real time for each step without affecting the driving of the surrounding vehicles, thereby improving the merging success rate.

[0069] (2) Action space

[0070] The action space of an intelligent vehicle consists of longitudinal acceleration and front wheel steering angle. Its action type is continuous action, represented as A = {a, θ}, where a is longitudinal acceleration and θ is front wheel steering angle.

[0071] (3) Reward function

[0072] The reward for each time step includes at least the first reward in section (3.1) below, or the sum of the first and second rewards in sections (3.1) and (3.2).

[0073] (3.1) Design different reward functions for different road segments with different impact levels according to the different stages of intelligent vehicle merging.

[0074] Within the control area, as intelligent vehicles merge from the entrance ramp into the merging zone and then into the merging point, the distance from the merging point after projection onto the main road varies, affecting the degree of influence on the intelligent vehicle's successful merging decision. Therefore, within the control area, different reward functions with varying degrees of influence are designed for intelligent vehicles at different merging stages and different road segments on the entrance ramp. These functions provide accurate and efficient rewards and penalties for merging decisions, thereby improving the success rate of intelligent vehicles merging into the highway from the ramp and reducing traffic accidents.

[0075] like Figure 1 As shown, when the intelligent vehicle is in the AB merging stage, it is relatively far from the merging point, and the impact on the intelligent vehicle's successful merging decision is small. The first reward function is designed as follows:

[0076] The intelligent vehicle's projection position should be between the preceding and following vehicles, with a speed close to the average speed of the two vehicles. The corresponding penalty and reward definitions for road segments A and B are as follows.

[0077]

[0078] In the formula k ab The influence factor for the A / B phase is 0.5, with an example value of 0.5. m This is the weight of the incoming location reward, with an example value of 0.015, Δv max The maximum permissible speed difference between the speed of the intelligent vehicle and the average speed of the two vehicles in front and behind, with an example value of 5 m / s, v m For intelligent vehicle speed, d m v is the distance from the center point of the intelligent vehicle to the merging point. p1 v is the speed of the first vehicle in front of the intelligent vehicle. f1 Let w be the speed of the first vehicle behind the intelligent vehicle. The value of w ranges from [0, 1], where 0 indicates that the distance between the intelligent vehicle and the first vehicle in front and the intelligent vehicle and the first vehicle behind are the same, and 1 indicates that the distance between the intelligent vehicle and the first vehicle in front or the first vehicle behind is zero. The definition of w is as follows:

[0079]

[0080] In the formula, l p1 l m These are the first vehicle preceding the intelligent vehicle and the actual length of the intelligent vehicle, d. p1 d is the distance d is from the center point of the first vehicle to the merging point. f1 This is the distance from the center point of the first following vehicle to the merging point.

[0081] like Figure 2 As shown, when the intelligent vehicle is in section BC, it is relatively close to the merging point. This is a crucial stage for the intelligent vehicle to choose the right time and action to successfully merge. The first reward function is designed as follows:

[0082]

[0083] In the formula k bc This is the impact factor for stage BC, with a value of 1.

[0084] like Figure 3 As shown, when the intelligent vehicle is in section C and D, it is in the merging phase and closest to the merging point. At this time, the intelligent vehicle's action output is to avoid collision and achieve a successful merge. The first reward function is designed as follows:

[0085]

[0086] In the formula k bc The CD stage impact factor has a value of 1.5.

[0087] (3.2) The second reward function remains unchanged throughout the entire inflow process, including the following parts:

[0088] ① To improve the impact of intelligent vehicle merging on passenger comfort, the corresponding reward function is defined as follows:

[0089]

[0090] In the formula, w j The weight of the comfort penalty reward is 0.012, for example. max The maximum impact force permissible for passenger comfort, exemplarily 3 m / s². 3 . Let j be the acceleration derivative of the intelligent vehicle. m The degree of impact on passengers.

[0091] ② When the intelligent vehicle collides with any other vehicle, a penalty is imposed, and the corresponding reward function is defined as follows:

[0092]

[0093] In the formula r col Indicates vehicle collision penalty, r collision The penalty value for a vehicle collision is -1.

[0094] ③ my country's highway regulations stipulate that the minimum driving speed for vehicles is 60 km / h. This minimum speed is set to prevent traffic congestion caused by slow-moving vehicles. Therefore, a high-speed driving reward is set to encourage intelligent vehicles to drive quickly while ensuring safety. The corresponding reward function is defined as follows:

[0095]

[0096] In the formula: r v (v) represents the high-speed bonus for the intelligent vehicle at its current speed v. This is the maximum speed for intelligent vehicles. The value is at least greater than or equal to the vehicle's minimum driving speed of 60 km / h, r h_s This is the reward value for high-speed vehicle travel; an example value is 0.5.

[0097] ④ When the intelligent vehicle successfully reaches 100 meters before the merging point, an arrival reward is given. The corresponding reward function is defined as follows:

[0098]

[0099] In the formula r a The reward indicates that the vehicle has successfully arrived. arrivalThis is the reward value for a vehicle's successful arrival; an example value is 1.

[0100] ⑤ When the intelligent vehicle fails to reach the designated destination, a penalty is imposed, terminating the current round during simulation. The corresponding reward function is defined as follows:

[0101]

[0102] In the formula r s The reward indicates that the vehicle has successfully arrived. stop The reward value for a vehicle's successful arrival is -0.5.

[0103] In summary, the overall reward function is as follows:

[0104] R = r m +r j +r col +r v (v)+r a +r s (1)

[0105] As a further improvement, incorporating a temporal neural network into the DDPG can effectively avoid merging failures such as collisions and driving off the road. Specifically, during the merging process at a highway entrance ramp, intelligent vehicles maximize long-term rewards through interaction with the environment to find the optimal driving strategy. The merging process involves interaction with other vehicles on the main road, and the behavioral state information of these other vehicles influences the merging vehicle's optimal driving strategy. Therefore, integrating a temporal neural network into the DDPG model framework leverages the network's ability to process temporal information to predictively avoid merging failures such as collisions and driving off the road, thus improving the training efficiency of the DDPG merging decision model. The input to the temporal neural network is the historical driving state information of the intelligent vehicle and surrounding vehicles over several consecutive time steps, i.e., historical information of both the intelligent vehicle and surrounding vehicles. The temporal neural network can be any of LSTM, GRU, Bi-LSTM, or RNN.

[0106] Taking the temporal neural network LSTM as an example, one implementation method for integrating LSTM into the DDPG framework is as follows: Figure 4As shown, the Actor network in the DDPG framework consists of three layers: two LSTM layers and one fully connected layer. The LSTM has an input size of 44 and 256 hidden units, followed by a fully connected layer. Taking 256-dimensional data as input, it calculates and generates a 2-dimensional action output. The LSTM network in the Critic network is configured identically to that in the Actor network, with input being the historical driving state information of the intelligent vehicle and the surrounding vehicles over four consecutive time steps (i.e., intelligent vehicle information and surrounding vehicle information over four consecutive time steps). For the Action output from the Actor network, it is concatenated with the output of the LSTM network in the Critic network to form a vector, which is then passed to the next fully connected layer. This fully connected layer calculates the Q-value for the Action. The LSTM can be replaced by any of GRU, Bi-LSTM, or RNN.

[0107] As a further improvement, the experience replay pool is equally divided into a positive sample experience replay pool and a negative sample experience replay pool. When the agent samples samples from the experience pool, it samples equally from both experience replay pools according to the set number of samples. Then, the sampled data is combined for training. This overcomes the problem that the imbalance between positive and negative samples generated in the initial stage of DDPG algorithm training makes it difficult for the agent to sample positive and negative samples evenly during training, causing network instability at the beginning of training. Positive samples are those from successfully integrated intelligent vehicles, and negative samples are those from those from failed integrated intelligent vehicles.

[0108] Typical reinforcement learning models include Figure 6 As shown, the training steps of the reinforcement learning DDPG model incorporating a temporal neural network are largely the same, except that training of the temporal neural network is added to each training cycle of each DDPG. An exemplary training step includes:

[0109] S1. Randomly initialize the network parameters θ of the Actor network and the Critic network. μ and θ Q And initialize the experience pool D and the target network parameters θ of the Actor network and Critic network. μ’ and θ Q’ , so that θ μ’ and θ μ The same, so that θ Q’ and θ Q same.

[0110] S2, the distance from the ramp entrance to the merging point is specified as one training cycle, and the training cycle is set to M.

[0111] In each training cycle, the intelligent vehicle needs T rounds of single-step training to travel the specified distance from the ramp entrance to the merging point. Before each round of single-step training, random noise N is initialized for action exploration, and the initial state of X consecutive time steps is obtained as the current state, where X is a set value.

[0112] S3. In each round of single-step training, obtain the target network parameters θ of the current Actor network. μ’ and random noise N t Select Action A t = {a, θ}, where t is the current time step, a is the acceleration, and θ is the front wheel steering angle.

[0113] S4. Obtain the current location of the intelligent vehicle on the ramp and select the desired action A. t And receive instant reward R t Enter the next state S t+1 Store state-action pairs {S} in experience pool D. t A t R t S t+1}

[0114] S5. Subsequently, N historical state-action pairs {S} are randomly sampled from the experience replay pool D. t A t R t S t+1}, where N is a set value, and the Adam optimizer is used to optimize the objective function Q(s, a; θ). Q Network parameters θ Q Optimization is performed by updating the network parameters θ of the Critic network by minimizing the loss function L. Q The objective function is as follows;

[0115]

[0116] r t This represents the instant reward calculated according to formula (1) at step t, where γ is the discount factor and y is the instant reward calculated according to formula (1) at step t. t Let Q' be the objective Q-value function, N be the number of samples, and Q' be the objective network parameter θ in the Actor network. μ′ And the target network parameters θ of the Critic network Q′ The Q value under μ′(s) t+1 ;θ μ′ ) represents the Actor network in the target network parameters θ μ′ Next pair of states s t+1 The strategy function for selecting actions.

[0117] S6. The Adam optimizer is used to optimize the action policy function μ(s; θ) of the Actor network.μ The parameter θ μ Optimization is performed by updating the network parameters θ of the Actor network by maximizing the Q-value estimated from the Critic network. μ gradient of the objective function As shown in the following formula;

[0118]

[0119] S7. Update the target network parameters for the Actor and Critic networks:

[0120] θ μ′ ←τθ μ +(1-τ)θ μ′

[0121] θ Q′ ←τθ Q +(1-τ)θ Q′

[0122] In the formula: τ is the approximation coefficient, and its value is 0.01.

[0123] S8. After completing T rounds of single-step training (i.e., t = T), m = m + 1, and the next training cycle begins. After M training cycles are completed (i.e., m = M), the action policy function μ′(s; θ) of the Actor network is... μ′ The training results can be directly used for the integration decision of intelligent vehicles.

[0124] In one implementation, the trained action policy function μ′(s; θ) is... μ′ Further, a controller for intelligent vehicle merging decision-making is implemented, which controls the longitudinal acceleration θ and front wheel steering angle θ of the intelligent vehicle to achieve successful merging of the intelligent vehicle on the highway ramp.

[0125] Table 1 shows a pseudocode description of an exemplary training process.

[0126] Table 1

[0127]

[0128]

[0129] As can be seen from the training process, in a training cycle, the intelligent vehicle may walk from the starting point of the ramp to the end point, or it may be terminated due to factors such as a collision, or it may successfully merge. However, the intelligent vehicle moves step by step, and a T-step is used in a merging process to ensure that the intelligent vehicle successfully reaches its destination.

[0130] In summary, as can be seen from the above embodiments, the merging decision method of the present invention, for a single intelligent vehicle, controls its merging from the ramp entrance onto the main highway. It designs a reward function considering the impact of changes in the distance between the vehicle's projected position on the main highway and the merging point on the success of the merging, thereby improving the merging success rate and reducing traffic accidents. Furthermore, it considers the impact of decision-making actions on passenger comfort and vehicle safety during the merging process, enhancing the practicality of the intelligent vehicle. By incorporating a temporal neural network into the reinforcement learning DDPG model, the robustness and stability of model training are improved, thus enhancing the reliability of the method.

[0131] Through the above description of the embodiments, those skilled in the art can clearly understand that this disclosure can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for this disclosure, software implementation is more often a preferred implementation method.

[0132] It should be noted that the terms "one embodiment," "another embodiment," and "embodiment" used in this specification refer to specific features, structures, or characteristics described in connection with that embodiment, which are included in at least one embodiment described in the general description of this application. The appearance of the same expression in multiple places in the specification does not necessarily refer to the same embodiment. Furthermore, when a specific feature, structure, or characteristic is described in connection with any embodiment, the intention is to suggest that implementing such a feature, structure, or characteristic in conjunction with other embodiments also falls within the scope of this invention.

[0133] Although embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A reinforcement learning-based intelligent vehicle highway ramp merging decision method, characterized in that: The method sets the intersection of the highway ramp and the main road as a merging point, sets the ramp and the main road located S1 meters behind the merging point and S2 meters in front of the merging point as a control area, and S1 and S2 are set values; In the control area, the intelligent vehicle is projected onto the target lane of the main road, and the projected intelligent vehicle is equal to the intelligent vehicle on the ramp in terms of distance to the merging point; Determine the front and rear vehicles of the projected intelligent vehicle and obtain their speed and position as environmental vehicle information; Obtain the distance of the intelligent vehicle from the merging point, the speed and acceleration of the intelligent vehicle as intelligent vehicle information; According to the intelligent vehicle information and the environmental vehicle information, the intelligent vehicle uses the reinforcement learning DDPG model to adjust the acceleration and front wheel steering angle, and gradually realizes successful merging; The agent of the reinforcement learning DDPG model obtains at least a first reward, and the calculation steps include: Divide the merging process into several stages, and set an impact factor for each stage; According to the distance of the projected intelligent vehicle to the merging point, determine its stage, and calculate the first reward using the impact factor of the stage, the calculation formula is as follows: In the formula: is the impact factor of the i-th stage, i = 1, 2, …, n, n is the total number of stages set, is the weight of the inflow position reward, is the maximum allowable speed difference between the speed of the intelligent vehicle and the average speed of the two vehicles in front and behind, is the speed of the first vehicle in front of the intelligent vehicle, is the speed of the first vehicle behind the intelligent vehicle, is the speed of the intelligent vehicle, is the distance from the center point of the intelligent vehicle to the merging point, w the value range of is [0, 1], where 0 indicates that the distance gap between the intelligent vehicle and the first vehicle in front and the distance gap between the intelligent vehicle and the first vehicle behind are the same, and 1 indicates that the distance gap between the intelligent vehicle and the first vehicle in front or the first vehicle behind is zero, w is defined as follows: In the formula: is the distance from the first vehicle in front of the intelligent vehicle to the merging point, is the distance from the first vehicle behind the intelligent vehicle to the merging point, , are the vehicle lengths of the first vehicle in front of the intelligent vehicle and the intelligent vehicle, respectively.

2. The method of claim 1, characterized in that: The reward obtained by the agent of the reinforcement learning DDPG model is the sum of the first reward and the second reward, and the second reward is the value of any one of the following, or the sum of any two, or the sum of multiple: Collision penalty and non-collision reward, safe driving speed reward, non-destination stop penalty, destination reward, passenger comfort reward; wherein: The collision penalty and the non-collision reward are calculated as follows: In the formulae: represents the penalty for a vehicle collision, is a penalty value set for a vehicle collision; The safe driving speed reward is calculated as follows: wherein: represents the current vehicle speed of the intelligent vehicle a high vehicle speed reward, is the maximum vehicle speed of the intelligent vehicle, is the reward value for high-speed driving of the vehicle; Non-destination stop penalty: wherein: represents a reward for the vehicle successfully reaching, is a reward value for the vehicle successfully reaching. The destination reward is calculated as follows: wherein: represents a reward for the vehicle successfully reaching, a reward value set for the vehicle successfully reaching the destination; The passenger comfort is calculated as follows: wherein: is the weight of the comfort penalty reward, is the maximum allowed impact level for passenger comfort, is the derivative of the acceleration of the intelligent vehicle, is is the impact level experienced by the passenger.

3. The method of claim 1, characterized in that: The agent of the reinforcement learning DDPG model selects actions through an Actor network and evaluates the Q value of the selected action through a Critic network, both the Actor network and the Critic network are composed of a time neural network layer and a fully connected layer; wherein: Both the Actor network and the Critic network take the historical state information of the intelligent vehicle information and the environmental vehicle information as input, the action Action output by the Actor network is concatenated with the output of the time neural network layer in the Critic network, and then passed to the fully connected layer in the Critic network, which calculates and generates the evaluation Q value of the action Action.

4. The method of claim 3, characterized in that: Both the Actor network and the Critic network have network parameters and target network parameters, which are the same when initialized; At the time of updating, from the experience replay pool D randomly sample N a historical state-action pair data, N Set the value, first update the network parameters of the Critic network by minimizing the loss function θ Q Then update the network parameters of the Actor network by maximizing the value estimated from the Critic network Q θ μ ;​ target network parameters of the actor network and critic network , the target network parameters of the actor network and critic network are updated target network parameters of the actor network θ μ’ , for the approximation coefficients, the target network parameters of the critic network are updated target network parameters of the critic network θ Q’ ; When the reinforcement learning DDPG model training is completed, the Actor network uses the target network parameters to make decisions for the intelligent vehicle merging.

5. The method of claim 4, characterized in that: The experience replay pool is divided into two experience replay pools: a positive sample experience replay pool and a negative sample experience replay pool. When the agent samples samples in the experience pool, according to the set number of samples, data is sampled from the two experience replay pools respectively and then combined together for training. The positive sample is a sample in which the intelligent vehicle successfully merges, and the negative sample is a sample in which the intelligent vehicle fails to merge.

6. The method of claim 3, wherein: The timing neural network is any one of LSTM, GRU, Bi-LSTM and RNN.

7. An intelligent vehicle, characterized by: The intelligent vehicle has a system implemented according to any one of the methods of claims 1 to 6, and the intelligent vehicle successfully merges from the ramp onto the main road of the expressway through the system control.

8. A computer-readable storage medium, characterized in that: The computer program capable of being loaded and executed by the processor to perform any one of the methods of claims 1 to 6 is stored.

Images