A control method for single-vehicle lane-changing and merging into a platoon based on curriculum reinforcement learning

Abstract

The present invention relates to a control method for a single vehicle lane change and merging into a convoy based on curriculum reinforcement learning. The method comprises setting reinforcement learning training courses from simple to complex according to the difficulty of the task; determining the state space and action space of the reinforcement learning based on the surrounding vehicle information and the control object obtained by the on-board sensor; and setting a reward function in a multi-dimensional and hierarchical manner according to the vehicle state and the target task to guide the vehicle to complete the pursuit and lane change and enter the convoy; building an Actor network and a Critic network, and building an environment exploration module based on the proximal policy optimization algorithm, and updating the network using the Adam optimizer; selecting a corresponding reward function for training, performing a random number seed experiment on the network after each course training, and performing a scenario test on the policy neural network finally trained by the course learning to verify the completion of the target task of the model. The present invention solves the problem of slow and difficult training convergence caused by sparse rewards in the process of single vehicles merging into a convoy under complex traffic flow scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and mainly relates to a single-vehicle lane-changing merging into a vehicle fleet control method based on curriculum reinforcement learning. Background Art

[0002] Single-vehicle lane-changing and merging into a platoon is a key research topic in multi-vehicle cooperative driving. Its primary task is for individual vehicles (hereinafter referred to as "single vehicles") to interactively and collaboratively complete a series of driving maneuvers, such as obstacle avoidance, lane-changing, and joining a platoon, based on varying road conditions and based on a wireless ad hoc network. Platooning reduces the complexity of traffic control and management, effectively alleviating congestion. It also improves the consistency and continuity of traffic flow, reducing environmental pollution while ensuring road safety.

[0003] There are three main approaches to cooperative driving control both domestically and internationally: those based on optimal control theory, those based on potential functions, and those based on communication network topology. Single-vehicle lane-changing merging is often achieved through rule-based trajectory planning and tracking control. This approach is simple to implement and highly operational when there are no free-flow vehicles. However, in traffic flow or saturated traffic, single-vehicle merging requires considering the relative states of multiple vehicles, including those in the current lane, the preceding and following lanes, and the merging convoy. This makes path calculation more complex and challenging. Furthermore, planning and control often optimize the design of their respective modules in isolation, making it difficult to achieve optimal overall system performance. Furthermore, path planning errors propagate to the tracking control module, increasing its unreliability.

[0004] Reinforcement learning in machine learning uses continuous interactive learning between the intelligent agent and the environment to obtain a policy function that maximizes cumulative rewards. This type of method directly controls the sensor end to the control end. It has the advantage of being easier to develop in complex scenarios, and its end-to-end nature makes it easier to ensure global optimality without error amplification between modules. It has great application prospects in complex scenarios such as controlling a single vehicle merging into a fleet under saturated traffic flow. Summary of the Invention

[0005] The purpose of the present invention is to provide a single-vehicle lane-changing and merging convoy control method based on curriculum reinforcement learning, establish a neural network from the vehicle state space to the continuous action space, and use the proximal policy optimization algorithm (PPO) combined with curriculum learning to train and optimize the established policy network, so as to achieve safe, efficient and globally optimal control of the intelligent body to complete obstacle avoidance, lane-changing and merging operations in a complex saturated traffic flow environment.

[0006] The technical solution of the present invention is:

[0007] A method for controlling a single vehicle lane-changing and merging into a convoy based on curriculum reinforcement learning includes the following steps:

[0008] S1: During the training of a bicycle chasing across lanes, reinforcement learning training courses are set up according to the difficulty of the tasks from simple to complex;

[0009] S2: Determine the state space and action space of reinforcement learning based on the surrounding vehicle information and control objects obtained by the on-board sensors;

[0010] S3: Based on the vehicle status and target task, a multi-dimensional and hierarchical reward function is set to guide the vehicle to complete the pursuit and lane change into the convoy;

[0011] S4: Build an environment exploration module based on the proximal policy optimization algorithm, including environment iteration, collection of state-action rewards, and network update using the Adam optimizer;

[0012] S5: Select the corresponding reward function for training based on the set courses, and conduct random number seed experiments on the network after each course training. Conduct scenario tests on the policy neural network finally trained by the course learning to verify the completion of the model's target tasks.

[0013] Preferably, the course settings in S1 are specifically the following 4 courses:

[0014] Lesson 1: Longitudinal Control: In the absence of other traffic participants, the agent learns to control the longitudinal speed within the speed limit;

[0015] Lesson 2: Lateral Control: In a platoon environment, the agent is placed 30 meters behind the platoon in the adjacent lane. The agent learns to control its lateral and longitudinal directions to enter the gap between the platoons.

[0016] Course 3: Random Position Entry, a platoon environment where the agent is trained to join the platoon at a random position within 100 meters behind the platoon. The agent learns to control the lateral and longitudinal directions to enter the platoon.

[0017] Lesson 4: Free-flow and random position joining, free-flow + platoon environment: The agent is randomly positioned within 100m behind the platoon, surrounded by other free-flow vehicles; the agent learns to bypass free-flow vehicles and enter the platoon through lateral and longitudinal control.

[0018] Preferably, the state vector S in step S2 is obtained by splicing and secondary processing the position, speed and heading angle information of the vehicle and surrounding vehicles obtained by the on-board sensors in the vehicle network environment;

[0019] State vector S=[V1,V2,…,V i ,D], where the vector V i =[x i ,y i ,v xi ,vyi ,cos(h i ),sin(h i )] represents the two trigonometric function values ​​of the longitudinal position, lateral position, longitudinal velocity, lateral velocity and heading angle h of the i-th vehicle, and the vector D = [v p ,gap1,gap2,gap3,cos(d),sin(d)] represents the state related to the target position in the platoon, specifically including the platoon speed, the gap between two vehicles from the head vehicle to the tail vehicle in the platoon, and the two trigonometric function values ​​of the required heading angle from the vehicle position to the platoon gap;

[0020] The action space is a continuous action space, with the action vector A = [δ, a], where δ is the steering wheel angle and a is the vehicle acceleration, both normalized to [-1, 1]. A negative acceleration represents deceleration, while a positive represents acceleration, and the acceleration range is [-2, 2] m / s². A negative steering wheel angle represents a left turn, while a positive steering wheel angle represents a right turn, and the steering wheel angle range is [-pi / 36, pi / 36].

[0021] Preferably, the reward function r in step S3 is expressed as follows, specifically including: distance reward, speed reward, state reward, collision penalty and out-of-bounds penalty, constraining vehicle behavior from multiple dimensions to guide the vehicle to complete the target task;

[0022] r total =r distance +r speed +r state +r collision +r out_of_road

[0023] where r total represents the total reward; r distance Represents distance reward; r speed represents speed reward; r state Indicates status reward;

[0024] r collision represents collision penalty; r out_of_road Indicates an out-of-bounds penalty.

[0025] Preferably, the distance reward r distance Set to multi-level reward, the distance reward is expressed as follows:

[0026]

[0027] Where l is the distance between the ego vehicle and the target point. When the vehicle is within 30 meters of the platoon gap, the reward increases linearly with the decrease of l; within 6 meters of the target point, the reward increases exponentially with the decrease of l; when l is greater than or equal to 30 meters, the ego vehicle is penalized with -10.

[0028] Preferably, speed reward r speed It is also set as a graded reward, which is expressed as follows:

[0029]

[0030] where δ speed =|vv p |, represents the speed of the vehicle v and the speed of the convoy v p When the speed difference is within 2m / s, the speed reward is 10; when the speed difference is between 2-5m / s, the smaller δ is, the faster the speed reward increases linearly; when the speed difference is greater than 5m / s, a penalty of -50 is given to the ego vehicle.

[0031] Preferably, the state reward r state It is expressed as follows:

[0032]

[0033] Used to measure the current state of the vehicle S ego and the target state S after becoming a team member destination The greater the difference between the two states, the smaller the reward.

[0034] Preferably, the collision penalty r collision and out-of-bounds penalty out_of_road They are represented as follows:

[0035]

[0036]

[0037] Preferably, the objective function of the proximal strategy optimization algorithm in S4 is as follows:

[0038]

[0039] in, It is the first term of the minimization function min, which represents the probability ratio of the new and old strategies; π θ (a t |s t ) represents the updated strategy π θ In s t Make a t The probability of an action; π θ (a t |s t ) represents the updated strategy In s t Make a t The probability of the action; θ represents the weight and bias parameters of the policy neural network;

[0040] is the advantage function, indicating that in state s t Next action a t Deviation from the mean;

[0041] It is the second term of the minimization function and is an alternative objective function formed by clipping ratio(θ) by [1-ε,1+ε];

[0042] Finally, the parameters θ of the policy network are updated by performing multiple gradient ascents through the Adam optimizer. The update rule is as follows:

[0043]

[0044] Where θ′ is the new parameter and η is the learning rate.

[0045] Preferably, the goal of the random number seed experiment in S5 is to fully consider the impact of the uncertainty of the model initialization parameters on the reward convergence time during network training and the performance that the network can achieve after training, so as to more comprehensively evaluate the model; the purpose of the scenario test in S5 is to fully test the generalization ability of the model and test the ability of the ego vehicle to join the team by changing the team gap, the initialization speed of the team and the ego vehicle, and the position of the free-flow vehicle.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1) This paper fully utilizes the end-to-end characteristics of deep reinforcement learning control methods to innovatively solve the problem of controlling the merging process of a single vehicle in complex saturated traffic flow. It avoids the complicated analysis and calculation of vehicle posture and relative position required by traditional planning control methods, and provides a new approach to solving the control of single-vehicle lane change merging.

[0048] 2) Modeling real control problems as reinforcement learning often leads to sparse rewards. This paper designs reinforcement learning courses with different task difficulties for the complex saturated traffic flow and single-vehicle merging scenario for the first time. It overcomes the problems of sparse rewards, such as difficulty in convergence and easy divergence, and guides the intelligent agent to ultimately complete operations such as obstacle avoidance, catching up, and lane changing.

[0049] Figures in the specification

[0050] Figure 1 This is a flow chart of the control method for a bicycle changing lanes and merging into a vehicle fleet based on course reinforcement learning;

[0051] Figure 2 It is a diagram illustrating the course setting;

[0052] Figure 3 This is a diagram illustrating the basic elements of reinforcement learning;

[0053] Figure 4 This is the PPO algorithm training flow chart. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the embodiments and drawings. The protection scope of the present invention is not limited to the embodiments, and any changes made by those skilled in the art within the scope defined by the claims also fall within the protection scope of the present invention.

[0055] The present invention is based on the course reinforcement learning of the bicycle lane change and merging into the vehicle team control method. Figure 1 As shown, the following steps are included:

[0056] S1: During the training of a bicycle chasing across lanes, reinforcement learning training courses are set up according to the difficulty of the tasks from simple to complex;

[0057] S2: Determine the state space and action space of reinforcement learning based on the surrounding vehicle information and control objects obtained by the on-board sensors;

[0058] S3: Based on the vehicle status and target task, a multi-dimensional and hierarchical reward function is set to guide the vehicle to complete the pursuit and lane change into the convoy;

[0059] S4: Build an environment exploration module based on the Proximal Policy Optimization (PPO) algorithm, including environment iteration, collection of state-action rewards, and network update using the Adam optimizer.

[0060] S5: Select the corresponding reward function for training based on the set course, and conduct random number seed experiments on the network after each course training. Conduct scenario testing on the policy neural network trained by the course learning to verify the completion of the model's target task.

[0061] The specific embodiment implements Python programming based on the Pytorch deep learning framework, uses the open source Highway-env environment to interact with the policy neural network to gain experience, and uses the PPO algorithm framework to implement model updates. In addition, in order to facilitate the analysis of the effectiveness of the model, the matplotlib third-party library is used to visualize the results. Specific embodiment:

[0063] S1: During the training of the bicycle cross-lane pursuit team, the reinforcement learning training course is set up from simple to complex according to the difficulty of the task. The specific course setting includes 4 stages, such as Figure 2 As shown:

[0064] Lesson 1: Longitudinal Control: In the absence of other traffic participants, the agent learns to control the longitudinal speed within the speed limit;

[0065] Lesson 2: Lateral Control, in a platoon environment, the agent is placed 30 meters behind the platoon in the adjacent lane, and learns to maneuver horizontally and vertically into the gap between the platoons.

[0066] Course 3: Random Position Entry, a platoon environment where the agent is trained to join the platoon at a random position within 100 meters of the platoon. The agent learns to control the lateral and longitudinal directions to enter the platoon.

[0067] Lesson 4: Free-flow and platooning: The agent is randomly positioned within 100 meters behind a platoon, surrounded by other free-flow vehicles. The agent learns to avoid free-flow vehicles and enter the platoon through lateral and longitudinal control.

[0068] S2: Determine the state space and action space of reinforcement learning. The state vector S is obtained by splicing and secondary processing the position, speed and heading angle information of the vehicle and surrounding vehicles obtained by the on-board sensors in the Internet of Vehicles environment. State vector S = [V1, V2, ..., V i ,D], where the vector V i =[x i ,y i ,v xi ,v yi ,cos(h i ),sin(h i )] represents the two trigonometric function values ​​of the longitudinal position, lateral position, longitudinal velocity, lateral velocity and heading angle h of the i-th vehicle, and the vector D = [v p , gap1, gap2, gap3, cos(d), sin(d)] represents the state related to the target position in the platoon, specifically including the platoon speed, the gap between each vehicle from the head vehicle to the tail vehicle in the platoon, and the two trigonometric function values ​​of the required heading angle from the vehicle position to the platoon gap.

[0069] The action space is a continuous action space, and the action vector A = [δ, a], where δ is the steering wheel angle and a is the vehicle acceleration, both normalized to [-1, 1]. A negative acceleration represents deceleration, and a positive acceleration represents acceleration. The acceleration range is [-2, 2] m / s. 2 ; A negative steering wheel angle represents a left turn, and a positive steering wheel angle represents a right turn. The steering wheel angle range is [-pi / 36, pi / 36].

[0070] S3: Determine the reward function for reinforcement learning. The reward function r is expressed as follows. It includes distance reward, speed reward, state reward, collision penalty, and out-of-bounds penalty. It constrains vehicle behavior from multiple dimensions to guide the vehicle to complete the target task.

[0071] r total =r distance +r speed +rstate +r collision +r out_of_road

[0072] where r total represents the total reward; r distance Represents distance reward; r speed represents speed reward; r state Indicates status reward;

[0073] r collision represents collision penalty; r out_of_road Indicates an out-of-bounds penalty.

[0074] 1) Distance reward r distance A multi-level reward is set, where l is the distance between the ego vehicle and the target point. When the vehicle is within 30 meters of the platoon, the reward increases linearly as l decreases; within 6 meters of the target point, the reward increases exponentially as l decreases; when l is greater than or equal to 30 meters, the ego vehicle is penalized with -10. The distance reward is expressed as follows:

[0075]

[0076] 2) Speed ​​reward r speed Also set as a graded reward, where δ speed =|vv p | represents the difference between the speed of the ego vehicle and the speed of the group. When the speed difference is within 2 m / s, the speed reward is 10. When the speed difference is between 2-5 m / s, the smaller δ is, the faster the speed reward increases linearly. When the speed difference is greater than 5 m / s, the ego vehicle is penalized with -50. The speed reward is expressed as follows:

[0077]

[0078] 3) Status reward r state Used to measure the current state of the vehicle S ego and the target state S after becoming a team member destination The greater the difference between the two states, the smaller the reward. The state reward is expressed as follows:

[0079]

[0080] 4) Collision penalty r collision and out-of-bounds penalty out_of_road Certain penalties will be imposed when a vehicle collides or goes out of bounds, as shown below:

[0081]

[0082] So far, step S2 and step S3 have jointly determined the elements of reinforcement learning. The elements of reinforcement learning are shown in the figure below: Figure 3shown.

[0083] S4: The policy neural network (Actor network) and critic network built using PyTorch each consist of one input layer, two fully connected layers, and one output layer. The dimension of the policy neural network (Actor network) input layer is equal to the state space dimension determined in step 1, 6*(i+1)×1, where i represents the total number of vehicles on the road. The dimensions of the two fully connected layers are 128×1 and 64×1, respectively. The dimension of the output layer is determined by the action space, 2×1. The output layer of the critic network is 1×1. The number of input and hidden layers in the remaining networks is the same as that of the actor network.

[0084] The specific training process is as follows: First, initialize the Actor network and Critic network, and initialize the environment. Then, start training, and the agent observes the environment to obtain the state s t , Actor network output action a t , the environment gives the agent reward r t , and whether this round is done, the environment changes to the next state s t+1 , obtaining a four-tuple [state, action, reward, done]. The agent then continuously interacts with the environment highway-env to obtain multiple four-tuples. These four-tuples are stacked to form a database for updating the neural network, with the number of stack layers being horizon_length = 1024. Finally, the proximal policy optimization algorithm is used to update the parameters of the policy neural network multiple times, with a number of repeat_times = 6. The objective function of the proximal policy optimization algorithm is as follows:

[0085]

[0086] in, It is the first term of the minimization function min, which represents the probability ratio of the new and old strategies; π θ (a t |s t ) represents the updated strategy π θ In s t Make a t The probability of an action; π θ (a t |s t ) represents the updated strategy In s t Make a t The probability of the action; θ represents the weight and bias parameters of the policy neural network;

[0087] is the advantage function, indicating that in state s t Next action at Deviation from the mean;

[0088] is the second term of the minimization function, which is an alternative objective function formed by clipping ratio(θ) by [1-ε,1+ε]. Finally, the parameters θ of the policy network are updated by performing multiple gradient ascents with the Adam optimizer. The update rule is as follows:

[0089]

[0090] Where θ′ is the new parameter and η is the learning rate.

[0091] The training methods for a single course are as follows: Figure 4 As shown, the specific PPO algorithm hyperparameter values ​​are selected as follows:

[0092] Hyperparameters Parameter meaning Parameter value ε Clipping range in PPO objective function 0.25 η Learning rate of Adam optimizer <![CDATA[6×10 -5 ]]> horizon_length The number of samples collected at a time 1024 repeat_times The number of times the sample library updates the Actor each time 6

[0093] The training process is optimized through course learning and hyperparameter tuning to obtain a strategic neural network that outputs continuous actions.

[0094] S5: Select the appropriate reward function for training based on the set curriculum. After each course, perform random seed experiments on the trained network. Finally, perform scenario testing on the policy neural network trained from the course. The goal of the random seed experiments is to fully consider the impact of model initialization parameter uncertainty on the reward convergence time during network training and the performance achieved after training, thereby providing a more comprehensive evaluation of each course-trained model. The scenario testing aims to fully verify the model's generalization capabilities. By varying the platoon spacing, the initialization speeds of the platoon and ego vehicle, and the position of free-flow vehicles, the ego vehicle's ability to join a platoon is tested.

Claims

1. A single-vehicle lane-changing merging control method based on curriculum reinforcement learning, characterized by: The following steps are involved: S1: During the training of a bicycle chasing across lanes, reinforcement learning training courses are set up according to the difficulty of the tasks from simple to complex; S2: Determine the state vector S and action space of reinforcement learning based on the surrounding vehicle information and control objects obtained by the on-board sensors; S3: Based on the vehicle status and target task, a multi-dimensional and hierarchical reward function is set to guide the vehicle to complete the pursuit and lane change into the convoy; S4: Build an environment exploration module based on the proximal policy optimization algorithm, including environment iteration, collection of state-action rewards, and network update using the Adam optimizer; S5: Select the corresponding reward function for training based on the set courses, and conduct random number seed experiments on the network after each course training. Conduct scenario tests on the policy neural network finally trained by the course learning to verify the completion of the model's target tasks; The course settings in S1 are specifically the following 4 courses: Lesson 1: Longitudinal Control: In the absence of other traffic participants, the agent learns to control the longitudinal speed within the speed limit; Lesson 2: Lateral Control: In a platoon environment, the agent is placed 30 meters behind the platoon in the adjacent lane. The agent learns to control its lateral and longitudinal directions to enter the gap between the platoons. Course 3: Random Position Entry, a platoon environment where the agent is trained to join the platoon at a random position within 100 meters behind the platoon. The agent learns to control the lateral and longitudinal directions to enter the platoon. Lesson 4: Joining the platoon at a random position with free flow, free flow and platoon environment: The agent is randomly positioned within 100 meters behind the platoon, surrounded by other free flow vehicles. The agent learns to bypass free flow vehicles and enter the platoon through lateral and longitudinal control. The step S3 specifically includes: distance reward, speed reward, state reward, collision penalty and out-of-bounds penalty, which constrains the vehicle behavior from multiple dimensions to guide the vehicle to complete the target task. The reward function r is expressed as follows: r total ＝r distance +r speed +r state +r collision +r out_of_road where r total represents the total reward; r distance Represents distance reward; r speed represents speed reward; r state Indicates status reward; r collision represents collision penalty; r out_of_road Indicates an out-of-bounds penalty.

2. The method for controlling a single vehicle lane change and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: The state vector S in step S2 is obtained by splicing and secondary processing the position, speed and heading angle information of the vehicle and surrounding vehicles obtained by the on-board sensors in the vehicle network environment; State vector S=[V1,V2,···,V i ,D], where the vector V i =[x i ,y i ,v xi ,v yi ,cos(h i ),sin(h i )] represents the two trigonometric function values ​​of the longitudinal position, lateral position, longitudinal velocity, lateral velocity and heading angle h of the i-th vehicle, and the vector D = [v p ,gap1,gap2,gap3,cos(d),sin(d)] represents the state related to the target position in the platoon, specifically including the platoon speed, the gap between two vehicles from the head vehicle to the tail vehicle in the platoon, and the two trigonometric function values ​​of the required heading angle from the vehicle position to the platoon gap; The action space is a continuous action space, and the action vector A = [δ, a], where δ is the steering wheel angle and a is the vehicle acceleration, both normalized to [-1, 1]. A negative acceleration represents deceleration, and a positive acceleration represents acceleration. The acceleration range is [-2, 2] m / s. 2 ; A negative steering wheel angle represents a left turn, and a positive steering wheel angle represents a right turn. The steering wheel angle range is [-pi / 36, pi / 36].

3. The method for controlling a single vehicle lane change and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: Distance reward r distance Set to multi-level reward, the distance reward is expressed as follows: Where l is the distance between the ego vehicle and the target point. When the vehicle is within 30 meters of the platoon gap, the reward increases linearly with the decrease of l; within 6 meters of the target point, the reward increases exponentially with the decrease of l; when l is greater than or equal to 30 meters, the ego vehicle is penalized with -10.

4. The method for controlling a single vehicle lane change and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: Speed ​​Bonus speed It is also set as a graded reward, which is expressed as follows: where δ speed =|vv p |, represents the speed of the vehicle v and the speed of the convoy v p When the speed difference is within 2m / s, the speed reward is 10; when the speed difference is between 2-5m / s, the smaller δ is, the faster the speed reward increases linearly; when the speed difference is greater than 5m / s, a penalty of -50 is given to the ego vehicle.

5. The method for controlling a single vehicle lane change and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: Status reward r state It is expressed as follows: Used to measure the current state of the vehicle S ego and the target state S after becoming a team member destination The greater the difference between the two states, the smaller the reward.

6. The method for controlling a single vehicle lane-changing and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: Collision Penalty collision and out-of-bounds penalty out_of_road They are represented as follows:

7. The method for controlling a single vehicle lane-changing and merging into a vehicle fleet based on curriculum reinforcement learning according to claim 1, characterized in that: The goal of the random number seed experiment in S5 is to fully consider the impact of the uncertainty of the model initialization parameters on the reward convergence time during network training and the performance that the network can achieve after training, thereby more comprehensively evaluating the model. The purpose of the scenario test in S5 is to fully verify the generalization ability of the model and test the ego vehicle's ability to join the team by changing the team spacing, the initialization speed of the team and the ego vehicle, and the position of free-flow vehicles.

Images