Reinforcement learning multi-lane driving decision-making method for dynamic traffic environment

Abstract

The invention relates to a reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment. The method comprises the following steps: establishing a decision-making neural network model by using a state space, an action space and track sampling information in a multi-lane driving decision-making scene; performing reinforcement learning training on the decision neural network model by using a comprehensive reward function; the method comprises the following steps: continuously and deeply sensing the surrounding environment through a sensor array arranged on a vehicle, and capturing sensed environment information; the perceived environment information is input into the decision neural network model, and the decision neural network model predicts a vehicle track and recommended driving operation within a period of time according to the current environment information; and the recommended driving operation output by the decision neural network model is converted into a specific control instruction, and the instruction is sent to a drive-by-wire chassis and an actuator of the vehicle. By means of the method, multi-lane automatic driving of the vehicle can be achieved, dependence on external data and a high-precision map is not needed, and independence and stability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and more specifically, to a reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment. Background Art

[0002] The core functions of autonomous vehicles include environmental perception, autonomous decision-making, path planning, and motion control. Environmental perception involves the use of on-board sensors and communication systems to perceive surrounding traffic conditions and obstacles. Autonomous decision-making includes identifying the intentions of other traffic participants and predicting trajectories to formulate driving behaviors and plan reference trajectories. Path planning is responsible for selecting the best driving route, taking into account road conditions, traffic conditions, and destinations to ensure efficient and safe navigation. Motion control converts decision results into low-level control instructions and masters key elements such as throttle, brakes, and steering wheels. Since autonomous decision-making and motion control can be modeled as optimization problems of dynamic processes and are interdependent, they can be integrated into a unified functional module. This module plays a role similar to the brain in autonomous vehicles, and its intelligence level affects the autonomous driving performance of the vehicle.

[0003] The existing assisted driving decision-making methods rely on urban area road maps, which may lead to geographical limitations. In areas that are not fully mapped or where road data is not updated in a timely manner, decision-making may be limited. The existing technology builds a global path planning solution that can only serve as an auxiliary to driving and cannot provide support for local autonomous driving. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a reinforcement learning multi-lane driving decision-making method for dynamic traffic environments, so that the vehicle can achieve multi-lane automatic driving without relying on external data and high-precision maps.

[0005] The technical solution adopted by the present invention to solve the technical problem is: constructing a reinforcement learning multi-lane driving decision method for a dynamic traffic environment, comprising:

[0006] A decision neural network model is established using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and reinforcement learning training is performed on the decision neural network model using a comprehensive reward function;

[0007] The sensor array equipped on the vehicle continuously and deeply perceives the surrounding environment and captures the perceived environmental information, including lane lines, the position, speed and expected behavior of traffic participants;

[0008] The perceived environmental information is input into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations within a future period of time based on the current environmental information;

[0009] The recommended driving operations output by the decision neural network model are converted into specific control instructions, and the instructions are sent to the vehicle's wire-controlled chassis and actuators.

[0010] In the above scheme, the state space includes the self-vehicle state, reference path information and surrounding vehicle state;

[0011] The self-vehicle state includes vehicle speed and heading angular velocity; the reference path information includes a series of path points on a continuous route, and each path point contains information including longitudinal position, lateral position and heading angle; the surrounding vehicle state includes a given number of surrounding vehicles, and the state of each vehicle includes longitudinal position, lateral position, heading angle, body length and body width.

[0012] In the above scheme, the action space includes the longitudinal acceleration of the vehicle and the steering wheel angle. The increment of the control amount is used as the strategy output, and the actual control amount is obtained by adding it to the current control amount of the vehicle.

[0013] In the above scheme, the trajectory sampling information samples N possible trajectories, assuming τ i is the i-th trajectory, where i = 1, 2, ..., N;

[0014] τ i ={s t ,a t ,s t+1 ,a t+1 ,...,s T}

[0015] Where T is the end time of the trajectory, and a t is the action taken at time t.

[0016] In the above scheme, the A2C reinforcement learning method is adopted, and the Critic network and the Actor network are introduced to train the decision neural network model;

[0017] The structure of the Critic network includes three fully connected layers, the activation function is ReLU, and the part that processes time series information uses LSTM. The input is the current state and the corresponding trajectory, and the output is the value of the trajectory. The score of the trajectory is based on the expected future reward;

[0018] The structure of the Actor network includes two fully connected layers, the activation function is tanh, and the input is the current state and the optimal trajectory determined by the Critic network. The Actor network directly generates corresponding control instructions to track the optimal trajectory based on the input state and optimal trajectory.

[0019] In the above scheme, the comprehensive reward function takes into account safety, efficiency and comfort;

[0020] For safety; if the distance between the predicted trajectory and other vehicles is lower than the predefined safety threshold, a strong negative reward will be applied; the distance between the vehicle and the center of the lane is calculated, and the greater the distance, the smaller the reward; for violations of traffic rules, such as changing lanes in a prohibited lane change area, strict negative rewards will be given;

[0021] Targeting efficiency; when a vehicle successfully overtakes a slow-speed vehicle ahead under appropriate conditions, a positive reward is given; the difference between the vehicle’s actual speed and the ideal speed of its lane is compared, the smaller the difference, the greater the reward;

[0022] For comfort; negative rewards are applied for frequent lane changes or sharp steering behaviors; sudden acceleration or braking is detected and negative rewards are applied.

[0023] In the above scheme, a constraint function is used to penalize the collision between the ego vehicle and surrounding vehicles in order to reward safety. The constraint function uses the double circle method to calculate the distance between the ego vehicle and surrounding vehicles. This method uses two circles to cover the plane shape of the vehicle body and converts the distance constraint between vehicles into the distance constraint between the centers of the circles. When using the double circle method to calculate the vehicle-to-vehicle distance, it is necessary to calculate the distance between the two centers of the ego vehicle and the two centers of each surrounding vehicle, and then take the minimum distance as the vehicle-to-vehicle distance. To ensure the safety of the driving strategy, the safety distance is increased by an amount based on the sum of the radii of the two circles, which is called the redundant safety distance. For all surrounding vehicles whose vehicle-to-vehicle distance is less than the safety distance, it is necessary to calculate the collision constraint penalty, and then add these penalty items to obtain the final constraint function.

[0024] In the above scheme, the method for the decision neural network model to predict the vehicle trajectory and recommended driving operation in the future period of time based on the current environmental information includes:

[0025] According to the identified lane line information, determine whether there are optional roads on both sides of the vehicle. Based on the optional lanes, perform trajectory sampling and design several alternative trajectories.

[0026] Use the Critic network to calculate the value function of each optional trajectory, and select the trajectory with the highest value, which is recorded as the optimal lane; if the lane where the optimal trajectory is located is the currently selected lane, then the selected lane is maintained and the lane selection process ends, otherwise the judgment process continues;

[0027] If the selected lane is not a safe lane, that is, a collision will occur, the optimal trajectory will be selected immediately and the process will end, otherwise it will continue. This judgment condition is used to handle emergency situations. If the selected lane is about to collide, there is no time for subsequent lane keeping and lane change waiting operations, and it must immediately switch to a safe lane;

[0028] When considering lane change decisions, the safety of the currently selected lane is first evaluated. Lane change is considered only if the vehicle has been in the current lane for a certain period of time and the optimal lane evaluated remains stable over a continuous period of time. Once the above constraints are met, the vehicle will select the optimal lane according to the evaluation results.

[0029] The selected optimal trajectory is used as the reference trajectory, and the differences and similarities between it and the current vehicle state and environmental state are input into the Actor network to obtain the control instructions, which are then sent to the controller for execution.

[0030] The present invention also provides a reinforcement learning multi-lane driving decision-making device for dynamic traffic environment, comprising:

[0031] A decision neural network model training module is used to establish a decision neural network model using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and train the decision neural network model using a comprehensive reward function;

[0032] The information perception module is used to continuously and deeply perceive the surrounding environment through the sensor array equipped by the vehicle, and capture the perceived environmental information, including lane lines, the position, speed and expected behavior information of other traffic participants;

[0033] A decision module, used to input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations within a future period of time based on the current environmental information;

[0034] The control module is used to convert the recommended driving operations output by the decision neural network model into specific control instructions and send the instructions to the vehicle's wire-controlled chassis and actuators.

[0035] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the steps of the reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment.

[0036] The present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of the reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment.

[0037] The implementation of the reinforcement learning multi-lane driving decision-making method for dynamic traffic environment of the present invention has the following beneficial effects:

[0038] 1. The present invention uses reinforcement learning, and the vehicle can self-learn and adjust according to the ever-changing road environment, which has good adaptability. By adding rule constraints, while ensuring adaptability, it also ensures that the vehicle's driving decision has a certain degree of certainty and standardization.

[0039] 2. The present invention mainly relies on the vehicle's sensor data, reinforcement learning technology and some basic rule constraints, and does not involve complex map information or other external resources, reducing the dependence on external information and improving the independence and stability of the system. It saves a part of the map cost. In the absence of maps, the vehicle can better adapt to various environments, such as in some areas where maps have not yet been covered or updated in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0041] Figure 1 It is a flow chart of the reinforcement learning multi-lane driving decision method for dynamic traffic environment of the present invention;

[0042] Figure 2 is a schematic diagram of the sampling trajectory;

[0043] Figure 3 It is a schematic diagram of the multi-lane decision process of the decision neural network model. DETAILED DESCRIPTION

[0044] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0045] Embodiment 1

[0046] like Figure 1-3 As shown, the reinforcement learning multi-lane driving decision method for dynamic traffic environment of the present invention comprises the following steps:

[0047] S1. Establish a decision neural network model using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and perform reinforcement learning training on the decision neural network model using a comprehensive reward function.

[0048] In the scenario of multi-lane driving decision, the state space provides the necessary input parameters for vehicle decision making. The state space consists of three parts: the state of the vehicle, the reference path information, and the state of the surrounding vehicles, expressed as s = [s ego ,s ref ,s sur ], the attributes of the vehicle state are v x (Unit: m·s -1 ) and angular velocity ω (unit: rad·s-1 ),Right now:

[0049] s ego =[v x ,ω] T .

[0050] The reference path information consists of a series of path points on a continuous route, namely:

[0051]

[0052] Where N = 30; each path point contains information including longitudinal position x (unit: m), lateral position y (unit: m) and heading angle θ (unit: rad), namely:

[0053] p i ={x i ,y i ,θ i}

[0054] These quantities are all relative quantities in the vehicle coordinate system.

[0055] The weekly car state contains a given number of weekly cars, namely:

[0056] s other ={c1,c2,…,c M}

[0057] The state of each vehicle includes the longitudinal position x (unit: m), the lateral position y (unit: m), the heading angle θ (unit: rad), the body length L (unit: m) and the body width W (unit: m), namely:

[0058] c i ={x i ,y i ,θ i ,L i ,W i},

[0059] In the scenario of multi-lane driving decision, the action space includes two control quantities: the vehicle longitudinal acceleration a x (Unit: m·s 2 ) and steering wheel angle δ (unit: rad). In order to improve the smoothness of driving strategy output, the increment of control amount is used as strategy output, and the actual control amount is obtained by adding it to the current control amount of the vehicle. x and Δδ represent the increments of longitudinal acceleration and steering wheel angle respectively, then the action can be written as:

[0060] u=[Δa x ,Δδ] T .

[0061] like Figure 2 As shown, the method for obtaining trajectory sampling information includes: for a given state s t , sample N possible trajectories. Let τ i is the i-th trajectory, where i=1, 2,…,N.

[0062] τ i ={s t ,a t ,s t+1 ,a t+1 ,...,s T}

[0063] Where T is the end time of the trajectory, and a t is the action taken at time t. These trajectories can be based on some known heuristics, based on previous experience, or in other appropriate ways.

[0064] The sampling trajectory can be described by a quartic function: i =ax 4 +bx 3 +cx 2 +dx+e.

[0065] In the training model, the A2C (Advantage Actor-Critic) reinforcement learning method was adopted, and a dual network structure, namely the Critic network and the Actor network, was introduced.

[0066] The structure of the Critic network includes three fully connected layers, the activation function is ReLU, the part that processes time series information uses LSTM, the input is the state s and the corresponding trajectory τ, and the output is the value of the trajectory V(τ). The score of the trajectory should be based on the expected future rewards, which may include the distance to other vehicles, the smoothness of driving, compliance with traffic rules, etc. The total expected reward of the trajectory is:

[0067]

[0068] where r t′ is the reward obtained at time t′, and γ is the discount factor of the reward.

[0069] The structure of the Actor network consists of two fully connected layers, the activation function is tanh, and the input is the current state s and the optimal trajectory τ determined by the Critic network * ,

[0070]

[0071] The Actor network is based on the input state s and the optimal trajectory τ *, directly generate corresponding control instructions to track τ * , as shown below:

[0072] a t =π(s,τ * )

[0073] Among them, a t is a vector containing the values ​​of acceleration and steering wheel angle, defined as:

[0074] a t =[Δa x ,Δδ]=π(s,τ * ).

[0075] When the vehicle is driving in the simulation scene, the reward r t To update the Actor and Critic network. The Actor’s goal is to maximize the expected reward evaluated by the Critic, so its loss function L A for:

[0076]

[0077] For Critic, its loss function L C is the mean squared error between the true reward and its prediction:

[0078] L C =(r t +γV(s t+1 )-V(s t )) 2 .

[0079] The reward function design method includes: The reward function in multi-lane driving decision-making involves multiple key parameters, aiming to ensure that the algorithm achieves the most optimized decision in actual scenarios. Considering the complexity of multi-lane driving scenarios, a comprehensive reward function is designed, combining factors such as safety, efficiency and comfort.

[0080] Safety factors include: collision prevention, where a strong negative reward is applied if the predicted trajectory is below a predefined safety threshold from other vehicles. Lane center driving, where the system calculates the distance between the vehicle and the center of the lane it is in, with the greater the distance, the smaller the reward. Traffic rule compliance, where strict negative rewards are applied for violations of traffic rules, such as changing lanes in a no-change zone.

[0081] Efficiency factors include: Overtaking reward, which gives a positive reward when the vehicle successfully overtakes a slow-speed vehicle ahead under appropriate conditions; Speed ​​maintenance, which compares the difference between the vehicle's actual speed and the ideal speed of the lane it is in, with smaller differences giving larger rewards.

[0082] Comfort factors include: driving smoothness, where negative rewards are applied for frequent lane changes or sharp steering behaviors to ensure smooth driving; acceleration and braking continuity, where the system can detect sudden acceleration or braking and apply negative rewards to encourage smooth driving behavior.

[0083] In order to integrate the above elements, the reward function adopts the form of weighted summation:

[0084] R=w1R 安全 +w2R 效率 +w3R 舒适

[0085] Among them, w1, w2, and w3 are predefined weights that can be adjusted according to the needs of the actual application scenario to achieve the desired decision balance.

[0086] Here, each part of the reward item is designed in more detail. The efficiency reward can be set as a reward for tracking error r track , which mainly includes items based on tracking errors, including the lateral position error, heading angle error and speed error of the currently selected path, which are used to improve the tracking performance of the strategy, namely:

[0087]

[0088] Among them, the coefficients w p ,w φ ,w v are all positive real numbers. In this experiment, w p =20,w φ =100,w v =0.2.

[0089] The comfort-related reward can be set here to constrain the control amount. In order to prevent the driving strategy from overfitting and causing the output control amount to be too large, it can be regarded as a regular term of the driving strategy:

[0090]

[0091] Among them, the coefficients w a ,w δ are all positive real numbers. In this experiment, w a =0.1,w δ =20.

[0092] Here, the safety reward uses a constraint function to penalize the collision between the ego vehicle and surrounding vehicles, which ensures the safety of the driving strategy.

[0093] Specifically, the double circle method is used to calculate the distance between the self-vehicle and the surrounding vehicles. This method uses two circles to cover the plane shape of the vehicle body, converting the distance constraint between the vehicles into the distance constraint between the centers of the circles, thereby simplifying the distance calculation between complex shapes. When using the double circle method to calculate the vehicle-to-vehicle distance, it is necessary to calculate the distance between the two centers of the self-vehicle and the two centers of each surrounding vehicle, that is, each surrounding vehicle needs to calculate 4 distances, and then take the minimum distance as the vehicle-to-vehicle distance, that is

[0094]

[0095] in, is the center coordinate of the vehicle, is the coordinate of the center of the circle of the i-th vehicle. i is greater than the sum of the radii of the two cars, then the two cars have not collided; if d i If the distance between the centers of the two circles is less than or equal to the sum of the radii of the two cars, the two cars collide. The minimum distance between the centers of the circles is called the safety distance d. safe To ensure the safety of the driving strategy, the safety distance is increased by an amount based on the sum of the two car circle radii, which is called the redundant safety distance, that is: d safe =r e +r i +d0, in this experiment d0 = 2m. For all vehicles whose vehicle distance is less than the safe distance, the collision constraint penalty needs to be calculated, and then these penalty terms are added together to obtain the final constraint function:

[0096]

[0097] Among them, I collision is the set of surrounding vehicles that collided, I collision ={id i ≤d safe}, coefficient w c is a positive real number. In this experiment, w c =5.

[0098] S2. The sensor array equipped by the vehicle continuously and deeply perceives the surrounding environment and captures the perceived environmental information, including lane lines, the position, speed and expected behavior of other traffic participants. All data is pre-processed and unified into the vehicle's coordinate system to ensure data consistency and accuracy.

[0099] To ensure effective and accurate environmental perception, it is recommended that the vehicle be equipped with at least:

[0100] The front camera is designed to capture the road conditions and environmental image data in front of the vehicle, thereby supporting the recognition of lane lines, traffic signs and other important target objects.

[0101] Surround millimeter-wave radar devices are designed to accurately detect and collect information about obstacles around the vehicle, such as their relative distance and speed.

[0102] According to the design of the present invention, the basic perception data should at least include:

[0103] Lane data, including the lane type and the mathematical equation used to describe the lane's position, curvature, and length relative to the vehicle.

[0104] Obstacle data, including relative distance to obstacles and lateral and longitudinal speeds.

[0105] Vehicle status data, such as vehicle speed, yaw angle and other key driving parameters.

[0106] It should be emphasized that the above description is the most basic perception data requirement. As the data increases and becomes richer, such as being able to recognize traffic lights, pedestrians, non-motor vehicles, etc., the system's model effect and prediction accuracy will be further improved, making the autonomous driving function more powerful and safer.

[0107] S3. Input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations in the future period of time based on the current environmental information.

[0108] like Figure 3 As shown in the figure, since reinforcement learning may have uncertainty, combining rule constraints can provide clearer driving decisions in specific situations. The decision-making method includes the following steps:

[0109] S31, calculating optional lanes. According to the identified lane line information, it is determined whether there are optional roads on both sides of the vehicle.

[0110] S32, based on the optional lanes, perform trajectory sampling and design several alternative trajectories.

[0111] S33, using the Critic network to calculate the value function of each optional trajectory, and select the trajectory with the highest value, which is recorded as the optimal lane. If the lane where the optimal trajectory is located is the currently selected lane, the selected lane is maintained and the lane selection process ends, otherwise the judgment process continues.

[0112] S34: If the selected lane is not a safe lane, that is, a collision will occur, the optimal trajectory is immediately selected and the process ends, otherwise it continues. This judgment condition is used to handle emergency situations. If the selected lane is about to collide, there is no time for subsequent lane keeping and lane change waiting operations, and it is necessary to switch to a safe lane immediately.

[0113] S35, when considering a lane change decision, first evaluate the safety of the currently selected lane. Based on industry standards and driving experience, we set a constraint module: if the vehicle has been in the current lane for a certain period of time, such as more than 3 seconds, and the optimal lane evaluated remains stable for a continuous period of time, such as 1 second, then consider changing lanes. Such constraints are intended to prevent frequent or unreasonable lane changes due to instantaneous misjudgments. After the above constraints are met, the vehicle will select the optimal lane according to the evaluation results.

[0114] S36, the selected optimal trajectory is used as a reference trajectory, and the differences and similarities between it and the current vehicle state and environment state are input into the Actor network to obtain a control instruction, which is then sent to the controller for execution.

[0115] S4. Convert the recommended driving operation output by the decision neural network model into specific control instructions, and send the instructions to the vehicle's wire-controlled chassis and actuators.

[0116] Furthermore, when designing a multi-lane driving decision framework, a simulation environment is needed to simulate real road scenarios to verify and tune the reinforcement learning model.

[0117] The method of the present invention first focuses on model training based on perception data to ensure that the model can fully understand and process data in the real world. The choice of simulation environment is not limited to a certain simulator, but it must meet the basic requirements of road construction and traffic flow design to simulate the real traffic environment.

[0118] Road design not only needs to simulate real road scenes, but also needs to take into account the generalization of scenes. To this end, a multi-lane road with a specific range of curvature is randomly generated. This design ensures that during training, the model can learn in a variety of different road scenes, thereby improving its generalization performance.

[0119] The present invention combines the advantages of real data and simulated data when processing traffic flow data. The real traffic flow data extracted from the NGSIM data set provides the model with real-world traffic behavior, thereby improving the fidelity of the model. At the same time, some car agents with pre-set routes are randomly generated in the simulation scene. These agents are driven by the intelligent driving model IDM and can interact with the self-vehicle in training, further enhancing the learning ability of the model.

[0120] Embodiment 2

[0121] The present invention also provides a reinforcement learning multi-lane driving decision-making device for dynamic traffic environment, comprising:

[0122] A decision neural network model training module is used to establish a decision neural network model using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and train the decision neural network model using a comprehensive reward function;

[0123] The information perception module is used to continuously and deeply perceive the surrounding environment through the sensor array equipped by the vehicle, and capture the perceived environmental information, including lane lines, the position, speed and expected behavior information of other traffic participants;

[0124] A decision module, used to input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations within a future period of time based on the current environmental information;

[0125] The control module is used to convert the recommended driving operations output by the decision neural network model into specific control instructions and send the instructions to the vehicle's wire-controlled chassis and actuators.

[0126] Embodiment 3

[0127] The present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment.

[0128] Embodiment 4

[0129] The present invention also provides a computer-readable storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0131] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0132] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0134] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A reinforcement learning multi-lane driving decision method for dynamic traffic environment, characterized in that: include: A decision neural network model is established using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and reinforcement learning training is performed on the decision neural network model using a comprehensive reward function; The sensor array equipped on the vehicle continuously and deeply perceives the surrounding environment and captures the perceived environmental information, including lane lines, the position, speed and expected behavior of traffic participants; The perceived environmental information is input into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations within a future period of time based on the current environmental information; The recommended driving operations output by the decision neural network model are converted into specific control instructions, and the instructions are sent to the vehicle's wire-controlled chassis and actuators.

2. The reinforcement learning multi-lane driving decision-making method for dynamic traffic environment according to claim 1 is characterized in that: The state space includes the self-vehicle state, reference path information and surrounding vehicle state; The self-vehicle state includes vehicle speed and heading angular velocity; the reference path information includes a series of path points on a continuous route, and each path point contains information including longitudinal position, lateral position and heading angle; the surrounding vehicle state includes a given number of surrounding vehicles, and the state of each vehicle includes longitudinal position, lateral position, heading angle, body length and body width.

3. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, characterized in that: The action space includes the longitudinal acceleration of the vehicle and the steering wheel angle. The increment of the control amount is used as the strategy output and added to the current control amount of the vehicle to obtain the actual control amount.

4. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, characterized in that: The trajectory sampling information samples N possible trajectories, assuming τ i is the i-th trajectory, where i = 1, 2, ..., N; t i ={s t ,a t ,s t+1 ,a t+1 ,...,s T } Where T is the end time of the trajectory, and a t is the action taken at time t.

5. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, characterized in that: The A2C reinforcement learning method is adopted, and the Critic network and the Actor network are introduced to train the decision neural network model; The structure of the Critic network includes three fully connected layers, the activation function is ReLU, and the part that processes time series information uses LSTM. The input is the current state and the corresponding trajectory, and the output is the value of the trajectory. The score of the trajectory is based on the expected future reward; The structure of the Actor network includes two fully connected layers, the activation function is tanh, and the input is the current state and the optimal trajectory determined by the Critic network. The Actor network directly generates corresponding control instructions to track the optimal trajectory based on the input state and optimal trajectory.

6. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, characterized in that: The comprehensive reward function takes into account safety, efficiency and comfort; For safety; if the distance between the predicted trajectory and other vehicles is lower than the predefined safety threshold, a strong negative reward will be applied; the distance between the vehicle and the center of the lane is calculated, and the greater the distance, the smaller the reward; for violations of traffic rules, such as changing lanes in a prohibited lane change area, strict negative rewards will be given; Targeting efficiency; when a vehicle successfully overtakes a slow-speed vehicle ahead under appropriate conditions, a positive reward is given; the difference between the vehicle’s actual speed and the ideal speed of its lane is compared, the smaller the difference, the greater the reward; For comfort; negative rewards are applied for frequent lane changes or sharp steering behaviors; sudden acceleration or braking is detected and negative rewards are applied.

7. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 6, characterized in that: In order to reward safety, a constraint function is used to penalize the collision between the ego vehicle and surrounding vehicles. The constraint function uses the two-circle method to calculate the distance between the ego vehicle and surrounding vehicles. This method uses two circles to cover the plane shape of the vehicle body and converts the distance constraint between vehicles into the distance constraint between the centers of the circles. When using the two-circle method to calculate the vehicle-to-vehicle distance, it is necessary to calculate the distance between the two centers of the ego vehicle and the two centers of each surrounding vehicle, and then take the minimum distance as the vehicle-to-vehicle distance. To ensure the safety of the driving strategy, the safety distance is increased by an amount based on the sum of the radii of the two circles, which is called the redundant safety distance. For all surrounding vehicles whose vehicle-to-vehicle distance is less than the safety distance, it is necessary to calculate the collision constraint penalty, and then add these penalty items to obtain the final constraint function.

8. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 5, characterized in that: The decision neural network model predicts the vehicle trajectory and recommended driving operations in the future based on the current environmental information, including: According to the identified lane line information, determine whether there are optional roads on both sides of the vehicle. Based on the optional lanes, perform trajectory sampling and design several alternative trajectories. Use the Critic network to calculate the value function of each optional trajectory, and select the trajectory with the highest value, which is recorded as the optimal lane; if the lane where the optimal trajectory is located is the currently selected lane, then the selected lane is maintained and the lane selection process ends, otherwise the judgment process continues; If the selected lane is not a safe lane, that is, a collision will occur, the optimal trajectory will be selected immediately and the process will end, otherwise it will continue. This judgment condition is used to handle emergency situations. If the selected lane is about to collide, there is no time for subsequent lane keeping and lane change waiting operations, and it must immediately switch to a safe lane; When considering lane change decisions, the safety of the currently selected lane is first evaluated. Lane change is considered only if the vehicle has been in the current lane for a certain period of time and the optimal lane evaluated remains stable over a continuous period of time. Once the above constraints are met, the vehicle will select the optimal lane according to the evaluation results. The selected optimal trajectory is used as the reference trajectory, and the differences and similarities between it and the current vehicle state and environmental state are input into the Actor network to obtain the control instructions, which are then sent to the controller for execution.

9. A reinforcement learning multi-lane driving decision-making device for dynamic traffic environment, characterized in that: include: A decision neural network model training module is used to establish a decision neural network model using state space, action space and trajectory sampling information in a multi-lane driving decision scenario; and train the decision neural network model using a comprehensive reward function; The information perception module is used to continuously and deeply perceive the surrounding environment through the sensor array equipped by the vehicle, and capture the perceived environmental information, including lane lines, the position, speed and expected behavior information of other traffic participants; A decision module, used to input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operations within a future period of time based on the current environmental information; The control module is used to convert the recommended driving operations output by the decision neural network model into specific control instructions and send the instructions to the vehicle's wire-controlled chassis and actuators.

10. The reinforcement learning multi-lane driving decision-making device for dynamic traffic environment according to claim 9, characterized in that: The state space includes the self-vehicle state, reference path information and surrounding vehicle state; The self-vehicle state includes vehicle speed and heading angular velocity; the reference path information includes a series of path points on a continuous route, and each path point contains information including longitudinal position, lateral position and heading angle; the surrounding vehicle state includes a given number of surrounding vehicles, and the state of each vehicle includes longitudinal position, lateral position, heading angle, body length and body width.

11. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instruction is executed by the processor, the processor implements the steps of the reinforcement learning multi-lane driving decision-making method for dynamic traffic environment as described in any one of claims 1 to 8.

12. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory stores a computer program, and when the program is executed by the processor, the processor executes the steps of the reinforcement learning multi-lane driving decision-making method for a dynamic traffic environment as described in any one of claims 1 to 8.

Images