Dynamic traffic environment-oriented multi-lane driving decision-making method based on reinforcement learning

Abstract

A dynamic traffic environment-oriented multi-lane driving decision-making method based on reinforcement learning, comprising: using state space, action space, and trajectory sampling information in a multi-lane driving decision-making scenario to establish a decision-making neural network model; using a comprehensive reward function to perform reinforcement learning training on the decision-making neural network model; continuously and deeply sensing the surrounding environment by means of a sensor array with which a vehicle is equipped, and capturing sensed environment information; inputting the sensed environment information into the decision-making neural network model, and, on the basis of current environment information, the decision-making neural network model predicting a vehicle trajectory and a recommended driving operation within a period of time; and converting the recommended driving operation outputted by the decision-making neural network model into a specific control instruction, and sending the instruction to a drive-by-wire chassis and an actuator of the vehicle.

Description

Reinforcement learning multi-lane driving decision method for dynamic traffic environment Cross-reference to related applications

[0001] The embodiments of the present application are based on and claim priority from Chinese Patent Application No. 202411384652.X filed on September 30, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of autonomous driving, more specifically, to a reinforcement learning multi-lane driving decision method for dynamic traffic environment. BACKGROUND

[0003] The core functions of autonomous vehicles include environment perception, autonomous decision-making, path planning, and motion control. Environment perception involves using on-board sensors and communication systems to perceive the surrounding traffic conditions and obstacles. Autonomous decision-making includes identifying the intentions and predicting the trajectories of other traffic participants to formulate driving behaviors and plan reference trajectories. Path planning is responsible for selecting the best travel route, taking into account road conditions, traffic situations, and destinations, to ensure efficient and safe navigation. Motion control then translates the decision results into low-level control instructions, mastering key elements such as throttle, brake, and steering wheel. Since autonomous decision-making and motion control can be modeled as optimization problems of dynamic processes, and they are interdependent, they can be integrated into a unified functional module. This module plays a role similar to the brain in autonomous vehicles, and its intelligence level affects the autonomous driving performance of the vehicle.

[0004] The existing auxiliary driving decision method relies on city area road maps, which may lead to geographical limitations. In areas where the mapping is not complete or the road data updates are not timely, the decision-making may be limited. The existing technology constructs a global path planning scheme, which can only serve as an auxiliary for driving and cannot provide support for local autonomous driving. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a reinforcement learning multi-lane driving decision method for dynamic traffic environment, so that the vehicle can realize multi-lane autonomous driving without relying on external data and high-precision maps.

[0006] According to a first aspect of the present application, a reinforcement learning multi-lane driving decision method for dynamic traffic environment is provided, comprising:

[0007] A decision neural network model is established using state space, action space, and trajectory sampling information in the scene of multi-lane driving decision; and reinforcement learning training is performed on the decision neural network model using a comprehensive reward function.

[0008] capturing the perceived environment information, the environment information including lane lines, positions of traffic participants, speeds and their expected behavior information;

[0009] inputting the perceived environment information into the decision neural network model, the decision neural network model predicting a vehicle trajectory and a recommended driving operation in a future period of time according to the current environment information;

[0010] converting the recommended driving operation output by the decision neural network model into specific control instructions, and sending the instructions to a drive-by-wire chassis and an actuator of the vehicle.

[0011] According to a second aspect of the present application, there is provided a reinforcement learning multi-lane driving decision device for a dynamic traffic environment, comprising:

[0012] a decision neural network model training module configured to establish a decision neural network model using state space, action space and trajectory sampling information in a multi-lane driving decision scenario, and train the decision neural network model using a comprehensive reward function;

[0013] an information perception module configured to continuously and deeply perceive a surrounding environment through a sensor array equipped on a vehicle, and capture perceived environment information, the environment information including lane lines, positions of other traffic participants, speeds and their expected behavior information;

[0014] a decision module configured to input the perceived environment information into the decision neural network model, the decision neural network model predicting a vehicle trajectory and a recommended driving operation in a future period of time according to the current environment information;

[0015] a control module configured to convert the recommended driving operation output by the decision neural network model into specific control instructions, and send the instructions to a drive-by-wire chassis and an actuator of the vehicle.

[0016] According to a third aspect of the present application, there is provided a computer readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the steps of the reinforcement learning multi-lane driving decision method for a dynamic traffic environment.

[0017] According to a fourth aspect of the present application, there is provided an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory has stored thereon a computer program, which, when executed by the processor, causes the processor to execute the steps of the reinforcement learning multi-lane driving decision method for a dynamic traffic environment.

[0018] The multi-lane driving decision method for dynamic traffic environment based on reinforcement learning has the following beneficial effects:

[0019] 1、The present application utilizes reinforcement learning, and the vehicle can self-learn and adjust according to the changing road environment, and has good adaptability. By adding rule constraints, the adaptability is ensured, and the driving decision of the vehicle also has certain certainty and standardization.

[0020] 2、The present application mainly relies on sensor data of the vehicle, reinforcement learning technology and some basic rule constraints, and does not involve complex map information or other external resources, reduces the dependence on external information, improves the independence and stability of the system, and saves part of the map cost. In the absence of a map, the vehicle can better adapt to various environments, such as in some areas where the map has not been covered or updated in time. BRIEF DESCRIPTION OF DRAWINGS

[0021] The present application will be further described below in conjunction with the drawings and embodiments, wherein:

[0022] Fig. 1 is a flow chart of the multi-lane driving decision method for dynamic traffic environment based on reinforcement learning of the present application;

[0023] Fig. 2 is a schematic diagram of a sampled trajectory;

[0024] Fig. 3 is a schematic diagram of a multi-lane decision process of a decision neural network model. DETAILED DESCRIPTION

[0025] In order to have a clearer understanding of the technical features, objects and effects of the present application, the specific embodiments of the present application will be described in detail with reference to the drawings.

[0026] Embodiment one

[0027] As shown in Figs. 1-3, the multi-lane driving decision method for dynamic traffic environment based on reinforcement learning of the present application includes the following steps:

[0028] S1, a decision neural network model is established by using state space, action space and trajectory sampling information in the scene of multi-lane driving decision; and reinforcement learning training is performed on the decision neural network model by using a comprehensive reward function.

[0029] In the scene of multi-lane driving decision, the state space provides necessary input parameters for vehicle decision. The state space is composed of three parts: ego state, reference path information and surrounding vehicle state, represented as The attributes included in the ego state are v x (unit: m⋅ s −1 ) and yaw rate ω (unit: rad ⋅ s −1), i.e.

[0030] .

[0031] The reference path information consists of a series of path points on a continuous route, i.e.

[0032]

[0033] where N = 30; each path point contains information of longitudinal position x (unit: m), lateral position y (unit: m) and heading angle θ (unit: rad), i.e.

[0034] These quantities are all relative quantities in the ego vehicle coordinate system.

[0035] The platoon state contains a given number of platoon vehicles, i.e.

[0036]

[0037] The state of each vehicle includes longitudinal position x (unit: m), lateral position y (unit: m), heading angle θ (unit: rad), vehicle body length L (unit: m) and vehicle body width W (unit: m), i.e.

[0038]

[0039] In the scenario of multi-lane driving decision, the action space includes two control quantities: vehicle longitudinal acceleration a x (unit: m·s 2 ) and steering wheel angle δ (unit: rad). In order to improve the smoothness of the driving strategy output, the increment of the control quantity is taken as the strategy output, and the actual control quantity is obtained by adding the current control quantity of the ego vehicle. Let Δa x and Δδ represent the increments of longitudinal acceleration and steering wheel angle respectively, then the action can be written as:

[0040]

[0041] As shown in FIG. 2, the method for obtaining trajectory sampling information includes: for a given state s t , sampling N possible trajectories. Let s be the i-th trajectory, where i = 1, 2, …, N.

[0042]

[0043] where T is the end time of the trajectory, and a tare actions taken at time t. These trajectories can be based on some known heuristics, based on previous experiences, or other appropriate ways.

[0044] The sampled trajectories can be described by a quartic function: 。

[0045] In the training model, the A2C (Advantage Actor-Critic) reinforcement learning method is adopted, and a double network structure, namely the Critic network and the Actor network, is introduced.

[0046] The structure of the Critic network includes three fully connected layers with ReLU as the activation function, and the part processing the timing information adopts LSTM, with the input being the state s and the corresponding trajectory , and the output being the value of the trajectory . The score of the trajectory should be based on the expected future reward, which can include the distance from other vehicles, the smoothness of driving, the compliance with traffic rules, etc. The total expected reward of the trajectory is:

[0047]

[0048] where r t ′ is the reward obtained at time t', and γ is the discount factor of the reward.

[0049] The structure of the Actor network includes two fully connected layers with tanh as the activation function, with the input being the current state s and the optimal trajectory determined by the Critic network ,

[0050] 。

[0051] The Actor network directly generates the corresponding control instructions to track according to the input state s and the optimal trajectory , as follows:

[0052]

[0053] where a t is a vector containing the values of acceleration and steering angle, defined as:

[0054] 。

[0055] When the vehicle drives in the simulation scene, the reward r t is used to update the Actor and Critic networks. The goal of the Actor is to maximize the expected reward evaluated by the Critic, so its loss function L A is:

[0056]

[0057] For Critic, the loss function L C is the mean squared error between the true reward and its prediction:

[0058] 。

[0059] The reward function design method includes: the reward function in multi-lane driving decision involves multiple key parameters, aiming to ensure that the algorithm makes optimal decisions in real-world scenarios. Considering the complexity of multi-lane driving scenarios, a comprehensive reward function is designed, combining safety, efficiency, and comfort factors.

[0060] Safety factors include: collision prevention, if the predicted trajectory is within a predefined safety threshold from other vehicles, a strong negative reward is applied. Lane center driving, the system calculates the distance between the vehicle and the center of the lane it is in, the larger the distance, the smaller the reward. Traffic rule compliance, strict negative rewards are given for behaviors that violate traffic rules, such as changing lanes in prohibited areas.

[0061] Efficiency factors include: overtaking reward, when a vehicle successfully overtakes a slower vehicle in front under appropriate conditions, a positive reward is given. Speed maintenance, the system compares the actual speed of the vehicle with the ideal speed of the lane it is in, the smaller the difference, the larger the reward.

[0062] Comfort factors include: driving smoothness, frequent lane changes or sharp steering behaviors are penalized with negative rewards to ensure smooth driving. Acceleration and brake continuity, the system can detect sudden acceleration or braking and apply negative rewards to encourage smooth driving behavior.

[0063] To integrate the above elements, the reward function adopts a weighted sum form:

[0064]

[0065] where w1, w2, w3 are predefined weights that can be adjusted according to the needs of the actual application scenario to achieve the desired decision balance.

[0066] Here, each part of the reward is designed in more detail. The efficiency reward can be set as an item about tracking error reward r track , which mainly contains items based on tracking error, including lateral position error, heading angle error and speed error of the current selected path, to improve the tracking performance of the strategy, that is:

[0067]

[0068] wherein each coefficient w p ,w Φ , w v is a positive real number, in the experiment w p = 20, w Φ = 100, w v = 0.2.

[0069] The reward related to comfort, which can be set to constrain the control amount here. In order to prevent the output control amount from being too large due to overfitting of the driving strategy, it can be regarded as a regularization term of the driving strategy:

[0070]

[0071] wherein each coefficient w a , w δ is a positive real number, in the experiment w a = 0.1, w δ = 20.

[0072] Here, the reward in the safety aspect adopts a constraint function to punish the collision between the ego vehicle and the surrounding vehicles, which guarantees the safety of the driving strategy.

[0073] Specifically, the distance between the ego vehicle and the surrounding vehicles is calculated by using the double circle method, which covers the planar shape of the vehicle body with two circles, converts the distance constraint between vehicles into the distance constraint between the centers of the circles, and thus simplifies the distance calculation between complex shapes. When the double circle method is used to calculate the vehicle-vehicle distance, the distance between the two centers of the ego vehicle and the two centers of each surrounding vehicle needs to be calculated, i.e. four distances need to be calculated for each surrounding vehicle, and then the minimum distance is taken as the vehicle-vehicle distance, i.e.

[0074]

[0075] wherein is the center coordinate of the ego vehicle, is the center coordinate of the i-th surrounding vehicle. If is greater than the sum of the radii of the two vehicles, the two vehicles do not collide; if is less than or equal to the sum of the radii of the two vehicles, the two vehicles collide. The minimum distance between the centers of the circles is called the safety distance d safe To guarantee the safety of the driving strategy, the safety distance is increased by a certain amount, called the redundant safety distance, based on the sum of the radii of the two vehicles, i.e. , in the experiment d0= 2 m. For all the surrounding vehicles whose vehicle-vehicle distance is less than the safety distance, the collision constraint penalty needs to be calculated, and then the penalty terms are added to obtain the final constraint function:

[0076]

[0077] wherein, is a set of vehicles that have collided, is a coefficient w c is a positive real number, and in the present experiment w c = 5.

[0078] S2, continuously and deeply perceive the surrounding environment through the sensor array equipped by the vehicle, capture the perceived environmental information, and the environmental information includes lane lines, positions of other traffic participants, speeds, and expected behavior information. All data are preprocessed and unified into the coordinate system of the vehicle to ensure the consistency and accuracy of the data.

[0079] To ensure effective and accurate environmental perception, it is recommended that the vehicle be equipped with at least:

[0080] a front camera, designed to capture road conditions and environmental image data in front of the vehicle, thereby supporting the identification of lane lines, traffic signs, and other important target objects.

[0081] a surround millimeter wave radar device, designed to accurately detect and collect relevant information of obstacles around the vehicle, such as their relative distance and speed.

[0082] According to the design of the present application, the basic perception data should at least include:

[0083] lane line data, including the type of lane line and the mathematical equation used to describe the position, curvature, and length of the lane line relative to the vehicle.

[0084] obstacle data, covering the relative distance and speed of the obstacle in the lateral and longitudinal directions.

[0085] self-vehicle state data, such as the speed, yaw angle, and other key driving parameters of the vehicle.

[0086] It should be emphasized that the above description is the most basic perception data requirement. With the increase and richness of data, such as the ability to identify traffic lights, pedestrians, and non-motor vehicles, the model effect and prediction accuracy of the system will be further improved, thereby making the automatic driving function more powerful and safe.

[0087] S3, input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operation in a future period of time according to the current environmental information.

[0088] As shown in FIG. 3, due to the uncertainty of reinforcement learning, combining rule constraints can provide more explicit driving decisions in specific situations, and the decision method includes the following steps:

[0089] S31, calculate optional lanes. According to the recognized lane line information, it is judged whether there is an optional road on both sides of the vehicle.

[0090] S32, based on the optional lane, trajectory sampling is carried out, and a plurality of optional trajectories are designed.

[0091] S33, the value function of each optional trajectory is calculated by using the Critic network, and the trajectory with the highest value is selected as the optimal lane. If the lane of the optimal trajectory is the currently selected lane, the selected lane is maintained, and the lane selection process ends, otherwise the judgment process continues.

[0092] S34, if the selected lane is not a safe lane, i.e. a collision will occur, the optimal trajectory is immediately selected, and the process ends, otherwise it continues. The role of this judgment condition is to deal with emergency situations. If the selected lane is about to collide, there is no time to perform subsequent lane keeping and lane changing waiting operations, and it must be immediately switched to a safe lane.

[0093] S35, when considering the lane change decision, the safety of the currently selected lane is first evaluated. Based on industry standards and driving experience, we set a constraint module: if the vehicle has been kept in the current lane for a certain period of time, for example, more than 3 seconds, and the optimal lane has been stable for a continuous period of time, for example, 1 second, the lane change is considered. Such constraints aim to prevent frequent or unreasonable lane changes due to temporary misjudgments. After meeting the above constraints, the vehicle will select the optimal lane according to the evaluation results.

[0094] S36, the selected optimal trajectory is taken as a reference trajectory, and the Actor network is input with the current vehicle state and environmental state, to obtain control instructions and deliver them to the controller for execution.

[0095] S4, the recommended driving operation output by the decision neural network model is converted into specific control instructions, and the instructions are sent to the line control chassis and actuators of the vehicle.

[0096] Further, when designing a multi-lane driving decision framework, a simulation environment is needed to simulate real road scenes, so as to verify and optimize the reinforcement learning model.

[0097] The method of the application first focuses on model training based on perception data, ensuring that the model can fully understand and process real-world data. The choice of simulation environment is not limited to a certain simulator, but it must meet the basic requirements of road construction and traffic flow design to simulate real traffic environments.

[0098] The road design not only simulates the real road scene, but also considers the generalization of the scene. To this end, a multi-lane road with a specific range of curvature is randomly generated. This design ensures that the model can learn in various road scenes during training, thereby improving its generalization performance.

[0099] The present application combines the advantages of real data and simulation data when processing traffic flow data. Real traffic flow data extracted from the NGSIM dataset provides real-world traffic behavior for the model, thereby improving the realism of the model. At the same time, some pre-set route car agents are randomly generated in the simulation scene. These agents are driven by the intelligent driving model IDM and can interact with the ego car during training, further enhancing the learning ability of the model.

[0100] Embodiment two

[0101] The present application also provides a reinforcement learning multi-lane driving decision device for dynamic traffic environment, comprising:

[0102] A decision neural network model training module is used to establish a decision neural network model using state space, action space and trajectory sampling information in the scene of multi-lane driving decision; and the decision neural network model is trained using a comprehensive reward function;

[0103] An information perception module is used to continuously and deeply perceive the surrounding environment through a sensor array equipped on the vehicle, and capture the perceived environmental information, including lane lines, positions of other traffic participants, speeds and expected behavior information of the other traffic participants;

[0104] A decision module is used to input the perceived environmental information into the decision neural network model, and the decision neural network model predicts the vehicle trajectory and recommended driving operation in the future period of time according to the current environmental information;

[0105] A control module is used to convert the recommended driving operation output by the decision neural network model into specific control instructions, and send the instructions to the drive-by-wire chassis and actuators of the vehicle.

[0106] Embodiment three

[0107] The present application also provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory stores a computer program, and when the program is executed by the processor, the processor executes the steps of the reinforcement learning multi-lane driving decision method for dynamic traffic environment.

[0108] Embodiment four

[0109] The application further provides a computer readable storage medium, which stores executable instructions, and the instructions enable a processor to implement the dynamic traffic environment oriented reinforcement learning multi-lane driving decision method.

[0110] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer usable program code.

[0111] The present application is described with reference to the flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0112] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction means, which implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide steps for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0114] The embodiments of the present application are described above in combination with the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not limiting, and those skilled in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these all belong to the protection of the present application.

Claims

1. A reinforcement learning multi-lane driving decision method for dynamic traffic environment, comprising: establishing a decision neural network model using state space, action space and trajectory sampling information in the scene of multi-lane driving decision; training the decision neural network model using a comprehensive reward function; capturing the perceived environmental information through continuous and in-depth perception of the surrounding environment by the sensor array equipped on the vehicle, the environmental information including lane lines, positions of traffic participants, speeds and their expected behavior information; inputting the perceived environmental information into the decision neural network model, which predicts the vehicle trajectory and recommended driving operation in the future period of time according to the current environmental information; converting the recommended driving operation output by the decision neural network model into specific control instructions and sending the instructions to the drive-by-wire chassis and actuators of the vehicle.

2. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, wherein, The state space includes the ego vehicle state, reference path information and surrounding vehicle state; The ego vehicle state includes vehicle speed and yaw rate; the reference path information includes a series of path points on the continuous route, each path point containing information such as longitudinal position, lateral position and yaw angle; the surrounding vehicle state includes a given number of surrounding vehicles, and the state of each vehicle includes longitudinal position, lateral position, yaw angle, vehicle length and vehicle width.

3. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, wherein, The action space includes vehicle longitudinal acceleration and steering wheel angle, and the increment of control quantity is taken as the policy output, which is added to the current control quantity of the ego vehicle to obtain the actual control quantity.

4. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, wherein, The trajectory sampling information samples N possible trajectories, assuming is the i-th trajectory, where i = 1, 2, …, N; , where T is the end time of the trajectory, and a t is the action taken at time t.

5. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, wherein, An A2C reinforcement learning method is adopted, and Critic network and Actor network are introduced to train the decision neural network model; The structure of Critic network includes three fully connected layers, and the activation function is ReLU; the part processing time series information adopts LSTM, and the input is the current state and the corresponding trajectory, and the output is the value of the trajectory, and the score of the trajectory is based on the expected future reward; The structure of Actor network includes two fully connected layers, and the activation function is tanh; the input is the current state and the optimal trajectory determined by Critic network, and Actor network directly generates the corresponding control instruction to track the optimal trajectory according to the input state and the optimal trajectory.

6. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 1, wherein, The comprehensive reward function considers safety, efficiency and comfort; For safety; if the predicted trajectory is less than the predefined safety threshold from other vehicles, a strong negative reward will be applied; the distance between the vehicle and the center of the lane is calculated, and the larger the distance, the smaller the reward; strict negative reward is given for behaviors that violate traffic rules, such as changing lanes in prohibited areas; For efficiency; when the vehicle successfully overtakes the front low-speed vehicle under appropriate conditions, a positive reward will be given; the difference between the actual speed of the vehicle and the ideal speed of the lane is compared, and the smaller the difference, the greater the reward; For comfort; negative reward is applied to frequent lane changes or sharp turning behaviors; negative reward is applied to sudden acceleration or sudden braking.

7. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 6, wherein, The reward for safety is to punish the collision between the ego vehicle and the surrounding vehicles by using a constraint function; the constraint function calculates the distance between the ego vehicle and the surrounding vehicles by using a double circle method, which covers the planar shape of the vehicle body with two circles, and converts the distance constraint between vehicles into the distance constraint between the centers of the circles; when calculating the vehicle-vehicle distance by using the double circle method, the distance between the two centers of the ego vehicle and the two centers of each surrounding vehicle needs to be calculated, and then the minimum distance is taken as the vehicle-vehicle distance; in order to ensure the safety of the driving strategy, the safety distance is increased by a certain amount, called the redundant safety distance, based on the sum of the radii of the two circles; for all surrounding vehicles with a vehicle-vehicle distance less than the safety distance, the collision constraint penalty needs to be calculated, and then the penalty terms are added to obtain the final constraint function.

8. The reinforcement learning multi-lane driving decision method for dynamic traffic environment according to claim 5, wherein, The method for predicting the vehicle trajectory and the recommended driving operation in a future period of time according to the current environmental information comprises the following steps: determining whether there is a selectable road on both sides of the ego vehicle according to the recognized lane line information, sampling trajectories based on the selectable lane, and designing a plurality of candidate trajectories; calculating the value function of each selectable trajectory by using a Critic network, and selecting the trajectory with the highest value as the optimal lane; if the optimal lane is the currently selected lane, the selected lane is maintained, and the lane selection process ends; otherwise, the process continues; if the selected lane is not a safe lane and a collision will occur, the optimal trajectory is immediately selected, and the process ends; otherwise, the process continues. The purpose of this judgment condition is to handle emergency situations. If the selected lane is about to collide, there is no time to perform subsequent lane maintenance and lane switching waiting operations, and the vehicle must be immediately switched to a safe lane; when considering the lane switching decision, the safety of the currently selected lane is first evaluated; if the vehicle has been maintained in the current lane for a certain period of time, and the optimal lane remains stable in a continuous period of time, the vehicle will switch to the optimal lane according to the evaluation result; the selected optimal trajectory is taken as a reference trajectory, which is input into an Actor network together with the current vehicle state and the environmental state, so as to obtain a control instruction and send the control instruction to a controller for execution.

9. A reinforcement learning multi-lane driving decision device for a dynamic traffic environment, comprising: a decision neural network model training module configured to establish a decision neural network model by using state space, action space and trajectory sampling information in a multi-lane driving decision scene, and train the decision neural network model by using a comprehensive reward function; an information perception module configured to continuously and deeply perceive the surrounding environment by using a sensor array of a vehicle, and capture the perceived environmental information, which includes lane lines, positions of other traffic participants, speeds of the other traffic participants, and expected behavior information of the other traffic participants; a decision module configured to input the perceived environmental information into the decision neural network model, and predict a vehicle trajectory and a recommended driving operation in a future period of time according to the current environmental information; a control module configured to convert the recommended driving operation output by the decision neural network model into a specific control instruction, and send the control instruction to a drive-by-wire chassis and an actuator of the vehicle.

10. The dynamic traffic environment oriented reinforcement learning multilane driving decision apparatus according to claim 9, wherein, The state space includes ego vehicle state, reference path information and surrounding vehicle state; The ego vehicle state includes vehicle speed and yaw rate; the reference path information includes a series of path points on the continuous route, each path point contains longitudinal position, lateral position and yaw angle; the surrounding vehicle state includes a given number of surrounding vehicles, each vehicle state includes longitudinal position, lateral position, yaw angle, vehicle length and vehicle width. 11.A computer readable storage medium having stored thereon executable instructions that, when executed by a processor, enable the processor to implement the steps of the method for multi-lane driving decision based on reinforcement learning in dynamic traffic environment according to any one of claims 1-8.

12. An electronic device comprising: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory has stored therein a computer program, which, when executed by the processor, causes the processor to execute the steps of the method for multi-lane driving decision based on reinforcement learning in dynamic traffic environment according to any one of claims 1-8.

Images