Intelligent vehicle roundabout decision-making rapid landing method and system based on reinforcement learning

Abstract

The application belongs to the technical field of automobiles, and particularly relates to a method for quickly landing an intelligent automobile based on a deep reinforcement learning decision under a roundabout, which comprises the following steps: step one, designing a decision based on deep reinforcement learning under a roundabout scene; step two, designing a decision based on a rule algorithm under a roundabout scene; and step three, designing a decision based on offline reinforcement learning. The system comprises a vehicle-mounted sensor module, a sensor driving module, a map module, a positioning module, a perception module, a navigation path module, a fusion decision module and a control module. The application can adapt to a complex interactive scene of a roundabout, ensure safe decision-making in the training process of a deep reinforcement learning algorithm, and realize quick landing of the reinforcement learning algorithm.

Description

Technical Field

[0001] This invention belongs to the field of automotive technology, specifically a method and system for implementing intelligent vehicle roundabout decision-making based on reinforcement learning. Background Technology

[0002] With the rapid development of computer technology, autonomous driving has become the mainstream direction of development in the automotive field, and decision-making and planning are among the key technologies. Ensuring vehicle safety in highly interactive urban scenarios (such as roundabouts and intersections) has become a crucial issue in the development of autonomous driving technology. Driving in roundabouts with heavy traffic flow, whether entering, driving in, or exiting the roundabout, requires interaction with other vehicles, posing a greater challenge for autonomous vehicles. Meanwhile, the application of reinforcement learning algorithms in autonomous driving decision-making has attracted considerable attention. While it offers significant advantages in avoiding cumbersome rule design and online iteration, the black-box nature of deep reinforcement learning, the uncertainty of algorithm performance improvement leading to safety issues, and the significant gap between simulation and reality pose major obstacles to its practical application in intelligent vehicle driving decision-making. Therefore, improving the safety of reinforcement learning algorithms during the iteration process, reducing training costs, and minimizing the impact of simulation and real-world environments on reinforcement learning are essential for their practical implementation. Summary of the Invention

[0003] To address the aforementioned issues, this invention provides a method and system for implementing intelligent vehicle roundabout decision-making based on reinforcement learning. This method can adapt to complex interactive scenarios in roundabouts, ensure safe decision-making during the training process of deep reinforcement learning algorithms, and enable rapid implementation of reinforcement learning algorithms.

[0004] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0005] Firstly, a method for implementing intelligent vehicle roundabout decision-making based on reinforcement learning includes the following steps:

[0006] Step 1: Design a decision-making system based on deep reinforcement learning for a roundabout scenario;

[0007] Step 2: Design a rule-based algorithm for decision-making in a roundabout scenario;

[0008] Step 3: Design a policy evaluation method based on offline reinforcement learning to shield dangerous outputs during deep reinforcement learning training, thereby achieving security during reinforcement learning training.

[0009] Furthermore, the specific method for step one is as follows:

[0010] 11) Design the motion space;

[0011] The action space is a two-dimensional continuous vector p containing the desired acceleration a and the desired rotation angle δ. e =(a,δ);

[0012] The relationship between the desired acceleration 'a' and the desired rotation angle 'δ' corresponds to the actual control command as shown in the formula:

[0013]

[0014] 12) Design the state space;

[0015] The state space is the driving task vector V. T Vehicle state vector V e and environmental information vector V E The combination of three parts;

[0016] The driving task vector V T Defined as:

[0017] V T =[△P End ,△L End ,d tag ,△θ tag ];

[0018] In the formula, △P End Here are the coordinates of the vehicle's position relative to the endpoint; △L End For relative lane departure; d tag Δθ represents the distance between the vehicle and the target point ahead. tag This refers to the vehicle's orientation deviation relative to the target point ahead.

[0019] The vehicle state vector V e Defined as:

[0020] V e =[v e ,a e ,h e ,ω e ,p e ,△θ e ] T ;

[0021] In the formula, v e =(v x ,v y ) represents the velocity vector; a e =(a x ,a y ) represents the acceleration vector; h e =(cosθ) e sinθ e ) represents the vehicle body attitude vector; ω e p is the yaw rate.e = (a, δ) is the action vector at the current moment; Δθ e =(△θ) 0m ,△θ 0.5m ,△θ 1.2m ,△θ 2.1m ) is the aiming angle vector, which is the angle difference between the road direction at 0m, 0.5m, 1.2m, and 2.1m in front of the vehicle and the current vehicle orientation;

[0022] The environmental information vector V E Including the status of surrounding vehicles and lanes;

[0023] The rectangular grid map centered on the vehicle is divided into eight regions according to the vehicle's positions: front left, left, rear left, front, rear, front right, right, and rear right. These regions are denoted as P1 to P8, and each region is defined with an 8-dimensional vector S. i Let i = {1, ..., 8}, to represent the vehicle state within this region; the region where the vehicle is located is determined by the vehicle's position coordinates on the rectangular grid map, and the vehicle's state vector is calculated as follows:

[0024] S i =[△L,d,△P,△θ,△v,△y];

[0025] In the formula, △L∈{-1, 0, 1} is the left and right position of the other vehicle relative to the vehicle itself; d is the closest distance between the other vehicle and the vehicle itself; △P is the relative coordinate of the other vehicle and the vehicle itself; △θ is the relative facing angle between the other vehicle and the vehicle itself; △v is the speed of the other vehicle relative to the vehicle itself; △y is the distance of the front of the vehicle from the center of the road, which is used to indicate the vehicle's intention to change lanes.

[0026] When entering a roundabout, the rectangular grid map is laterally offset; while driving on the roundabout, the rectangular grid map is redefined into a ring shape.

[0027] The lane status is determined by the road curvature κ and road orientation of the vehicle lane and the left and right lanes, obtained through a semantic map. The information is structured as follows:

[0028]

[0029] Therefore, the total environmental state vector is defined as:

[0030] V E =[S1,S2,S3,L1,S4,S5,L2,S6,S7,S8,L3];

[0031] The driving task vector V T and the vehicle's state vector V e Repeat this process three times and store the data in the state space.

[0032] In summary, the vectorized state space at time t is defined as follows:

[0033] V = [V T *3,V e *3,V E ];

[0034] 13) Design a reward function based on the artificial potential field method;

[0035] The reward function is defined as a field to guide the agent's exploration; at the same time, the reward function based on the artificial potential field method is established through the processing of sensor data.

[0036] The reward function U is determined by the obstacle potential field U. O Road potential field U L Velocity potential field U S and target potential field U T The composition is as follows:

[0037] U APF =ω1U O +ω2U L +ω3U S +ω4U T ;

[0038] In the formula, ω1, ω2, ω3, and ω4 are divided into four types of potential field coefficients;

[0039] Among them, the obstacle potential field U O It consists of other vehicles on the road; the obstacle potential field is defined as a two-dimensional Gaussian distribution function with the vehicle center as the origin, as follows:

[0040]

[0041] In the formula, △P=(△x,△y) are relative coordinates; μ∈R 2 The mean; Σ∈R 2×2 It is the covariance matrix; The model of the relative speed between the other vehicle and the vehicle itself;

[0042] The formula for the mean μ is defined as follows:

[0043]

[0044] In the formula, μ0 = (0,0) is the vehicle center position; v is the vehicle velocity; a is the acceleration; and t is the single-step running time.

[0045] The formula for the covariance matrix Σ is:

[0046]

[0047] In the formula, L is the vehicle length; W is the vehicle width; △L is the longitudinal expansion relative to the vehicle body; △W is the lateral expansion relative to the vehicle body.

[0048] Road potential field U L To create a repulsive force field, the road potential field is modeled with the center of the two-lane road as the origin, as shown below:

[0049]

[0050] In the formula, y is the lateral coordinate of the vehicle, γ1 is the weight of the dashed road line, and γ2 is the weight of the solid road boundary.

[0051] Velocity potential field U S The gravitational field is used to propel the vehicle forward at normal speed, as shown below:

[0052]

[0053] In the formula, v e It is the velocity vector;

[0054] Target potential field U T The combined gravitational potential field of the target point and the endpoint is given by the formula:

[0055]

[0056] In the formula, d tag ≥2 represents the distance from the vehicle to the target point, when d tag When the time is less than 2, the target point is automatically updated forward, △P End Σ represents the distance between the vehicle and the endpoint; Σ = diag(25,25) is the covariance matrix.

[0057] Furthermore, the specific method for step two is as follows:

[0058] 21) Design a static obstacle avoidance module;

[0059] Using a mixture of A * The algorithm is a static obstacle avoidance algorithm;

[0060] Define the expansion size as half the length of the vehicle body, and define the grid value occupied by the obstacle vehicle and its expansion range as infinity in the rectangular grid map; Hybrid A * The algorithm's planning range is the intersection of the grid map and the road semantic map within the angle range of [-π / 2, π / 2] in front of the vehicle. The starting point of the planning is the center of the rear axle of the vehicle, and the planning endpoint is determined by traversing the navigation path points in front of obstacles. When a path point satisfies the hybrid A... *The algorithm's planning requirements are to complete trajectory planning under the conditions of satisfying vehicle dynamics constraints, not exceeding the planning range of the semantic map, and avoiding collisions. This path point is the planning endpoint. If planning cannot be completed within the specified number of traversals, the planning fails, and the road following module continues to be executed.

[0061] 22) Design a road following module;

[0062] The road following module consists of an intelligent driver model and a pure tracking algorithm, which are used to calculate the vehicle’s expected acceleration and expected turning angle in the roundabout, respectively.

[0063] The intelligent driver model needs to incorporate three scenarios into the vehicle's acceleration control: entering a roundabout, passing through a roundabout merging point, and a vehicle about to change lanes in front of the vehicle. The formula for calculating the desired acceleration 'a' is defined as follows:

[0064]

[0065] In the formula, v is the vehicle's speed; a0 is the maximum expected acceleration; v0 is the vehicle's expected speed; ψ is the acceleration exponent; s is the actual relative distance between the vehicle and the vehicle in front; θ represents the angle between the line connecting the centers of the two vehicles and the direction of the vehicle's speed; ξ(n) is shown below:

[0066]

[0067] In the formula, n is the number of vehicles at positions S1, S4, and S6; when ξ(n) = 1, s(ξ(n)(cosθ-1)+1) = s·cosθ = s', which means that the relative distance between the two vehicles is the projection distance of the actual distance between the two vehicles in the longitudinal velocity direction of the vehicle. The minimum acceptable distance between the vehicle and the vehicle in front is shown below:

[0068]

[0069] In the formula, The difference between the projected velocity of the other vehicle in the vehicle's coordinate system and the longitudinal velocity of the vehicle itself; s0 = 2m is the stationary safety distance parameter; T = 2s is the safety time distance; b = 2m / s 2 Deceleration for comfort

[0070] The pure tracking algorithm uses a target point 2m in front of the vehicle as the tracking point for vehicle steering control to calculate the desired steering angle δ, as shown in the formula:

[0071]

[0072] In the formula, L is the wheelbase; a is the angle between the vehicle body posture and the target path point; vector Let h be the vector pointing from the center of the rear axle of the wheel to the target point; eThe unit vector for vehicle body attitude;

[0073] The desired acceleration and desired steering angle are calculated using an intelligent driver model and a pure tracking algorithm. The difference between these values ​​and the vehicle's actual acceleration and front wheel steering angle at the current moment is then calculated. A PID algorithm is used to optimize the control of this difference, thereby achieving the road following requirement.

[0074] Furthermore, in step 21), the mixed A is performed. * The conditions for the algorithm to complete the static obstacle avoidance algorithm are as follows:

[0075] If an obstacle exists within 15 meters of the vehicle's trajectory, and the obstacle's average speed is below 0.1 m / s for a continuous 3 seconds, and this is consistent with the previous execution of Hybrid A... * The algorithm's time interval exceeds 2 seconds, and the previously successfully planned trajectory has already been executed by the vehicle.

[0076] Furthermore, the specific method for step three is as follows:

[0077] 31) Collect offline roundabout trajectory sets; perceive the driving behavior of human experts in real roundabout scenarios and the surrounding environment through sensors, obtain sensor data in real time and model it as the state space and action space of the Markov decision process, calculate the reward value, and save it as an offline dataset for training offline reinforcement learning algorithms.

[0078] 32) Offline reinforcement learning training: Use a trajectory set consisting of states, actions, and rewards collected in real-world scenarios to train offline reinforcement learning until convergence;

[0079] 33) Decision evaluation method based on offline reinforcement learning; the decision evaluation module consists of two parts: a state evaluation module and a policy action evaluation module; the state evaluation module consists of the state-value network V of the offline reinforcement learning algorithm. ORL The input is a state space constructed from the perception module, map module and localization module, and the output is a scalar used to evaluate the quality of the current state.

[0080] The policy action evaluation module consists of the state-action value network Q of an offline reinforcement learning algorithm. ORL The network consists of (s,a), with inputs comprising the state space and action space, and outputs the network's evaluation of the policy's effectiveness in the current state. The overall decision evaluation module is defined as follows:

[0081]

[0082] In the formula, v target The threshold for determining the quality of a state is determined by calculating the expected discount of the reward from the collected offline trajectory sets; a RULE This is the output of the rule-based algorithm; aRL To enhance learning decision output; a best For the final decision output;

[0083] 34) Combining two evaluation networks to evaluate the two output strategies of the decision output module. When the current state environment is poor, i.e., the state evaluation value V... ORL (s) <v target When the environment is favorable, reinforcement learning and rule-based algorithms are used for joint decision-making, and the optimal policy among the two policies is output through the policy evaluation module; when the environment state is good, i.e., V ORL (s)≥v target At that time, reinforcement learning algorithms are used alone to explore the solution space.

[0084] Secondly, this invention provides a rapid deployment system for intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments, used to implement a method for rapid deployment of intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments, including:

[0085] Vehicle-mounted sensor modules are used to collect various types of information;

[0086] The sensor driver module is used to establish communication with the vehicle-mounted sensor module and transmit the collected information to other modules.

[0087] The map module includes a point cloud map, a semantic map, and a grid map; the point cloud map is used for vehicle localization and obstacle recognition; the semantic map is used to realize vehicle lane position perception and local planning; the grid map is used for local planning of rule-based algorithms and state perception of reinforcement learning algorithms.

[0088] The positioning module is used to acquire and output the current vehicle attitude and position;

[0089] The perception module is used to sense and identify the specific location and size of road obstacles, and to track and estimate the speed of identified obstacles.

[0090] The navigation path module is used to provide intelligent vehicles with global navigation paths;

[0091] The fusion decision module is used to process the data obtained from each module, make safe and effective decisions based on the known vehicle status and surrounding environment, and output the results to the control module.

[0092] The control module receives the desired acceleration and desired steering angle commands from the decision module and converts the commands into actual control commands for the vehicle to control its movement.

[0093] The bus module is used to acquire vehicle status and transmit control commands;

[0094] The vehicle-mounted sensor module is connected to the sensor driver module; the sensor driver module is connected to the map module, positioning module, perception module, navigation path module, and fusion decision module; the fusion decision module is connected to the control module; and the control module is connected to the vehicle chassis via a bus module.

[0095] Furthermore, the fusion decision module includes a strategy output module and a strategy evaluation module;

[0096] The strategy evaluation module includes a state evaluation module and a strategy action evaluation module;

[0097] The policy output module includes a reinforcement learning decision module and a rule-based decision module.

[0098] Furthermore, the vehicle-mounted sensor module includes LiDAR, inertial navigation, GPS, and wheel speed sensors.

[0099] The beneficial effects of this invention are as follows:

[0100] 1) This invention designs and improves a Markov decision process that can be quickly embedded into existing intelligent driving decision systems, enabling the rapid implementation and continuous improvement of reinforcement learning algorithms;

[0101] 2) This invention improves the fusion decision-making based on the multi-interaction characteristics of urban roundabouts, making it applicable to conditions such as entering and exiting roundabouts and driving within roundabouts;

[0102] 3) This invention uses the artificial potential field method to construct the reward function in the Markov decision process, which can modularly design the merits and demerits of driving behavior during driving, simplify the design difficulty of multi-objective reward functions, and improve the learning efficiency of reinforcement learning algorithms.

[0103] 4) This invention uses offline reinforcement learning based on human round-island driving trajectory training for decision evaluation, which improves the accuracy of policy evaluation, reduces the difficulty of dataset preparation, avoids dangerous actions in the reinforcement learning exploration process, and ensures its exploration ability and efficiency, thereby further improving the safety and stability of decision-making.

[0104] 5) This invention replaces the decision-making module in the original autonomous driving system with a fusion decision-making module, realizing the rapid implementation and successful application of fusion decision-making based on deep reinforcement learning.

[0105] 6) As an important part of the autonomous driving system that undertakes perception and output actions, the decision-making module adopts the data output by the vehicle's real perception system as the only prior condition for constructing the state space and reward function, which meets the requirements of system operation.

[0106] 7) This invention designs the action space as a real control quantity, reduces intermediate processing, optimizes the inverse mapping relationship from the action space to the state space, and matches the rule output to keep the policy evaluation consistent.

[0107] 8) This invention provides an application process for rapidly embedding reinforcement learning into a real autonomous driving framework and implementing it, while ensuring the safety, efficiency, and comfort of the autonomous driving system in a real roundabout environment. Attached Figure Description

[0108] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0109] Figure 1 This is a framework diagram of an autonomous driving system suitable for a roundabout environment as described in this invention;

[0110] Figure 2 This is a framework diagram of the fusion decision-making module for improving decision-making security as described in this invention;

[0111] Figure 3 Schematic diagrams of grid maps under different roundabout conditions;

[0112] Figure 4 This is a diagram illustrating the operational framework of the rule module described in this invention.

[0113] Figure 5 This is a logic framework diagram of a rule-based static obstacle avoidance algorithm. Detailed Implementation

[0114] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0115] Example 1

[0116] See Figure 2 This embodiment provides a method for the rapid deployment of intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments, including the following steps:

[0117] Step 1: Design a decision-making system based on deep reinforcement learning for a roundabout scenario;

[0118] To meet the specific requirements of urban roundabout scenarios and the need to directly transfer the algorithm to real-vehicle experiments without changing the reinforcement learning input and output, this invention designs a Markov decision process that considers different roundabout conditions, and directly uses parameters that can be obtained from the perception, localization, and mapping modules for the representation design of the state space, action space, and reward function.

[0119] A deep reinforcement learning algorithm is essentially a network whose input is a state vector in the state space, and whose output is an action. The reward function is used to calibrate performance metrics such as whether performing this action in the current state is safe, or whether vehicle control is comfortable. This process is called a Markov decision process; simultaneously, the vectors of actions, states, and rewards are saved for network training.

[0120] Various deep reinforcement learning algorithms are essentially different processes that utilize the collected states, actions, and rewards to train the network and ensure its convergence. Regardless of the algorithm, only one Markov decision process needs to be constructed.

[0121] Specifically as follows:

[0122] 11) Design the motion space;

[0123] This invention only controls the speed and steering angle of the intelligent vehicle, thus mainly including three control parameters: drive, braking, and steering angle. To ensure that the control actions output by reinforcement learning meet the vehicle control requirements, the vehicle parameters are calibrated before the experiment begins. The maximum acceleration 'a' of the vehicle is obtained through the IMU and the steering column angle sensor. max Maximum deceleration a min Maximum left turn angle δ Lmax and maximum right turn angle δ Rmax Therefore, the action space defined in this invention is a two-dimensional continuous vector p containing the desired acceleration a and the desired rotation angle δ. e = (a, δ); Meanwhile, to avoid excessive vehicle acceleration / deceleration and steering angle exceeding limits, the ranges for acceleration and steering angle are defined as a ∈ (0.8a) and (δ). min 0.5a max ) and δ∈(-0.9δ max 0.9δ max ),δ max =min(δ) Lmax ,δ Rmax ).

[0124] The relationship between the desired acceleration 'a' and the desired rotation angle 'δ' corresponds to the actual control command as shown in the formula:

[0125]

[0126] 12) Design the state space;

[0127] The state space should be able to fully represent the vehicle's surrounding environment, driving task, and the vehicle's relevant states. Therefore, in this invention, the state space is defined as the driving task vector V. T Vehicle state vector V e and environmental information vector V E The invention combines three parts; it simplifies and upgrades the state space, enabling state space parameters to be directly obtained from real sensor data processing, thus facilitating modeling in both real and simulation environments.

[0128] The driving task vector V T This is used to assist the vehicle in completing a predetermined driving task. In a roundabout driving task, the vehicle needs to follow the navigation path to enter the roundabout from one intersection and exit from another. Therefore, a driving task vector V is defined. T for:

[0129] V T =[△P End ,△L End ,d tag ,△θ tag ];

[0130] In the formula, △P End Here are the coordinates of the vehicle's position relative to the endpoint; △L End For relative lane departure; d tag Δθ represents the distance between the vehicle and the target point ahead. tag This represents the vehicle's orientation deviation relative to the target point ahead; the target point is the navigation path point, which is continuously updated as the vehicle moves forward, guiding it to complete the predetermined task. These parameters can be directly calculated using the global path and semantic map.

[0131] Vehicle state vector V e To help the autonomous vehicle understand the impact of different driving behaviors on changes in the vehicle's state and the environment's state, a vehicle state vector V is defined. e for:

[0132] V e =[v e ,a e ,h e ,ω e ,p e ,△θ e ] T ;

[0133] In the formula, v e =(v x ,v y ) represents the velocity vector; a e =(a x ,a y) represents the acceleration vector; h e =(cosθ) e sinθ e ) represents the vehicle body attitude vector; ω e This refers to the yaw rate; all of the above parameters can be obtained directly or indirectly through devices such as GPS, IMU, and wheel speed sensors. △θ e =(△θ) 0m ,△θ 0.5m ,△θ 1.2m ,△θ 2.1m ) represents the pre-aiming turning angle vector, which is the angle difference between the road direction at 0m, 0.5m, 1.2m, and 2.1m in front of the vehicle and the current vehicle orientation. It is used to perceive the driving direction on roundabout roads and can also be obtained through map information and IMU sensors; p e = (a, δ) is the action vector at the current moment;

[0134] The environmental information vector V E Including the condition of surrounding vehicles and lanes; as attached Figure 3 As shown, the rectangular grid map centered on the vehicle is first divided into eight regions according to the vehicle's front left, left, rear left, front, rear, front right, right, and rear right positions, denoted as P1 to P8 respectively. Each region is defined with an 8-dimensional vector S. i Let i = {1, ..., 8}, to represent the vehicle state within this region; the region where the vehicle is located is determined by the vehicle's position coordinates on the rectangular grid map, and the vehicle's state vector is calculated as follows:

[0135] S i =[△L,d,△P,△θ,△v,△y];

[0136] In the formula, △L∈{-1, 0, 1} represents the left and right positions of the other vehicle relative to the vehicle itself; d represents the closest distance between the other vehicle and the vehicle itself; △P represents the relative coordinates of the other vehicle and the vehicle itself; △θ represents the relative facing angle between the other vehicle and the vehicle itself; △v represents the speed of the other vehicle relative to the vehicle itself; all of the above information can be obtained through clustering and tracking algorithms of LiDAR; △y represents the distance of the vehicle's front from the center of the road, used to indicate the vehicle's lane-changing intention;

[0137] When entering a roundabout, the rectangular grid map is laterally offset so that it can detect vehicles approaching from within the roundabout earlier. While driving on a roundabout, the rectangular grid map is redefined into a ring shape so that it can better fit the road shape and increase the coverage of vehicles around the vehicle.

[0138] The lane status is determined by the road curvature κ and road orientation of the vehicle lane and the left and right lanes, obtained through a semantic map. The information is structured as follows:

[0139]

[0140] Therefore, the total environmental state vector is defined as:

[0141] V E =[S1,S2,S3,L1,S4,S5,L2,S6,S7,S8,L3];

[0142] To avoid insufficient network understanding of driving tasks and vehicle states due to excessively high environmental information dimensionality, the task vector and vehicle state vector are repeated three times in the state space. Furthermore, all the above parameters can be calculated using existing map and sensor information, eliminating the need for additional sensors and achieving a unified state space representation in both simulation and real-world environments. In summary, the vectorized state space at time t is defined as:

[0143] V = [V T *3,V e *3,V E ];

[0144] 13) Design a reward function based on the artificial potential field method;

[0145] The artificial potential field method treats the target and obstacles as objects with attractive and repulsive forces, respectively, to guide the vehicle's movement. Therefore, the reward function can be defined as a field to guide the agent's exploration, simplifying the design of reward functions for reinforcement learning while providing complete constraint capabilities. Furthermore, the reward function established based on the artificial potential field method can also be easily constructed through the processing of sensor data.

[0146] The reward function U is determined by the obstacle potential field U. O Road potential field U L Velocity potential field U S and target potential field U T The composition is as follows:

[0147] U APF =ω1U O +ω2U L +ω3U S +ω4U T ;

[0148] In the formula, ω1, ω2, ω3, and ω4 are divided into four types of potential field coefficients;

[0149] Among them, the obstacle potential field U O It consists of other vehicles on the road; the obstacle potential field is defined as a two-dimensional Gaussian distribution function with the vehicle center as the origin, as shown in the following formula. In the calculation of the reward function, only the influence of the vehicle in front of the vehicle on the vehicle is considered.

[0150]

[0151] In the formula, △P=(△x,△y) are relative coordinates; μ∈R 2 The mean; Σ∈R 2×2 It is the covariance matrix; Let μ be the magnitude of the relative velocity between the vehicle and another vehicle, and let μ be the repulsive force generated if and only if the two vehicles are close to each other. Vehicles at different speeds on the road have different effects on their surroundings; therefore, the center point of the vehicle's potential field is offset, i.e., the mean μ is defined by the formula:

[0152]

[0153] In the formula, μ0 = (0,0) is the vehicle center position; v is the vehicle velocity; a is the acceleration; and t is the single-step running time.

[0154] The formula for the covariance matrix Σ is:

[0155]

[0156] In the formula, L is the vehicle length; W is the vehicle width; △L and △W are the longitudinal and lateral expansions relative to the vehicle body, which have different values ​​in different roundabout scenarios;

[0157] Road potential field U L The repulsive field is used to prevent vehicles from going out of the road and to suppress serpentine driving behavior. The road potential field is modeled with the center of the two-lane road as the origin, as shown below:

[0158]

[0159] In the formula, y is the lateral coordinate of the vehicle, γ1 is the weight of the dashed road line, and γ2 is the weight of the solid road boundary.

[0160] Velocity potential field U S The gravitational field is used to propel the vehicle forward at normal speed, as shown below:

[0161]

[0162] In the formula, v e It is the velocity vector;

[0163] Target potential field U T The combined gravitational potential field of the target point and the endpoint is given by the formula:

[0164]

[0165] In the formula, d tag ≥2 represents the distance from the vehicle to the target point, when d tag When the time is less than 2, the target point is automatically updated forward, △PEnd Σ represents the distance between the vehicle and the endpoint; Σ = diag(25,25) is the covariance matrix.

[0166] Once the Markov decision process design for the island-circling scenario is completed, the next part of the design can proceed. This invention does not restrict specific reinforcement learning algorithms, but recommends using state-of-the-art reinforcement learning algorithms such as SAC as the foundation.

[0167] Step Two, Refer to Figure 4 Design a rule-based algorithm for decision-making in a roundabout scenario;

[0168] To unify the strategy output of the decision-making framework, the output of the rule-based algorithm is also defined as control commands for vehicle acceleration, deceleration, and steering. Furthermore, to enhance the decision-making safety of the rule-based algorithm in real-vehicle experiments, the algorithm constructed in this invention primarily focuses on road-following, only performing local trajectory planning to bypass static obstacles. Therefore, the rule-based algorithm is divided into a static obstacle avoidance module and a road-following module, as detailed below:

[0169] 21) Design a static obstacle avoidance module; static obstacle avoidance is used for local planning, and road following is used to control the vehicle's forward movement and steering. In step two, only static obstacles are avoided; when encountering dynamic obstacles, only following is performed.

[0170] Using a mixture of A * The algorithm is a static obstacle avoidance algorithm; see attached. Figure 5 To avoid frequent calls to hybrid A * The algorithm causes the path to continuously change, therefore an activation hybrid A was designed. * The basic logic of the static obstacle avoidance algorithm is as follows: First, obstacle avoidance only occurs when there is an obstacle within 15m of the vehicle's trajectory points in front of it, and the average speed of the obstacle is less than 0.1m / s over a continuous 3-second period. Simultaneously, it is compared with the previous execution of the hybrid A algorithm. * The algorithm's time interval exceeds 2 seconds, and the previously successfully planned trajectory has already been executed by the vehicle. Finally, only after these conditions are met will the hybrid A algorithm be executed. * An algorithm is used to perform static obstacle avoidance.

[0171] To avoid collisions, all detected obstacles are expanded to half the length of the vehicle itself, and the grid value occupied by the obstacle vehicle and its expanded area is defined as infinity in the grid map. Hybrid A * The algorithm's planning range is defined as the intersection of the grid map and the road semantic map within the angular range of [-π / 2, π / 2] in front of the vehicle. The planning starting point is the center of the vehicle's rear axle, and the planning endpoint is determined by traversing the navigation path points in front of obstacles. When a path point satisfies the hybrid A... *The algorithm's planning requirements are that trajectory planning can be completed under the conditions of satisfying vehicle dynamics constraints, not exceeding the planning range of the semantic map, and avoiding collisions. This path point is the planning endpoint. If planning cannot be completed within the specified number of traversals, the planning fails, and the road following module continues execution.

[0172] 22) Design a road following module;

[0173] The road following module consists of an intelligent driver model and a pure tracking algorithm, used to calculate the vehicle's desired acceleration and desired turning angle in a roundabout, respectively. The intelligent driver model needs to incorporate three scenarios into its acceleration control: the vehicle entering the roundabout, passing the roundabout merging point, and a vehicle about to change lanes in front of it. Therefore, to enable the intelligent driver model to adapt to the complex conditions of a roundabout, the relative speed and relative distance between the vehicle and the vehicle in front are rewritten in the model formula, and the formula for calculating the desired acceleration 'a' is redefined as follows:

[0174]

[0175] In the formula, v is the vehicle's speed; a0 is the maximum expected acceleration; v0 is the vehicle's expected speed; ψ is the acceleration exponent; s is the actual relative distance between the vehicle and the vehicle in front; θ represents the angle between the line connecting the centers of the two vehicles and the direction of the vehicle's speed; ξ(n) is shown below:

[0176]

[0177] In the formula, n is the number of vehicles at positions S1, S4, and S6; when ξ(n) = 1, s(ξ(n)(cosθ-1)+1) = s·cosθ = s', which means that the relative distance between the two vehicles is the projection distance of the actual distance between the two vehicles in the longitudinal velocity direction of the vehicle. The minimum acceptable distance between the vehicle and the vehicle in front is shown below:

[0178]

[0179] In the formula, The difference between the projected velocity of the other vehicle in the vehicle's coordinate system and the longitudinal velocity of the vehicle itself; s0 = 2m is the stationary safety distance parameter; T = 2s is the safety time distance; b = 2m / s 2 Deceleration for comfort

[0180] The pure tracking algorithm uses a target point 2m in front of the vehicle as the tracking point for vehicle steering control to calculate the desired steering angle δ, as shown in the formula:

[0181]

[0182] In the formula, L is the wheelbase; α is the angle between the vehicle body posture and the target path point; vector Let h be the vector pointing from the center of the rear axle of the wheel to the target point; e The unit vector for vehicle body attitude;

[0183] Static obstacle avoidance is designed to allow a vehicle to follow a planned path around a static obstacle, while road following involves controlling the vehicle to follow the pre-planned path to avoid the obstacle. Without static obstacle avoidance, the vehicle will wait until the obstacle disappears.

[0184] The desired acceleration and desired steering angle are calculated using an intelligent driver model and a pure tracking algorithm. The difference between these values ​​and the vehicle's actual acceleration and front wheel steering angle at the current moment is then calculated. A PID algorithm is used to optimize the control of this difference, thereby achieving the road following requirement.

[0185] Furthermore, in step 21), the mixed A is performed. * The conditions for the algorithm to complete the static obstacle avoidance algorithm are as follows:

[0186] If an obstacle exists within 15 meters of the vehicle's trajectory, and the obstacle's average speed is below 0.1 m / s for a continuous 3 seconds, and this is consistent with the previous execution of Hybrid A... * The algorithm's time interval exceeds 2 seconds, and the previously successfully planned trajectory has already been executed by the vehicle.

[0187] Step 3: Design a decision evaluation module based on offline reinforcement learning;

[0188] 31) The decision evaluation module based on offline reinforcement learning first needs to train the offline reinforcement learning. Several human experts drive vehicles equipped with lidar, IMU, GPS and other sensors to collect data in real urban roundabout and straight road environments. The sensor data is obtained in real time and modeled as the state space and action space of the Markov decision process, and the reward value is calculated.

[0189] 32) A series of trajectories consisting of states, actions, and rewards are saved as the training dataset for offline reinforcement learning. Since the modeling of the state space and reward function achieves the unification of reality and simulation, driving data from several human experts in the simulation environment are collected in the same way and added to the offline reinforcement learning training data to improve the diversity of the dataset and the accuracy of policy evaluation.

[0190] 33) The decision evaluation module consists of two parts: state evaluation and policy action evaluation. The state evaluation module uses the state-value network V, which is an offline reinforcement learning algorithm. ORL The system consists of (s), which takes as input a state space constructed from the perception, mapping, and localization modules, and outputs a scalar to evaluate the quality of the current state. The policy action evaluation module is composed of the state-action value network Q, built using an offline reinforcement learning algorithm. ORLThe network consists of (s,a) modules, whose inputs are the state space and action space, and whose output is the network's evaluation of the policy's quality in the current state. The overall decision evaluation module is defined as follows:

[0191]

[0192] In the formula, v target The threshold for the quality of a state; a RULE For the output of the rule algorithm, a RL To enhance learning decision output, a best This is the final decision output. The above formula combines two evaluation networks to evaluate the two output strategies of the decision output module. That is, only when the state environment is poor, reinforcement learning algorithm and rule algorithm are used for joint decision-making to avoid the reinforcement learning algorithm outputting dangerous exploration behavior, thereby improving the overall decision safety of the framework; when the environment state is good, reinforcement learning algorithm is used alone to explore the solution space to ensure the exploration capability and efficiency of reinforcement learning.

[0193] Once the modeling and design of the above three parts are completed, the deep reinforcement learning can be trained and further improved in the island scenario by relying on the fusion decision module.

[0194] In summary, the specific implementation scheme of the intelligent vehicle rapid deployment method based on deep reinforcement learning decision-making in roundabouts described in this invention is not only applicable to roundabout sections, but also fully adaptable to general straight roads; therefore, the application of the fusion decision-making method and concept described in this invention on different types of roads all fall within the protection scope of this invention.

[0195] Example 2

[0196] See Figure 1 This embodiment provides a rapid deployment system for intelligent vehicles based on deep reinforcement learning decision-making in roundabouts, including an onboard sensor module, a sensor driver module, a map module, a positioning module, a perception module, a navigation path module, a fusion decision-making module, a control module, and a bus module.

[0197] The vehicle-mounted sensor module includes LiDAR, inertial navigation, GPS, and wheel speed sensors. It is used to collect various types of information.

[0198] The sensor driver module is used to establish communication with the vehicle-mounted sensor module and transmit the collected information to other modules.

[0199] The map module includes a point cloud map, a semantic map, and a grid map; the point cloud map is used for vehicle localization and obstacle recognition; the semantic map is used to realize vehicle lane position perception and local planning; the grid map is used for local planning of rule-based algorithms and state perception of reinforcement learning algorithms.

[0200] The positioning module is used to acquire and output the current vehicle attitude and position;

[0201] The perception module is used to sense and identify the specific location and size of road obstacles, and to track and estimate the speed of identified obstacles.

[0202] The navigation path module is used to provide intelligent vehicles with global navigation paths;

[0203] The fusion decision module is used to process the data obtained from each module, make safe and effective decisions based on the known vehicle status and surrounding environment, and output the results to the control module.

[0204] The control module receives the desired acceleration and desired steering angle commands from the decision module and converts the commands into actual control commands for the vehicle to control its movement.

[0205] The bus module is used to acquire vehicle status and transmit control commands;

[0206] The fusion decision module includes a strategy output module and a strategy evaluation module.

[0207] The policy output module includes a reinforcement learning decision module and a rule decision module; because the policy output module uses two algorithms to output the same result, an algorithm is needed to evaluate which output is better, hence the policy evaluation module. The rule decision module includes a static obstacle avoidance module and a road following module;

[0208] The strategy evaluation module includes a state evaluation module and a strategy action evaluation module;

[0209] The vehicle-mounted sensor module is connected to the sensor driver module; the sensor driver module is connected to the map module, positioning module, perception module, navigation path module, and fusion decision module; the fusion decision module is connected to the control module; and the control module is connected to the vehicle chassis via a bus module.

[0210] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for rapid deployment of intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments, characterized in that, Includes the following steps: Step 1: Design a decision-making system based on deep reinforcement learning for a roundabout scenario; Step 2: Design a rule-based algorithm for decision-making in a roundabout scenario; Step 3: Design decision-making based on offline reinforcement learning; The specific method for step two is as follows: 21) Design a static obstacle avoidance module; Using a hybrid The algorithm is a static obstacle avoidance algorithm; Define the expansion size as half the length of the vehicle body, and define the grid value occupied by the obstacle vehicle and its expansion zone as infinity in the rectangular grid map; hybrid The algorithm's planning range is the area in front of the vehicle. The intersection of the grid map and the road semantic map within the angular range constitutes the planning range; the starting point of the planning is the center of the vehicle's rear axle, and the planning endpoint is determined by traversing the navigation path points in front of obstacles. When a path point satisfies the hybrid... The algorithm's planning requirements are to complete trajectory planning under the conditions of satisfying vehicle dynamics constraints, not exceeding the planning range of the semantic map, and avoiding collisions. This path point is the planning endpoint. If planning cannot be completed within the specified number of traversals, the planning fails, and the road following module continues to be executed. Perform a blend The conditions for the algorithm to complete the static obstacle avoidance algorithm are as follows: When there is an obstacle within 15m of the vehicle's trajectory point in front of it, and the average speed of the obstacle is less than 0.1m / s over a continuous 3-second period; simultaneously, it is combined with the previous execution... The algorithm's time interval exceeds 2 seconds, and the previously successfully planned trajectory has already been executed by the vehicle. 22) Design a road following module; The road following module consists of an intelligent driver model and a pure tracking algorithm, which are used to calculate the vehicle’s expected acceleration and expected turning angle in the roundabout, respectively. The intelligent driver model needs to incorporate three scenarios into the vehicle's acceleration control: entering a roundabout, passing through a roundabout merging point, and a vehicle about to change lanes in front of the vehicle; and define the desired acceleration. The calculation formula is: ； In the formula, For vehicle speed; The maximum expected acceleration; The desired speed of the vehicle; The acceleration index; This represents the actual relative distance between the vehicle and the vehicle in front. It represents the angle between the line connecting the centers of the two vehicles and the direction of the vehicle's velocity; As shown below: ； In the formula, for The number of vehicles at the location; when hour, This indicates that the relative distance between the two vehicles is the projected distance of the actual distance between the two vehicles in the longitudinal velocity direction of the vehicle itself; The minimum acceptable distance between the vehicle and the vehicle in front is shown below: ； In the formula, The difference between the projected velocity of the other vehicle in the vehicle's coordinate system and the longitudinal velocity of the vehicle itself; This refers to the stationary safety distance parameter; For safe time intervals; Deceleration for comfort The pure tracking algorithm uses a target point 2m in front of the vehicle as the tracking point for vehicle steering control to determine the desired steering angle. The calculation is as follows: ； In the formula, Wheelbase; The angle between the vehicle body posture and the target path point; vector The vector pointing from the center of the rear axle of the wheel to the target point; This is the vehicle body attitude vector; The desired acceleration and desired steering angle are calculated using an intelligent driver model and a pure tracking algorithm. The difference between these values ​​and the vehicle's actual acceleration and front wheel steering angle at the current moment is then calculated. A PID algorithm is used to optimize the control of this difference, thereby achieving the road following requirement.

2. The method for rapid deployment of intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments according to claim 1, characterized in that, The specific method for step one is as follows: 11) Design the motion space; The motion space includes the desired acceleration. and expected corner Two-dimensional continuous vector ; The desired acceleration and expected corner The corresponding relationship between actual control commands is shown in the formula: ； 12) Design the state space; The state space is a driving task vector. Vehicle state vector and environmental information vector The combination of three parts; The driving task vector Defined as: ； In the formula, These are the coordinates of the vehicle's position relative to the endpoint; This refers to relative lane departure; This represents the distance between the vehicle and the target point ahead. This refers to the vehicle's orientation deviation relative to the target point ahead. The vehicle state vector Defined as: ； In the formula, It is the velocity vector; It is the acceleration vector; This is the vehicle body attitude vector; This refers to the yaw rate; This is the action vector at the current moment; This is the aiming angle vector; The environmental information vector Including the status of surrounding vehicles and lanes; The rectangular grid map centered on the vehicle is divided into eight regions according to the vehicle's positions: front left, left, rear left, front, rear, front right, right, and rear right. These regions are then divided into eight areas using... This means that each region defines an 8-dimensional vector. This is used to represent the vehicle status within the area; the area where the vehicle is located is determined by the vehicle's position coordinates on the rectangular grid map, and the vehicle's state vector is calculated as follows: ； In the formula, The left and right positions of the other car relative to your own car; The closest distance between your car and another car; The coordinates are the relative coordinates of the other vehicle and the vehicle itself. The relative orientation angle between the other vehicle and the vehicle itself; The speed of the other vehicle relative to its own. This represents the distance the front of the vehicle deviates from the center of the road, used to indicate the vehicle's intention to change lanes; When entering a roundabout, the rectangular grid map is laterally offset; while driving on the roundabout, the rectangular grid map is redefined into a ring shape. The lane status is determined by the road curvature of the vehicle lane and the left and right lanes, obtained through a semantic map. and road direction The information is structured as follows: ； Therefore, the total environment state vector is defined as: ； driving task vector and vehicle state vector Repeat this process three times and store the data in the state space. In summary, t The state space vectorized at time step is defined as follows: ； 13) Design a reward function based on the artificial potential field method; The reward function is defined as a field to guide the agent's exploration; at the same time, the reward function based on the artificial potential field method is established through the processing of sensor data. reward function By obstacle potential field Road surface Velocity potential field and target potential field The composition is as follows: ； In the formula, These are four types of potential field coefficients; Among them, the obstacle potential field It consists of other vehicles on the road; the obstacle potential field is defined as a two-dimensional Gaussian distribution function with the vehicle center as the origin, as follows: ； In the formula, Relative coordinates; The mean; It is the covariance matrix; The model of the relative speed between the other vehicle and the vehicle itself; Define the mean The formula is: ； In the formula, The center position of the vehicle; For the desired acceleration; This refers to the single-step execution time; covariance matrix The formula is: ； In the formula, For the train commander; For vehicle width; This refers to the expansion relative to the longitudinal direction of the vehicle body; This refers to the amount of expansion relative to the lateral direction of the vehicle body; Road potential field To create a repulsive force field, the road potential field is modeled with the center of the two-lane road as the origin, as shown below: In the formula, Let the vehicle's lateral coordinates be... The weight of the dashed road; The weight of the solid line road boundary; Velocity potential field The gravitational field is used to propel the vehicle forward at normal speed, as shown below: ； In the formula, It is the velocity vector; Target potential field The combined gravitational potential field of the target point and the endpoint is given by the following formula: ； In the formula, ,when The target point is automatically updated forward. Let be the covariance matrix.

3. The method for rapid deployment of intelligent vehicles based on deep reinforcement learning decision-making in roundabout environments according to claim 1, characterized in that, The specific method for step three is as follows: 31) Collect offline roundabout trajectory sets; perceive the driving behavior of human experts and the surrounding environment in real roundabout scenarios through sensors, obtain sensor data in real time and model it as the state space and action space of the Markov decision process, calculate the reward value, and save it as an offline dataset for training offline reinforcement learning algorithms. 32) Offline reinforcement learning training; Offline reinforcement learning is trained using a trajectory set consisting of states, actions, and rewards collected in real-world scenarios until convergence. 33) Decision evaluation method based on offline reinforcement learning; the decision evaluation module consists of two parts: a state evaluation module and a policy action evaluation module; the state evaluation module consists of the state-value network of the offline reinforcement learning algorithm. It consists of a state space constructed from the perception module, map module, and localization module as input, and a scalar output used to evaluate the quality of the current state. The policy action evaluation module consists of a state-action value network derived from an offline reinforcement learning algorithm. The system consists of a state space and an action space as input, and outputs the network's evaluation of the policy's effectiveness in the current state. The overall decision evaluation module is defined as follows: ； In the formula, The threshold for good or bad status is determined by calculating the expected discount value of the rewards collected from offline trajectories. This is the output of the rule algorithm; To enhance learning decision output; For the final decision output; 34) Combining two evaluation networks to evaluate the two output strategies of the decision output module, when the current state environment is poor, i.e., the state value network... When the environment is favorable, reinforcement learning and rule-based algorithms are used for joint decision-making, and the optimal policy among the two policies is output through the policy evaluation module; when the environment is favorable, i.e. At that time, reinforcement learning algorithms are used alone to explore the solution space.

4. A rapid deployment system for intelligent vehicles based on deep reinforcement learning decision-making in roundabouts, used to implement the rapid deployment method for intelligent vehicles based on deep reinforcement learning decision-making in roundabouts as described in claim 1, characterized in that, include: Vehicle-mounted sensor modules are used to collect various types of information; The sensor driver module is used to establish communication with the vehicle-mounted sensor module and transmit the collected information to other modules. The map module includes a point cloud map, a semantic map, and a grid map; the point cloud map is used for vehicle localization and obstacle recognition; the semantic map is used to realize vehicle lane position perception and local planning; and the grid map is used for local planning of rule-based algorithms and state perception of reinforcement learning algorithms. The positioning module is used to acquire and output the current vehicle attitude and position; The perception module is used to sense and identify the specific location and size of road obstacles, and to track and estimate the speed of identified obstacles. The navigation path module is used to provide intelligent vehicles with global navigation paths; The fusion decision module is used to process the data obtained from each module, make safe and effective decisions based on the known vehicle status and surrounding environment, and output the results to the control module. The control module receives the desired acceleration and desired steering angle commands from the decision module and converts the commands into actual control commands for the vehicle to control its movement. The bus module is used to acquire vehicle status and transmit control commands; The vehicle-mounted sensor module is connected to the sensor driver module; the sensor driver module is connected to the map module, positioning module, perception module, navigation path module, and fusion decision module; the fusion decision module is connected to the control module; and the control module is connected to the vehicle chassis via a bus module.

5. A rapid deployment system for intelligent vehicles based on deep reinforcement learning decision-making in a roundabout, as described in claim 4, is characterized in that... The vehicle-mounted sensor module includes lidar, inertial navigation, GPS, and wheel speed sensors.

6. The rapid deployment system for intelligent vehicles based on deep reinforcement learning decision-making in roundabouts according to claim 4, characterized in that, The fusion decision-making module includes a strategy output module and a strategy evaluation module; The strategy evaluation module includes a state evaluation module and a strategy action evaluation module; The policy output module includes a reinforcement learning decision module and a rule-based decision module.

Images