Humanoid feature-based automatic driving reinforcement learning decision and planning method and system

Abstract

This invention discloses an autonomous driving reinforcement learning decision-making and planning method and system based on human-like characteristics. The method analyzes the driving styles of traffic participants to determine and quantify the driving style suitable for the autonomous vehicle under the current conditions; trains the autonomous vehicle's behavioral strategies, namely lane keeping / following, left lane changing, and right lane changing, using an A2C algorithm; and generates a human-like safe trajectory by adhering to the driving style and behavioral decisions in a path-speed decoupling manner. This invention captures the attributes of other traffic participants to guide the autonomous vehicle in designing a socially compatible, safe, and efficient trajectory similar to that of humans, achieving a balance between safety and efficiency in complex multi-scenario environments. Hierarchical behavior and motion planning establishes the driving task as a high-level behavioral decision-making process emphasizing efficiency, and a low-level motion planning method prioritizing safety.

Description

Technical Field

[0001] This invention relates to the fields of reinforcement learning and motion planning technology, specifically to a method and system for decision-making and planning of autonomous driving based on human-like features. Background Technology

[0002] Generating safe and efficient human-like trajectories has always been one of the main research topics in autonomous driving decision-making and planning, especially in highly interactive scenarios without clear rules, such as continuous overtaking, unprotected left turns, and intersections without traffic lights. Autonomous vehicles need to consider the uncertainty and interactivity of traffic scenarios, ensuring their own safety while maximizing traffic efficiency. At the same time, the public also expects autonomous vehicles to be able to weigh the demands of different driving tasks like humans do.

[0003] Currently, common autonomous driving planning methods mainly fall into two categories: heuristic prior rule-based methods and learning-based methods. Prior rule-based methods offer clear interpretability and stable safety, but require manually designing a large number of parameters. Faced with ever-changing and uncertain scenarios, they struggle to construct a general heuristic rule base and tend to be overly conservative. Learning-based methods, such as deep learning or reinforcement learning, extract complex or even novel knowledge from data or the environment, reducing the need for manual rule design and improving model generalization. However, these methods suffer from a "black box" problem, exhibiting very low interpretability. Based on the analysis of these limitations, an increasing number of works attempt to combine the two approaches, using reinforcement learning to learn safer operations from interactions with the environment within the constraints of rule-based models. However, while methods combining rules and learning often perform well in certain challenging single-feature scenarios, a static driving style struggles to balance the comprehensive needs of multiple driving tasks and fails to exhibit human-like driving characteristics when dealing with other road users. Summary of the Invention

[0004] To address the problems existing in current technologies, this invention provides a novel adaptive socially compatible hierarchical behavior and motion planning method. In highly interactive scenarios without explicit rules, it learns flexible, human-like behavioral decisions to generate feasible and reliable motion trajectories, offering a solution to structured multi-scenario decision-making and planning problems. This invention combines a reinforcement learning-based behavior planner and a sampling-based trajectory planner, considering road feasibility to generate flexible and effective action strategies. Simultaneously, the trajectory planner ensures the safety of decision-making, maximizing complementary advantages. Building upon this, a higher-level adaptive socially compatible module is constructed to help the behavior planning layer adaptively adjust the reward function, while enabling the motion planning layer to generate motion trajectories with distinctive styles, achieving real-time adjustment of safety and efficiency weights.

[0005] Based on the characteristics of autonomous driving planning tasks, several improvements are proposed. This invention is completed within a hierarchical behavior planning and motion planning framework (HBMP). An adaptive socially compatible module is designed to guide behavior and motion planning, simulating human driving methods. The behavior planning part uses reinforcement learning to select the optimal action strategy, while the motion planning part employs a state sampling method to ensure the safety of the framework.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: an autonomous driving reinforcement learning decision-making and planning method based on human-like features, the process of which is as follows:

[0007] By analyzing the driving styles of traffic participants, the appropriate driving styles for autonomous vehicles under the current circumstances can be determined and quantified.

[0008] The autonomous vehicle's behavior strategy is trained using the A2C algorithm, namely, lane keeping / following, left lane changing, and right lane changing.

[0009] Following the aforementioned driving style and behavioral decisions, a human-like safe trajectory is generated in a path-speed decoupling manner.

[0010] Furthermore, the analysis of the driving style of traffic participants from the motion information of surrounding traffic participants includes:

[0011] A VAE+RNN network is constructed to encode the historical trajectories of surrounding traffic participants as potential driving features. The collected historical trajectories are rotated and aligned. Based on the VAE+RNN network, the potential driving style features in the historical trajectories are learned to obtain the driving styles of other traffic participants.

[0012] Based on the distances of surrounding vehicles to the autonomous vehicle, the predicted trajectory is used to determine whether it has valid intersections, which is considered as the driving style of the surrounding vehicles. i weight W i Calculate the driving style that the autonomous vehicle should select. AV If the weighted average potential driving style of surrounding vehicles is conservative, then the autonomous vehicle will be selected as "aggressive," as calculated below:

[0013]

[0014] s = {0, 1, 2} corresponds to "aggressive", "mild", and "conservative" driving styles, respectively. i ≠1, n is the number of surrounding traffic participants, and W∈[0,n] is the weight of driving style.

[0015] Furthermore, when analyzing the driving styles of traffic participants, a nested hash table is used to collect real-time motion information of surrounding vehicles. This real-time motion information includes longitudinal distance, longitudinal acceleration, distance and speed difference with the vehicle in front, and the ratio to the desired speed in the Frenet coordinate system.

[0016] Furthermore, when training the behavior strategy of autonomous vehicles using the A2C algorithm, the features contained in the state space mainly include socially compatible style strategy, autonomous vehicle features, features of surrounding traffic participants, and road information features. The reward function adaptively adjusts the behavior decision through the socially compatible style strategy, and outputs the decision based on the observed state and style features as the reward structure. The decision includes lane keeping, left lane changing, and right lane changing.

[0017] Furthermore, the reward function includes: (1) Collision penalty: The definition of collision includes collision with other traffic participants and collision with non-travelable lanes. When a collision occurs, the agent is given the maximum cost and the training ends; (2) Lane change penalty: There are two situations that lead to lane change. Lane change following global guidance is a passive lane change without penalty. Lane change taken to obtain less penalty is an active lane change that requires a penalty based on style, so as to avoid the vehicle changing lanes frequently for no reason; (3) Low speed penalty: It is always hoped that the speed of the autonomous vehicle will reach the desired speed under ideal conditions. The difference between the current speed and the desired speed is used to form a low speed penalty to push the vehicle to increase its driving speed as much as possible.

[0018] Furthermore, a state sampling method is used to generate a candidate path set in the Frenet coordinate system;

[0019] Simultaneously, the optimal path is selected from the candidate path set, taking into account lateral offset, collision constraints, maximum curvature, current vehicle speed, and driving style.

[0020] Speed ​​curves are generated based on the optimal path, taking into account acceleration and abrupt changes, lane speed limits, vehicle kinematics, information on obstacles and their trajectory predictions, and socially compatible driving styles.

[0021] Furthermore, when generating the speed curve based on the optimal path, it is assumed that the vehicle acceleration is constant within the boundaries of speed and acceleration limits. The desired acceleration is determined by combining obstacle information and driving style to generate an appropriate speed distribution.

[0022] Specifically, this includes: assessing the predicted area occupied by vehicles and obstacles under different accelerations to determine the critical acceleration value to ensure a safe distance; assessing reasonable predictions based on the behavior of traffic participants and the control of safe distance through different driving methods to obtain a feasible intersection of acceleration intervals;

[0023] Based on the driving style and behavior strategy described, and considering the current vehicle state, motion constraints, and comfort, the most suitable value is selected from the feasible acceleration range as the desired acceleration, and the speed curve is calculated.

[0024] Based on the above-mentioned concept, the present invention can provide an autonomous driving reinforcement learning decision-making and planning system based on human-like features, including an adaptive social compatibility module, a behavior planning module, and a motion planning module; the adaptive social compatibility module is used to determine and quantify the driving style suitable for autonomous vehicles under the current situation by analyzing the driving styles of traffic participants.

[0025] The behavior planning module trains the autonomous vehicle's behavior strategy through the A2C algorithm, namely, straight lane keeping / following, left lane changing, and right lane changing;

[0026] The motion planning module follows the driving style and behavior strategy to generate a human-like safe trajectory in a path-speed decoupling manner.

[0027] Simultaneously, a computer device is provided, including a processor and a memory. The memory is used to store a computer executable program. The processor reads part or all of the computer executable program from the memory and executes it. When the processor executes part or all of the computer executable program, it can realize the human-like feature-based autonomous driving reinforcement learning decision and planning method.

[0028] Alternatively, a computer-readable storage medium may be provided, in which a computer program is stored, which, when executed by a processor, enables the aforementioned human-feature-based autonomous driving reinforcement learning decision-making and planning method.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects:

[0030] This invention proposes a path planning method based on an adaptive socially compatible model (ASCM), ensuring socially compatible interaction with other traffic participants and facilitating human-like decision-making and planning for autonomous vehicles. A high-level behavioral decision-making and low-level motion planning method is customized for multiple scenarios, operating under the guidance of ASCM to effectively balance safety and efficiency considerations, designing human-like, safe, and efficient trajectories. The behavioral planning method employs reinforcement learning, considering socially compatible style strategies, traffic rules, surrounding traffic participants, and autonomous vehicle information, displaying tactical performance decisions. Complex lane-changing decisions are trained using reinforcement learning on the SUMO simulation platform without any fine-tuning, and co-simulated and visualized on another simulation platform, ROS, verifying the robustness of the method and model described in this invention. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of an adaptive social compatibility model.

[0032] Figure 2 A network architecture for an adaptive, socially compatible hierarchical behavior and motion planning framework.

[0033] Figure 3 For path planning with stylistic characteristics.

[0034] Figure 4 This is a diagram illustrating the vehicle's interactive state.

[0035] Figure 5 This is an ablation test diagram of ASCM in an unprotected left-turn interaction scenario.

[0036] Figure 6 This is an ablation test diagram of ASCM in an interaction scenario at a signalless intersection.

[0037] Figure 7 This is an ablation test diagram of ASCM in a continuous overtaking interaction scenario. Detailed Implementation

[0038] The exemplary embodiments of this application are described in detail below with reference to the accompanying drawings and specific implementations, including various details of the embodiments of this application to aid understanding. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art fall within the scope defined by the appended claims.

[0039] This invention proposes a novel Adaptive Socially Compatible Hierarchical Behavior and Motion Planning (ASC-HBMP) framework, specifically designed for highly interactive scenarios. The method involves identifying and quantifying driving styles as reward features, then using these features to train a reinforcement learning (RL) model to generate human-like action strategies. A state sampling method is employed to ensure the generation of high-quality trajectories that meet temporal and spatial feasibility requirements. Finally, extensive experiments were conducted in various traffic scenarios, and cross-platform SUMO-ROS joint simulations were performed to verify the cross-platform effectiveness and robustness of the proposed method. The method specifically includes the following steps:

[0040] Step 1: Construct an adaptive social compatibility model. Figure 1 This is a schematic diagram of the adaptive social compatibility model of the present invention. The adaptive social compatibility model (ASCM) is the highest-level module of the present invention, responsible for determining and quantifying the driving style suitable for autonomous vehicles under the current circumstances. This is achieved by analyzing the driving styles of other traffic participants. Specific details are as follows:

[0041] 1) Recognition of Driving Styles of Surrounding Traffic Participants. This invention encodes the historical trajectories of surrounding traffic participants as potential driving features for autonomous vehicles to determine their own style strategies. To reduce computational load, a nested hash table is used to collect real-time motion information of surrounding vehicles, including longitudinal distance, longitudinal acceleration, distance and speed difference with the vehicle in front in the Frenet coordinate system, and the ratio to the desired speed. The observed state of the vehicle is defined as... in, 'a' represents the longitudinal distance in the Frenet coordinate system, and 'a' represents the longitudinal acceleration. Let d be the longitudinal distance to the vehicle in front in the Frenet coordinate system. v This is the speed difference with the vehicle in front, where v is the ratio of the autonomous vehicle's speed to the desired speed. The driving style at the current moment is predicted using the historical trajectory from the previous round. Furthermore, the data processing module rotates and aligns the collected trajectories, allowing the VAE+RNN network to focus on learning the latent driving style features within the trajectory. Alignment is achieved by rotating the vector and subtracting a vector equal in length to the first point of the trajectory from the vector along the vertical axis. Figure 2 As shown in (b), the VAE+RNN network consists of gated recurrent units (GRUs) forming the encoder and decoder. In the encoding stage, the original state s∈S is fed into the GRU through the embedding layer for compression, and then a two-dimensional latent feature vector distribution Z is generated through the fully connected layer. In the decoding stage, the GRU combines z∈Z with the s′ reconstructed in the previous time step and feeds it back into the GRU in the next time step. The encoder performance is optimized by the loss function L to infer the driving style features of the surrounding drivers without the need for real style labels.

[0042] Loss=αD KL (N(μ,σ)||N(0,I))+βd(s,s′) (1)

[0043] Where α and β are adjustable weight parameters, N(μ, σ) represents a normal function with mean μ and variance σ, N(0, I) is the latent feature vector distribution, and D KL (N(μ, σ)||N(0, I)) represents the divergence D KL The distance between the latent eigenvector distribution and the standard normal distribution is measured. d is a function that calculates the error distance between the original state s and the reconstructed state s′.

[0044] 2) Autonomous Vehicle Socially Compatible Style Decision. After identifying the driving styles of other traffic participants, the autonomous vehicle's trajectory is predicted to have valid intersections based on the distances of surrounding vehicles to the autonomous vehicle, which serves as the weight W for the driving styles of other vehicles. i Calculate the driving style that the autonomous vehicle should select. AV If the weighted average potential driving style of surrounding vehicles is conservative, the autonomous vehicle will choose "aggressive", and vice versa.

[0045]

[0046] Where s = {0, 1, 2} corresponds to "aggressive", "mild", and "conservative" driving styles, respectively. ego For the autonomous vehicle driving style to be calculated, s i To identify the driving styles of surrounding vehicles, and s i ≠1, s max =2. n is the number of surrounding traffic participants, w i ∈[0, n] represents the weight of the driving style, ε is the minimum value to prevent overflow, and d and p are the distances from surrounding vehicles to the autonomous vehicle and the existence of valid intersections of the predicted trajectories, respectively. Finally, the calculated driving style decision results are output to the downstream layer.

[0047] Socially compatible driving style is an important indicator for assessing driving task requirements and a possible means of incorporating human-like driving characteristics. As the highest level of the framework, ASCM plays a central role in the subsequent behavioral planning and motion planning phases.

[0048] Step 2: Constructing Reinforcement Learning-Based Behavior Planning. Behavior planning is the second layer of the hierarchical framework. The autonomous vehicle's behavior policies are trained using the A2C algorithm, namely, lane keeping / following, left lane changing, and right lane changing. Specific details are as follows:

[0049] 1) State Space. In this invention, the state space mainly includes socially compatible style strategy, autonomous vehicle features, surrounding traffic participant features and road information features. As an example, this invention selects up to 12 vehicles that are closest to its own vehicle and within 100 meters as surrounding traffic participants, and refers to the global path to form road information features, forming a 56-dimensional observation space, as follows: (1) Socially compatible style strategy: ASCM provides direct input to adjust the weight of the reward function to adapt to the current scenario; (2) Autonomous vehicle features: including the vehicle's position, orientation angle and speed, all measured in the vehicle coordinate system; (3) Surrounding traffic participant features: including position, orientation angle and the ratio of the speed of each vehicle to the speed of the autonomous vehicle, the measurement is still performed in the vehicle coordinate system; (4) Road information features: including the feasibility of each road under the guidance of the global path.

[0050] 2) Reward function. In order to train a behavioral strategy with driving style, a reward function that can adaptively adjust the weight preference according to the driving style is provided, which includes: (1) Collision penalty: The definition of collision includes collision with other traffic participants and collision with non-driving lanes. When a collision occurs, the agent is given the maximum cost and the training ends. The collision reward function is shown in Equation (3); (2) Lane change penalty: There are actually two situations that lead to lane change. Lane change following global guidance is a passive lane change without penalty, while lane change to obtain less penalty is an active lane change that requires penalty to be formulated according to style, so as to avoid the vehicle changing lanes frequently for no reason. The lane change reward function is shown in Equation (4); (3) Low speed penalty: It is always hoped that the speed of the autonomous vehicle can reach the expected speed under ideal conditions. Therefore, the difference between the current speed and the expected speed is used to form a low speed penalty to push the vehicle to increase the driving speed as much as possible. The speed reward function is shown in Equation (5).

[0051]

[0052]

[0053] r vel =-|v ref -v ego |(-0.5s+1.5) (5)

[0054] 3) Action Strategies. The trained behavioral strategy network uses observed state and style features as reward structures to execute lane keeping and straight-ahead, left-lane change, or right-lane change action decisions.

[0055] In formulas (3)-(5), 's' represents the driving style decision output by the upper layer. Different driving styles lead to different weight biases in the agent's learning. Under the aggressive style, the agent tends to change lanes to improve traffic efficiency; under the conservative style, the agent tends to keep the lane to ensure safety and comfort during driving; the moderate style is in the middle.

[0056] Step 3: Construct motion planning based on state sampling. Motion planning is the final layer of the layered framework, following the driving style and behavioral decisions of the upper layers, and generating a human-like safe trajectory in a path-velocity decoupling manner. Specific details are as follows:

[0057] 1) Path Planning. In this invention, a state sampling method is used to generate a set of candidate paths in the Frenet coordinate system. The selection of the optimal path with style characteristics considers various factors, including, for example, lateral offset, collision constraints, maximum curvature, current vehicle speed, and driving style. A schematic diagram of the path planning is provided below. Figure 3At the same speed, vehicles employing a conservative strategy tend to travel along longer paths with lower curvature, prioritizing passenger comfort. Similarly, higher speeds require longer paths to ensure passenger comfort, safety, and compliance with vehicle kinematic constraints. In collision-free conditions, smaller lateral offsets and mean curvature can be achieved at lower costs. Furthermore, the first and second derivatives of spline curves are continuously differentiable, ensuring the path remains intact and smooth despite changes in style and motion.

[0058] 2) Speed ​​Planning. The generation of the speed curve considers five key factors: (1) acceleration and abrupt change limits, (2) lane speed limits, (3) vehicle kinematics considerations, such as curvature and rate of change of curvature, (4) information on obstacles and their trajectory predictions, and (5) socially compatible driving styles. During a round of speed planning, assuming that the vehicle acceleration is constant within the boundaries of speed and acceleration limits, the desired acceleration is determined by combining obstacle information and driving style to generate an appropriate speed distribution, including two main steps: solving for feasible acceleration intervals and selecting the optimal acceleration. First, the predicted area occupied by the vehicle and obstacles under different accelerations is evaluated to determine the critical acceleration value that ensures a certain safe distance, and the reasonable prediction based on the behavior of traffic participants and the control of the safe distance through different driving styles is evaluated. Two interactive states corresponding to two feasible acceleration intervals may occur: passive waiting and active overtaking. Figure 4 This concept is illustrated by the requirement that, in both states, one side must maintain a minimum safe distance from the other side when approaching the intersection, denoted as d. safe The calculation method is as follows:

[0059] d safe =max{d brake d follow} (6)

[0060] Both distances in formula (6) must satisfy certain conditions, and the specific calculation methods are shown in formulas (7) and (8). brake This indicates the minimum distance required for both vehicles to brake simultaneously without colliding in the event of a sudden emergency. follow This indicates the expected distance during the interaction period, allowing for a certain reaction time for waiting vehicles.

[0061]

[0062] d follow =d gap +t c ×v pas +(v pas -v act ) 2 -sign(v pas -vact (8)

[0063] Among them, v pas and v act These are vehicles that are passively waiting and those that are actively overtaking, t c It is a reaction time-related parameter, d gap For the minimum spacing between workshops, a max For maximum deceleration, all three are related to driving style.

[0064] After calculating and obtaining the intersection of feasible acceleration intervals, the speed planner listens to the driving style and behavior strategy from upstream, considers the current vehicle state, motion constraints and comfort, selects the most suitable value from the feasible acceleration interval as the expected acceleration, and calculates the speed curve.

[0065] Based on the verification of the method or system described in this invention, a simulation environment can be deployed, as detailed below:

[0066] To verify the cross-platform robustness and multi-scenario feasibility of the ASC-HBMP framework, joint simulations of SUMO and ROS were performed in a busy urban scenario, such as... Figure 2 As shown in (a), the simulator-generated map is collected from real-world data. In SUMO, a reinforcement learning model is initialized and quickly trained without a motion planner, providing a traffic flow environment for ASCM and motion planning. Once the behavior planning achieves a socially compatible driving style and satisfactory performance through training, SUMO is connected to the motion planner in ROS, enabling the generation of safe and comfortable human-like trajectories through action decisions. Finally, the method of this invention was verified using the Gazebo simulation platform and visualized in Rviz. More information about the SUMO simulation parameters is shown in the table below:

[0067]

[0068] The deployment of a human-feature-based reinforcement learning decision-making and planning framework for autonomous driving is detailed below:

[0069] exist Figure 5 , Figure 6 , Figure 7 In this paper, we analyze three strong interaction scenarios in which an autonomous vehicle uses different driving style decision modules to interact with two other vehicles with different styles, and show the speed information of the vehicles in the scenario and the corresponding trajectory at time t0. The specific details in the figure are as follows: (1) Interaction of unprotected left turn: the blue line represents the autonomous vehicle, and the red and yellow lines represent other vehicles with aggressive and conservative styles, respectively. Figure 5(a) The background color of the speed map symbolizes the current style of the autonomous vehicle: red is the aggressive style stage, where the aggressiveness of the strategy and speed are greater than other styles; yellow is the mild style stage, where the strategy and speed are in the middle; green is the conservative style stage, similar to no style, where the vehicle will try to give way to ensure its own safety during the interaction; (2) Interaction in the intersection scene: the green line represents other vehicles with a conservative style. Autonomous vehicles with varied styles adopt a mild strategy when interacting with mild style vehicles, and adopt an aggressive strategy by default when there is no interaction. Therefore, at the same time, the vehicle travels a greater distance than the vehicle that only adopts a mild strategy, and its driving efficiency is higher. Aggressive autonomous vehicles compete for right-of-way by accelerating; (3) Interaction during continuous overtaking: autonomous vehicles with no style will not change lanes regardless of the style of the vehicle in front. Compared with autonomous vehicles with a mild style, vehicles with an aggressive style will overtake more decisively to increase their own speed, while vehicles with a conservative style will hesitate more and maintain a greater safe distance from the vehicle in front.

[0070] The above examples demonstrate that autonomous vehicles with no particular driving style employ the most conservative strategy. One major reason for this is the failure to consider potential surrounding vehicle prediction information within the driving style framework, leading to the assumption of maximum malice towards other vehicles in order to ensure safety. Autonomous vehicles with aggressive, mild, or conservative driving styles adhere to a single strategy; however, what were originally strengths may become weaknesses in different situations. Figure 6 In (d), conservative autonomous vehicles are compared to Figure 6 (b) The more aggressive autonomous vehicles in this design offer higher safety, preventing them from falling into dangerous situations, but... Figure 7 In (d), prioritizing safety resulted in an overly conservative strategy, compromising traffic efficiency. Autonomous vehicles with adaptive driving styles generally outperformed other strategies, avoiding collisions in all three cases and maintaining driving efficiency second only to the aggressive style.

[0071] Furthermore, the method described in this invention was compared with three advanced rule-based baseline methods in three highly interactive scenarios, and the performance of the planning was quantitatively evaluated from three dimensions: safety, efficiency, and comfort, as shown in the table below:

[0072]

[0073] The method described in this invention achieves significantly higher average speeds than other baseline methods in all situations, while strictly adhering to road regulations, with a maximum average collision rate of only 1.7%, consistent with the most conservative CL2013+IDM method. This demonstrates the superior efficiency and safety of ASC-HBMP. In terms of comfort, our method outperforms MMBPL in scenarios such as unsignaled intersections and continuous overtaking, providing a better driving experience.

[0074] Based on the above-mentioned concept, the present invention can provide an autonomous driving reinforcement learning decision-making and planning system based on human-like features, including an adaptive social compatibility module, a behavior planning module, and a motion planning module; the adaptive social compatibility module is used to determine and quantify the driving style suitable for autonomous vehicles under the current situation by analyzing the driving styles of traffic participants.

[0075] The behavior planning module trains the autonomous vehicle's behavior strategy through the A2C algorithm, namely, straight lane keeping / following, left lane changing, and right lane changing;

[0076] The motion planning module follows the driving style and behavior strategy to generate a human-like safe trajectory in a path-speed decoupling manner.

[0077] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the human-like feature-based autonomous driving reinforcement learning decision-making and planning method described in the present invention.

[0078] The computer equipment may be a laptop, desktop computer, workstation, or vehicle-mounted computer.

[0079] The processor described in this invention may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0080] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, workstation, or vehicle-mounted computer, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0081] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the human-like feature-based autonomous driving reinforcement learning decision and planning method described in the present invention when executing the computer executable program.

[0082] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. A reinforcement learning-based decision-making and planning method for autonomous driving based on human-like features, characterized in that, The process is as follows: The appropriate driving style for autonomous vehicles under current conditions is determined and quantified by analyzing the driving styles of traffic participants. This analysis involves examining the driving styles of surrounding traffic participants, specifically including: A VAE+RNN network is constructed to encode the historical trajectories of surrounding traffic participants as potential driving features. The collected historical trajectories are rotated and aligned. Based on the VAE+RNN network, the potential driving style features in the historical trajectories are learned to obtain the driving styles of other traffic participants. Based on the distances of surrounding vehicles to the autonomous vehicle, the predicted trajectory's potential intersections are used to determine the driving styles of the surrounding vehicles. weight Calculate the driving style that an autonomous vehicle should select. If the weighted average potential driving style of surrounding vehicles is conservative, then the autonomous vehicle will select "aggressive," as calculated below: These correspond to "aggressive," "mild," and "conservative" driving styles, respectively. , where n is the number of traffic participants in the surrounding area. Weighting of driving style; The autonomous vehicle's behavior strategy is trained by the A2C algorithm, namely, straight lane keeping / following, left lane changing and right lane changing. When training the autonomous vehicle's behavior strategy by the A2C algorithm, the state space contains features including socially compatible style strategy, autonomous vehicle features, surrounding traffic participant features and road information features. The reward function adaptively adjusts the behavior decision through the socially compatible style strategy and outputs the decision based on the observed state and style features as the reward structure. The decision includes lane keeping, left lane changing and right lane changing. The reward function includes: (1) Collision penalty: The definition of collision includes collision with other traffic participants and collision with non-feasible lanes. When a collision occurs, the agent is given the maximum cost and the training ends. (2) Lane changing penalty: There are two situations that lead to lane changing. Following the global guidance lane changing is a passive lane changing without penalty. Lane changing to obtain less penalty is an active lane changing that requires penalty based on style, so as to avoid the vehicle changing lanes frequently without meaning. (3) Low speed penalty: It is always hoped that the speed of the autonomous vehicle will reach the expected speed under ideal conditions. The difference between the current speed and the expected speed is used to form a low speed penalty to push the vehicle to increase the driving speed as much as possible. Following the aforementioned driving style and behavioral decisions, a human-like safe trajectory is generated in a path-speed decoupling manner.

2. The autonomous driving reinforcement learning decision-making and planning method based on human-like features according to claim 1, characterized in that, When analyzing the driving styles of traffic participants, a nested hash table is used to collect real-time motion information of surrounding vehicles. This real-time motion information includes longitudinal distance, longitudinal acceleration, distance and speed difference with the vehicle in front, and the ratio to the desired speed in the Frenet coordinate system.

3. The autonomous driving reinforcement learning decision-making and planning method based on human-like features according to claim 1, characterized in that, A candidate path set is generated in the Frenet coordinate system using a state sampling method; Simultaneously, the optimal path is selected from the candidate path set, taking into account lateral offset, collision constraints, maximum curvature, current vehicle speed, and driving style. Speed ​​curves are generated based on the optimal path, taking into account acceleration and abrupt changes, lane speed limits, vehicle kinematics, information on obstacles and their trajectory predictions, and socially compatible driving styles.

4. The autonomous driving reinforcement learning decision-making and planning method based on human-like features according to claim 1, characterized in that, When generating speed curves based on the optimal path, it is assumed that the vehicle acceleration is constant within the boundaries of speed and acceleration limits. The desired acceleration is determined by combining obstacle information and driving style to generate an appropriate speed distribution. Specifically, this includes: assessing the predicted area occupied by vehicles and obstacles under different accelerations to determine the critical acceleration value to ensure a safe distance; assessing reasonable predictions based on the behavior of traffic participants and the control of safe distance through different driving methods to obtain a feasible intersection of acceleration intervals; Based on the driving style and behavior strategy described, and considering the current vehicle state, motion constraints, and comfort, the most suitable value is selected from the feasible acceleration range as the desired acceleration, and the speed curve is calculated.

5. An autonomous driving reinforcement learning decision-making and planning system based on human-like features, characterized in that, The method for implementing the human-like feature-based autonomous driving reinforcement learning decision-making and planning method according to any one of claims 1-4 includes an adaptive social compatibility module, a behavior planning module, and a motion planning module. The adaptive social compatibility module is used to determine and quantify the driving style suitable for autonomous vehicles under the current circumstances by analyzing the driving styles of traffic participants. The behavior planning module trains the autonomous vehicle's behavior strategy through the A2C algorithm, namely, straight lane keeping / following, left lane changing, and right lane changing; The motion planning module follows the driving style and behavior strategy to generate a human-like safe trajectory in a path-speed decoupling manner.

6. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading part or all of the computer-executable program from the memory and executing it, and when the processor executes part or all of the computer-executable program, it can implement the human-like feature-based autonomous driving reinforcement learning decision and planning method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the human-like feature-based reinforcement learning decision-making and planning method for autonomous driving as described in any one of claims 1 to 4.

Images