A longitudinal flexible safe driving regulation method based on constraint strategy optimization

Abstract

The present invention relates to the field of autonomous driving technology, and specifically discloses a longitudinal flexible safe driving protocol method based on constraint strategy optimization, including obtaining key safety events of a natural driving vehicle based on natural driving data; constructing the parameter optimization problem in the longitudinal safety distance formula in a responsibility-sensitive safety model into a constrained Markov decision process model, building a neural network based on a deep reinforcement learning algorithm, evaluating and optimizing the strategy neural network, the reward value neural network and the cost value neural network, and extracting the key safety events of the natural driving vehicle by designing dangerous scene trigger conditions in real natural driving data, thereby solving the problem that the current responsibility-sensitive safety model parameters deviate from the actual scene design and are difficult to use in practice. In safety-critical scenarios that conform to natural driving, the longitudinal driving safety of the vehicle can be maintained, and the triggering protocol caused by overly conservative estimates can be avoided to the greatest extent, thereby achieving a balance between safety and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a longitudinal flexible safety driving regulation method based on constraint strategy optimization. Background Art

[0002] With the increasing popularity of advanced automated driving assistance systems, the safety of autonomous vehicles has become one of the core issues of concern to the research community. People have high safety expectations for autonomous driving technology, but fatal accidents involving autonomous driving have occurred repeatedly, leading to a crisis of trust in autonomous driving technology. Among them, the consequences of longitudinal safety accidents are the most frequent and serious. The existing technology for protecting the longitudinal safety of autonomous vehicles is most famous for adaptive cruise control, but there are also problems such as unsafe and unstable false alarms in high-speed driving scenarios. Among them, the unpredictable and overly urgent braking problem is the most fatal, which makes it difficult to gain the trust of users and potentially violates the driver's usage habits, making it impossible to provide reliable longitudinal safety protection for autonomous vehicles.

[0003] In the face of the safety issues of autonomous driving, Mobileye recently proposed the Responsibility Sensitive Safety (RSS) model, which aims to transform human concepts of safe driving and judgments of accident responsibility into mathematical models, and provide flexible constraints as key parameters for decision-making and control. The RSS model is based on five basic safety common senses and clearly stipulates the safe lateral and longitudinal distances of autonomous vehicles, as well as the appropriate responses in various driving scenarios. The model not only distinguishes various driving environments, such as the topology of the road, traffic signals, and the occlusion of obstacles, but also carefully considers the potential improper behavior of other traffic participants, the exercise of priority, and the specific division of responsibilities. The RSS model is widely used in passenger cars, commercial vehicles, shared travel vehicles, and autonomous driving test vehicles.

[0004] Through these refined parameters and definitions, the RSS model strives to achieve an ideal state, that is, the self-driving car will never actively cause a collision or accident due to its own reasons. The effect of the RSS model depends on the parameters preset in the formula. If the parameters are overestimated, the calculated safety distance will be too large. In extreme cases, the vehicle will always be unable to meet the safety distance, and thus be judged as a dangerous situation, triggering an emergency response. Therefore, calculating the optimal parameters based on the real driving data set is a necessary prerequisite for the application of RSS to ensure that the autonomous driving system operates safely and efficiently.

[0005] Existing RSS parameter optimization methods usually select the median value of the parameter range or use heuristic search, but these methods are difficult to achieve the best mapping from parameter selection to optimization results. Although these methods have achieved certain results in optimizing RSS model parameters, they are still overly conservative or insufficiently safe when applied to autonomous driving, and the parameters are difficult to balance driving safety and efficiency.

[0006] Secondly, the current RSS parameter optimization setting method mainly relies on the simulation environment. Although this method provides a reference basis to a certain extent, the actual driving scenes are complex and changeable. This method is difficult to fully reflect the real driving environment, which limits its practical application. Summary of the invention

[0007] The purpose of the present invention is to provide a longitudinal flexible safe driving regulation method based on constraint strategy optimization to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a longitudinal flexible safe driving regulation method based on constraint strategy optimization, the method comprising:

[0009] Acquire key safety events of natural driving vehicles based on natural driving data;

[0010] Building a responsibility-sensitive safety model based on critical safety events of natural driving vehicles;

[0011] The parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model is constructed as a constrained Markov decision process model. The state space in the constrained Markov decision process model is defined as the statistical mean and standard deviation of the relative distance and relative speed between the front and rear vehicles; the behavior space is defined as the parameters to be optimized in the safety distance formula, including the response time of the rear vehicle, the maximum deceleration of the front vehicle, the maximum acceleration of the rear vehicle, and the minimum deceleration of the rear vehicle.

[0012] Building a neural network based on a deep reinforcement learning algorithm, wherein the neural network includes a strategy neural network, a reward value neural network, and a cost value neural network;

[0013] The strategy neural network, reward value neural network and cost value neural network are evaluated and optimized, the neural network is iteratively trained multiple times to obtain a trained neural network model, and the parameters in the longitudinal safety distance formula are optimized based on the trained neural network model.

[0014] As a further solution of the present invention, the step of obtaining key safety events of natural driving vehicles based on natural driving data specifically includes:

[0015] Acquiring natural driving data, wherein the natural driving data includes kinematic data and video data;

[0016] Determine the triggering conditions of dangerous scenarios;

[0017] Extract initial safety-critical scenario events based on natural driving data and dangerous scenario trigger conditions;

[0018] Screening the initial safety-critical scenario events to generate the final safety-critical scenario events;

[0019] Kinematic features are extracted based on the final safety-critical scenario events.

[0020] As a further solution of the present invention, the step of constructing the parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model into a constrained Markov decision problem model specifically includes:

[0021] Construct a cost function based on the objective of minimizing the collision time integral;

[0022] The reward function is constructed based on the goal of minimizing the sum of conservative safety distances.

[0023] As a further solution of the present invention, the cost function Designed for:

[0024] ;

[0025] ;

[0026] in, Indicates the number of safety critical events The duration of time below a certain threshold, N is the number of events that trigger TTC below the threshold in each round during each policy update; Indicates safety critical period value; for Threshold value; Indicates Period in time In the simulation step value; is the time interval between two simulation steps.

[0027] As a further solution of the present invention, the reward function Designed for:

[0028] ;

[0029] ;

[0030] in, Representative Conservative measures for the period; Indicates In the period Greater than No. a moment; and Represent the minimum reasonable distance and the actual distance respectively.

[0031] As a further solution of the present invention, the step of evaluating and optimizing the policy neural network, the reward value neural network and the cost value neural network specifically includes:

[0032] Determine the expected return and constraints based on the reward function and the cost function;

[0033] Update the policy neural network based on the approximate solution method;

[0034] The reward value neural network and the cost value neural network are updated.

[0035] As a further solution of the present invention, the step of determining the expected return and the constraint conditions based on the reward function and the cost function includes:

[0036] Define the expected discounted return of a policy relative to the cost function;

[0037] Determine the expected discounted return of the policy relative to the reward function;

[0038] Determine a strategy that satisfies the constraints.

[0039] As a further solution of the present invention, strategy Relative to the cost function Expected discounted return Defined as:

[0040] ;

[0041] Representation strategy Relative to the cost function The expected discounted return, is the discount factor.

[0042] As a further solution of the present invention, strategy With respect to the reward function Expected discounted return It is expressed as:

[0043] .

[0044] As a further solution of the present invention, a policy set that satisfies all cost constraints is: ;

[0045] In the policy constraint set can maximize Strategy :

[0046] .

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows: by designing dangerous scene trigger conditions in real natural driving data, extracting safety-critical events of natural driving vehicles and counting event features, the problem that the current responsibility-sensitive safety model parameters deviate from the actual scene design and are difficult to use in practice is solved. The deep reinforcement learning method is used to optimize the parameters, and the nonlinear and complex relationship between the dangerous situation and the parameter setting is mapped using a neural network, which solves the problem of under-optimization of parameters. In safety-critical scenarios that conform to natural driving, this method can not only maintain the longitudinal driving safety of the vehicle, but also avoid the triggering regulations caused by overly conservative estimates to the greatest extent, achieving a balance between safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0049] Figure 1 A flowchart of a longitudinal flexible safe driving regulation method based on constraint strategy optimization provided in an embodiment of the present invention.

[0050] Figure 2 A framework diagram of a longitudinal flexible safe driving regulation method based on constraint strategy optimization provided in an embodiment of the present invention.

[0051] Figure 3 A flowchart of the steps for obtaining key safety events of natural driving vehicles based on natural driving data provided in an embodiment of the present invention.

[0052] Figure 4 A flowchart of the steps of constructing the parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model into a constrained Markov decision process model is provided in an embodiment of the present invention.

[0053] Figure 5 A flowchart of steps for evaluating and optimizing a policy neural network, a reward value neural network, and a cost value neural network provided in an embodiment of the present invention.

[0054] Figure 6A training curve diagram of the reward function provided by an embodiment of the present invention.

[0055] Figure 7 A training curve diagram of the cost function provided by an embodiment of the present invention.

[0056] Figure 8 A scatter plot of dangerous vehicle distance time for autonomous driving using RSS parameters of a traditional algorithm and RSS parameters of a deep reinforcement learning algorithm provided in an embodiment of the present invention.

[0057] Fig. 9 A scatter plot of dangerous vehicle distance for autonomous driving between the RSS parameters selected by the machine provided in an embodiment of the present invention and the RSS parameters of the deep reinforcement learning algorithm.

[0058] Fig.10 A scatter plot of the conservatism of autonomous driving using the RSS parameters of a traditional algorithm and the RSS parameters of a deep reinforcement learning algorithm provided in an embodiment of the present invention.

[0059] Fig.11 A scatter plot of the conservatism of autonomous driving between the RSS parameters selected by the machine in the steps provided in an embodiment of the present invention and the RSS parameters of the deep reinforcement learning algorithm.

[0060] Fig.12 The vehicle kinematic parameters of the deep reinforcement learning algorithm training process provided by the embodiment of the present invention vary with the scene. Figure 1 .

[0061] Fig.13 The vehicle kinematic parameters of the deep reinforcement learning algorithm training process provided by the embodiment of the present invention vary with the scene. Figure 2 . DETAILED DESCRIPTION

[0062] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] See also Figure 1 , Figure 2 , Figure 1 The flowchart of the longitudinal flexible safe driving regulation method based on constraint strategy optimization is shown in the following figure. In the embodiment of the present invention, the longitudinal flexible safe driving regulation method based on constraint strategy optimization includes:

[0064] Acquire key safety events of natural driving vehicles based on natural driving data;

[0065] Building a responsibility-sensitive safety model based on critical safety events of natural driving vehicles;

[0066] The parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model is constructed as a constrained Markov decision process model. The state space in the constrained Markov decision process model is defined as the statistical mean and standard deviation of the relative distance and relative speed between the front and rear vehicles; the behavior space is defined as the parameters to be optimized in the safety distance formula, including the response time of the rear vehicle, the maximum deceleration of the front vehicle, the maximum acceleration of the rear vehicle, and the minimum deceleration of the rear vehicle.

[0067] Building a neural network based on a deep reinforcement learning algorithm, the neural network includes a policy neural network, a reward value neural network and a cost value neural network, the policy neural network is used to map the vehicle state space of the current time step to the behavior space of the next time step; the reward value neural network and the cost value neural network are used to evaluate the expected performance of the current strategy in terms of reward and safety cost;

[0068] The strategy neural network, reward value neural network and cost value neural network are evaluated and optimized, the neural network is iteratively trained multiple times to obtain a trained neural network model, and the parameters in the longitudinal safety distance formula are optimized based on the trained neural network model.

[0069] By optimizing the formulated parameters used to calculate the longitudinal safe distance between the ego vehicle and surrounding vehicles, the safety and conservatism of the safety emergency response of the autonomous driving vehicle under dangerous situations are balanced, including designing dangerous scenario triggering conditions in real natural driving data, extracting key safety events of natural driving vehicles, and statistically analyzing event characteristics; the parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model is constructed as a constrained Markov decision problem, in which the state space is defined as the statistical mean and standard deviation of the relative distance and relative speed between the front and rear vehicles; the behavior space is defined as the parameters to be optimized in the safety distance formula, including the response time of the rear vehicle, the maximum deceleration of the front vehicle, the maximum acceleration of the rear vehicle, and the minimum deceleration of the rear vehicle. The cost function takes into account the duration of the rear vehicle collision time below a certain threshold, namely the collision time integral TIT; the reward function is measured by minimizing the conservatism during all safety-critical events. The sum of negative rewards and the TIT negative reward of the following car are used to comprehensively quantify the conservatism of the model; the constrained strategy optimization algorithm is used to solve the constrained Markov decision problem, and the strategy is maintained under the premise of satisfying the cost constraint to seek the maximization of the reward;

[0070] like Figure 2 As shown, the overall architecture of the present invention includes four steps: environmental data preparation based on NDS; modeling of the longitudinal safety distance parameter optimization method of the RSS model based on CMDP; longitudinal parameter set optimization and strategy solution of the RSS model based on the CPO algorithm; simulation platform verification and parameter optimization of the longitudinal safety driving flexible specification method based on the constraint strategy optimization algorithm.

[0071] like Figure 3 As shown, as a preferred embodiment of the present invention, the step of obtaining key safety events of natural driving vehicles based on natural driving data specifically includes:

[0072] Acquiring natural driving data, wherein the natural driving data includes kinematic data and video data;

[0073] Determine the triggering conditions of dangerous scenarios;

[0074] Extract initial safety-critical scenario events based on natural driving data and dangerous scenario trigger conditions;

[0075] Screening the initial safety-critical scenario events to generate the final safety-critical scenario events;

[0076] Kinematic features are extracted based on the final safety-critical scenario events.

[0077] The present invention adopts natural driving data to evaluate and optimize the performance of the RSS model, and utilizes the on-board data acquisition system to collect a large amount of high-precision data during daily driving, so as to reconstruct safety-critical events in the simulation platform of the RSS model and reflect the driving environment more realistically.

[0078] The Shanghai Natural Driving Study is the first natural driving study in China, jointly conducted by Tongji University, General Motors and Virginia Tech Transportation Research Institute. The 57 drivers participating in the study include 45 men and 12 women, ranging in age from 25 to 59 years old, with driving experience ranging from 1 to 23 years. The research vehicles are equipped with advanced data collection systems, including four camera views of the front, face, hand, and rear, GPS, speedometer, 3D accelerometer and radar. Data collection started in December 2012 and ended in December 2015, recording a total of 19,133 trips with a total mileage of 161,055 kilometers.

[0079] The rich kinematic and video data collected in the natural driving research project provides a basis for the effective extraction of safety-critical scene events. Safety-critical events are usually identified by abnormal vehicle kinematic characteristics, such as high lateral and longitudinal acceleration and low collision time TTC. In this study, the kinematic trigger criteria were initially set based on the 100-car natural driving study and then adjusted to balance the false positive rate and false negative rate. The final trigger criteria are as follows:

[0080] Trigger condition 1: lateral acceleration is greater than or equal to 0.7g;

[0081] Trigger condition 2: Longitudinal deceleration is greater than or equal to -0.5g;

[0082] Trigger condition 3: The driver presses the dashboard button to activate event recording when he believes a critical event has occurred;

[0083] Trigger condition 4: lateral acceleration is greater than or equal to 0.5g and the front collision time TTC is 4 seconds or less;

[0084] Trigger condition 5: Longitudinal deceleration is greater than or equal to -0.45g and the forward collision time TTC is 4 seconds or less.

[0085] By combining the above trigger criteria and manual verification, 591 safety-critical scenarios were finally identified, including 583 impending collision scenarios and 8 collision scenarios.

[0086] Of the 591 safety-critical events identified, 249 were due to conflicts with the lead vehicle, as verified manually in the video. This type of conflict accounted for the highest proportion of all conflict types, at 42%; in comparison, in a 100-car natural driving study, conflicts with the lead vehicle accounted for 48%. Due to the high frequency of conflicts in these cases, this study selected safety-critical scenarios for car tracking.

[0087] The analysis of each event starts from 5 seconds before the trigger moment and 10 seconds after the trigger moment, for a total of 15 seconds of analysis. Of the 249 events, 26 were omitted because the sensor data was missing for more than 10 seconds, making it almost impossible to track the trajectory of the lead vehicle in the subsequent virtual scene transition. The remaining 223 safety-critical events involved a crash. A total of 223 safety-critical car tracking events were summarized for environmental conditions, including surface, weather, lighting conditions, and road type.

[0088] The distribution of vehicle kinematics during 223 events was statistically analyzed. The subject vehicle is the research vehicle. The average maximum deceleration of the subject vehicle is -0.62g, and the initial speed is 8.97m / s; the average maximum deceleration of the lead vehicle is -0.67g. The average distance between the subject vehicle and the lead vehicle is 11.96 meters, and the initial relative speed is -0.6m / s (the negative sign indicates that the subject vehicle has a higher speed); the average minimum TTC is 1.31 seconds.

[0089] This study uses scenarios created from 223 safety-critical car-following events to calibrate the RSS model and replace the human-driven test vehicle with the RSS-embedded autonomous driving algorithm, which is defined as the rear vehicle. The goal is to find the best combination of RSS parameters to ensure that the RSS-embedded autonomous driving algorithm achieves the best safety performance in 223 simulated car-following scenarios.

[0090] like Figure 4As shown, as a preferred embodiment of the present invention, the step of constructing the parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model into a constrained Markov decision process model specifically includes:

[0091] Construct a cost function based on the objective of minimizing the collision time integral;

[0092] The reward function is constructed based on the goal of minimizing the sum of conservative safety distances.

[0093] This paper proposes a modeling method based on constrained Markov decision process for optimizing longitudinal safety distance parameters. Constrained Markov decision process can formally express the decision-making process of a single agent in an environment with safety constraints. In this framework, the agent aims to maximize the total expected reward of its behavior strategy while satisfying specific safety constraints. The model consists of a tuple Definition, where:

[0094] represents the global state space;

[0095] Represents the action space of the agent;

[0096] Represents the reward function, describing the transition from state By taking joint action To the next state Instant rewards;

[0097] Represents a cost function that maps a transformation tuple to a function with a safe cost threshold. Each cost function Corresponding to an upper limit ;

[0098] Indicates from the state By taking joint action to discrete time steps The next state The transition probability distribution of

[0099] Represents the initial state distribution, which defines the initial probability of each state at time step t=0.

[0100] Through the constrained Markov decision process model, the longitudinal safety distance can be effectively optimized to ensure the safety and efficiency of the intelligent agent in a dynamic environment.

[0101] The vertical safety regulation formula in the responsibility-sensitive safety model based on the constrained Markov decision process model is:

[0102] ;

[0103] Among them, the response time of the following vehicle is , the maximum deceleration of the front vehicle is , the maximum acceleration of the rear vehicle is , the minimum deceleration of the rear vehicle is , is the speed of the rear vehicle, is the speed of the front vehicle, The minimum safe distance.

[0104] The state space needs to include the speed information of each vehicle and the distance information between the two vehicles in the same direction. Since the two vehicles in front and behind are driving on a fixed road planned by the autonomous driving algorithm, the lateral displacement and deviation can be ignored, which makes the trajectory changes of each group of vehicles consistent during the test and reduces the redundant space dimension.

[0105] The state space not only covers the basic parameters of vehicle driving, but also fully considers the complexity and uncertainty in actual traffic. It is specifically expressed as:

[0106] The initial relative distance between the front and rear vehicles: ; Relative speed of front and rear vehicles: .

[0107] The initial relative distance and relative speed of the two vehicles in front and behind follow the normal distribution and This setting not only takes into account the diversity of vehicle spacing in actual traffic, but also ensures the versatility and extensibility of the model.

[0108] Behavior space is an important part of the autonomous driving system. Through in-depth analysis of state space parameters, appropriate parameters are output to control the model operation in the scene. According to the longitudinal safety specification formula of the RSS model, it is assumed that the autonomous driving vehicle has a high degree of prediction and response capabilities, and can effectively avoid collisions even in emergencies. Specific parameters include: Response time of the following vehicle: 、Maximum deceleration of the vehicle ahead: 、Maximum acceleration of the following vehicle: 、Minimum deceleration of the following vehicle: These parameters are finely controlled and optimized to ensure that autonomous vehicles maintain a high degree of safety and stability in complex scenarios.

[0109] like Figure 3 As shown, as a preferred embodiment of the present invention, the cost function for:

[0110] ;

[0111] ;

[0112] in, Indicates the number of safety critical events The duration of time below a certain threshold is expressed as The unit is , N is the number of events that trigger TTC below the threshold in each round during each strategy update process; Indicates safety critical period value, used as a safety measure; for Threshold, set to 4 seconds; Indicates Period in time In the simulation step value; is the time interval between two simulation steps, which is equal to 0.1 seconds in the present invention.

[0113] in, The objective function of the algorithm is to maximize safety and ensure that the autonomous vehicle avoids collisions as much as possible during the safety-critical period. It does this by minimizing the TIT value of all safety-critical periods. TTC (Time to Collision) is an important indicator for measuring the relative safety between two vehicles. The higher the TTC value, the better the relative safety between the two vehicles; conversely, the lower the TTC value, the greater the risk of collision. The objective function ensures that the autonomous vehicle has the highest possible safety during the safety-critical period by minimizing the sum of TITs of all critical periods.

[0114] On the basis of confirming the state and behavior space, according to the constrained Markov decision process, the present invention optimizes the selection of the autonomous driving algorithm embedded in RSS by constructing a reward function that can reflect the degree of optimization of system performance to characterize the optimal state of the system, and describes the performance optimality of the system by constructing a cost function that can reflect the degree of violation of the system's safety performance.

[0115] The cost function selects sub-goal A in the research goal, that is, minimizing the collision time integral , which aims to minimize the duration of the rear vehicle collision time TTC below a certain threshold. Specifically, a deep reinforcement learning method is used to define the minimum is a cost function, and by optimizing this function, it is determined whether the two vehicles have alleviated the potential unsafe conflict.

[0116] like Figure 3 As shown, as a preferred embodiment of the present invention, the reward function for:

[0117] ;

[0118] ;

[0119] in, Representative Conservative measures for the period; Indicates In the period Greater than No. a moment; and Represent the minimum reasonable distance and the actual distance respectively. Greater than If there is no moment Greater than ,but is 0.

[0120] in, The objective function is to minimize the conservatism of the model. Conservatism refers to the degree to which the safety distance is excessively maintained when it is not necessary. An overly conservative model will cause vehicles to decelerate or stagnate frequently, which will affect the efficiency of traffic flow and the comfort of passengers. Therefore, this objective function reduces conservatism by minimizing the sum of relative distance deviations (Δdi) during all critical periods. Specifically, if the safety distance dmin is greater than the actual detected relative distance drelative, the RSS model will be triggered, causing the AV to decelerate with the minimum braking deceleration. The more conservative the model, the earlier the RSS model is triggered, and the larger the Δdi.

[0121] In order to optimize the driving strategy of the autonomous vehicle, the present invention defines the conservative measure as the sub-goal B of the reward function, that is, minimizing the conservative safety distance The function aims to minimize the conservative measure during all safety-critical events The sum of Representative in the Minimum safety distance during safety critical events and relative distance Specifically, when Greater than When , the RSS model will be activated, forcing the autonomous vehicle to slow down, thereby increasing the conservatism of the model. The value of can reduce the conservatism of the model and improve the operating efficiency of the vehicle. If during a safety-critical event, Never greater than ,but will be set to 0. By minimizing this reward function, the conservatism of the autonomous vehicle can be reduced and its operating efficiency can be improved while ensuring safety.

[0122] like Figure 5 As shown, as a preferred embodiment of the present invention, the step of evaluating and optimizing the policy neural network, the reward value neural network and the cost value neural network specifically includes:

[0123] Determine the expected return and constraints based on the reward function and the cost function;

[0124] Update the policy neural network based on the approximate solution method;

[0125] The reward value neural network and the cost value neural network are updated.

[0126] In autonomous driving, algorithms based on the RSS protocol have great potential in longitudinal scenarios. However, traditional optimization algorithms calculate fewer scenarios and have poor results when obtaining RSS longitudinal parameters. Deep reinforcement learning algorithms have not yet been widely used in this field, but they have great potential. The solution of the present invention is to optimize the strategy by maximizing the reward while satisfying the RSS longitudinal safety scenario constraints. Next, the present invention will use the constraint policy optimization (CPO) algorithm to solve the RSS longitudinal parameter optimization problem previously formally defined by the constrained Markov decision process.

[0127] In the aforementioned steps, the present invention has established a framework for modeling the RSS parameter optimization problem as a Markov decision process. In this embodiment, the CPO algorithm is used to solve this problem by using an advanced algorithm that maintains safety constraints during the learning process. Specifically, the time when the two vehicles enter the dangerous distance is used as a reward, and a relatively safe distance is ensured between the two vehicles as a constraint. The present invention adopts the CPO algorithm to ensure that the agent follows safety constraints during the learning process and avoids dangerous behaviors. CPO guides the agent to optimize task performance through score rewards and constraint losses, while restricting its behavior to ensure safety. The algorithm takes the constraints as part of the optimization problem and optimizes them through Lagrange multipliers, aiming to maximize the expected cumulative reward and limit the difference between the new and old strategies to ensure the stability of strategy updates.

[0128] In an environment based on a constrained Markov decision process model, the agent interacts with the environment in discrete time steps. , the intelligent experience first understands the environment and reservation strategy Select Behavior ; After performing this action, you will get a reward and the corresponding expenses The environment state then shifts to At the end of each episode, the environment changes from an arbitrary initial state Reset. In this way, the trajectories collected from consecutive episodes Used for policy updating, the goal is to guide the agent to learn the optimal strategy to maximize the expected reward within the cost limit .

[0129] As a preferred embodiment of the present invention, the step of determining the expected return and the constraint conditions based on the reward function and the cost function includes:

[0130] Define the expected discounted return of a policy relative to the cost function;

[0131] Determine the expected discounted return of the policy relative to the reward function;

[0132] Determine a strategy that satisfies the constraints.

[0133] In the framework of constrained policy optimization, policy updating aims to enable the agent to learn the optimal strategy that maximizes the expected reward within the cost constraint through continuous environment interaction and policy iteration. To this end, the present invention first defines the strategy Relative to the cost function The expected discounted return is Representation strategy Relative to the cost function The expected discounted return is:

[0134] ;

[0135] in, is a discount factor, which represents the current value of future rewards. Under this setting, the present invention defines a feasible strategy set: , is the set of policies that satisfy all cost constraints:

[0136] ;

[0137] Strategy The expected discounted return with respect to the reward function is expressed as:

[0138] ;

[0139] Ultimately, the present invention seeks can maximize Strategy :

[0140] ;

[0141] These definitions and conditions form the basic framework for policy iteration, which aims to help the agent optimize its strategy through approximate solution methods to achieve the best performance while complying with safety and cost constraints. To find the optimal feasible strategy that satisfies the cost and reward constraints, the approximate solution strategy relies on finding the optimal feasible strategy that satisfies the cost and reward constraints. and parameterized strategies The intersection of the KL trust regions This search involves solving a constrained Markov decision process problem and maximizing the expected discounted reward in a local neighborhood. To iterate the update strategy:

[0142] ;

[0143] ；

[0144] ;

[0145] in, The constraint ensures that the new policy remains within a trust region in terms of KL divergence with the previous policy, thereby balancing exploration and exploitation and ensuring the stability of policy iteration. However, it is difficult to implement this update in practice because it is necessary to evaluate the safety constraint function to determine the policy Is it feasible? When using When calculating constraint policy updates based on collected samples, policy evaluation is considered challenging. In this study, we replace the objective and safety constraints with easily evaluated proxy functions developed by CPO to obtain the equation A good local approximation of . The policy-based value function representing the reward Indicates in status According to the strategy Initial expected return: , indicating that from the state Begin joint action After that, follow the strategy expected return. , action value function Indicates from the state Begin joint action After that, follow the strategy The expected return, advantage function Measure and act relative to the average It is defined as the difference between the action-value function and the value function ,Similarly, the policy-based value function, action-value function and advantage function of the auxiliary cost are defined as , , ,make represents the discounted future state distribution, defined as ,equation So it is approximated as:

[0146] ;

[0147] ;

[0148] ;

[0149] The above formula can be used to satisfy the specified constraints While increasing the expected reward return. However, for neural network strategies with high-dimensional parameter space, the above equation may not be directly solvable due to computational overhead. For this reason, we consider using a small step size To approximate the strategy These constraints can be set by the current policy The linear function around it can be effectively approximated, and the KL divergence constraint can be approximated by its second-order expansion, so it is approximated as:

[0150] ;

[0151] ;

[0152] ;

[0153] in, represents the parameters of the neural network, represents the linear approximation of the objective function under parameter variation, express Right The Hessian matrix of is used to approximate the second-order term of KL divergence. This approximation allows us to ensure the security and performance of the new strategy while maintaining computational efficiency. is always semi-positive, and the policy optimization problem is convex. When a convex optimization problem is feasible, it can be solved efficiently through duality:

[0154] ;

[0155] in: and is a dual variable; and are the parameters in the corresponding dual problem; Difference vector representing the safety constraint function.

[0156] The above equations show the solution of the dual problem using the dual variables and , and the corresponding parameters and . represents the difference vector associated with the safety constraint. This form of the dual problem expresses how to simplify the solution of the original problem through duality while maintaining the convexity of the problem. If and is the solution to the dual problem, then the solution to the original problem is:

[0157] ;

[0158] That is, it provides specific steps for updating the strategy, including and is the optimal solution to the dual problem. However, due to the approximation error, CPO may take wrong steps and produce infeasible iterations , sometimes the aforementioned approximate equation is still feasible, and CPO can automatically recover from the wrong step, but for the unfeasible case, a recovery method is necessary. Therefore, backtracking linear search is used to obtain To ensure that the sample estimate satisfies the constraints in the previous approximation equation:

[0159] .

[0160] Reward-based value neural network parameter update rule:

[0161] ;

[0162] in, Represents at time The predicted reward value of , which is an estimate of the upcoming reward.

[0163] Cost-based value neural network parameter update rule:

[0164] ;

[0165] Here, It's in time The predicted cost value of the current strategy assesses the cost or adverse consequences of taking a specific action, reflecting the potential negative impact of the current strategy.

[0166] See also Figures 6 to 13, the present invention uses the Carla simulator to create a Python script for the RL environment, and the main components and functions are as follows: First, the code construction starts with importing the necessary libraries, including matplotlib, numpy, gym, carla, etc. These libraries are used for visualization, numerical calculation, creation of reinforcement learning environment, and interaction with the Carla simulator, respectively. Secondly, the Carla environment is set and the CarlaEnv class is defined, which inherits from the gym.Env class as the environment for reinforcement learning. In the constructor init, various parameters related to the Carla simulator environment are initialized, including client connection, world settings, blueprint library, vehicle settings, observation space, and action space. Next, the observation space and action space are defined using gym.spaces.Box, clarifying the range of observations and actions that the reinforcement learning agent can perform in the environment. Subsequently, the reset function is set for environment initialization and reset. This method resets the simulation environment to the initial state, including generating vehicles, setting initial positions, speeds, and other parameters. The execution of actions and time step functions step method is the core part of the reinforcement learning agent to perform actions and interact with the environment. This method receives the agent's action, updates the environment based on this action, calculates the reward, and returns the new state, reward, and other relevant information. The script also includes logic for handling collisions and terminations based on certain conditions (such as collisions or reaching the maximum episode length). Finally, the code logs various data, including logging data to CSV files for analysis and evaluation, such as collisions, relative speeds of the front and rear vehicles, relative distances between the front and rear vehicles, and other relevant information.

[0167]

[0168] The present invention implements a training function for training an agent model for a safety longitudinal scenario to solve specific environmental problems. The main steps and functions are as follows: First, various parameters required for training are defined, including agent name, environment name, number of training cycles, etc. Next, an agent model is initialized and a random seed is set to ensure the repeatability of the experiment. Subsequently, each training cycle is iterated through a loop. In each cycle, the agent model interacts with the environment and is trained based on the results of the interaction. During the training process, various training indicators and results, such as scores, losses, etc., are recorded, and the trained model is saved regularly. After the training is completed, the trained model is obtained and saved to a file. The entire training process records the training data in an academic manner and realizes the preservation and management of the training model.

[0169] In the simulation platform verification phase, the present invention adopts the following experimental environment hardware configuration and software environment: the processor is the 13th generation Intel (R) Core (TM) i7-13700K 3.40 GHz, and the graphics card is NVIDIA GeForce RTX 3080Ti. The software environment includes Pycharm-Community Edition-2023.1.1 and Carla 0.9.13 simulator, running on the Windows 11 platform. The intersection scene is constructed using the Carla simulator based on the UE4 engine, and a parameter optimization model of a longitudinal safe driving flexible specification method based on a constraint strategy optimization algorithm is established under the Pytorch framework. Carla's internal sensors are used to monitor the vehicle status in real time, while the Basic Agent method is used to track road vehicles. The internal PID controller converts speed information into control instructions such as acceleration and braking. The program is written in Python 3.7 and trained with CUDA 11.7 using a GPU. 2048 steps were collected in the experiment, and the CPO optimization algorithm was used to adjust the strategy. After 1024 cycles, the learning rate was gradually reduced from 0.003 to 0. The node speed is limited to 0 to 25 m / s, the acceleration is limited to -2.5 to 2.5 m / s, and the expected speed of the main vehicle is 12 m / s. The weight factors of speed and acceleration are set to 1, the conflict risk factor is set to 1000, and the time lag is 0.005 seconds. Carla's "Town05" map is selected as the simulation environment. The straight-line distance of this map is long enough and the slope change is small. Two vehicles (including the main vehicle and the interference vehicle) are set in a straight section of about 400 meters long, and the vehicles are arranged in front and behind. When the main or interference vehicle reaches the destination, the training ends and the scene is reset. If a collision occurs during training, the next round of training will be restarted. The initial positions of the vehicles are randomly distributed between 5 and 10 meters, and the speed of the leading vehicle ranges from 4.68 to 17.24 m / s. Braking points are randomly set along the entire road section, and the collision probability is set to 40% to test the emergency braking performance of the main vehicle.

[0170] The score and cost values ​​are used as evaluation indicators. The total score corresponds to the reward function, and the initial reward is set to 0 points. If the main vehicle and the interference vehicle successfully reach the end point, the reward is increased by 100 points to guide the vehicle to learn the strategy to reach the end point; if the rear vehicle collides with the front vehicle, 50 points are deducted and the scene is reset for the next round of testing. In addition, after each time step, the reward value is subtracted by 0.10 points to encourage the neural network to choose a more aggressive strategy and improve traffic efficiency, so as to better use deep reinforcement learning to optimize the parameters that conform to the RSS longitudinal formula.

[0171] In the 0-200 training cycles, the reward function value quickly increased from about -60 to -40. In the 200-600 cycles, the reward value continued to increase but the volatility increased; at 600 cycles, it reached a maximum value of close to -20. In the 600-1200 cycles, the reward function showed high-frequency fluctuations, and the score fluctuated between -20 and -30, with a slight overall decrease.

[0172] In terms of overhead, during the initial 0 to 200 training cycles, the overhead increased slightly from about 17.5 and fluctuated. During 200 to 600 cycles, the overhead gradually decreased to between 12 and 15. Subsequently, during 600 to 900 cycles, the overhead increased slightly and reached a peak of about 17. After more than 900 cycles, the overhead decreased rapidly to close to 0. After a total of about 1,300 trainings, the overhead was close to 0.

[0173] The research results show that the car had no collision during the test, the tracking effect was good, and the control algorithm had good convergence. Figure 6 and Figure 7 The training curves of the reward function and the cost function are shown respectively. Each training takes about 10 to 15 minutes.

[0174] In this study, RSS parameters found and calibrated by deep reinforcement learning were embedded in the autonomous driving system, and longitudinal safety-critical scenarios were tested in Carla. The results were compared with the autonomous driving with RSS parameters embedded in the traditional algorithm and the autonomous driving without RSS model embedded. In the original data, one collision occurred in 223 safety-critical scenarios, while no collision occurred in the two RSS-embedded autonomous driving schemes.

[0175] We further use more than 100 extreme scenarios in Carla to evaluate the TIT and conservatism of these autonomous driving methods. , the safety of autonomous driving is further evaluated by measuring the potential collision probability. A set of sample data is randomly selected to embed the calibrated RSS model ACC-controlled autonomous driving vehicle to ensure that the parameter values ​​of the deep reinforcement learning algorithm are more appropriate. The specific values ​​are shown in the table below.

[0176]

[0177] In 100 extreme scenarios, it takes about 120 seconds for all vehicles to complete a round of testing. The proportion of vehicles with a distance less than the safe distance within 2048 time steps is about 1%, with a unit level of 0.1 meters, and the number of collisions is 0%. The time for the main vehicle to enter the dangerous distance is basically less than the limit value. For specific TIT comparisons, see Figure 8 as well as Fig. 9 The horizontal axis is the number of experiments and the vertical axis is the TIT value. Fig.10 as well as Fig.11 A scatter plot of .

[0178] from Figures 8 to 11 It is observed that the change in TIT of the autonomous driving technology using deep reinforcement learning embedded in the calibrated RSS model is significantly smaller than that of the other two methods. TIT is a key indicator for evaluating the reaction speed and safety of the autonomous driving system, reflecting the time it takes for the system to take intervention measures after detecting potential danger. Specifically, the TIT of the deep reinforcement learning method is smaller than the TTC of the other methods, indicating that the system can respond faster and reduce the probability of accidents.

[0179] In addition, this study found that the autonomous driving system trained by deep reinforcement learning is less conservative and tends to make flexible decisions rather than being overly conservative. Conservatism is an important indicator for evaluating the decision-making tendency of autonomous driving systems, reflecting the decision-making tendency of the system when facing complex traffic scenarios. Although higher conservatism can lead to more cautious behavior, it may limit the ability to cope with complex environments.

[0180] This study adopts a flexible regulation method for longitudinal safe driving based on a constraint strategy optimization algorithm, which enables the autonomous driving system to show higher flexibility while ensuring safety. This method uses deep reinforcement learning training to find the parameters of the RSS longitudinal regulation formula. This method has significant advantages: it not only improves the system's response speed and reduces the probability of accidents, but also enhances the system's flexibility and comfort. Therefore, the flexible regulation method for longitudinal safe driving based on a constraint strategy optimization algorithm plays an important role in the decision-making optimization of the autonomous driving system.

[0181] The comparison results of the RSS (responsibility-sensitive safety) model trained and calibrated by deep reinforcement learning (DRL) and other algorithms (such as genetic algorithm and random data) in different experimental scenarios were compared with the operation model of the Carla traffic scenario. By establishing an autonomous driving control model embedded in RSS, the following mode of the main vehicle is realized. The kinematic data corresponding to each scenario can be found in Fig.12 as well as Fig.13 , Fig.12 This is a line graph of the vehicle kinematic parameters changing with the scene during deep reinforcement learning training, including the relative speed and relative distance in the first 50 time steps. The line graph of the relative speed and relative distance changing with the scene in the first 50 time steps can help observe the behavior changes of the vehicle in the early stages of training. The relative speed of the vehicle fluctuates greatly, but the relative distance remains at a relatively stable level. This shows that in the early stages of training, the model tried multiple strategies to control the speed while maintaining a safe distance from the vehicle in front. We can see how the autonomous driving system gradually learns better strategies. There are usually more fluctuations and instabilities in the early stages, reflecting the process of adaptation to the environment during model exploration and learning.

[0182] Fig.13 This is a line graph of the vehicle kinematic parameters changing with the scene during deep reinforcement learning training, including the relative speed and relative distance for the first 100 time steps. Expanding the graph to 100 steps can help evaluate the performance of the model over a longer training time. As training progresses, the change in relative speed begins to show a more regular pattern, and the relative distance gradually tends to be stable. This shows that the model gradually finds a more effective and stable control strategy during training, which can better balance speed and safe distance. The 100-step data can show the convergence and stability of the strategy, as well as the model's ability to adapt to different scenarios and situations.

[0183] Changing in different time steps can reflect how the autonomous driving system responds to different scenarios. For example, when approaching the vehicle in front, how the system adjusts the speed and maintains a safe distance. Separating these two time steps can specifically observe the vehicle's response to different situations in the short and medium term.

[0184] In the first 50 steps, we can analyze the effect of the initial training of the model to see if there are large deviations or insufficient safety. In the 0-100 step chart, we can evaluate whether the performance of the model is gradually optimized as the training time increases, and whether there are overly aggressive or conservative adjustments. This helps provide more specific feedback for algorithm design in deep reinforcement learning, and helps adjust the reward function, policy update method, or other hyperparameter settings to achieve better training results. Through these two pictures, we can observe how the autonomous driving control model gradually learns better strategies during training. Changes in relative speed and relative distance indicate how the vehicle adjusts its speed to maintain a safe distance. These data show that the model has learned how to accelerate and decelerate to avoid collisions when dealing with different traffic conditions.

[0185] By dividing the data into two step intervals, the learning speed and convergence of different algorithms or models in the same environment can be compared. Models that converge quickly or show obvious learning ability will show a stable control strategy in 0-50 steps, while models with poor performance may take longer (such as 100 steps) to reach a similar stable state.

[0186] In CARLA, a scenario that conforms to the statistical law of safety-critical time is built, and the strategy is continuously trained through the longitudinal safety driving flexible specification method based on constraint strategy optimization to learn and optimize the parameters of the safety distance calculation formula until the model converges. The test results show that the optimal parameters proposed by the present invention can not only maintain the longitudinal driving safety of the vehicle in the safety-critical scenario that conforms to natural driving, but also avoid triggering the specification due to over-conservative estimation to the greatest extent, and have both safety and efficiency.

[0187] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A longitudinal flexible safe driving regulation method based on constraint strategy optimization, It is characterized in that The method comprises: Acquire key safety events of natural driving vehicles based on natural driving data; Building a responsibility-sensitive safety model based on critical safety events of natural driving vehicles; The parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model is constructed as a constrained Markov decision process model. The state space in the constrained Markov decision process model is defined as the statistical mean and standard deviation of the relative distance and relative speed between the front and rear vehicles; the behavior space is defined as the parameters to be optimized in the safety distance formula, including the response time of the rear vehicle, the maximum deceleration of the front vehicle, the maximum acceleration of the rear vehicle, and the minimum deceleration of the rear vehicle. Building a neural network based on a deep reinforcement learning algorithm, wherein the neural network includes a strategy neural network, a reward value neural network, and a cost value neural network; Evaluate and optimize the strategy neural network, reward value neural network and cost value neural network, perform multiple iterations of neural network training to obtain a trained neural network model, and optimize the parameters in the longitudinal safety distance formula based on the trained neural network model; The step of obtaining key safety events of natural driving vehicles based on natural driving data specifically includes: Acquiring natural driving data, wherein the natural driving data includes kinematic data and video data; Determine the triggering conditions of dangerous scenarios; Extract initial safety-critical scenario events based on natural driving data and dangerous scenario trigger conditions; Screening the initial safety-critical scenario events to generate the final safety-critical scenario events; Extract kinematic features based on the final safety-critical scenario events; The step of constructing the parameter optimization problem in the longitudinal safety distance formula in the responsibility-sensitive safety model into a constrained Markov decision problem model specifically includes: Construct a cost function based on the objective of minimizing the collision time integral; Construct a reward function based on the goal of minimizing the sum of conservative safety distances; The cost function C i Designed for: Among them, T, T i represents the duration of time that the TTC in each safety-critical event is lower than a certain threshold, N is the number of events that trigger the TTC to be lower than the threshold in each round during each policy update process; TTC i represents the TIT value of the i-th safety critical period; TTC* is the TTC threshold; TTC i (t) represents the TTC value in the simulation step of the i-th period at time t; τ bc is the time interval between two simulation steps; The reward function R is designed as: Where, Δd i represents the conservative measure of the i-th period; Indicates the number of d in the i-th period min Greater than d xdjl The i-th moment of min and d xdjl Represent the minimum reasonable distance and the actual distance respectively; The steps of evaluating and optimizing the policy neural network, the reward value neural network and the cost value neural network specifically include: Determine the expected return and constraints based on the reward function and the cost function; Update the policy neural network based on the approximate solution method; Update the reward value neural network and the cost value neural network; The step of determining the expected return and the constraint conditions based on the reward function and the cost function includes: Define the expected discounted return of a policy relative to the cost function; Determine the expected discounted return of the policy relative to the reward function; Determine a strategy that satisfies the constraints; Strategy π relative to the cost function C i Expected discounted return Defined as: in, Denotes the strategy π relative to the cost function C i The expected discounted return, γ∈[1,0) is the discount factor; The expected discounted return / (π) of policy π relative to reward function R is expressed as: J(π)=E τ～π ∑ t c t R(s t ,a t ,s t+1 ); A policy set that satisfies all cost constraints: In the policy constraint set Π C The strategy π that can maximize J(π) * :

Images