Behavior decision and security verification method for ADS in uncertain environment

Abstract

The invention discloses an ADS behavior decision and safety verification method in an uncertain environment, and belongs to the field of automatic driving. According to the method, the problems of poor behavior decision performance and poor accuracy of behavior decision security verification of an existing method are solved. According to the method, the state space, the action space and the reward function of the intelligent agent in the depth deterministic strategy gradient method are designed, and the reward function is designed by fully considering multiple safety constraints in reality, so that the behavior decision performance of the intelligent agent on the automatic driving vehicle can be improved. According to the method, the automatic driving vehicle scene is constructed into the random hybrid automaton model, the safety constraint of the automatic driving vehicle scene is expressed in a formalized mode through the method, the accuracy of behavior decision safety verification can be improved, and therefore the ADS behavior decision safety is guaranteed. The method can be applied to behavior decision and security verification of the ADS.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and specifically relates to a behavior decision and safety verification method of an ADS (autonomous driving system) in an uncertain environment. Background Art

[0002] In recent years, with the rapid development of sensor technology, computer technology and 5G technology, autonomous driving cars have gradually entered our daily transportation life (L.Liu et al.,'Computing Systems for Autonomous Driving:State of the Art and Challenges',IEEE Internet Things J.,vol.8,no.8,pp.6469–6486,Apr.2021,doi:10.1109 / JIOT.2020.3043716.). ADS is a complex system of integrated sensing, decision-making and control, designed to enable vehicles to complete driving tasks autonomously with minimal human intervention. The safety of ADS has also received widespread attention, especially in driving scenarios with complex traffic conditions and large passenger flow, such as highways, urban roads, and intersections. The accuracy and safety verification of system behavior decisions (G. Li, Y. Yang, S. Li, X. Qu, N. Lyu, and S. E. Li, 'Decision making of autonomous vehicles in lane change scenarios: Deeper instruction learning approaches with risk awareness', Transp. Res. Part CEmerg. Technol., vol. 134, p. 103452, Jan. 2022, doi: 10.1016 / j.trc.2021.103452.) are particularly important. Unlike traditional manual driving, ADS collects scene data through interaction with the surrounding environment, analyzes the data within a controllable time and makes accurate and safe decision actions (H. Chen, Y. Zhang, U. A. Bhatti, and M. Huang, 'Safe Decision Controller for Autonomous Driving Based on Deep Reinforcement Learning in Nondeterministic Environment', Sensors, vol. 23, no. 3, p. 1198, Jan. 2023, doi: 10.3390 / s23031198.).Therefore, how to ensure the accuracy and safety of ADS behavior decisions in uncertain environments is a current challenge (S. Noh,'Decision-Making Framework for Autonomous Driving at Road Intersections:Safeguarding Against Collision,Overly Conservative Behavior,and Violation Vehicles',IEEE Trans. Ind. Electron.,vol.66,no.4,pp.3275–3286,Apr.2019,doi:10.1109 / TIE.2018.2840530.).

[0003] The deep deterministic policy gradient algorithm (DDPG) is a combination of reinforcement learning method and deterministic policy gradient algorithm. It combines the advantages of policy gradient and reinforcement learning and is suitable for solving continuous action space problems. In recent years, it has been widely used in autonomous driving path planning, behavioral decision-making, safe obstacle avoidance, etc. (X.Xu, L.Zuo, X.Li, L.Qian, J.Ren, and Z.Sun, 'A Reinforcement Learning Approach to Autonomous Decision Making ofIntelligent Vehicles on Highways', IEEE Trans.Syst.Man Cybern.Syst., pp.1–14, 2019, doi:10.1109 / TSMC.2018.2870983.). Since the behavioral decision-making of autonomous driving vehicles usually involves continuous control, such as acceleration, steering, braking, etc., the application of DDPG in autonomous driving is very advantageous. In the process of autonomous driving decision-making, the longitudinal control actions and lateral control actions of the vehicle are continuous rather than discrete. Traditional reinforcement learning algorithms, such as Q-learning or DQN, are good at processing discrete action spaces, but perform poorly in continuous action spaces. DDPG can directly process continuous action spaces and output a certain action through a policy network, which gives it a clear advantage in behavioral decision-making in autonomous driving, especially in the control level (H.Shu, T.Liu, X.Mu, and D.Cao, 'Driving Tasks Transfer Using Deep Reinforcement Learning for Decision-Making of Autonomous Vehicles in Unsignalized Intersection', IEEE Trans.Veh.Technol., vol.71, no.1, pp.41–52, Jan.2022, doi:10.1109 / TVT.2021.3121985.). Behavioral decisions in autonomous driving must be real-time and consistent. The vehicle needs to continuously perceive the surrounding environment and make real-time adjustments during driving. DDPG can help the system make smooth action selections in continuous time, avoiding the discontinuity problem in discrete decision-making algorithms, thereby achieving smooth control behavior. In autonomous driving, vehicles must not only consider a single task, such as reaching the destination quickly, but also need to comprehensively consider multiple tasks or goals, such as safety, driving comfort, and compliance with traffic rules. DDPG can optimize multiple tasks by designing a multi-objective reward function.For example, it can simultaneously consider vehicle speed, path following, obstacle avoidance, etc. in different driving scenarios, helping the system to make the best decision based on weighing multiple factors. An et al. (D.An, J.Liu, M.Zhang, X.Chen, M.Chen, and H.Sun, 'Uncertaintymodeling and runtime verification for autonomous vehicles driving control: A machine learning-based approach', J.Syst.Softw., vol.167, p.110617, Sep.2020, doi:10.1016 / j.jss.2020.110617.) use machine learning classification algorithms to analyze environmental data, classify the styles of surrounding driving vehicles, and use the classification results as input to the statistical model detection tool UPPAAL, which verifies quantitative properties to give a more intelligent strategy. Kong et al. (Q. Kong, L. Zhang, and X. Xu, 'Constrained Policy Optimization Algorithm for Autonomous Driving via Reinforcement Learning', in 2021 6th International Conference on Image, Vision and Computing (ICIVC), Qingdao, China: IEEE, Jul. 2021, pp. 378–383. doi: 10.1109 / ICIVC52351.2021.9526928.) used the constrained policy optimization algorithm to update the policy and model the Markov decision process.Li et al. (G. Li, S. Lin, S. Li, and X. Qu, 'Learning Automated Driving in Complex Intersection Scenarios Based on Camera Sensors: A Deep Reinforcement Learning Approach', IEEE Sens. J., vol. 22, no. 5, pp. 4687–4696, Mar. 2022, doi: 10.1109 / JSEN.2022.3146307.) not only considered traffic safety in intersection scenarios, but also considered driving efficiency and passenger comfort issues. They proposed a deep reinforcement learning method combined with a long short-term memory network, which uses traffic images of different time steps to reflect the relative motion information between vehicles, thereby obtaining the optimal decision-making action for comprehensive driving safety, driving efficiency and passenger comfort.

[0004] However, when the existing DDPG algorithm gives the corresponding reward function based on the safety constraints in the decision-making process, it does not fully consider the safety constraints in reality. Therefore, the behavior decision-making performance of the existing DDPG algorithm is still poor. Formal methods can accurately describe the safety constraints in actual scenarios using linear temporal logic, probabilistic computation tree logic, etc. Therefore, it is necessary to construct the driving scenario as a formal model to verify the safety of the behavior decision. Stochastic Hybrid Automaton SHA (Stochastic Hybrid Automaton) is a mathematical model that combines the characteristics of hybrid systems and random processes and can effectively model and analyze the uncertainty and complexity in actual systems. Gao et al. (Gao Fei, Luo Cheng, Shi Fangyuan, et al. Online Safety Verification of Autonomous Driving Decision-Making Based on Dynamic Reachability Analysis [J]. IEEE Access, 2023) proposed an online verification safety method, which verifies the safety of autonomous driving behavior decisions by establishing a reasonable area that can be reached in the short term for the vehicle in the driving scenario and observing whether there are overlapping areas. Xu et al. (Xu Bingqing, Li Qin, Guo Tong, et al. A quantitative safety verification approach for the decision-making process of autonomous driving [C] / / Proc of IEEE Int Symposium on Theoretical Aspects of Software Engineering (TASE). NJ: IEEE, 2019: 128-135) proposed an operation verification method based on random hybrid automata to verify the safety of behavioral decisions. Du et al. (Du Dehui, Chen Jiena, Zhang Mingzhuo, et al. Towards Verified Safety-critical Autonomous Driving Scenario with ADSML [C] / / Proc of the 45th Annual Computers, Software, and Applications Conf (COMPSAC). NJ: IEEE, 2021: 1333-1338) used random hybrid automata to specify the dynamic behavior in the scenario and used the formal verification tool UPPAAL-SMC to verify its safety.Philippe Ledent et al. (Ledent P, Paigwar A, Renzaglia A, et al. Formal validation of probabilistic collision risk estimation for autonomous driving [C] / / Proc of IEEE Int Conf on Cybernetics and Intelligent Systems (CIS) and Conf on Robotics, Automation and Mechatronics (RAM). NJ: IEEE, 2019: 433-438) focused on the risk assessment generated by the probabilistic perception system, selected key appropriate indicators for the verification of the collision probability risk estimation, and performed formal verification using the XTL model checker of the CADP toolbox. Selvaraj et al. (Selvaraj Y, Ahrendt W, Fabian M. Verification of decision making software in an autonomous vehicle: An industrial case study [C] / / Proc of the 24th Int Conf Formal Methods for Industrial Critical Systems. Berlin: Springer, 2019: 143-159) verified through a variety of formal techniques and concluded that the safety of autonomous driving behavior decisions requires the support of many safety verification tools. An et al. (An Ddongdong, Liu Jing, Zhang Min, et al. Uncertaintymodeling and runtime verification for autonomous vehicles driving control: A machine learning-based approach [J]. Journal of Systems and Software, 2020, 167: 110617) proposed a parameterized modeling language that takes the driver's driving style as input, and converted it into NPTA for safety verification using UPPAAL-SMC.Wang et al. (Wang Jinyong, Huang Zzhiqiu, Zhu Yi, et al. Statistical Model Checking for Stochastic and Hybrid Autonomous Driving Based on Spatio-Clock Constraints [J]. International Journal of Software Engineering and Knowledge Engineering, 2022, 32 (04): 553-582) proposed a multi-lane spatiotemporal clock random specification language to describe the safety invariance of autonomous driving in specific scenarios, and designed a spatiotemporal clock random hybrid automaton based on this language to simulate random and hybrid behaviors. In summary, although the prior art has studied the safety verification of behavioral decisions, the accuracy of the existing methods for safety verification of behavioral decisions is still poor. Therefore, it is an urgent problem to propose a behavioral decision and safety verification method to solve the problems of poor performance of behavioral decisions and poor accuracy of safety verification of behavioral decisions in existing methods. Summary of the invention

[0005] The purpose of the present invention is to solve the problems of poor behavior decision performance and poor accuracy of behavior decision safety verification in existing methods, and to propose a behavior decision and safety verification method for ADS in an uncertain environment.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a behavior decision and safety verification method of ADS in an uncertain environment, the method specifically comprising the following steps:

[0007] Step 1: Define the state space, action space and reward function of the deep deterministic policy gradient method, and train the agent of the deep deterministic policy gradient method based on the reward function;

[0008] The trained deep deterministic policy gradient method agent outputs the action of the current state autonomous driving vehicle, that is, obtains the behavioral decision result of the current state autonomous driving vehicle;

[0009] Step 2: Construct the autonomous driving vehicle scenario as a random hybrid automaton model and formally express the safety constraints of the autonomous driving vehicle scenario;

[0010] Step 3: Under the random hybrid automaton model and safety constraints, verify the safety of the behavior decision result in step 1;

[0011] If the safety verification is passed, the autonomous driving vehicle executes the behavioral decision result in step 1;

[0012] If the security verification fails, proceed to step 4;

[0013] Step 4: Feedback the safety verification results to the intelligent agent, use the intelligent agent to re-output the actions of the autonomous driving vehicle, and then return to execute step 3.

[0014] Further, the state space of the deep deterministic policy gradient method includes the relative position of the current autonomous driving vehicle and the vehicle in front, the speed of the autonomous driving vehicle, the speed of vehicles around the autonomous driving vehicle, the acceleration of the autonomous driving vehicle, the acceleration of vehicles around the autonomous driving vehicle, and the steering angle of the autonomous driving vehicle;

[0015] The action space of the deep deterministic policy gradient method includes the acceleration and steering angle of the autonomous vehicle.

[0016] Furthermore, the reward function of the deep deterministic policy gradient method is:

[0017] R=R1+R2+R3+R4+R5

[0018] Among them, R represents the reward function, R1 is the collision penalty, R2 is the penalty for deviating from the center line of the lane, R3 is the speeding penalty, R4 is the penalty for deviating from the expected speed within the speed limit, and R5 is the turning penalty.

[0019] Furthermore, the collision penalty R1 is:

[0020]

[0021] Furthermore, the penalty R2 for deviating from the lane centerline is:

[0022]

[0023] Among them, l c represents the centerline position of the lane where the autonomous driving vehicle is located, l h is the lateral position of the autonomous vehicle in the lane, σ is used to control the width of the Gaussian function, and abs(dis) represents l c With l h The distance between out is the distance threshold set, and e is the base of the natural logarithm.

[0024] Furthermore, the speeding penalty R3 is:

[0025]

[0026] Among them, v egorepresents the speed reached by the autonomous vehicle after executing the output action, v max Indicates the maximum speed limit, v min Indicates the minimum speed limit.

[0027] Furthermore, the penalty R4 for deviating from the desired speed within the speed limit is:

[0028]

[0029] Among them, v x Indicates v ego The component in the x direction, v y Indicates v ego The component in the y direction, v desire Indicates the expected speed.

[0030] Furthermore, the turning penalty R5 is:

[0031] R5=-s 2

[0032] Where s represents the steering angle in the output action.

[0033] Furthermore, the states of the autonomous driving vehicle defined by the random hybrid automaton model include normal driving, preparing to overtake, and changing lanes to overtake;

[0034] The system variables defined by the random hybrid automaton model include the detection range of the autonomous vehicle, the speed of the autonomous vehicle, the acceleration of the autonomous vehicle, and the distance between the autonomous vehicle and the obstacle;

[0035] The transition conditions between different states of the autonomous driving vehicle defined by the random hybrid automaton model are:

[0036] 1) When the relative distance between the autonomous driving vehicle and the vehicle ahead is equal to the threshold value b, the autonomous driving vehicle switches from the normal driving state to the state of preparing to overtake;

[0037] Where b represents the minimum safe distance;

[0038] 2) When the speed of the autonomous driving vehicle is within the normal speed range, the distance between the autonomous driving vehicle and the right vehicle is greater than or equal to the threshold value b, and the distance between the autonomous driving vehicle and the left vehicle is less than the threshold value b, the autonomous driving vehicle switches from the preparing to overtake state to the right overtaking state;

[0039] 3) When the speed of the autonomous driving vehicle is within the normal speed range, the distance between the autonomous driving vehicle and the left vehicle is greater than or equal to the threshold value b, and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold value b, the autonomous driving vehicle switches from the preparing to overtake state to the left overtaking state;

[0040] 4) When ①, ② or ③ are met, the autonomous driving vehicle switches from the overtaking preparation state to the normal driving state;

[0041] ① The relative distance between the autonomous driving vehicle and the vehicle in front is less than the threshold c, where c represents the minimum dangerous distance;

[0042] ② The speed of the autonomous vehicle is greater than the maximum speed limit;

[0043] ③ The distance between the autonomous driving vehicle and the left vehicle is less than the threshold b and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold b.

[0044] Furthermore, the safety constraints of the autonomous driving vehicle scenario include:

[0045] (1) Within the next 100 clock units, the probability that the distance between the autonomous driving vehicle and the vehicle ahead is always greater than 100 meters is recorded as P1, and P1 is greater than the threshold P'1;

[0046] (2) Within the next 100 clock units, the probability that the distance between the autonomous driving vehicle and the vehicle in front is less than or equal to 5 meters is recorded as P2, and P2 is less than the threshold value P'2;

[0047] (3) The probability that the autonomous vehicle successfully changes lanes to the left and overtakes within 10 clock units is recorded as P3, and P3 is greater than the threshold value P'3;

[0048] (4) The autonomous driving vehicle is finally in a normal driving state.

[0049] The beneficial effects of the present invention are:

[0050] The present invention proposes a behavior decision and safety verification method for ADS in an uncertain environment. By designing the state space, action space and reward function of the agent in the deep deterministic policy gradient method, and the reward function is designed with full consideration of multiple safety constraints in reality, the behavior decision performance of the agent for autonomous driving vehicles can be improved. By constructing the autonomous driving vehicle scenario as a random hybrid automaton model and formally expressing the safety constraints of the autonomous driving vehicle scenario through the method of the present invention, the accuracy of the behavior decision safety verification can be improved, thereby ensuring the safety of ADS behavior decision.

[0051] At the same time, the method of the present invention can solve the problem that environmental uncertainty can lead to poor system security through a deep deterministic policy gradient method, thereby ensuring the accuracy and security of the ADS decision results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a behavior decision and safety verification method of ADS in an uncertain environment of the present invention. DETAILED DESCRIPTION

[0053] Specific implementation method 1: Combination Figure 1 This embodiment describes a behavior decision and safety verification method for ADS in an uncertain environment, and the method specifically includes the following steps:

[0054] Step 1: Define the state space, action space and reward function of the deep deterministic policy gradient method, and train the agent of the deep deterministic policy gradient method based on the reward function;

[0055] The trained deep deterministic policy gradient method agent outputs the action of the current state autonomous driving vehicle, that is, obtains the behavioral decision result of the current state autonomous driving vehicle;

[0056] Step 2: Construct the autonomous driving vehicle scenario as a random hybrid automaton model, and use PCTL (Probabilistic Computation Tree Logic) to formally express the safety constraints of the autonomous driving vehicle scenario;

[0057] Step 3: Under the random hybrid automaton model and safety constraints, use UPPAAL to verify the safety of the behavior decision results in step 1;

[0058] If the safety verification is passed, the autonomous driving vehicle executes the behavioral decision result in step 1;

[0059] If the security verification fails, proceed to step 4;

[0060] Step 4: Feedback the safety verification results to the intelligent agent, use the intelligent agent to re-output the actions of the autonomous driving vehicle, and then return to execute step 3.

[0061] The deep deterministic policy gradient method constructs scenarios on the autonomous driving simulation tool CARLA and performs training to obtain decision-making behavior actions. Assume that the state of the vehicle at time t is s t , the action taken is a t ,θ μ and θ Q are all network parameters. The operations are as follows: First, the self-driving car interacts with the environment, and the Actor-online network gives the real-time action μ(s t |θ Q ) and pass it to the Critic-online network. The Critic-online network is based on the state s t and action a t Calculate Q(s t ,a t |θ Q), which is used to evaluate the value of the current state-behavior. t Execute action a t Then it is converted to state s t+1 . Actor-online network identification status t+1 The best follow-up to a t+1 , and μ'(s t+1 |θ μ' ) is passed to the Critic-target network. The Critic-target network generates Q'(s t+1 ,μ'(s t+1 |θ μ′ )|θ Q ), and the self-driving car receives environmental rewards to derive the target Q value.

[0062] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the state space of the deep deterministic policy gradient method includes the relative position of the current autonomous driving vehicle and the vehicle in front, the speed of the autonomous driving vehicle, the speed of vehicles around the autonomous driving vehicle, the acceleration of the autonomous driving vehicle, the acceleration of vehicles around the autonomous driving vehicle, and the steering angle of the autonomous driving vehicle;

[0063] The action space of the deep deterministic policy gradient method includes the acceleration and steering angle of the autonomous vehicle.

[0064] The other steps and parameters are the same as those in the first embodiment.

[0065] The deep deterministic policy gradient method is suitable for problems in continuous action space, so the action space is set to continuous action space, the acceleration range can be set to [-3, 3], and the acceleration unit is m / s 2 , the steering angle interval can be set to [-0.3, 0.3], the unit of the steering angle interval is degree, and the steering angle interval here refers to the steering angle interval based on the current state, that is, with the current state as the center, and the limit of no more than 0.3 degrees to the left and no more than 0.3 degrees to the right, each decision outputs an acceleration value within the acceleration interval and a steering angle value within the steering angle interval. When the scene is a highway scene, the value range of the action space can be appropriately expanded. When the scene is an urban road or a scene with complex traffic flow, an action space with a relatively small change interval can be selected.

[0066] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the reward function of the deep deterministic policy gradient method is:

[0067] R=R1+R2+R3+R4+R5

[0068] Among them, R represents the reward function, R1 is the collision penalty, R2 is the penalty for deviating from the center line of the lane, R3 is the speeding penalty, R4 is the penalty for deviating from the expected speed within the speed limit, and R5 is the turning penalty.

[0069] The other steps and parameters are the same as those in the first or second embodiment.

[0070] During the driving process of the self-driving car, collision is a very dangerous behavior, and the collision will have a serious impact on the surrounding vehicles. Therefore, the present invention gives a penalty R1 for the collision. According to the driving habits of human drivers, the vehicle should be driven in the center of the road as much as possible. If the self-driving car deviates from the center line of the lane too much, it may collide with the vehicle on the left or right. Therefore, the present invention gives a penalty R2. For road types with a large speed limit such as highway scenes, if the actual speed of the car exceeds the maximum speed limit, it is difficult to take countermeasures when encountering emergencies, so the speed of the self-driving car cannot exceed the maximum speed limit. Therefore, the present invention imposes a penalty R3 for speeding. In order for the self-driving car to reach the destination quickly, the speed is required to be as close to the speed limit as possible. Therefore, the present invention imposes a penalty R4 for deviation from the speed limit. Finally, the comfort of the passengers is considered. Frequent steering of the car or excessive steering angle will reduce the comfort of the passengers, so a penalty R5 is given for oversteering. The reward function set in the present invention can enable the self-driving car to accurately avoid surrounding obstacles and drive safely.

[0071] The scope of the reward function can be modified appropriately according to different scenarios. The thresholds and changes of speed and acceleration are different for different scenarios and traffic rules. The speed limit in the highway scenario is larger and changes more frequently, so it is necessary to punish changes in speed. The highway meets the requirements of high-speed driving, so penalties are imposed on low speeds to achieve the purpose of high-speed driving.

[0072] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the collision penalty R1 is:

[0073]

[0074] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0075] The occurrence of a collision in the present invention refers to a collision occurring after the autonomous driving vehicle executes the output action, and the non-occurrence of a collision refers to no collision occurring after the autonomous driving vehicle executes the output action.

[0076] Specific implementation method 5: This implementation method is different from any one of specific implementation methods 1 to 4 in that the lane centerline deviation penalty R2 is:

[0077]

[0078] Among them, l c represents the centerline position of the lane where the autonomous driving vehicle is located, l h is the lateral position of the autonomous driving vehicle in the lane, σ is used to control the width of the Gaussian function. The larger the influence range, the slower the reward decays when the vehicle deviates from the given conditions. In this invention, the value of σ is 10; abs(dis) represents l c With l h The distance between the autonomous vehicle and the center line of the current lane (i.e., the shortest distance between the autonomous vehicle and the center line of the current lane), l out is the set distance threshold (which can be set according to actual conditions), and e is the base of the natural logarithm.

[0079] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0080] The lateral position of the autonomous driving vehicle in the lane in the present invention refers to the position reached at the next moment after the autonomous driving vehicle performs the output action.

[0081] Specific implementation method 6: This implementation method is different from any one of the specific implementation methods 1 to 5 in that the speeding penalty R3 is:

[0082]

[0083] Among them, v ego represents the speed reached by the autonomous vehicle after executing the output action, v max Indicates the maximum speed limit, v min Indicates the minimum speed limit.

[0084] The other steps and parameters are the same as those in Specific Implementations 1 to 5.

[0085] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the penalty R4 for deviating from the expected speed within the speed limit is:

[0086]

[0087] Among them, v x Indicates v ego The component in the x direction, v y Indicates v ego The component in the t direction, v desire Indicates the expected speed.

[0088] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0089] The x direction and the y direction in the present invention refer to the x direction and the y direction of a spatial rectangular coordinate system.

[0090] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the turning penalty R5 is:

[0091] R5=-s 2

[0092] Where s represents the steering angle in the output action.

[0093] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0094] During the training process, the present invention calculates the reward function based on the state value reached after the autonomous driving vehicle executes the action output by the intelligent agent, and feeds back the reward function value to the intelligent agent to guide the next decision.

[0095] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the autonomous driving vehicle states defined by the random hybrid automaton model include normal driving (Norma_driving), preparing for overtaking (prepare_overtaking), and changing lanes to overtake (overtaking);

[0096] The system variables defined by the random hybrid automaton model include the detection range of the autonomous vehicle, the speed of the autonomous vehicle, the acceleration of the autonomous vehicle, and the distance between the autonomous vehicle and the obstacle;

[0097] The transition conditions between different states of the autonomous driving vehicle defined by the random hybrid automaton model are:

[0098] 1) When the relative distance between the autonomous driving vehicle and the vehicle ahead is equal to the threshold value b, the autonomous driving vehicle switches from the normal driving state to the state of preparing to overtake;

[0099] Wherein, b represents the minimum safe distance;

[0100] 2) When the speed of the autonomous vehicle is within the normal speed range (i.e., less than the maximum speed limit and greater than the minimum speed limit), the distance between the autonomous vehicle and the right vehicle is greater than or equal to the threshold value b, and the distance between the autonomous vehicle and the left vehicle is less than the threshold value b, the autonomous vehicle switches from the ready to overtake state to the right overtaking state;

[0101] 3) When the speed of the autonomous driving vehicle is within the normal speed range (i.e., less than the maximum speed limit and greater than the minimum speed limit), the distance between the autonomous driving vehicle and the left vehicle is greater than or equal to the threshold value b, and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold value b, the autonomous driving vehicle switches from the preparing to overtake state to the overtaking to the left state;

[0102] 4) When ①, ② or ③ are met, the autonomous driving vehicle switches from the overtaking preparation state to the normal driving state;

[0103] ① The relative distance between the autonomous driving vehicle and the vehicle in front is less than the threshold c, where c represents the minimum dangerous distance;

[0104] ② The speed of the autonomous vehicle is greater than the maximum speed limit;

[0105] ③ The distance between the autonomous driving vehicle and the left vehicle is less than the threshold b and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold b.

[0106] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0107] Specific implementation method 10: This implementation method is different from any one of specific implementation methods 1 to 9 in that the safety constraints of the autonomous driving vehicle scenario include:

[0108] (1) Within the next 100 clock units (one clock unit is 1 ms), the probability that the distance between the autonomous driving vehicle and the vehicle ahead is always greater than 100 meters is recorded as P1, and P1 is greater than the threshold P'1;

[0109] (2) Within the next 100 clock units, the probability that the distance between the autonomous driving vehicle and the vehicle in front is less than or equal to 5 meters is recorded as P2, and P2 is less than the threshold value P'2;

[0110] (3) The probability that the autonomous vehicle successfully changes lanes to the left and overtakes within 10 clock units is recorded as P3, and P3 is greater than the threshold value P'3;

[0111] (4) The autonomous driving vehicle is finally in a normal driving state.

[0112] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0113] The thresholds P'1, P'2 and P'3 in the present invention can be set according to actual conditions. For example, in relatively dangerous scenarios, the thresholds P'1 and P'3 can be set as large as possible, while the threshold P'2 can be set as small as possible to meet safe driving requirements.

[0114] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A behavior decision and safety verification method for ADS in an uncertain environment, characterized in that: The method specifically comprises the following steps: Step 1: Define the state space, action space and reward function of the deep deterministic policy gradient method, and train the agent of the deep deterministic policy gradient method based on the reward function; The trained deep deterministic policy gradient method agent outputs the action of the current state autonomous driving vehicle, that is, obtains the behavioral decision result of the current state autonomous driving vehicle; Step 2: Construct the autonomous driving vehicle scenario as a random hybrid automaton model and formally express the safety constraints of the autonomous driving vehicle scenario; Step 3: Under the random hybrid automaton model and safety constraints, verify the safety of the behavior decision result in step 1; If the safety verification is passed, the autonomous driving vehicle executes the behavioral decision result in step 1; If the security verification fails, proceed to step 4; Step 4: Feedback the safety verification results to the intelligent agent, use the intelligent agent to re-output the actions of the autonomous driving vehicle, and then return to execute step 3.

2. According to the method for behavior decision and safety verification of ADS in an uncertain environment in claim 1, it is characterized in that: The state space of the deep deterministic policy gradient method includes the relative position of the current autonomous driving vehicle and the vehicle in front, the speed of the autonomous driving vehicle, the speed of vehicles around the autonomous driving vehicle, the acceleration of the autonomous driving vehicle, the acceleration of vehicles around the autonomous driving vehicle, and the steering angle of the autonomous driving vehicle; The action space of the deep deterministic policy gradient method includes the acceleration and steering angle of the autonomous vehicle.

3. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 2, characterized in that: The reward function of the deep deterministic policy gradient method is: R=R1+R2+R3+R4+R5 Among them, R represents the reward function, R1 is the collision penalty, R2 is the penalty for deviating from the center line of the lane, R3 is the speeding penalty, R4 is the penalty for deviating from the expected speed within the speed limit, and R5 is the turning penalty.

4. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 3, characterized in that: The collision penalty R1 is:

5. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 4, characterized in that: The penalty R2 for deviation from the lane centerline is: Among them, l c represents the centerline position of the lane where the autonomous driving vehicle is located, l h is the lateral position of the autonomous vehicle in the lane, σ is used to control the width of the Gaussian function, and abs(dis) represents l c With l h The distance between out is the distance threshold set, and e is the base of the natural logarithm.

6. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 5, characterized in that: The speeding penalty R3 is: Among them, v ego represents the speed reached by the autonomous vehicle after executing the output action, v max Indicates the maximum speed limit, v min Indicates the minimum speed limit.

7. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 6, characterized in that: The penalty R4 for deviating from the desired speed within the speed limit is: Among them, v x Indicates v ego The component in the x direction, v y Indicates v ego The component in the y direction, v desire Indicates the expected speed.

8. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 7, characterized in that: The turning penalty R5 is: R5=-s 2 Where s represents the steering angle in the output action.

9. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 8, characterized in that: The states of the autonomous driving vehicle defined by the random hybrid automaton model include normal driving, preparing to overtake, and changing lanes to overtake; The system variables defined by the random hybrid automaton model include the detection range of the autonomous vehicle, the speed of the autonomous vehicle, the acceleration of the autonomous vehicle, and the distance between the autonomous vehicle and the obstacle; The transition conditions between different states of the autonomous driving vehicle defined by the random hybrid automaton model are: 1) When the relative distance between the autonomous driving vehicle and the vehicle ahead is equal to the threshold value b, the autonomous driving vehicle switches from the normal driving state to the state of preparing to overtake; Wherein, b represents the minimum safe distance; 2) When the speed of the autonomous driving vehicle is within the normal speed range, the distance between the autonomous driving vehicle and the right vehicle is greater than or equal to the threshold value b, and the distance between the autonomous driving vehicle and the left vehicle is less than the threshold value b, the autonomous driving vehicle switches from the preparing to overtake state to the right overtaking state; 3) When the speed of the autonomous driving vehicle is within the normal speed range, the distance between the autonomous driving vehicle and the left vehicle is greater than or equal to the threshold value b, and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold value b, the autonomous driving vehicle switches from the preparing to overtake state to the left overtaking state; 4) When ①, ② or ③ are met, the autonomous driving vehicle switches from the overtaking preparation state to the normal driving state; ① The relative distance between the autonomous driving vehicle and the vehicle in front is less than the threshold c, where c represents the minimum dangerous distance; ② The speed of the autonomous vehicle is greater than the maximum speed limit; ③ The distance between the autonomous driving vehicle and the left vehicle is less than the threshold b and the distance between the autonomous driving vehicle and the right vehicle is less than the threshold b.

10. The behavior decision and safety verification method of ADS in an uncertain environment according to claim 9, characterized in that: The safety constraints of the autonomous driving vehicle scenario include: (1) Within the next 100 clock units, the probability that the distance between the autonomous driving vehicle and the vehicle ahead is always greater than 100 meters is recorded as P1, and P1 is greater than the threshold P'1; (2) Within the next 100 clock units, the probability that the distance between the autonomous driving vehicle and the vehicle in front is less than or equal to 5 meters is recorded as P2, and P2 is less than the threshold value P'2; (3) The probability that the autonomous vehicle successfully changes lanes to the left and overtakes within 10 clock units is recorded as P3, and P3 is greater than the threshold value P'3; (4) The autonomous driving vehicle is finally in a normal driving state.

Images