A Safety-Efficiency Decision-Making Method for Autonomous Driving Based on Soft Risk Constraints

Abstract

This invention discloses a safety-efficiency decision-making method for autonomous driving based on risk soft constraints, belonging to the field of autonomous driving decision-making technology. It solves the problems of poor adaptability to extreme scenarios and lagging risk response in traditional autonomous driving decision-making methods in the prior art. This invention constructs an autonomous driving safety-efficiency decision-making architecture, inputting the environmental state from the environmental perception layer into a multi-expert strategy layer. Three experts are used to set the multi-expert strategy layer, which outputs candidate following actions. A risk adjustment mechanism is embedded in the loss function of each expert in the multi-expert strategy layer, resulting in the Actor network loss function for each expert. Based on the output candidate following actions, the basic reward function is used as the basic behavioral constraint for all experts, generating an autonomous driving safety-efficiency decision. This invention effectively improves the interpretability and robustness of the following strategy in autonomous driving decision-making methods and can be applied to autonomous driving decision-making.

Description

Technical Field

[0001] This invention relates to a safety-efficiency decision-making method for autonomous driving, and more particularly to a safety-efficiency decision-making method for autonomous driving based on risk soft constraints, belonging to the field of autonomous driving decision-making technology. Background Technology

[0002] The car-following decision system is the core module of longitudinal control in autonomous driving. Its core task is to achieve efficient vehicle driving while ensuring safety. As the complexity of open-world scenarios increases (such as extreme weather, perceived noise, and uncertainty in the behavior of traffic participants), the car-following decision system should be able to simultaneously meet the following three objectives: 1) Safety objective: The decisions made should enable the vehicle to avoid collisions and comply with traffic rules; 2) Efficiency objective: The decisions made should enable the vehicle to maintain a reasonable speed and minimize travel time; 3) Robustness objective: The decisions made should enable the vehicle to maintain stable performance under conditions of perception degradation or extreme scenarios.

[0003] Traditional car-following decision-making methods are divided into rule-based (such as IDM models) and data-driven (such as deep reinforcement learning) approaches. Rule-based methods offer strong interpretability but have limited generalization ability; data-driven methods are highly adaptable but lack explicit risk modeling and are prone to failure in extreme scenarios. Existing technologies have explored multi-objective optimization for car-following decisions: Wen et al. used IQ-Learn combined with SAC to simulate human driving behavior, discovering differences between human and machine car-following strategies, but failed to translate these differences into a response mechanism for autonomous driving, lacking dynamic adjustment capabilities; Zhang et al. proposed a "two-stage car-following strategy," embedding safety as a precondition into energy consumption optimization, which reduces the risk of rear-end collisions, but cannot achieve strategy reconstruction in emergency situations; Zhu et al. used trajectory fitting... Reward functions predict car-following actions, improving performance, but long-term predictions suffer from error accumulation and insufficient stability. Bennajeh et al. designed a two-layer fuzzy decision system that embeds "safety first" into speed control, improving safety and interpretability, but exhibiting poor adaptability in extreme scenarios such as rain, fog, and nighttime. Yao et al. introduced a MoE structure that integrates IDM rules and DRL strategies, improving efficiency in high-speed scenarios, but expert scheduling relies on heuristic conditions and cannot dynamically respond to risk signals. Liang et al. proposed a dynamic accumulative braking model that introduces dynamic evidence accumulation of approximation information, demonstrating human-like risk control advantages, but it has not been extended to multi-objective optimization scenarios.

[0004] Therefore, existing autonomous driving following decision-making methods have the following key shortcomings in real-world complex scenarios: 1) Most multi-strategy or hybrid expert-based methods do not introduce dynamic risk-driven mechanisms, leading to lag in strategy switching response or behavioral instability (e.g., inability to quickly switch from efficiency-first to safety-first when encountering sudden obstacles); 2) Under conditions of perception degradation (e.g., sensor failure) or extreme weather (e.g., rain, fog, nighttime), existing models lack robust strategy adjustment mechanisms, making it difficult to achieve stable following in time-varying environments. They do not explicitly model risk constraints such as severe weather and perception failures. Due to the lack of targeted adjustment mechanisms, safety and efficiency are compromised. 3) In multi-objective optimization, there is a conflict between safety and efficiency objectives. The traditional method of integrating multiple objectives through reward functions is prone to inconsistencies in policy gradient directions and difficulty in convergence of loss functions, leading to training instability and policy oscillations (such as frequent switching between acceleration and braking actions); 4) When facing emergencies (such as red light collisions or sudden obstacles), there is a lack of policy reconstruction and dynamic response capabilities. It is impossible to achieve an adaptive trade-off between safety and efficiency in extreme scenarios. Risk constraints are not embedded in the optimization process in a differentiable form. It is impossible to explain the policy's response logic to risks through gradient directions, which is not conducive to debugging and verification.

[0005] In summary, a safety-efficiency decision-making method for autonomous driving based on soft risk constraints is needed. Summary of the Invention

[0006] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0007] In view of this, in order to solve the problems of poor adaptability to extreme scenarios and delayed risk response in traditional autonomous driving decision-making methods in the prior art, this invention provides an autonomous driving safety-efficiency decision-making method based on risk soft constraints.

[0008] The technical solution is as follows: A safety-efficiency decision-making method for autonomous driving based on soft risk constraints, comprising the following steps:

[0009] S1. Construct an autonomous driving safety-efficiency decision-making architecture, which includes an environmental perception layer, a multi-expert policy layer, a λ-Risk soft constraint layer, and a reward function layer;

[0010] Specifically: The scene state input to the environmental perception layer includes the vehicle's speed, distance to the vehicle in front, lane departure, weather conditions, and perception confidence.

[0011] S2. Input the environmental state of the environmental perception layer into the multi-expert strategy layer, adopt three complementary deep reinforcement learning experts, set up the multi-expert strategy layer, and output candidate following actions based on the scene state through the multi-expert strategy layer.

[0012] S3. Embed the risk adjustment mechanism into the loss function of each expert in the multi-expert strategy layer, adjust the weight of risk in the loss function through the risk adjustment factor, obtain the Actor network loss function of each expert, and complete the setting of the λ-Risk soft constraint layer.

[0013] S4. Based on the candidate following actions output, combined with collision penalties, driving efficiency rewards and comfort rewards are calculated. The obtained basic reward function is used as the basic behavioral constraint for all experts to complete the reward function layer setting.

[0014] S5. Generate autonomous driving safety-efficiency decisions based on the autonomous driving safety-efficiency decision architecture set in steps S2-S5.

[0015] Furthermore, step S2 includes the following steps:

[0016] S21. Employing the DDPG strategy approach as a security expert;

[0017] In S11, the training process of the DDPG policy method consists of an Actor-Critic dual network.

[0018] Loss function of Critic network Represented as:

[0019]

[0020]

[0021] in, This indicates the number of training samples in a mini-batch. , They represent the first The state and actions of each sample For the first The next state of a sample. Indicates an immediate reward. As a discount factor, , For state Deterministic policy network, For the first The target action value for each sample The action value function output by the Critic network;

[0022] The update process of the Actor network is represented as follows:

[0023]

[0024] in, , These represent the policy network parameters of the Critic and Actor networks, respectively. Representing state Deterministic policy network; Represents the function Regarding the status ,action gradient, Indicates the network parameters of the policy Regarding its output action, that is The gradient;

[0025] The equivalent objective of the DDPG strategy method is to maximize Value, i.e., minimizing the negative The value is used to obtain the objective function of the DDPG strategy method. ;

[0026] Objective function of the DDPG strategy method Represented as:

[0027]

[0028] in, This indicates taking the expected value. This represents the experience replay pool. Represents the action value function. Indicates the first The state of each sample For state Deterministic policy network;

[0029] S22. Employ the SAC strategy as an adaptive expert;

[0030] In S12, the objective function of the SAC strategy method is... Represented as:

[0031]

[0032] in, This indicates taking the expected value. Indicates the network parameters of the policy Given the action distribution, This represents the entropy regularization weight coefficient. , Indicates action ,state The entropy value under;

[0033] Entropy regularization weight coefficient Adaptive adjustments are made to obtain the updated entropy regularization weight coefficients. ;

[0034] Updated entropy regularization weight coefficients Represented as:

[0035]

[0036] in, The temperature represents the entropy learning rate. Represents the target entropy. Describe the objective function Regarding the entropy regularization weight coefficient The partial derivatives, Indicates action ,state The entropy value under;

[0037] The goal of the SAC strategy method is to maximize... The sum of the value and the policy entropy is converted into the Actor network loss Actor-Loss in a minimization form, yielding the loss function of the SAC policy method. ;

[0038] Loss function of SAC strategy method Represented as:

[0039]

[0040] in, Indicates action ,state The action value function below;

[0041] S23. Employing the PPO strategy as an efficiency expert;

[0042] In S13, the loss function of the PPO strategy method Represented as:

[0043]

[0044] Where min represents the minimum value. This represents the probability ratio between the current policy and the old policy. , Indicates the current policy in state Take action below The probability, Indicates the old strategy in state Take action below The probability, Representing state-action pairs The advantage estimate, clip (·) indicates that the policy ratio is clipped within the interval. Functions between This represents the pruning threshold for policy updates. Represents all time steps in a batch The empirical average calculated from the sample.

[0045] Furthermore, in S3, a risk cost function is set based on the key factors that lead to high driving risks in actual driving scenarios. The risk cost was calculated. ;

[0046] Risk Cost Represented as:

[0047]

[0048] in, Indicates whether a collision has occurred. This indicates the degree of lateral deviation of the vehicle relative to the center line of its lane at the current moment. Indicates the distance between your vehicle and the nearest obstacle;

[0049] Based on risk cost Construct a risk assessment function It is directly embedded into the policy optimization process based on the Actor-Critic dual network as a regularization term of the Actor network to participate in the loss calculation, and a risk regularization weight coefficient is introduced. We obtain the loss function of the Actor network;

[0050] Actor network loss function Represented as:

[0051]

[0052] Among them, risk assessment function Indicates the current time step The observed risk cost Take the expected value. For risk regularization weighting coefficients, Let represent the original Actor network loss function for each;

[0053] Refer to the Actor network loss function Risk assessment function By introducing the Actor network loss functions of three experts in the multi-expert strategy layer, we obtain the Actor network loss functions of the safety expert, the adaptation expert, and the efficiency expert.

[0054] Loss function of Actor network for security experts Represented as:

[0055]

[0056] Loss function of Actor network for adaptive experts Represented as:

[0057]

[0058] Loss function of Actor network for efficiency experts Represented as:

[0059]

[0060] Among them, different experts use differentiated risk regularization weighting coefficients. To achieve stylistic adjustments to risk control capabilities, safety-oriented experts are given higher weights, adaptation-oriented experts are given medium weights, and efficiency-oriented experts are given weaker adjustments.

[0061] Furthermore, in S4, a collision penalty is set... Calculated driving efficiency bonus and comfort rewards Construct the basic reward function at the reward function layer. As a fundamental behavioral constraint for all experts;

[0062] Basic reward function Represented as:

[0063]

[0064] in, Indicates a collision penalty. This indicates a reward for improved driving efficiency. Indicates a comfort reward;

[0065] Collision Penalty Represented as:

[0066]

[0067] Driving efficiency reward Represented as:

[0068]

[0069] in, Indicates the current speed of the main vehicle. This indicates the forward travel progress of the vehicle in the current step. This represents the task-oriented weights, where a and b represent the weights of speed and distance traveled, respectively.

[0070] Comfort reward Represented as:

[0071]

[0072]

[0073] in, This represents the rate of change of vehicle acceleration. express The penalty weight, This represents the total number of time steps in the current simulation round. For the first The acceleration of the vehicle in step, For the first The acceleration of the vehicle.

[0074] The beneficial effects of this invention are as follows: This invention aims to propose an autonomous driving safety-efficiency decision-making method based on risk soft constraints, construct a dynamic risk-driven mechanism, and realize real-time response and stable switching of the strategy; it can maintain the robustness and adaptability of following decision-making in extreme scenarios; by decoupling safety and efficiency objectives, it avoids conflicts in multi-objective optimization and improves training stability; it quantifies risk constraints and embeds them into the strategy optimization process, realizing a dynamic trade-off between safety and efficiency.

[0075] This invention addresses the shortcomings of existing technologies in the following ways: 1) It constructs a regularized loss function that integrates risk weight coefficients λ-Risk. By differentiating λ values, it achieves functional differentiation of safety-oriented, adaptive, and efficiency-oriented experts. The strategy can be adjusted in real time according to the scenario risk (e.g., automatically increasing the λ value in rainy weather to strengthen safety preference). Continuously differentiable risk constraints (e.g., extreme weather, perception failure, red light collision) are incorporated into the strategy optimization in the form of gradients, realizing explicit risk modeling and dynamic response, thus solving the problem of response lag in traditional methods; 2) It designs a multi-expert collaborative architecture (DDPG, SAC, PPO), which achieves functional differentiation of experts (safety-oriented, adaptive, and efficiency-oriented) by differentiating λ-Risk values, thereby improving scenario adaptability; 3) It decouples safety and efficiency goals: safety is embedded in the loss function through a risk cost term, while efficiency and comfort are modeled through a reward function, avoiding gradient conflicts and improving training stability; 4) In open-world extreme scenarios, safety is regulated through the risk cost term in the loss function, while efficiency and comfort are modeled through a reward function, avoiding gradient conflicts and achieving an adaptive trade-off between safety and efficiency, thus improving the robustness and stability of the follow-the-carry strategy;

[0076] Furthermore, this invention achieves the goals of decoupling safety and efficiency by relying on training stability, avoiding gradient conflicts caused by reward function integration. Experiments show that the training convergence speed is improved by 30%, and the policy oscillation frequency is reduced by 40%. The explicit modeling of extreme scenario risk constraints in this invention reduces the rear-end collision rate by 65% ​​in the CARLA rain and fog scenario, while the traffic efficiency only decreases by 10% (existing methods have an efficiency decrease of more than 30%). This invention has strong interpretability, and the physical meaning of the risk cost function and λ-Risk is clear. The policy's response logic to risk can be explained through the gradient direction (e.g., when λ increases, the gradient proportion of braking action increases). Attached Figure Description

[0077] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0078] Figure 1 This is a flowchart illustrating a safety-efficiency decision-making method for autonomous driving based on soft risk constraints.

[0079] Figure 2 This is a schematic diagram of an embodiment of a risk-based soft constraint-based autonomous driving safety-efficiency decision-making method. Detailed Implementation

[0080] To make the technical solutions and advantages of the embodiments of the present invention clearer, the exemplary embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0081] refer to Figures 1-2 This embodiment describes a safety-efficiency decision-making method for autonomous driving based on soft risk constraints, specifically including the following steps:

[0082] S1. Construct an autonomous driving safety-efficiency decision-making architecture, which includes an environmental perception layer, a multi-expert policy layer, a λ-Risk soft constraint layer, and a reward function layer;

[0083] Specifically, the scene states input to the environmental perception layer include the vehicle's speed, distance to the vehicle in front, lane departure, weather conditions (such as friction coefficient in rainy weather), and perception confidence (such as camera failure probability).

[0084] S2. Input the environmental state of the environmental perception layer into the multi-expert strategy layer, adopt three complementary deep reinforcement learning experts, set up the multi-expert strategy layer, and output candidate following actions based on the scene state through the multi-expert strategy layer.

[0085] S3. Embed the risk adjustment mechanism into the loss function of each expert in the multi-expert strategy layer, adjust the weight of risk in the loss function through the risk adjustment factor, obtain the Actor network loss function of each expert, complete the setting of the λ-Risk soft constraint layer, and realize cross-expert differentiated weight control.

[0086] S4. Based on the candidate following actions output, combined with collision penalties, driving efficiency rewards and comfort rewards are calculated. The obtained basic reward function is used as the basic behavioral constraint for all experts to complete the reward function layer setting.

[0087] S5. Generate autonomous driving safety-efficiency decisions based on the autonomous driving safety-efficiency decision architecture set in steps S2-S5.

[0088] Specifically, in this implementation, the expert model can be replaced by TD3 (Twin Delayed DDPG) as the security expert, thereby improving the stability of the strategy in extreme scenarios.

[0089] Using A2C (Advantage Actor-Critic) instead of PPO as the efficiency expert is suitable for scenarios with limited computing resources;

[0090] Risk cost function extension: More risk indicators (such as pedestrian crossing probability and traffic light recognition confidence) can be introduced, and the weights of α, β, and γ can be dynamically adjusted through an attention mechanism to enhance scene adaptability.

[0091] λ-Risk adaptive mechanism: It can dynamically optimize the value of λ based on reinforcement learning (such as using λ as a learnable parameter), instead of taking a fixed value, to further improve the generalization ability in multiple scenarios.

[0092] λ-Risk: Risk regularization weight coefficient, used to control the impact of risk cost on strategy optimization, and to achieve a dynamic trade-off between safety and efficiency;

[0093] Following-the-vehicle decision: The core module of longitudinal control in autonomous driving, whose task is to maintain an appropriate distance and speed, and to achieve a dynamic balance between safety, efficiency and comfort in complex scenarios;

[0094] DDPG (Deep Deterministic Policy Gradient): A deterministic policy method applicable to continuous action spaces, used in this invention as a safety-oriented expert model;

[0095] SAC (Soft Actor-Critic): A soft actor-critic algorithm, a policy-abandoning algorithm based on the maximum entropy principle, used as an adaptive expert model in this invention;

[0096] PPO (Proximal Policy Optimization): The nearest neighbor policy optimization algorithm improves training stability by limiting the policy update magnitude. In this invention, it is used as an efficiency-oriented expert model.

[0097] IQ-Learn: An algorithm based on inverse reinforcement learning. Its core is to infer the reward function by imitating the behavior of human experts (such as human driving data), and then train the agent's policy based on the reward function. IQ-Learn is good at learning implicit decision preferences from human demonstrations and is often used to improve the "human-likeness" of autonomous driving policies.

[0098] MoE structure (Mixture of Experts): an integrated learning framework consisting of multiple "expert models" (responsible for handling specific sub-tasks) and a "gating network" (responsible for dynamically selecting or fusing expert outputs based on input). In autonomous driving, decision-making tasks for different scenarios (such as highways and traffic jams) can be assigned to different experts. The gating network selects the optimal expert strategy based on real-time road conditions, improving adaptability to complex scenarios.

[0099] IDM (Intelligent Driver Model): A classic rule-based car-following model that defines the acceleration / deceleration behavior of a vehicle based on physical rules and mathematical formulas (such as "the closer to the vehicle in front, the greater the deceleration"). Its core is to simulate human driving habits through preset parameters (such as safe distance threshold and expected speed). It has strong interpretability but limited generalization ability and is often used as a benchmark or supplement to data-driven methods.

[0100] DRL (Deep Reinforcement Learning) strategy: a decision-making strategy that combines deep learning and reinforcement learning. It uses deep neural networks (such as CNN and MLP) to fit the mapping relationship between environmental state and action, and at the same time uses the trial and error mechanism of reinforcement learning to optimize network parameters in order to achieve adaptive decision-making in complex environments. In autonomous driving, DRL strategy can interact with the simulation environment and autonomously learn complex behaviors such as following and changing lanes.

[0101] Actor-Critic architecture: A classic dual-network architecture in reinforcement learning. The Actor generates actions (such as throttle opening in autonomous driving) and outputs a policy (a mapping from state to action). The Critic evaluates the value of the actions generated by the Actor (such as the cumulative reward the action can bring) and outputs a value function. The Critic guides the Actor to optimize its policy by evaluating the quality of actions, while the Actor collects data by executing actions to update the Critic. Working together, they ultimately achieve iterative improvement of the policy.

[0102] Actor-Loss: A loss function used to optimize the Actor network, measuring the difference between the current policy and the "optimal policy". Its core is to guide Actors to increase the probability of selecting "high-value actions" and decrease the probability of selecting "low-value actions" through value assessments provided by the Critic (such as an advantage function). In autonomous driving scenarios, Actor-Loss can drive the policy towards optimization in a "safe and efficient" direction (e.g., reducing the probability of collisions and increasing the probability of reasonable acceleration).

[0103] Furthermore, step S2 includes the following steps:

[0104] S21. Set the DDPG strategy approach to a security expert;

[0105] In S11, DDPG (Deep Deterministic Policy Gradient) is a deterministic policy method applicable to continuous action spaces. The training process of the DDPG policy method consists of an Actor-Critic dual network.

[0106] Loss function of Critic network Represented as:

[0107]

[0108]

[0109] in, This indicates the number of training samples in a mini-batch. , They represent the first The state and actions of each sample For the first The next state of a sample. Indicates an immediate reward. This is a discount factor used to control the degree of decay in future rewards. , For state Deterministic policy network, For the first The target action value for each sample The action value function output by the Critic network;

[0110] The update process of the Actor network is represented as follows:

[0111]

[0112] in, , These represent the policy network parameters of the Critic network and the Actor network, respectively. Representing state Deterministic policy network, Represents the function Regarding the action gradient, Indicates the network parameters of the policy Regarding its output action, that is The gradient;

[0113] The equivalent objective of the DDPG strategy method is to maximize Value, i.e., minimizing the negative The value is used to obtain the loss function of the DDPG strategy method. ;

[0114] Loss function of DDPG strategy method Represented as:

[0115]

[0116] in, This indicates taking the expected value. This represents the experience playback pool (experience sampling distribution). Represents the action value function. Indicates the first The state of each sample For state Deterministic policy network;

[0117] S22. Set the SAC strategy approach as an adaptive expert;

[0118] In S12, SAC (Soft Actor-Critic) is a policy-determining algorithm based on the maximum entropy principle. It maximizes the entropy of the policy distribution while optimizing the reward to enhance exploratory behavior. The objective function of the SAC policy method is... Represented as:

[0119]

[0120] in, This indicates taking the expected value. Indicates the network parameters of the policy Given the action distribution, This represents the entropy regularization weighting coefficient, which is used to control the exploration intensity. , Indicates action ,state The entropy value under;

[0121] Entropy regularization weight coefficient Adaptive adjustments are made to obtain the updated entropy regularization weight coefficients. ;

[0122] Updated entropy regularization weight coefficients Represented as:

[0123]

[0124] in, The temperature represents the entropy learning rate. This represents the target entropy, which is typically a negative action space dimension. Describe the objective function Regarding the entropy regularization weight coefficient The partial derivative of , which is used to guide . The adaptive adjustment allows the entropy of the policy output to gradually approach the target entropy. This allows for dynamic adjustment of strategies in an exploratory manner. Indicates action ,state The entropy value under;

[0125] The goal of the SAC strategy method is to maximize... The sum of the value and the policy entropy is converted into the Actor network loss Actor-Loss in a minimization form, yielding the loss function of the SAC policy method. ;

[0126] Loss function of SAC strategy method Represented as:

[0127]

[0128] in, Indicates action ,state The action value function below;

[0129] S23. Set the PPO strategy approach as an efficiency-oriented expert;

[0130] In S13, Proximal Policy Optimization (PPO) is one of the most widely used policy gradient methods. Its core feature is the introduction of a policy update constraint mechanism to control the magnitude of each policy iteration, thereby improving the stability and robustness of the training process. The loss function of the PPO policy method... Represented as:

[0131]

[0132] Where min represents the minimum value. This represents the probability ratio between the current policy and the old policy. , Indicates the current policy in state Take action below The probability, Indicates the old strategy in state Take action below The probability, Representing state-action pairs The advantage estimate, clip (·) indicates that the policy ratio is clipped within the interval. Functions between This represents the pruning threshold for policy updates, typically set to 0.2. Represents all time steps in a batch The empirical average calculated from the sample.

[0133] Specifically, since the DDPG strategy method lacks policy entropy adjustment, the training process is more sensitive to environmental feedback, the policy update speed is fast, and it has strong responsiveness. Combined with its good adaptability to continuous control, this invention sets it as a safety expert, which is suitable for scenarios with many vehicles and people, complex scenes, and high-risk potential conflicts, such as collision avoidance at urban intersections, and strengthens its obstacle avoidance ability and safety guarantee role under risk-driven decision-making.

[0134] Given that the SAC strategy method can flexibly adjust the control strategy under different risk levels, it can achieve a dynamic balance between safety and efficiency in medium-risk scenarios such as medium-speed driving, congested lane changing, and urban interaction. Therefore, this invention uses it as an adaptive expert, which is activated when a dynamic trade-off between exploration and stability is required.

[0135] Since the PPO strategy method only allows small updates to the strategy parameters in each round, it has good convergence stability and efficient strategy optimization capabilities. Its output focuses more on improving traffic efficiency. Therefore, this invention uses it as an efficiency expert, which is suitable for low-risk scenarios with sparse traffic flow and clear road rules, in order to achieve fast passage and efficient driving strategies.

[0136] Furthermore, in S3, a risk cost function is set based on the key factors that lead to high driving risks in actual driving scenarios. The risk cost was calculated. ;

[0137] Risk Cost Represented as:

[0138]

[0139] in, Indicates whether a collision has occurred. This indicates the degree of lateral deviation of the vehicle relative to the center line of its lane at the current moment. Indicates the distance between your vehicle and the nearest obstacle;

[0140] Risk assessment function It is composed of continuous variables (minimum vehicle distance, lane departure degree, etc.) linearly combined with the ReLU function, and the whole is piecewise differentiable, which satisfies the basic requirements for gradient propagation in reinforcement learning training. Therefore, based on risk cost... Construct a risk assessment function It is directly embedded into the policy optimization process based on the Actor-Critic dual network as a regularization term of the Actor network to participate in the loss calculation, and a risk regularization weight coefficient is introduced. The Actor network loss function is obtained. Based on this design, this invention further constructs a cross-model general training framework, embedding a unified form of risk adjustment mechanism into the Actor network loss function of the three types of expert models, thereby achieving consistent risk guidance for the optimization process of DDPG, SAC and PPO strategies.

[0141] Actor network loss function Represented as:

[0142]

[0143] Among them, risk assessment function Indicates the current time step The observed risk cost Take the expected value. These are risk regularization weights, used to control the weights in the Actor network loss function. The degree of impact of "risk cost" on strategy optimization varies depending on the expert model's strategy preferences, reflecting the differences in the level of security concern among various experts. Let represent the original Actor network loss function for each;

[0144] Refer to the Actor network loss function Risk assessment function By introducing the Actor network loss functions of three experts in the multi-expert strategy layer, we obtain the Actor network loss functions of the safety expert, the adaptation expert, and the efficiency expert.

[0145] Loss function of Actor network for security experts Represented as:

[0146]

[0147] Loss function of Actor network for adaptive experts Represented as:

[0148]

[0149] Loss function of Actor network for efficiency experts Represented as:

[0150]

[0151] Among them, different experts use differentiated risk regularization weighting coefficients. To achieve a stylistic adjustment of risk control capabilities, security experts are given higher weighting, i.e. =0.8, emphasizing the significant suppression of potential risks to generate safe, conservative, and stable decision-making behavior; adaptive experts are assigned a medium weight, i.e. =0.4, balancing efficiency and safety to form a moderate strategy preference; efficiency-oriented experts, on the other hand, only introduce a weak adjustment, i.e. =0.1, which maximizes efficiency under the premise of basic risk control. Through this differentiated setting, the three types of experts exhibit behaviors with obvious style differentiation under a unified training framework.

[0152] Specifically, this invention introduces a risk-aware regularization mechanism, incorporating security as a soft constraint in the policy update process into the Actor network loss function, Actor-Loss, and using a risk adjustment factor. To achieve differentiated weighting across experts, this invention uses a multi-dimensional advanced safety index as a risk cost function at the loss level. We introduce this as a reward function, while retaining only efficiency rewards, comfort rewards, and the most basic collision penalties in the reward function part, thereby achieving direct gradient constraint control of safety.

[0153] Furthermore, in S4, a collision penalty is set... Calculated driving efficiency bonus and comfort rewards Construct the basic reward function at the reward function layer. As a fundamental behavioral constraint for all experts;

[0154] Basic reward function Represented as:

[0155]

[0156] in, Indicates a collision penalty. This indicates a reward for improved driving efficiency. Indicates a comfort reward;

[0157] Collision Penalty Represented as:

[0158]

[0159] Driving efficiency reward Represented as:

[0160]

[0161] in, Indicates the current speed of the main vehicle. This indicates the forward travel progress of the vehicle in the current step (estimated by distance increment). This represents the task-oriented weights, where a and b represent the weights of speed and distance traveled, respectively.

[0162] Comfort reward Represented as:

[0163]

[0164]

[0165] in, The rate of change of acceleration of a vehicle is represented by the first derivative of the acceleration per unit time. express The penalty weight is set to 2. This represents the total number of time steps in the current simulation round. For the first The acceleration of the vehicle in step, For the first The acceleration of the vehicle.

[0166] Specifically, refer to Table 1 for the setting of efficiency and speed reward weight parameters in various expert models. a and b are set according to the expert type. Among them, safety-type experts prioritize stability and risk avoidance, consider efficiency factors less, and maintain low motivation to move forward. Adaptive experts balance efficiency and safety, and take forward distance as the main indicator of task completion. Efficiency-type experts emphasize optimal accessibility, and therefore give higher weights to progress and speed.

[0167] Table 1

[0168]

[0169] The reward function layer and the λ-Risk soft constraint layer work together to decouple the safety and efficiency of the autonomous driving strategy. The reward function layer is designed based on basic traffic rules and is constructed through basic state variables such as collision, speed, and comfort. The three types of expert models proposed in this invention mainly model different safety behavior styles from the perspective of risk concern. Therefore, efficiency or comfort objectives are not explicitly included in the strategy optimization process. Correspondingly, factors such as vehicle speed and acceleration are uniformly modeled by the reward feedback function as basic behavioral constraints for all experts, which enhances the system's adaptability to conventional traffic rules while ensuring training stability.

[0170] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A safety-efficiency decision-making method for autonomous driving based on risk soft constraints, characterized in that, Includes the following steps: S1. Construct an autonomous driving safety-efficiency decision-making architecture, which includes an environmental perception layer, a multi-expert strategy layer, a λ-Risk soft constraint layer, and a reward function layer; Specifically: The scene state input to the environmental perception layer includes the vehicle's speed, distance to the vehicle in front, lane departure, weather conditions, and perception confidence. S2. Input the environmental state of the environmental perception layer into the multi-expert strategy layer, adopt three complementary deep reinforcement learning experts, set up the multi-expert strategy layer, and output candidate following actions based on the scene state through the multi-expert strategy layer. S3. Embed the risk adjustment mechanism into the loss function of each expert in the multi-expert strategy layer, adjust the weight of risk in the loss function through the risk adjustment factor, obtain the Actor network loss function of each expert, and complete the setting of the λ-Risk soft constraint layer. S4. Based on the candidate following actions output, combined with collision penalties, driving efficiency rewards and comfort rewards are calculated. The obtained basic reward function is used as the basic behavioral constraint for all experts to complete the reward function layer setting. S5. Generate autonomous driving safety-efficiency decisions based on the autonomous driving safety-efficiency decision architecture set in steps S2-S4; S2 includes the following steps: S21. Employing the DDPG strategy approach as a security expert; In S21, the training process of the DDPG policy method consists of an Actor-Critic dual network. Loss function of Critic network Represented as: in, This indicates the number of training samples in a mini-batch. , They represent the first The state and actions of each sample For the first The next state of a sample. Indicates an immediate reward. As a discount factor, , For state Deterministic policy network, For the first The target action value for each sample The action value function output by the Critic network; The update process of the Actor network is represented as follows: in, , These represent the policy network parameters of the Critic and Actor networks, respectively. Representing state Deterministic policy network; Represents the function Regarding the status ,action gradient, Indicates the network parameters of the policy Regarding its output action, that is The gradient; The equivalent objective of the DDPG strategy method is to maximize Value, i.e., minimizing the negative The value is used to obtain the objective function of the DDPG strategy method. ; Objective function of the DDPG strategy method Represented as: in, This indicates taking the expected value. This represents the experience replay pool. Represents the action value function. Indicates the first The state of each sample For state Deterministic policy network; S22. Employ the SAC strategy as an adaptive expert; In S22, the objective function of the SAC strategy method is... Represented as: in, This indicates taking the expected value. Indicates the network parameters of the policy Given the action distribution, This represents the entropy regularization weight coefficient. , Indicates action ,state The entropy value under; Entropy regularization weight coefficient Adaptive adjustments are made to obtain the updated entropy regularization weight coefficients. ; Updated entropy regularization weight coefficients Represented as: in, The temperature learning rate represents the entropy. Represents the target entropy. Describe the objective function Regarding the entropy regularization weight coefficient The partial derivatives, Indicates action ,state The entropy value under; The goal of the SAC strategy method is to maximize... The sum of the value and the policy entropy is converted into the Actor network loss Actor-Loss in a minimization form, yielding the loss function of the SAC policy method. ; Loss function of SAC strategy method Represented as: in, Indicates action ,state The action value function below; S23. Employing the PPO strategy as an efficiency expert; In S23, the loss function of the PPO strategy method Represented as: Where min represents the minimum value. This represents the probability ratio between the current policy and the old policy. , Indicates the current policy in state Take action below The probability, Indicates the old strategy in state Take action below The probability, Representing state-action pairs The advantage estimate, clip (·) indicates that the policy ratio is clipped within the interval. Functions between This represents the pruning threshold for policy updates. Represents all time steps in a batch The empirical average calculated from the sample.

2. The autonomous driving safety-efficiency decision-making method based on risk soft constraints according to claim 1, characterized in that, In step S3, a risk cost function is set based on the key factors that lead to high driving risks in actual driving scenarios. The risk cost was calculated. ; Risk Cost Represented as: in, Indicates whether a collision has occurred. This indicates the degree of lateral deviation of the vehicle relative to the center line of its lane at the current moment. Indicates the distance between your vehicle and the nearest obstacle; Based on risk cost Construct a risk assessment function It is directly embedded into the policy optimization process based on the Actor-Critic dual network as a regularization term of the Actor network to participate in the loss calculation, and a risk regularization weight coefficient is introduced. We obtain the loss function of the Actor network; Actor network loss function Represented as: Among them, risk assessment function Indicates the current time step The observed risk cost Take the expected value. For risk regularization weighting coefficients, Represent the original Actor network loss function for each; Refer to the Actor network loss function Risk assessment function By introducing the Actor network loss functions of three experts in the multi-expert strategy layer, we obtain the Actor network loss functions of the safety expert, the adaptation expert, and the efficiency expert. Loss function of Actor network for security experts Represented as: Loss function of Actor network for adaptive experts Represented as: Loss function of Actor network for efficiency experts Represented as: Among them, different experts use differentiated risk regularization weighting coefficients. To achieve stylistic adjustments to risk control capabilities, safety-oriented experts are given higher weights, adaptation-oriented experts are given medium weights, and efficiency-oriented experts are given weaker adjustments.

3. The autonomous driving safety-efficiency decision-making method based on risk soft constraints according to claim 2, characterized in that, In S4, the collision penalty is set. Calculated driving efficiency bonus and comfort rewards Construct the basic reward function at the reward function layer. As a fundamental behavioral constraint for all experts; Basic reward function Represented as: in, Indicates a collision penalty. This indicates a reward for improved driving efficiency. Indicates a comfort reward; Collision Penalty Represented as: Driving efficiency reward Represented as: in, Indicates the current speed of the main vehicle. This indicates the forward travel progress of the vehicle in the current step. This represents the task-oriented weights, where a and b represent the weights of speed and distance traveled, respectively. Comfort reward Represented as: in, This represents the rate of change of vehicle acceleration. express The penalty weight, This represents the total number of time steps in the current simulation round. For the first The acceleration of the vehicle in step, For the first The acceleration of the vehicle.

Images