Reinforcement learning automatic driving decision control method based on Lyapunov function

Abstract

The invention discloses a reinforcement learning automatic driving decision control method based on a Lyapunov function, and relates to the technical field of automatic driving automobile safety control. The method comprises the steps that driving state information of a self-vehicle and driving state information of surrounding environment vehicles are received, the driving state information of the self-vehicle comprises the position, the speed, the acceleration and the course angle, and the driving state information of the surrounding environment vehicles comprises the position, the speed, the course angle and lane line information of the environment vehicles relative to the self-vehicle; and a decision control model of the automatic driving task based on reinforcement learning is constructed, and the decision control model takes the driving state information of the vehicle and the surrounding environment vehicles as output and takes the optimal automatic driving strategy as output. According to the method, the reinforcement learning strategy updating process is constrained through the stability condition proved by theories, the unification of safety and decision-making efficiency is realized in dynamic scenes such as highway lane changing and emergency obstacle avoidance, and the method has a good application prospect for training the safety and strategy stability of automatic driving by adopting reinforcement learning.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving vehicle safety control technology, and specifically to a reinforcement learning autonomous driving decision control method based on Lyapunov function. Background Art

[0002] Reinforcement learning has emerged as a promising approach for end-to-end autonomous driving. However, due to its black-box nature, ensuring the safety of the training process and the stability of the learned policy remain open challenges. For example, due to environmental uncertainty, how can behavior lead to unstable vehicle dynamics, rendering the policy unusable? Furthermore, the reward function that defines the training objective can lead to unsafe driving behavior if safety-related components are improperly weighted.

[0003] Constrained policy optimization is a classic approach to ensuring safety in reinforcement learning. It defines safety constraints (e.g., collision-free distance) and constrains the learned policy using the Lagrangian duality principle. Another approach, such as trust region policy optimization, strikes a balance between exploring new policies and developing better-performing policies from early training stages. A key challenge in constrained policy optimization is that constrained Markov processes typically involve representations of future expectations. The Q-values ​​used in reinforcement learning are typically estimated via neural networks, meaning that the accuracy of these estimates is crucial to ensuring that constraints are satisfied. While trust region policy optimization adds constraint capabilities, its on-policy nature can lead to low sample efficiency, particularly in high-complexity tasks requiring extensive interaction data, where convergence to local optima is a common problem. Existing research has performed well in high-dimensional training environments, but it remains difficult to determine explicit policy objectives in multi-objective optimization driving tasks. Consequently, few feasible methods exist that can definitively guarantee that reinforcement learning will not fully satisfy constraints without adopting alternative policies. To this end, we propose a reinforcement learning decision-making and control method for autonomous driving based on Lyapunov functions. Summary of the Invention

[0004] The purpose of the present invention is to provide a reinforcement learning autonomous driving decision control method based on Lyapunov function to solve the problems raised in the above background technology.

[0005] According to a first aspect of the present invention, to achieve the above-mentioned purpose, the present invention provides the following technical solution: a reinforcement learning autonomous driving decision control method based on Lyapunov function, comprising the following steps:

[0006] Receive driving status information of the vehicle and surrounding vehicles, where the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle;

[0007] Construct a decision-making and control model for autonomous driving tasks based on reinforcement learning. The decision-making and control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision-making and control model includes a decision training network and a decision execution and optimization network.

[0008] The decision training network uses the SAC algorithm to make acceleration, deceleration and steering decisions, and specifically includes four sub-networks:

[0009] Actor network, used to output control actions and perform policy training;

[0010] Critic-Reward network, used to estimate the value of training data trajectories;

[0011] Critic-Cost network, used to estimate the constraint cost of travel speed;

[0012] Critic-Lyapunov network, used to estimate the collision risk stability of the actor output strategy;

[0013] The decision-making execution and optimization network aims to maximize traffic efficiency and keep the vehicle in the right lane, with collision as the constraint cost. It generates acceleration and front wheel steering angle control data for the controlled vehicle in real time. After execution by the underlying vehicle, the feedback data collected by the sensors is transmitted to the intelligent agent. Through continuous iteration and optimization of various network parameters, a safe and stable autonomous driving strategy is ultimately learned.

[0014] The trained and optimized decision control model is configured on the vehicle to execute autonomous driving decisions.

[0015] Furthermore, the driving state information of the vehicle itself is obtained through a vehicle speed sensor and CAN protocol communication, and the driving state information of the surrounding vehicles is obtained through a laser radar and a millimeter wave radar, namely the Beidou positioning system.

[0016] Furthermore, when constructing a decision-making and control model for autonomous driving tasks based on reinforcement learning, it is necessary to consider vehicle collision constraints and speed constraints. Based on the Lagrange duality principle, the vehicle collision constraint and speed constraint problems are transformed into unconstrained problems, and the collision and traffic efficiency are solved as follows:

[0017] (31) A constrained Markov decision process (CMDP) is established, and the six-tuple of the standard Markov decision process is extended to (S, A, P, R, C, S′), where S is the state space, A is the action space, P: S × A → Δ (S) is the transition dynamics, R: S × A → R is the reward function, and C = c i:S×A→R(i=1,…,k) is the constraint function, S′ is the state at the next moment; the constraint optimization objective is expressed as:

[0018]

[0019] Where π * Denote as the optimal strategy, Π c It is represented as the allowed strategy set, s represents the state, a represents the action, γ represents the discount coefficient, r(s t ,a t ) represents the state s at time t t and action a t The reward value when express expectations;

[0020] The constraints are:

[0021]

[0022] In the formula represents the constraint target, c i (s t ,a t ) represents state action s t ,a t The i-th cost under represents the threshold of the i-th cost;

[0023] (32) The constrained optimization problem is handled by the Lagrange duality principle, which facilitates the construction of the augmented objective function:

[0024]

[0025] Where L(π,λ) represents the Lagrangian augmentation function, λ=[λ1,…,λ k ] T ≥0 is the Lagrange multiplier;

[0026] (33) The constrained optimization problem in secure reinforcement learning is equivalently transformed into a minimax optimization problem, which can be expressed as follows:

[0027]

[0028] The solution is obtained through an alternating optimization method, which consists of two alternating stages:

[0029] (33.1) Strategy optimization, update the strategy parameter π to maximize the Lagrangian L(π,λ) while keeping λ unchanged;

[0030] (33.2) Multiplier update, adjust λ according to the following rules i :

[0031]

[0032] Where η α represents the learning rate;

[0033] (34) Adopting the trust region strategy optimization method, the Kullback-Leibler divergence is used to constrain the strategy update;

[0034] D KL (π||π old )≤δ

[0035] Where D KL (π||π old ) is represented by the new strategy π and the old strategy π old The KL divergence between them, δ is a predefined threshold.

[0036] Furthermore, when using the SAC algorithm for decision training, interaction data is collected and the Actor network is updated using the SAC algorithm:

[0037] (41) Collecting driving data of intelligent agent interactions as the initial learning buffer experience pool of the expert experience library;

[0038] (42) Based on the Bellman optimal iterative time difference objective of the Markov model, the target value estimated by the Critic-Reward network is calculated:

[0039]

[0040] Where R t represents the reward at time t, γ represents the discount coefficient, Q r (s t ,a t ) is the Q value estimate of the target Reward network, α is the entropy coefficient, π t+1 represents the strategy at time t+1;

[0041] (43) The Reward main network calculates the Q corresponding to the action-state data collected in the experience pool r value;

[0042] (44) According to the target reward Q target Value, calculate the partial loss of the boosting strategy:

[0043]

[0044] Furthermore, the construction of the decision training network is as follows:

[0045] (51) A critic-reward network (i.e., a Q-value function approximator) is constructed to evaluate the long-term rewards of state-action pairs and provide value guidance for policy optimization. This network also uses a multi-layer neural network structure, with the input being the combination of state s and action a, and the output being the corresponding Q-value estimate. The details are as follows:

[0046] (51) Construct a critic-reward network to estimate the value of the training data trajectory as follows:

[0047] (51.1) In each initial buffer trajectory, the parameters of the Critic-Cost network estimation are updated. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model:

[0048]

[0049] Among them, C t represents the cost value at time t;

[0050] (51.2) The time difference algorithm is used to update the Cost network, and the loss function is the current Q c Value and target value The mean square error of :

[0051]

[0052] Where, Λ cost Represents the loss function value of the Cost network, and the state s obeys μ π Strategy distribution;

[0053] (52) Construct a Critic-Lyapunov network to estimate the stability of the collision risk of the Actor network output strategy, as follows:

[0054] (52.1) Based on the Lyapunov stability criterion, we improve the stability of the reinforcement learning algorithm during training. Specifically, we:

[0055] (52.2) In the data-driven autonomous driving task environment, the system model is unknown and depends on sampled data. The stability goal is to design a controller, namely, a policy π, so that the system state converges to an equilibrium point, the state s t The expected cost of strategy π in is defined as:

[0056]

[0057] The instantaneous cost is represents the expected distance from the next state to the goal, and the goal is to minimize this value over time, ideally satisfying:

[0058]

[0059] Where c π (s t ) represents the state s under strategy π t Cost;

[0060] (52.3) Establish a Critic-Lyapunov estimation network and update the parameters of the Critic-Lyapunov network estimation in each initial buffer trajectory. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model:

[0061]

[0062] Among them, C t represents the cost value at time t, Q l represents the Q value estimated by the target Lyapunov network;

[0063] (52.4) The time difference algorithm is used to update the Lyapunov network, and the loss function is the current L c Value and target value The mean square error of :

[0064]

[0065] (53) Construct an Actor network, whose function is to input and output continuous actions (including longitudinal acceleration and lateral front wheel angle) based on the current state to achieve the autonomous driving control goal. The network adopts a parameterized strategy model, which is generally a multi-layer feedforward neural network in structure. The input includes the state feature information of the vehicle and the surrounding vehicles at the current moment, such as position, speed, heading angle, acceleration, and lane line coding; the output is the action distribution parameters (such as mean and variance) of the control command, and the action is generated by random sampling. The specific implementation process includes:

[0066] (53.1) Using the maximum entropy reinforcement learning-based policy update method in the SAC algorithm, the Actor network aims to maximize the weighted expected cumulative reward and policy entropy;

[0067] (53.2) By sampling the current policy action and combining the feedback from the Critic-Reward, Critic-Cost, and Critic-Lyapunov networks, the target loss function of the policy is calculated as follows:

[0068]

[0069] Among them, π θ(a|s) represents the action probability density of the current policy output, α is the entropy temperature coefficient, Δ represents the experience pool, and θ represents the actor network neural network parameters;

[0070] (53.3) By using the original SAC to update the Actor network, combined with the Critic Cost and Critic Lyapunov networks, the safe and stable policy gradient loss function is:

[0071]

[0072] where α L is the Lyapunov adjustment coefficient;

[0073] (51) Construct a critic-reward network to estimate the value of the training data trajectory as follows:

[0074] (51.1) In each initial buffer trajectory, the parameters of the Critic-Cost network estimation are updated. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model:

[0075]

[0076] Among them, C t represents the cost value at time t;

[0077] (51.2) The time difference algorithm is used to update the Cost network, and the loss function is the current Q c Value and target value The mean square error of :

[0078]

[0079] Where, Λ cost Represents the loss function value of the Cost network, and the state s obeys μ π Strategy distribution;

[0080] (52) Construct a Critic-Lyapunov network to estimate the stability of the collision risk of the Actor network output strategy, as follows:

[0081] (52.2) Based on the Lyapunov stability criterion, we can improve the stability of the reinforcement learning algorithm during training. Specifically, we can:

[0082] (52.3) In the data-driven autonomous driving task environment, the system model is unknown and depends on sampled data. The stability goal is to design a controller, namely, a policy π, so that the system state converges to an equilibrium point, the state s t The expected cost of strategy π in is defined as:

[0083]

[0084] The instantaneous cost is represents the expected distance from the next state to the goal, and the goal is to minimize this value over time, ideally satisfying:

[0085]

[0086] Where c π (s t ) represents the state s under strategy π t Cost;

[0087] (52.4) Establish a Critic-Lyapunov estimation network and update the parameters of the Critic-Lyapunov network estimation in each initial buffer trajectory. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model:

[0088]

[0089] Among them, C t represents the cost value at time t, Q l represents the Q value estimated by the target Lyapunov network;

[0090] (52.5) ​​The time difference algorithm is used to update the Lyapunov network, and the loss function is the current L c Value and target value The mean square error of :

[0091]

[0092] (53) Construct an Actor network, whose function is to input and output continuous actions based on the current state to achieve the autonomous driving control goal. The Actor network adopts a parameterized strategy model and is structurally a multi-layer feedforward neural network. The Actor network input includes the state feature information of the current vehicle and the surrounding vehicles, and the output is the action distribution parameter of the control instruction. The action is generated by random sampling. The specific implementation process includes:

[0093] (53.1) Using the maximum entropy reinforcement learning-based policy update method in the SAC algorithm, the Actor network aims to maximize the weighted expected cumulative reward and policy entropy;

[0094] (53.2) By sampling the current policy action and combining the feedback from the Critic-Reward, Critic-Cost, and Critic-Lyapunov networks, the target loss function of the policy is calculated as follows:

[0095]

[0096] Among them, π θ (a|s) represents the action probability density of the current policy output, α is the entropy temperature coefficient, Δ represents the experience pool, and θ represents the actor network neural network parameters;

[0097] (53.3) By using the original SAC to update the Actor network, combined with the Critic Cost and Critic Lyapunov networks, the safe and stable policy gradient loss function is:

[0098]

[0099] where α L is the Lyapunov adjustment coefficient.

[0100] Furthermore, the Reward network, Cost network, and Lyapunov network are collaboratively optimized to ensure security and stability, including:

[0101] (61) formulate the RL problem as a constrained optimization task to balance safety and stability:

[0102]

[0103] Where c threshold and l threshold are predefined thresholds for security and stability, respectively. L(s,a) is the Lyapunov function for evaluating the stability of the strategy, and its expected value follows the state distribution S and the strategy π(·│S) distribution.

[0104] (62) is transformed into the augmented Lagrangian objective through the primal-dual method:

[0105]

[0106] Where λ1 and λ2 are Lagrange multipliers, ∈1 and ∈2 are slack variables;

[0107] (63) Strategy π θ , parameterized as θ, and updated by gradient descent:

[0108]

[0109] The Lagrange multiplier is adjusted to:

[0110]

[0111] where i=1,2, η1 and η2 are the policy parameters and the learning rate multiplier respectively.

[0112] Furthermore, the maximum entropy reinforcement learning framework is integrated to update the Actor network through the SAC algorithm, including:

[0113] (71) Maximize the weighted sum of expected cumulative reward and policy entropy:

[0114]

[0115] where α represents the entropy temperature coefficient that controls the trade-off between exploration and exploitation, Indicates the state is s t The entropy of the policy π.

[0116] (72)Convert the constrained optimization problem into an augmented Lagrangian objective:

[0117]

[0118] (73) The target network parameters are updated as follows:

[0119]

[0120] Where τ represents the learning rate, ω, φ, and ψ are the parameters of the Reward, Cost, and Lyapunov networks, respectively.

[0121] According to a second aspect of the present invention, the present invention provides a reinforcement learning autonomous driving decision control system based on a Lyapunov function, which is used to implement the reinforcement learning autonomous driving decision control method based on a Lyapunov function described in the first aspect, comprising:

[0122] A status information acquisition module is used to receive driving status information of the vehicle and surrounding vehicles, wherein the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle;

[0123] The decision control training module is used to build a decision control model for autonomous driving tasks based on reinforcement learning. The decision control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision control model includes a decision training network and a decision execution and optimization network.

[0124] The control output module is used to configure the trained and optimized decision control model on the vehicle to execute autonomous driving decisions.

[0125] According to the third aspect of the present invention, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor loads and executes the computer program, it adopts a reinforcement learning autonomous driving decision control method based on Lyapunov function described in the first aspect.

[0126] According to the fourth aspect of the present invention, the present invention provides a storage medium containing computer-executable instructions, characterized in that the computer-executable instructions, when executed by a computer processor, are used to execute a reinforcement learning autonomous driving decision control method based on Lyapunov function described in the first aspect.

[0127] The present invention has at least the following beneficial effects:

[0128] 1) This paper proposes a reinforcement learning method based on Lyapunov stability, which realizes the stability training method of vehicles in autonomous driving tasks.

[0129] 2) This paper introduces the Lyapunov stability criterion in control theory into the data-driven stability judgment logic, effectively characterizing the stability of the data collected by reinforcement learning training.

[0130] 3) Based on the constrained Markov modeling process, the present invention models the collision constraints and traffic efficiency constraints of the autonomous driving task, uses the Lagrangian duality principle to deal with the reinforcement learning optimization problem with constraints, and combines it with the Lyapunov stability constraint to effectively ensure the stability of the training iterative optimization.

[0131] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0132] Figure 1 4 is a system flow chart of the method of the present invention.

[0133] Figure 2 This is a schematic diagram of the Lyapunov function reinforcement learning security and stability algorithm of the present invention.

[0134] Figure 3 This is a simulation effect diagram comparing the training process effects of the present invention and the algorithm without stable constraints. DETAILED DESCRIPTION

[0135] The following will be combined with the accompanying drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present disclosure.

[0136] Example 1:

[0137] The present embodiment is a reinforcement learning autonomous driving decision control method based on Lyapunov stability and Lyapunov function, which can be referred to Figure 1 . The computing system based on the invention includes a surrounding vehicle information acquisition module, a lane change decision and motion planning module, and a safe collision avoidance control module, wherein the surrounding vehicle information acquisition module includes the vehicle information obtained based on sensor measurement and CAN protocol communication, as well as other vehicle status information obtained through laser radar, millimeter wave radar, etc.; constructs a safety assessment network based on Lyapunov theory, and quantifies the stability boundary of the driving strategy through the potential energy function; establishes a dual constraint mechanism within the soft actor-critic (SAC) framework, wherein the critic network guides the strategy exploration through the reward signal, and the Lyapunov network simultaneously imposes stability constraints; designs a dynamic safety verification module, combines the vehicle kinematic model with the Lyapunov stability theorem, and generates a safety certification boundary for the strategy output; adopts a hybrid optimization algorithm to synchronously update the participant strategy parameters and the Lyapunov network parameters to ensure that the control instructions meet the requirements of optimality and asymptotic stability at the same time. The present invention realizes the unity of safety and decision-making efficiency in dynamic scenarios such as highway lane changing and emergency obstacle avoidance by constraining the reinforcement learning strategy update process through the theoretically proven stability condition, and has good application prospects for the safety and strategy stability of autonomous driving trained by reinforcement learning.

[0138] See also Figure 1-Figure 3 The present invention provides a technical solution: a reinforcement learning autonomous driving decision control method based on Lyapunov function, comprising the following steps:

[0139] Receive driving status information of the vehicle and surrounding vehicles, where the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle;

[0140] Construct a decision-making and control model for autonomous driving tasks based on reinforcement learning. The decision-making and control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision-making and control model includes a decision training network and a decision execution and optimization network.

[0141] The decision training network uses the SAC algorithm to make acceleration, deceleration and steering decisions, and specifically includes four sub-networks:

[0142] Actor network, used to output control actions and perform policy training;

[0143] Critic-Reward network, used to estimate the value of training data trajectories;

[0144] Critic-Cost network, used to estimate the constraint cost of travel speed;

[0145] Critic-Lyapunov network, used to estimate the collision risk stability of the actor output strategy;

[0146] The decision-making execution and optimization network aims to maximize traffic efficiency and keep the vehicle in the right lane, with collision as the constraint cost. It generates acceleration and front wheel steering angle control data for the controlled vehicle in real time. After execution by the underlying vehicle, the feedback data collected by the sensors is transmitted to the intelligent agent. Through continuous iteration and optimization of various network parameters, a safe and stable autonomous driving strategy is ultimately learned.

[0147] The trained and optimized decision control model is configured on the vehicle to execute autonomous driving decisions.

[0148] Next, the technical solution of the present invention is further described with reference to specific embodiments:

[0149] 1. Obtain status information of the vehicle and surrounding vehicles, including:

[0150] Based on the vehicle's speed sensor, CAN protocol communication, and other vehicle status information obtained through lidar, millimeter-wave radar, etc., combined with positioning systems such as Beidou, the vehicle and surrounding vehicle status information is obtained, including the position, speed, acceleration, heading angle, etc. of the vehicle, as well as the position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle.

[0151] 2. Build an autonomous driving task decision-making and control training framework based on reinforcement learning algorithms, specifically including:

[0152] Build a reinforcement learning decision control training framework and use the SAC algorithm for acceleration, deceleration, and steering decision training. The framework includes:

[0153] An Actor network that outputs control actions (acceleration and front wheel angle) and performs policy training;

[0154] A Critic-Reward network to estimate the value of training data trajectories;

[0155] A Critic-Cost network to estimate the constraint cost of travel speed;

[0156] A Critic-Lyapunov network to estimate the collision risk stability of the actor output policy;

[0157] With the mission objectives of maximizing traffic efficiency and staying in the right lane, and collision risk as the cost constraint, the framework in step 2 generates control data in real time. After execution by the underlying vehicle, the feedback data collected by the sensors is transmitted to the intelligent agent. Through continuous iterative optimization of various network parameters, a safe and stable autonomous driving strategy is ultimately learned. The details are as follows:

[0158] Handling vehicle collision and speed constraint problems through constrained reinforcement learning algorithms:

[0159] (1) Establish a constrained Markov decision process (CMDP) and extend the standard Markov decision process to a sextuple (S, A, P, R, C, S′), where S is the state space, A is the action space, P: S × A → Δ(S) is the transition dynamics, R: S × A → R is the reward function, and C = c i :S×A→R(i=1,…,k) is the constraint function, S′ is the state at the next moment; the constraint optimization objective is expressed as:

[0160]

[0161] Where π * Denote as the optimal strategy, Π c It is represented as the allowed strategy set, s represents the state, a represents the action, γ represents the discount coefficient, r(s t ,a t ) represents the state s at time t t and action a t The reward value when express expectations;

[0162] The constraints are:

[0163]

[0164] In the formula represents the constraint target, c i (s t ,a t ) represents state action s t ,a t The i-th cost under represents the threshold of the i-th cost;

[0165] (2) The constrained optimization problem is handled by the Lagrange duality principle, which facilitates the construction of the augmented objective function:

[0166]

[0167] Where L(π,λ) represents the Lagrangian augmentation function, λ=[λ1,…,λ k ]T ≥0 is the Lagrange multiplier;

[0168] (3) Convert the constrained optimization problem into a minimax problem:

[0169]

[0170] Solve via alternating optimization:

[0171] Policy optimization: fix λ and update the policy parameter π to maximize L(π,λ);

[0172] Multiplier update: adjust λ according to the following rules i :

[0173]

[0174] (4) Adopt the trust region strategy optimization method and update the strategy through the Kullback-Leibler divergence constraint:

[0175] D KL (π||π old )≤δ

[0176] Where δ is a predefined threshold to ensure iterative monotonicity and constraint satisfaction.

[0177] When using the SAC algorithm for decision training, collect interaction data and update the Actor network through the SAC algorithm:

[0178] (5) Collect driving data of agent interactions and build an initial experience pool (buffer);

[0179] (6) Based on the Bellman optimal iterative time difference objective of the Markov model, the target value estimated by the Critic-Reward network is calculated:

[0180]

[0181] Where R t represents the reward at time t, γ represents the discount coefficient, Q r (s t ,a t ) is the Q value estimate of the target Reward network, α is the entropy coefficient, π t+1 represents the strategy at time t+1;

[0182] (7) The Reward main network calculates the Q corresponding to the action-state data collected in the experience pool r value;

[0183] (8) According to the target reward Q target Value, calculate the partial loss of the boosting strategy:

[0184]

[0185] Integrate Cost network to ensure security:

[0186] (9) Calculate the target value of Cost network estimation:

[0187]

[0188] Among them, c t represents the cost value at time t, Q c (s t ,a t ) is the Q value estimation of the target Cost network.

[0189] (10) Use the time difference algorithm to update the Cost network, and the loss function is the current Q c Value and target value The mean square error of :

[0190]

[0191] Integrate Lyapunov network to ensure stability:

[0192] (11) Calculate the target value estimated by the Lyapunov network:

[0193]

[0194] Among them, Q L (s t ,a t ) is the value estimate of the target Lyapunov network.

[0195] (12) The time difference algorithm is used to update the Lyapunov network, and the loss function is the current L c Value and target value The mean square error of :

[0196]

[0197] (14) By sampling the current policy action and combining the feedback from the Critic-Reward, Critic-Cost, and Critic-Lyapunov networks, the target loss function of the policy is calculated as follows:

[0198]

[0199] Among them, π θ(a|s) represents the action probability density output by the current strategy, α is the entropy temperature coefficient, Δ represents the experience pool, and θ represents the neural network parameters of the actor network;

[0200] The Reward network, Cost network, and Lyapunov network are collaboratively optimized to ensure security and stability:

[0201] (15) Formulate the reinforcement learning problem as a constrained optimization task:

[0202]

[0203] The constraints are:

[0204]

[0205] Where c threshold and l threshold are predefined thresholds for security and stability, respectively;

[0206] (16) is transformed into the augmented Lagrangian objective through the primal-dual method:

[0207]

[0208] Where λ1 and λ2 are Lagrange multipliers, ∈1 and ∈2 are slack variables;

[0209] (17) Strategy π θ (parameterized as θ) and updated via gradient descent:

[0210]

[0211] The Lagrange multiplier is adjusted to:

[0212] λ i ←max(0,λ i +η λ (E s～s,a～π(│s) [c]-∈ i ))

[0213] where i=1,2, η1 and η2 are the policy parameters and the learning rate multiplier respectively.

[0214] Integrate the maximum entropy reinforcement learning framework and update the Actor network through the SAC algorithm:

[0215] (18) Maximize the weighted sum of expected cumulative reward and policy entropy:

[0216]

[0217] where α represents the entropy temperature coefficient that controls the trade-off between exploration and exploitation, Indicates the state is s t The entropy of the policy π.

[0218] (19)Convert the constrained optimization problem into an augmented Lagrangian objective:

[0219]

[0220] (20) The target network parameters are updated as follows:

[0221]

[0222] Where τ represents the learning rate, ω, φ, and ψ are the parameters of the Reward, Cost, and Lyapunov networks, respectively.

[0223] like Figure 2 As shown in the figure, the algorithm architecture consists of two main implementation parts. First, the controlled vehicle needs to collect vehicle status information and surrounding environment information based on its own sensors, which constitutes the necessary environmental data component for training. Second, there is the reinforcement learning agent learning part, which includes the actor network and the critic network, the experience cache, and the loss function calculation and update part.

[0224] In order to achieve practical application effects, this embodiment can use a virtual simulation model for training. Through the vehicle and environment modeling of the simulation software, the trained strategy can be deployed on the real vehicle using the method of the embodiment to achieve the purpose of safe passage in the actual traffic environment, such as Figure 3 The figure shows the simulation effect diagram of the stable reinforcement learning training process of this embodiment. According to the proposed method, the strategy can be continuously and stably stabilized within a certain range during training, reducing the probability of accidents and achieving stable strategy learning.

[0225] To summarize, the present invention first obtains the position, speed and other status information of the vehicle and surrounding vehicles and lane line information; secondly, it constructs a reinforcement learning decision control training framework based on the SAC algorithm, which includes an Actor network, a Critic-Reward network, etc., with the goal of maximizing traffic efficiency and driving on the right, and training strategies with collision as the constraint cost; then it uses the Lagrange duality principle and other principles to deal with constraint problems, introduces the Lyapunov stability criterion to improve the stability of the strategy, integrates the cost network and the Lyapunov network to ensure safety and stability, and also integrates the maximum entropy reinforcement learning framework to promote exploration; finally, by iteratively optimizing the network parameters, it generates a safe and stable autonomous driving decision control strategy. Therefore, the present invention combines Lyapunov stability with reinforcement learning to take into account both safety and effectiveness in autonomous driving decision control.

[0226] Example 2:

[0227] This embodiment provides a reinforcement learning autonomous driving decision control system based on a Lyapunov function, which is used to implement the reinforcement learning autonomous driving decision control method based on a Lyapunov function described in Example 1, including:

[0228] A status information acquisition module is used to receive driving status information of the vehicle and surrounding vehicles, wherein the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle;

[0229] The decision control training module is used to build a decision control model for autonomous driving tasks based on reinforcement learning. The decision control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision control model includes a decision training network and a decision execution and optimization network.

[0230] The control output module is used to configure the trained and optimized decision control model on the vehicle to execute autonomous driving decisions.

[0231] Example 3:

[0232] The present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores a computer program capable of running on the processor. When the processor loads and executes the computer program, the above-mentioned reinforcement learning autonomous driving decision control method based on Lyapunov function is adopted.

[0233] It should be noted that the terminal device can be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device can also include input and output devices, network access devices and buses, etc.

[0234] Furthermore, the processor may adopt a central processing unit (CPU). Of course, depending on the actual usage, other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. may also be adopted. The general-purpose processor may adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0235] Example 4:

[0236] The present invention provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute the above-mentioned reinforcement learning autonomous driving decision control method based on Lyapunov function.

[0237] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that computer-readable medium includes but is not limited to the above-mentioned components.

[0238] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0239] For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. When an element is referred to as being "assembled on", "installed on", "fixed on" or "set on" another element, it can be directly on the other element or there can be a central element. When an element is considered to be "connected" to another element, it can be directly connected to the other element or there can be a central element at the same time. The terms "vertical", "horizontal", "up", "down", "left", "right" and similar expressions used herein are for illustrative purposes only and are not intended to be the only embodiment.

[0240] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0241] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present disclosure. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Claims

1. A reinforcement learning autonomous driving decision control method based on Lyapunov function, characterized in that: The following steps are involved: Receive driving status information of the vehicle and surrounding vehicles, where the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle; Construct a decision-making and control model for autonomous driving tasks based on reinforcement learning. The decision-making and control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision-making and control model includes a decision training network and a decision execution and optimization network. The decision training network uses the SAC algorithm to make acceleration, deceleration and steering decisions, and specifically includes four sub-networks: Actor network, used to output control actions and perform policy training; Critic-Reward network, used to estimate the value of training data trajectories; Critic-Cost network, used to estimate the constraint cost of travel speed; Critic-Lyapunov network, used to estimate the collision risk stability of the actor output strategy; The decision-making execution and optimization network aims to maximize traffic efficiency and keep the vehicle in the right lane, with collision as the constraint cost. It generates acceleration and front wheel steering angle control data for the controlled vehicle in real time. After execution by the underlying vehicle, the feedback data collected by the sensors is transmitted to the intelligent agent. Through continuous iteration and optimization of various network parameters, a safe and stable autonomous driving strategy is ultimately learned. The trained and optimized decision control model is configured on the vehicle to execute autonomous driving decisions.

2. The Lyapunov function-based reinforcement learning autonomous driving decision-making and control method according to claim 1, characterized in that: The driving state information of the vehicle itself is obtained through a vehicle speed sensor and CAN protocol communication, and the driving state information of the surrounding vehicles is obtained through a laser radar and a millimeter wave radar, namely the Beidou positioning system.

3. The Lyapunov function-based reinforcement learning autonomous driving decision-making and control method according to claim 2, characterized in that: When building a decision-making and control model for autonomous driving tasks based on reinforcement learning, it is necessary to consider vehicle collision constraints and speed constraints. Based on the Lagrangian duality principle, the vehicle collision constraint and speed constraint problems are transformed into unconstrained problems, and the collision and traffic efficiency are solved as follows: (31) A constrained Markov decision process (CMDP) is established, and the six-tuple of the standard Markov decision process is extended to (S, A, P, R, C, S′), where S is the state space, A is the action space, P: S × A → Δ (S) is the transition dynamics, R: S × A → R is the reward function, and C = c i :S×A→R(i=1,…,k) is the constraint function, S′ is the state at the next moment; the constraint optimization objective is expressed as: Where π * Denote as the optimal strategy, Π c It is represented as the allowed strategy set, s represents the state, a represents the action, γ represents the discount coefficient, r(s t ,a t ) represents the state s at time t t and action a t The reward value when express expectations; The constraints are: In the formula represents the constraint target, c i (s t ,a t ) represents state action s t ,a t The i-th cost under represents the threshold of the i-th cost; (32) The constrained optimization problem is handled by the Lagrange duality principle, which facilitates the construction of the augmented objective function: Where L(π,λ) represents the Lagrangian augmentation function, λ=[λ1,…,λ k ] T ≥0 is the Lagrange multiplier; (33) The constrained optimization problem in secure reinforcement learning is equivalently transformed into a minimax optimization problem, which can be expressed as follows: The solution is obtained through an alternating optimization method, which consists of two alternating stages: (33.1) Strategy optimization, update the strategy parameter π to maximize the Lagrangian L(π,λ) while keeping λ unchanged; (33.2) Multiplier update, adjust λ according to the following rules i : Where η α represents the learning rate; (34) Adopting the trust region strategy optimization method, the Kullback-Leibler divergence is used to constrain the strategy update; D KL (p||p old )≤δ Where D KL (π||π old ) is represented by the new strategy π and the old strategy π old The KL divergence between them, δ is a predefined threshold.

4. The Lyapunov function-based reinforcement learning autonomous driving decision-making and control method according to claim 3, characterized in that: When using the SAC algorithm for decision training, interaction data is collected and the Actor network is updated by combining three evaluation networks: (41) Collecting driving data of intelligent agent interactions as the initial learning buffer experience pool of the expert experience library; (42) Based on the Bellman optimal iterative time difference objective of the Markov model, the target value estimated by the Critic-Reward network is calculated: Where R t represents the reward at time t, γ represents the discount coefficient, Q r (s t ,a t ) is the Q value estimate of the target Reward network, α is the entropy coefficient, π t+1 represents the strategy at time t+1; (43) The Reward main network calculates the Q corresponding to the action-state data collected in the experience pool r value; (44) The target Q value is constructed using the Bellman optimality equation, that is, the target network Generate TD target: Among them, r is the reward, γ is the discount factor, π θ For the current strategy; (45) Use the mean square error loss function for parameter optimization:

5. The Lyapunov function-based reinforcement learning autonomous driving decision-making and control method according to claim 4, characterized in that: The construction of the decision training network is as follows: (51) Construct a critic-reward network to estimate the value of the training data trajectory as follows: (51.1) In each initial buffer trajectory, the parameters of the Critic-Cost network estimation are updated. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model: Among them, C t represents the cost value at time t; (51.2) The time difference algorithm is used to update the Cost network, and the loss function is the current Q c Value and target value The mean square error of : Where, Λ cost Represents the loss function value of the Cost network, and the state s obeys μ π Strategy distribution; (52) Construct a Critic-Lyapunov network to estimate the stability of the collision risk of the Actor network output strategy, as follows: (52.1) Based on the Lyapunov stability criterion, we improve the stability of the reinforcement learning algorithm during training. Specifically, we: (52.2) In the data-driven autonomous driving task environment, the system model is unknown and depends on sampled data. The stability goal is to design a controller, namely, a policy π, so that the system state converges to an equilibrium point, the state s t The expected cost of strategy π in is defined as: The instantaneous cost is represents the expected distance from the next state to the goal, and the goal is to minimize this value over time, ideally satisfying: Where c π (s t ) represents the state s under strategy π t Cost; (52.3) Establish a Critic-Lyapunov estimation network and update the parameters of the Critic-Lyapunov network estimation in each initial buffer trajectory. The updated target is based on the TD difference target of the Bellman optimal iteration under the Markov model: Among them, C t represents the cost value at time t, Q l represents the Q value estimated by the target Lyapunov network; (52.4) The time difference algorithm is used to update the Lyapunov network, and the loss function is the current L c Value and target value The mean square error of : (53) Construct an Actor network, whose function is to input and output continuous actions based on the current state to achieve the autonomous driving control goal. The Actor network adopts a parameterized strategy model and is structurally a multi-layer feedforward neural network. The Actor network input includes the state feature information of the current vehicle and the surrounding vehicles, and the output is the action distribution parameter of the control instruction. The action is generated by random sampling. The specific implementation process includes: (53.1) Using the maximum entropy reinforcement learning-based policy update method in the SAC algorithm, the Actor network aims to maximize the weighted expected cumulative reward and policy entropy; (53.2) By sampling the current policy action and combining the feedback from the Critic-Reward, Critic-Cost, and Critic-Lyapunov networks, the target loss function of the policy is calculated as follows: Among them, π θ (a|s) represents the action probability density of the current policy output, α is the entropy temperature coefficient, Δ represents the experience pool, and θ represents the actor network neural network parameters; (53.3) By using the original SAC to update the Actor network, combined with the Critic Cost and Critic Lyapunov networks, the safe and stable policy gradient loss function is: where α L is the Lyapunov adjustment coefficient.

6. The Lyapunov function-based reinforcement learning autonomous driving decision-making and control method according to claim 5, characterized in that: The Reward network, Cost network, and Lyapunov network are collaboratively optimized to ensure security and stability, including: (61) formulate the RL problem as a constrained optimization task to balance safety and stability: Where c threshold and l threshold are predefined thresholds for security and stability, respectively. L(s,a) is the Lyapunov function for evaluating the stability of the strategy, and its expected value follows the state distribution S and the strategy π(·│S) distribution. (62) is transformed into the augmented Lagrangian objective through the primal-dual method: Where λ1 and λ2 are Lagrange multipliers, ∈1 and ∈2 are slack variables; (64) Strategy π θ , parameterized as θ, and updated by gradient descent: The Lagrange multiplier is adjusted to: where i=1,2, η1 and η2 are the policy parameters and the learning rate multiplier respectively.

7. The Lyapunov function-based reinforcement learning autonomous driving decision control method according to claim 6, characterized in that: Integrate the maximum entropy reinforcement learning framework and update the Actor network through the SAC algorithm, including: (71) Maximize the weighted sum of expected cumulative reward and policy entropy: Where α represents the entropy temperature coefficient that controls the trade-off between exploration and exploitation, and H represents the state s t The entropy of the policy π. (72)Convert the constrained optimization problem into an augmented Lagrangian objective: (73) The target network parameters are updated as follows: Where τ represents the learning rate, ω, φ, and ψ are the parameters of the Reward, Cost, and Lyapunov networks, respectively.

8. A reinforcement learning autonomous driving decision control system based on Lyapunov function, used to implement the reinforcement learning autonomous driving decision control method based on Lyapunov function according to any one of claims 1 to 7, characterized in that: include: A status information acquisition module is used to receive driving status information of the vehicle and surrounding vehicles, wherein the driving status information of the vehicle includes position, speed, acceleration, and heading angle, and the driving status information of surrounding vehicles includes position, speed, heading angle, and lane line information of surrounding vehicles relative to the vehicle; The decision control training module is used to build a decision control model for autonomous driving tasks based on reinforcement learning. The decision control model uses the driving status information of the vehicle and surrounding vehicles and outputs the optimal autonomous driving strategy. The decision control model includes a decision training network and a decision execution and optimization network. The control output module is used to configure the trained and optimized decision control model on the vehicle to execute autonomous driving decisions.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor loads and executes the computer program, it adopts a reinforcement learning autonomous driving decision control method based on Lyapunov function as described in any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to execute a reinforcement learning autonomous driving decision control method based on Lyapunov function as described in any one of claims 1 to 7.

Images