A green wave vehicle speed tracking control method and system based on safety critic

Abstract

The present invention relates to a green wave vehicle speed tracking control method based on a safety critic, and the method comprises the following steps: Step 1: The traffic information layer establishes a high-precision map through sensors and communication facilities, and collects road information and vehicle information; Step 2: The vehicle communicates with the infrastructure on the road side through V2I communication facilities, and the road side obtains the guiding green wave vehicle speed according to the traffic status of the road section and the traffic signal control information at the intersection; Step 3: The upper control layer performs vehicle speed tracking control based on the safety critic function reinforcement learning algorithm through the established reinforcement learning framework: Step 4: The lower control module performs comprehensive torque distribution that takes into account dynamics and energy saving according to the total demand torque, and outputs the distributed torque to the vehicle; Step 5: Each wheel hub engine of the vehicle is driven according to the distributed torque. Compared with the prior art, the present invention has the advantages of enabling the vehicle to keep tracking the guiding green wave vehicle speed and thus improving the traffic efficiency of the road section.

Description

Technical Field

[0001] The present invention relates to the field of automobile intelligent driving, and in particular to a green wave vehicle speed tracking control method and system based on a safety critic. Background Art

[0002] As the per capita car ownership continues to increase, the load on the transportation network is increasing, and more and more problems are beginning to be exposed in traditional road design methods. According to statistics from the National Bureau of Statistics, in 2021, there were 244,937 road traffic accidents across the country, causing 63,194 deaths, 258,532 injuries, and direct property losses of 1.38 billion yuan. The number of accidents has increased slightly compared with 10 years ago; in terms of energy consumption, the proportion of transportation energy consumption in developed countries is usually between 20% and 40% of terminal energy consumption, and it is still less than 20% in my country. With economic development and improvement of living standards, it is an inevitable trend for my country's transportation energy consumption to increase as a proportion of total social energy consumption, and it will become the main area of ​​energy consumption growth.

[0003] Traffic planning methods based on green waves may help solve these problems. Green wave traffic is an overall traffic control concept. It is a multi-point signal control technology. The intersections of roads are coordinated and controlled by traffic lights to ensure road safety and traffic efficiency. Traffic signal control is usually used to control and optimize the green wave traffic theory, which can provide a better travel environment for pedestrians and vehicles. Vehicles guided by good green wave speed are more likely to encounter green lights at intersections. Using the green wave traffic theory for safety, energy conservation and emission reduction, this paper conducts speed guidance, which is more conducive to road safety and energy conservation and emission reduction.

[0004] At present, there have been studies on dynamic speed planning algorithms for arterial traffic. With similar ideas, we propose a strategy that helps vehicles pass through intersections as non-stop as possible, called the green wave speed guidance strategy. The implementation of this guidance depends on supporting hardware. With the development of vehicle networking technology, traffic facilities can obtain vehicle location and speed information in real time. After processing this information, the recommended speed can be transmitted to the vehicle through V2I (vehicle to infrastructure) equipment, and the on-board electronic equipment can provide speed recommendations to the driver through human-machine interaction interfaces such as on-board displays, thereby achieving the expected driving style. Since it takes a certain amount of time for the driver to respond to the recommended speed and cannot continuously pay attention to the recommended speed, reinforcement learning technology can be used to automatically optimize speed tracking.

[0005] Reinforcement learning has shown great potential in the field of control in laboratory environments, but when deploying these systems in reality, safety is an important bottleneck that limits their performance. The intuitive method is to manually add constraints to the learning process to prevent the agent from taking unsafe actions or entering a state that is too risky; but the disadvantage of manually adding constraints is that it cannot be simply generalized to new scenarios or new environments; another important issue is that in the actual situation of high-speed vehicle driving, the torque distribution process must be completed quickly to ensure the responsiveness, power and safety of the drive. The time safety of training and testing is an important factor that needs to be considered in practical applications; in the reinforcement learning loop, because the agent needs to fully explore the environment to optimize its behavior, it will inevitably encounter dangerous situations. Although some training algorithms use complex techniques to reduce the probability of this happening, it is always difficult to avoid it. How to ensure the safety of training in this situation also needs to be considered. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a green wave vehicle speed tracking control method and system based on safety critics.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A green wave vehicle speed tracking control method based on a safety critic, the method comprising the following steps:

[0009] Step 1: The traffic information layer builds a high-precision map through sensors and communication facilities, and collects road and vehicle information;

[0010] Step 2: The vehicle communicates with the road infrastructure through V2I communication facilities. The road obtains the guidance green wave speed according to the road section traffic status and intersection traffic signal control information, and transmits the guidance green wave speed to the upper control layer through V2I communication facilities.

[0011] Step 3: The upper control layer performs vehicle speed tracking control based on the safety critic function reinforcement learning algorithm through the established reinforcement learning framework, and transmits the acquired target acceleration to the lower control module;

[0012] Step 4: The lower-level control module distributes the comprehensive torque that takes into account both dynamics and energy saving according to the total required torque, and outputs the distributed torque to the vehicle;

[0013] Step 5: Each wheel hub engine of the vehicle is driven according to the allocated torque to enable the vehicle to keep tracking the guidance green wave speed, thereby improving the traffic efficiency of the road section.

[0014] In the step 1, the communication facilities include communication facilities between vehicles, between vehicles and infrastructure, and between vehicles and external networks, and the road information includes the traffic status of road sections and traffic signal control information at intersections.

[0015] In step 3, the upper control layer performs the vehicle speed tracking control based on the safety critic function reinforcement learning algorithm through the established reinforcement learning framework as follows:

[0016] Step 301: input the guidance green wave speed, vehicle longitudinal speed, longitudinal acceleration, vehicle motion parameters and road condition information into the agent, and output the target acceleration to complete one iteration;

[0017] Step 302: The vehicle updates the driving torque according to the target acceleration, thereby completing the interaction between the intelligent agent and the environment;

[0018] Step 303: The agent receives new environmental information and feedback, and performs reinforcement learning based on the safety critic function reinforcement learning algorithm, that is, reinforcement learning is divided into two stages, namely, a pre-training stage and a fine-tuning stage. In the pre-training stage, a safety critic and a strategy are trained at the same time, so that the agent can freely explore unsafe behaviors, and the safety critic obtains the conditions for judging unsafe behaviors. In the fine-tuning stage, the new target training task training strategy is adopted, and the safety critic obtained in the pre-training stage is used to limit the update of the strategy and the selection of actions, that is, the strategy obtained in the pre-training stage is used for the target training task. For the target training task, the agent avoids accessing unsafe behaviors, and the strategy selects safe behaviors according to the trained safety critic and blocks unsafe behaviors. If there is no safe behavior, the behavior with the lowest failure probability is selected to avoid the local optimal strategy, and then correctly obtain the target acceleration according to the external information.

[0019] In step 303, the security critic function reinforcement learning algorithm is specifically:

[0020] The safety critic function reinforcement learning algorithm divides reinforcement learning into two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, a safety critic and a strategy are trained at the same time, so that the agent can freely explore unsafe behaviors, and the safety critic obtains the conditions for judging unsafe behaviors. In the fine-tuning stage, the strategy is trained with a new target training task, and the safety critic obtained in the pre-training stage is used to limit the update of the strategy and the selection of actions, that is, the strategy obtained in the pre-training stage is used for the target training task. For the target training task, the agent avoids accessing unsafe behaviors, and the strategy selects safe behaviors according to the trained safety critic and blocks unsafe behaviors. If there is no safe behavior, the behavior with the lowest probability of failure is selected.

[0021] The definition expression of the Markov decision process used in the pre-training phase is:

[0022] T pre = <S,A,P pre ,r pre ,μ pre ,I>

[0023] Among them, T pre is the pre-training environment, i.e. the pre-training stage, S is the state space, A is the action space, P pre is the state transition probability in the pre-training environment, r pre is the reward function in the pre-training environment, μ pre is the initial state distribution in the pre-training environment, I is the safety indicator, which is used to indicate whether a given state is safe;

[0024] The definition expression of the Markov decision process for the target training task in the fine-tuning phase is:

[0025] T target = <S,A,P target ,r target ,μ target ,I>

[0026] Among them, T taeget is the target training task, P target is the state transition probability in the target training task, r target is the reward function in the target training task, μ target is the initial state distribution in the target training task, I is the safety indicator;

[0027] In a safe pre-training environment pre After pre-training in , the agent must optimize its performance on the target training task T target In the expected return, while minimizing the visit to unsafe states, for the target training task, the strategy must ensure that the unsafe state S is not visited. unsafe ={s|I(s)=1}:

[0028]

[0029] Among them, I(s) = 1 represents an unsafe state, ∈ safe is the target safety threshold, T is the total number of steps,

[0030] is the total expected return function, I(s t ) is the current security status, and the specific goal is to ensure That is, under the premise that the probability of entering an unsafe state is less than the safety threshold, maximize the expected reward, st for

[0031] The state at time t, a t is the action at time t, ρ π Indicates the corresponding status compilation.

[0032] The process of the pre-training stage is specifically as follows:

[0033] Step a: Define the target safety threshold ∈=∈ safe ;

[0034] Step b: In each iteration, collect data from the current policy whose actions are constrained by the safety critic, put the data in the buffer, use the hybrid policy in the buffer to update the safety critic and update the policy using the maximum entropy reinforcement learning algorithm;

[0035] Step c: After the iteration, return to the pre-training strategy and security critics Assuming that the safety Q function obtained after pre-training is optimal and can correctly estimate the failure probability, the safety Q function

[0036] In the step b, the expression of the security critic is:

[0037]

[0038] in, is a security critic that evaluates the policy π from state s t Start taking action t The failure probability in the future, I(s t ) is the current safety state, P pre is the state transition probability, γ safe is the discount factor, t′ is the next moment after time t, T is the total number of steps, γ safe is the discount factor, I(s t′ ) is the future security state, Indicates that the state of the next moment at the current moment is determined by the state transition probability function in the pre-training phase, and the state at subsequent moments is determined by the strategy and the state transition probability function;

[0039] When the security critic is a binary classifier, dynamic programming is used to train the security critic, and the discount factor γ is used to safe Limit the propagation distance of past fault signals, the cumulative discounted failure probability is estimated by the Bellman equation, and the safety Q function The expression is:

[0040]

[0041] in, is the security Q function, s′ and a′ are a future state and action respectively, γ safe is the discount factor, Indicates that the state is determined by the state transition probability function, and the action at this moment is determined by the strategy sampling;

[0042] The secure Q function The parameter ψ is used to parameterize the neural network, and the resulting target is:

[0043]

[0044] Among them, J safe (ψ) is the expected return, The distribution of trajectories representing current and future state-action pairs is determined by the policy π, is the Q function corresponding to the delayed target network;

[0045] Based on the SAC algorithm, the agent is encouraged to explore by maximizing the reward and entropy of the strategy, and the safety critic is optimized under a hybrid strategy constrained by the safety critic. For a safe state-action pair in the data, the subsequent behavior from that point on will be constrained to be safe, thereby generating a reliable target label and avoiding the negativity of the safety critic.

[0046] During the pre-training phase, the security critic is trained with a random security constraint policy Simultaneously train to make the safety constraint policy For secure Q function The sampling probability of action a in is 0, and the set of all strategies that meet the conditions is The mapping from any policy to a set of policies that meet the conditions is established as follows:

[0047]

[0048] Among them, π is an arbitrary strategy, The strategy closest to strategy π that satisfies the constraint condition is the safety constraint strategy;

[0049] According to the definition of security critics, the expression of actions produced by the policy subject to the security Q-function is:

[0050]

[0051] in, is the action generated by the strategy constrained by the security Q function, π(a|s) is the action generated by the original strategy, ∈ is the security threshold of the security Q function, that is, the target security threshold, which means that if the action exceeds the range of the target security threshold, the action is rejected. The output distribution of the action is masked from π to only the sampled actions that meet the security threshold to ensure the constrained strategy. safety.

[0052] The process of the fine-tuning stage is specifically as follows:

[0053] The Safety Critic reinforcement learning algorithm initializes the policy to a pre-trained policy subject to safety constraints. And fine-tuned to a new safety-critical objective training task T traget To this end, the security critic obtained in the pre-training phase is used Generate a safety-constrained Markov decision process that uses security critics’ knowledge of the data collected during the pre-training phase when fine-tuning the target training task. The constrained strategy collects all data and updates the strategy according to the target training task reward function. In order to make the unconstrained strategy π′ The actions in the range are sampled, where To satisfy the policy distribution of security constraint loss ∈, In order to satisfy the strategy of safety constraint loss ∈, the safety constraint cost is added to the objective of the standard SAC strategy, and the expected return is optimized at the same time. The objective of the improved SAC strategy is:

[0054]

[0055] where α and v are the Lagrange multipliers of entropy and security constraints, respectively. is the target entropy, a′ is a future action, Indicates that the strategy is initialized to the pre-trained strategy, and the future action a′ is determined by the strategy π′ θ Sampling, θ is the parameter of the strategy.

[0056] In step 4, the process of the lower control module distributing the comprehensive torque taking into account both dynamics and energy saving according to the total demand torque is specifically as follows:

[0057] Obtain the real-time four-wheel speed and longitudinal acceleration of the target acceleration vehicle, and calculate the torque allocated to each wheel hub engine of the vehicle in combination with the vehicle's motion state, motor loss, and tire slip loss;

[0058] Establish tire slip energy loss model:

[0059] P s =F s V s

[0060] J s =2(P s,r +P s,f )

[0061] Among them, F sV is the longitudinal force in the sliding area of ​​the tire contact surface, i.e. the longitudinal sliding force, s is the longitudinal slip velocity of the vehicle, P s is the longitudinal sliding force power, J s is the energy lost by friction, P s,f is the power loss due to front wheel slip, P s,r Power loss due to rear wheel slip;

[0062] Establishing the hub motor loss model:

[0063]

[0064] J m =2(P loss,r +P loss,f )

[0065] Among them, η m is the efficiency of the motor, P m is the motor power, J m is the motor energy loss, P loss is the power loss of the hub motor, P loss,r Represents the power loss of the rear wheel hub motor, P loss,f Indicates the power loss of the front wheel hub motor;

[0066] The expression of total energy consumption is:

[0067] J=J s +J m

[0068] Where, J is the total energy consumption;

[0069] Set constraints on the comprehensive torque distribution that takes into account both dynamics and energy saving, and solve the driving torque of each wheel hub motor under the constraints.

[0070] The constraints are:

[0071] T req =2(T m,r +T m,f )

[0072] T m,mim (n m (t))≤T m ≤T m,max (n m (t))

[0073] n m,min ≤n m ≤n m,max

[0074] Among them, T m,r With Tm,f Represent the rear motor torque and the front motor torque respectively, T m,mim (n m (t)) and T m,max (n m (t)) are the minimum and maximum torques of the motor, n m,min With n m,max are the minimum and maximum speed of the motor, T req is the total motor torque, n m is the motor speed, T m is the torque of a single motor.

[0075] A green wave vehicle speed tracking control system for implementing the green wave vehicle speed tracking control method, the system comprising a traffic information layer, an upper control layer, a lower control layer, a data collection layer and a buffer zone;

[0076] The traffic information layer obtains the guiding green wave speed by comprehensively calculating the traffic status of the road section and the traffic signal control information at the intersection collected by the road side, and transmits the guiding green wave speed to the vehicle through the vehicle-road cooperative facility. Other information obtained by the traffic information layer is used to provide input parameters for the upper control layer.

[0077] The upper control layer receives the guiding green wave speed, road driving status information and vehicle dynamics parameters obtained in the traffic information layer, and uses the safety critic function reinforcement learning algorithm for training to obtain the target acceleration;

[0078] The lower control layer receives the vehicle dynamics parameters and the target acceleration transmitted by the upper control layer, distributes the comprehensive torque taking into account both dynamics and energy saving according to the total required torque, and the lower control layer outputs the distributed torque to the vehicle;

[0079] The data collection layer is used to collect data;

[0080] The buffer is used to store data and to perform updates throughout the training process.

[0081] Compared with the prior art, the present invention has the following beneficial effects:

[0082] 1. The present invention obtains the guiding green wave vehicle speed according to the traffic status of the road section and the control information of the traffic signal at the intersection, and performs vehicle speed tracking control based on the safety critic function reinforcement learning algorithm, performs comprehensive torque distribution that takes into account both dynamics and energy saving according to the total demand torque, and outputs the distributed torque to the vehicle to enable the vehicle to keep tracking the guiding green wave vehicle speed, thereby improving the traffic efficiency of the road section;

[0083] 2. Compared with the unconstrained SAC, the security critic function reinforcement learning algorithm adopted in the present invention can alleviate these problems by conducting a large amount of exploration in the safe behavior space, thereby producing more stable and effective learning, improving the safety of training, accelerating the training process, and reducing the probability of failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 The figure is a flow chart of the method of the present invention.

[0085] Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0086] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0087] The present invention considers the reinforcement learning problem on Markov decision process (MDP), which is described by the tuple:

[0088] <S,A,γ,r,P,μ>

[0089] Among them, S is the state space, A is the action space, γ is the discount factor, is the reward function, P(·|s,a) is the state transition probability, s is the state, a is the action, μ is the initial state distribution, and the purpose of reinforcement learning is to learn a strategy π θ :S×A→[0,1], so that the expected return J(θ) has the maximum value, and the expression of the expected return J(θ) is:

[0090]

[0091] Among them, H(π θ (·|s t )) represents the action entropy of the strategy, and α>0 is the adjustment parameter.

[0092] In dynamic control problems, states that are prone to accidents are difficult to fully know before learning; however, unsafe states are easy to judge, such as vehicle speed or torque exceeding the safe range. It is difficult to fully specify these safety constraints, and there may be bias or serious obstacles to reinforcement learning, especially when the potential reward function and safety constraints are uncertain. However, the conditions for accidents are relatively easy to determine. Therefore, the goal is to learn the best strategy for a task while minimizing the probability of accidents during training.

[0093] To achieve this goal, the present invention introduces a safe pre-training environment and a safety indicator, wherein the pre-training environment can tolerate safety accidents, and the safety indicator is used to prompt whether a given state is safe. In the pre-training environment and the target training task, the fault state is regarded as a terminal. This configuration enables the agent to learn safely and transfer strategies to new tasks without accumulating additional failure costs.

[0094] The present invention divides reinforcement learning into two stages:

[0095] The first stage is the pre-training stage, in which simpler / safer tasks are solved;

[0096] The second stage is the fine-tuning stage, which transfers the learned strategy to a target training task with higher security while ensuring security.

[0097] The definition expression of the Markov decision process used for pre-training is:

[0098] T pre = <S,A,P pre ,r pre ,μ pre ,I>

[0099] Among them, T pre is the pre-training environment, P pre is the state transition probability in the pre-training environment, r pre is the reward function in the pre-training environment, μ pre is the initial state distribution in the pre-training environment, I is the safety indicator;

[0100] The definition expression of the Markov decision process for the target training task is:

[0101] T target = <S,A,P target ,r target ,μ target ,I>

[0102] Among them, T taeget is the target training task, p target is the state transition probability in the target training task, r target is the reward function in the target training task, μ target is the initial state distribution in the target training task, I is the safety indicator;

[0103] In a safe pre-training environment pre After pre-training in , the agent must optimize its performance on the target training task T targetIn the expected return, while minimizing the visit to unsafe states, for the target training task, the strategy must ensure that the unsafe state S is not visited. unsafe ={s|I(s)=1}:

[0104]

[0105] Among them, ∈ safe is the target safety threshold, T is the total number of steps, is the total expected return function, I(s t ) is the current security status;

[0106] Introducing target safety threshold ∈ safe , which is used to represent the upper limit of the expected risk of a given policy. When used as a single-step constraint on actions sampled during policy training, it is used to ensure that the policy π is safe under this threshold under certain assumptions. This safety constraint needs to be imposed throughout the training process, not only at convergence.

[0107] To solve the above problems, the secure Q-function reinforcement learning algorithm (SQRL) is introduced. As shown in the above process, the secure Q-function divides reinforcement learning into two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, the strategy and security concepts are learned at the same time, and then the learned security measures are used to fine-tune the strategy of the target training task; the security of the training in the fine-tuning stage is guaranteed by the learning process itself, and the security critic Compute the failure probability of a safety-constrained policy given a state and action, via a safety critic The learned safety constraints ensure the safety of the strategy. The process of the safety Q function reinforcement learning algorithm is as follows:

[0108] 1. Pre-training stage:

[0109] The purpose of the pre-training phase is to learn an optimal security critic and optimal pre-training strategy The optimal pre-training strategy As the initialization of the target training task training, the pre-training stage T pre The best safety critic will evaluate the expectation of a given policy π, the safety critic The expression is:

[0110]

[0111] in, To evaluate the strategy π from state s t Start taking action t The failure probability in the future, I(s t ) is the failure probability at the current moment, P preis the state transition probability, γ safe is the discount factor;

[0112] Regarding the training method of the safety critic, if the safety critic is a binary classifier, supervised learning is used to evaluate the safety label of a single time step. However, in order to learn the future cumulative failure probability, the safety critic must reason about future time steps. Therefore, dynamic programming is used to train the safety critic:

[0113] Similar to general Q function learning, the discount factor γ is used safe Limiting the propagation distance of past fault signals, the cumulative discounted failure probability is estimated by the Bellman equation, and the expression of the safety Q function is:

[0114]

[0115] in, is the security Q function, s′ and a′ are a certain future state and action respectively;

[0116] The secure Q function Parameterized as a neural network with parameter ψ, the resulting target is:

[0117]

[0118] In order to learn new task objectives without failure, a series of different state-action pairs must be explored in the pre-training phase, including unsafe state-action pairs. To ensure sufficient exploration of the agent, the soft actor-critic (SAC) algorithm is selected, which encourages the agent to explore by maximizing the reward and entropy of the strategy.

[0119] Finally, the strategy used when optimizing the safety critic is to use all the experience from the SAC algorithm, which may cause the safety critic to be too negative because it contains mixed data from all encountered strategies. If the mixed strategy contains a random, unsafe strategy from the beginning, then even cautious behavior in one state may be considered unsafe because risky behavior will be observed later. Therefore, it is necessary to pay attention to both including diverse data from a range of unsafe conditions and avoiding learning a negative safety critic. To solve this problem, the safety critic is optimized under the mixed strategy constrained by the safety critic. For safe state-action pairs in the data, subsequent behaviors from that point on will be constrained to be safe, thereby producing reliable target labels and avoiding safety critic negativity.

[0120] During the pre-training phase, the security critic is trained with a random security constraint policy. Simultaneously train to make the safety constraint policy For secure Q function The sampling probability of action a in is 0, and the set of all strategies that meet the conditions is Order Mapping Map any policy to According to the definition of security critics, the action generated by the policy constrained by the security Q function is expressed as:

[0121]

[0122] in, is the action generated by the strategy constrained by the security Q function, π(a|s) is the action generated by the original strategy, ∈ is the security threshold of the security Q function, that is, the target security threshold, which means that if the action exceeds the range of the target security threshold, the action is rejected. The output distribution of the action is masked from π to only the sampled actions that meet the security threshold to ensure the constrained strategy. safety;

[0123] The steps of the pre-training phase are:

[0124] Step a: Define the target safety threshold ∈=∈ safe ,Actions that exceed the target safety threshold will be rejected as too risky;

[0125] Step b: In each iteration, collect data from the current policy whose actions are constrained by the safety critic, put the data in a buffer, update the safety critic using the hybrid policy in the buffer, and update the policy using the maximum entropy (MaxEnt) reinforcement learning algorithm;

[0126] Step c: After the iteration, return to the pre-training strategy and security critics Assuming that the safety Q function obtained after pre-training is optimal and can correctly estimate the failure probability, the safety critic With the secure Q function equal.

[0127] Fine-tuning phase:

[0128] In the fine-tuning phase, the safe Q-function reinforcement learning algorithm initializes the policy to the pre-trained safe constraint policy subject to safety constraints. And fine-tuned to a new safety-critical objective training task T traget To this end, a pre-trained security critic Generate a safety-constrained Markov decision process, where all data is trained on the target training task using the safety critic We collect data from the unconstrained strategy, follow the data collection method used in the pre-training phase, and update the strategy according to the reward function of the target training task. In order to encourage the unconstrained strategy π′ to The actions in the range are sampled, and the safety constraint cost is added to the policy objective while optimizing the expected return. The expression of the standard SAC policy objective is:

[0129]

[0130] Where α and ν are entropy and security constraint multipliers respectively, is the target entropy (a hyperparameter of SAC);

[0131] When applying the safety Q function reinforcement learning algorithm to train safety critics and policies on actual driving control problems, several approximations need to be made in the specific implementation. These approximations have little effect on performance. A data collection layer is added to the existing different-policy algorithm as an implementation. The data collection layer is used to collect data and store data in a buffer. The buffer is used to perform updates during the entire training process. The present invention stores the records in an offline replay buffer D offline In the example above, a small buffer D is kept while collecting samples for training. safe , used to store the offline training strategy n off After the step, the k trajectories generated by the latest strategy are used to train the safety critic. In the pre-training stage, the rejection sampling method is used to find actions with a failure probability just below the safety threshold, and the current safety critic is used to score them. This is equivalent to sampling safe but risky actions, allowing the agent to explore the safety boundary to correctly identify safe and unsafe actions in a given state. Methods such as cross entropy can also be used to perform importance-based sampling weighted by failure probability to provide the safety critic with the most information about safe and unsafe operations close to the safety threshold. The behavior of the final safety policy is mainly determined by the safety hyperparameter ∈ safe and γ safe These two parameters are set based on specific tasks and environments, and the security hyperparameter ∈ is adjusted during the pre-training phase. safe and γ safe to achieve the best pre-training performance, as it is unsafe to adapt when solving the target training task.

[0132] During the fine-tuning phase, the security sampling strategy is switched to the security critic Select a safe action, sample k actions again, mask unsafe actions, and then sample the remaining options with a probability proportional to their logarithm under the original distribution output π(·|s); if no safe action is found, select the action with the lowest failure probability.

[0133] In dynamic control problems, it is difficult to fully know the state that is likely to cause an accident before learning. However, unsafe states are easy to judge, such as vehicle speed or torque exceeding the safe range. However, it is difficult to fully obtain these safety constraints, and there may be bias or serious obstacles to reinforcement learning. This is more serious when the potential reward function and safety constraints are uncertain. However, the conditions for accidents are relatively easy to determine. Therefore, the purpose of the present invention is to learn the best strategy for a task while minimizing the probability of accidents during training. The present invention obtains safety precautions from previous experience to construct a framework for safety reinforcement learning and proposes a method for The safety critic reinforcement learning algorithm for reinforcement learning obtains a safety critic under a strategy constrained by the safety Q function itself. The safety critic is used to evaluate whether a state / action pair will cause unsafe behavior. The pre-training stage and the fine-tuning stage are carried out separately: in the pre-training stage, the safety critic and the strategy are trained at the same time, and the agent can explore and learn unsafe behaviors; in the fine-tuning stage, the strategy is trained on a new target training task, and the strategy update and selected behavior of the safety critic are adopted. Compared with the reinforcement learning algorithm and the safety reinforcement learning algorithm, SQRL shows higher efficiency in learning safety and new task learning efficiency.

[0134] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A green wave vehicle speed tracking control method based on safety critics, characterized in that: The method comprises the following steps: Step 1: The traffic information layer builds a high-precision map through sensors and communication facilities, and collects road and vehicle information; Step 2: The vehicle communicates with the road infrastructure through V2I communication facilities. The road obtains the guidance green wave speed according to the road section traffic status and intersection traffic signal control information, and transmits the guidance green wave speed to the upper control layer through V2I communication facilities. Step 3: The upper control layer performs vehicle speed tracking control based on the safety critic function reinforcement learning algorithm through the established reinforcement learning framework, and transmits the acquired target acceleration to the lower control module; Step 4: The lower control module distributes the comprehensive torque taking into account both dynamics and energy saving according to the total required torque, and outputs the distributed torque to the vehicle; Step 5: Each wheel hub engine of the vehicle is driven according to the allocated torque to enable the vehicle to keep tracking the guiding green wave speed, thereby improving the traffic efficiency of the road section; In step 3, the upper control layer performs the vehicle speed tracking control based on the safety critic function reinforcement learning algorithm through the established reinforcement learning framework as follows: Step 301: input the guidance green wave speed, vehicle longitudinal speed, longitudinal acceleration, vehicle motion parameters and road condition information into the agent, and output the target acceleration to complete one iteration; Step 302: The vehicle updates the driving torque according to the target acceleration, thereby completing the interaction between the intelligent agent and the environment; Step 303: the agent receives new environmental information and feedback, and performs reinforcement learning based on the safety critic function reinforcement learning algorithm, that is, reinforcement learning is divided into two stages, namely, a pre-training stage and a fine-tuning stage. In the pre-training stage, a safety critic and a strategy are trained at the same time, so that the agent can freely explore unsafe behaviors, and the safety critic obtains the conditions for judging unsafe behaviors. In the fine-tuning stage, the strategy is trained with a new target training task, and the safety critic obtained in the pre-training stage is used to limit the update of the strategy and the selection of actions, that is, the strategy obtained in the pre-training stage is used for the target training task. For the target training task, the agent avoids accessing unsafe behaviors, and the strategy selects safe behaviors according to the trained safety critic and shields unsafe behaviors. If there is no safe behavior, the behavior with the lowest failure probability is selected to avoid the local optimal strategy, and then correctly obtain the target acceleration according to the external information; In step 303, the security critic function reinforcement learning algorithm is specifically: The safety critic function reinforcement learning algorithm divides reinforcement learning into two stages, namely the pre-training stage and the fine-tuning stage. In the pre-training stage, a safety critic and a strategy are trained at the same time, so that the agent can freely explore unsafe behaviors, and the safety critic obtains the conditions for judging unsafe behaviors. In the fine-tuning stage, the strategy is trained with a new target training task, and the safety critic obtained in the pre-training stage is used to limit the update of the strategy and the selection of actions, that is, the strategy obtained in the pre-training stage is used for the target training task. For the target training task, the agent avoids accessing unsafe behaviors, and the strategy selects safe behaviors according to the trained safety critic and blocks unsafe behaviors. If there is no safe behavior, the behavior with the lowest probability of failure is selected. The definition expression of the Markov decision process used in the pre-training phase is: T pre =<S,A,P pre ,r pre ,μ pre ,I> Among them, T pre is the pre-training environment, i.e. the pre-training stage, S is the state space, A is the action space, P pre is the state transition probability in the pre-training environment, r pre is the reward function in the pre-training environment, μ pre is the initial state distribution in the pre-training environment, I is the safety indicator, which is used to indicate whether a given state is safe; The definition expression of the Markov decision process for the target training task in the fine-tuning phase is: T target =<S,A,P target ,r target ,μ target ,I> Among them, T taeget is the target training task, P target is the state transition probability in the target training task, r target is the reward function in the target training task, μ target is the initial state distribution in the target training task, I is the safety indicator; In a safe pre-training environment pre After pre-training in , the agent must optimize its performance on the target training task T target In the expected return, while minimizing the visit to unsafe states, for the target training task, the strategy must ensure that the unsafe state S is not visited. unsafe ={s|I(s)=1}: Among them, I(s) = 1 represents an unsafe state, s is the state, ∈ safe is the target safety threshold, T is the total number of steps, is the total expected return function, I(s t ) is the current security status, and the specific goal is to ensure That is, under the premise that the probability of entering an unsafe state is less than the safety threshold, maximize the expected reward, s t is the state at time t, a t is the action at time t, ρ π Indicates the corresponding state compilation; The process of the pre-training stage is specifically as follows: Step a: Define the target safety threshold ∈=∈ safe ; Step b: In each iteration, collect data from the current policy whose actions are constrained by the safety critic, put the data in the buffer, use the hybrid policy in the buffer to update the safety critic and update the policy using the maximum entropy reinforcement learning algorithm; Step c: After the iteration, return to the pre-training strategy and security critics Assuming that the safety Q function obtained after pre-training is optimal and can correctly estimate the failure probability, the safety Q function In the step b, the expression of the security critic is: in, is a security critic that evaluates the policy π from state s t Start taking action t The failure probability in the future, I(s t ) is the current safety state, P pre is the state transition probability, γ safe is the discount factor, t′ is the next moment after time t, T is the total number of steps, γ safe is the discount factor, I(s t′ ) is the future security state, Indicates that the state of the next moment at the current moment is determined by the state transition probability function in the pre-training phase, and the state at subsequent moments is determined by the strategy and the state transition probability function; When the security critic is a binary classifier, dynamic programming is used to train the security critic, and the discount factor γ is used to safe Limit the propagation distance of past fault signals, the cumulative discounted failure probability is estimated by the Bellman equation, and the safety Q function The expression is: in, is the security Q function, s′ and a′ are a future state and action respectively, γ safe is the discount factor, Indicates that the state is determined by the state transition probability function, and the action at this moment is determined by the strategy sampling; The secure Q function The parameter ψ is used to parameterize the neural network, and the resulting target is: Among them, J safe (ψ) is the expected return, The distribution of trajectories representing current and future state-action pairs is determined by the policy π, is the Q function corresponding to the delayed target network; Based on the SAC algorithm, the agent is encouraged to explore by maximizing the reward and entropy of the strategy, and the safety critic is optimized under a hybrid strategy constrained by the safety critic. For a safe state-action pair in the data, the subsequent behavior from that point on will be constrained to be safe, thereby generating a reliable target label and avoiding the negativity of the safety critic. During the pre-training phase, the security critic is trained with a random security constraint policy Simultaneously train to make the safety constraint policy For secure Q function The sampling probability of action a in is 0, and the set of all strategies that meet the conditions is The mapping from any policy to a set of policies that meet the conditions is established as follows: Among them, π is an arbitrary strategy, The strategy closest to strategy π that satisfies the constraint condition is the safety constraint strategy; According to the definition of security critics, the expression of actions produced by the policy subject to the security Q-function is: in, is the action generated by the strategy constrained by the security Q function, π(a|s) is the action generated by the original strategy, ∈ is the security threshold of the security Q function, that is, the target security threshold, which means that if the action exceeds the range of the target security threshold, the action is rejected. The output distribution of the action is masked from π to only the sampled actions that meet the security threshold to ensure the constrained strategy. safety; The process of the fine-tuning stage is specifically as follows: The Safety Critic reinforcement learning algorithm initializes the policy to a pre-trained policy subject to safety constraints. And fine-tuned to a new safety-critical objective training task T traget To this end, the security critic obtained in the pre-training phase is used Generate a safety-constrained Markov decision process that uses security critics’ knowledge of the data collected during the pre-training phase when fine-tuning the target training task. The constrained strategy collects all data and updates the strategy according to the target training task reward function. In order to make the unconstrained strategy π′ The actions in the range are sampled, where To satisfy the policy distribution of security constraint loss ∈, In order to satisfy the strategy of safety constraint loss ∈, the safety constraint cost is added to the objective of the standard SAC strategy, and the expected return is optimized at the same time. The objective of the improved SAC strategy is: where α and v are the Lagrange multipliers of entropy and security constraints, respectively. is the target entropy, a′ is a future action, Indicates that the strategy is initialized to the pre-trained strategy, and the future action a′ is determined by the strategy π′ θ Sampling, θ is the parameter of the strategy; In step 4, the process of the lower control module distributing the comprehensive torque taking into account both dynamics and energy saving according to the total demand torque is specifically as follows: Obtain the real-time four-wheel speed and longitudinal acceleration of the target acceleration vehicle, and calculate the torque allocated to each wheel hub engine of the vehicle in combination with the vehicle's motion state, motor loss, and tire slip loss; Establish tire slip energy loss model: P s =F s V s J s =2(P s,r +P s,f ) Among them, F s V is the longitudinal force in the sliding area of ​​the tire contact surface, i.e. the longitudinal sliding force, s is the longitudinal slip velocity of the vehicle, P s is the longitudinal sliding force power, J s is the energy lost by friction, P s,f is the power loss due to front wheel slip, P s,r Power loss due to rear wheel slip; Establishing the hub motor loss model: J m =2(P loss,r +P loss,f ) Among them, η m is the efficiency of the motor, P m is the motor power, J m is the motor energy loss, P loss is the power loss of the hub motor, P loss,r Represents the power loss of the rear wheel hub motor, P loss,f Indicates the power loss of the front wheel hub motor; The expression of total energy consumption is: J=J s +J m Where, J is the total energy consumption; Set constraints on the comprehensive torque distribution that takes into account both dynamics and energy saving, and solve the driving torque of each wheel hub motor under the constraints; The constraints are: T req =2(T m,r +T m,f ) T m,mim (n m (t))≤T m ≤T m,max (n m (t)) n m,min ≤n m ≤n m,max Among them, T m,r With T m,f Represent the rear motor torque and the front motor torque respectively, T m,mim (n m (t)) and T m,max (n m (t)) are the minimum and maximum torques of the motor, n m,min With n m,max are the minimum and maximum speed of the motor, T req is the total motor torque, n m is the motor speed, T m is the torque of a single motor.

2. The green wave vehicle speed tracking control method based on safety critic according to claim 1 is characterized in that: In the step 1, the communication facilities include communication facilities between vehicles, between vehicles and infrastructure, and between vehicles and external networks, and the road information includes the traffic status of road sections and traffic signal control information at intersections.

3. A green wave vehicle speed tracking control system for implementing the green wave vehicle speed tracking control method according to any one of claims 1 to 2, characterized in that: The system includes a traffic information layer, an upper control layer, a lower control layer, a data collection layer, and a buffer zone; The traffic information layer obtains the guiding green wave speed by comprehensively calculating the traffic status of the road section and the traffic signal control information at the intersection collected by the road side, and transmits the guiding green wave speed to the vehicle through the vehicle-road cooperative facility. Other information obtained by the traffic information layer is used to provide input parameters for the upper control layer. The upper control layer receives the guiding green wave speed, road driving status information and vehicle dynamics parameters obtained in the traffic information layer, and uses the safety critic function reinforcement learning algorithm for training to obtain the target acceleration; The lower control layer receives the vehicle dynamics parameters and the target acceleration transmitted by the upper control layer, distributes the comprehensive torque taking into account both dynamics and energy saving according to the total required torque, and the lower control layer outputs the distributed torque to the vehicle; The data collection layer is used to collect data; The buffer is used to store data and to perform updates throughout the training process.

Images