Operation speed safety control method based on reinforcement learning hybrid control strategy

Abstract

The invention discloses a running speed safety control method based on a reinforcement learning hybrid control strategy. The method comprises the following steps: S1, constructing a high-speed train running process model; s2, constructing a reward function of a hybrid control strategy based on reinforcement learning and fractional order PID; and S3, constructing a hybrid control strategy based on reinforcement learning and fractional order PID, and jointly generating a final control signal through superposition processing. According to the running speed safety control method based on the reinforcement learning hybrid control strategy, the problems that in the prior art, the anti-disturbance capacity is poor, and the system response speed is low are solved, accurate tracking of the high-speed train on the target running speed is achieved, and the purpose of safe running of the high-speed train is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of running speed control, and in particular to a running speed safety control method based on a reinforcement learning hybrid control strategy. Background Art

[0002] In strong wind conditions, the speed control of high-speed trains mainly relies on the driver's experience and operating skills. The driver refers to the train automatic protection curve for manual control, which requires high experience and operating skills of the driver and is difficult to meet the requirements of safe and efficient operation. The automatic train driving system can overcome the defects of manual operation. The automatic train driving system relies on the given target speed curve to automatically adjust the train's running speed in real time, realize accurate tracking of the train's running speed, and achieve the purpose of safe train operation.

[0003] In the prior art, the speed controller is the core component of the train automatic driving, and the PID controller or the adaptive PID controller is the most commonly used speed controller. However, the flexibility and adaptability of the PID controller are not strong. Especially in a strong wind environment (where there are disturbances and uncertainties), for a high-speed train that is a highly nonlinear and strongly coupled complex object, the PID controller has the problems of poor anti-disturbance capability and slow system response speed. Summary of the invention

[0004] The purpose of the present invention is to provide a running speed safety control method based on reinforcement learning hybrid control strategy, to solve the problems of poor anti-disturbance ability and slow system response speed in the prior art, to realize accurate tracking of the target running speed of the high-speed train, and to achieve the purpose of safe operation of the high-speed train.

[0005] To achieve the above object, the present invention provides a running speed safety control method based on a reinforcement learning hybrid control strategy, comprising the following steps:

[0006] S1. Construct a high-speed train operation process model;

[0007] S2. Construct a reward function for a hybrid control strategy based on reinforcement learning and fractional-order PID.

[0008] S3. Construct a hybrid control strategy based on reinforcement learning and fractional-order PID to jointly generate the final control signal through superposition processing.

[0009] Preferably, in step S1, the high-speed train operation process is subjected to dynamic analysis and mathematical description, and the dynamic equation is specifically as follows:

[0010]

[0011] Where v is the train speed, m is the acceleration coefficient, F is the resultant force on the train, n is the traction or braking force of the train, and f r is the basic resistance of the train, a0 is the rolling mechanical force coefficient of the train, a1 is the other mechanical resistance coefficient of the train, and a2 is the external air resistance coefficient of the train.

[0012] Preferably, in step S2, the reward function is as follows:

[0013]

[0014] Among them, e t represents the control error at time t, c is a constant reward, and when the control error e t When the absolute value of is less than 0.001, a constant reward c is added.

[0015] Preferably, S3 is specifically:

[0016] S31, designing the transfer function C(s) of the fractional-order PID of the main controller based on the definition of Grunwald-Letnikov fractional-order calculus;

[0017] S32, constructing the objective function of the auxiliary learning controller based on the truncation method of the reinforcement learning PPO algorithm, and updating the strategy parameter θ and the value function parameter φ until the strategy converges or reaches the maximum number of training steps;

[0018] S33, superimposing the outputs of the main controller and the auxiliary learning controller to jointly generate a final control signal.

[0019] Preferably, in step S31, the definition of Grunwald-Letnikov fractional calculus is as follows:

[0020]

[0021] Where α is the fractional order, α>0 means differentiation, α<0 means integration; h means the time step; is the generalized binomial coefficient, specifically

[0022] Based on the definition of Grunwald-Letnikoff fractional-order calculus, the transfer function of the fractional-order PID is as follows:

[0023]

[0024] Among them, K p is the proportional gain, K d is the differential gain, K i is the integral gain, s is the Laplace operator, λ is the integral order, usually 0≤λ≤1, and μ is the differential order, usually 0≤μ≤1.

[0025] Preferably, S32 is specifically:

[0026] S321, initialization: initializing the strategy parameters θ and value function parameters φ, as well as initializing the environment and experience storage;

[0027] S322, Collect trajectories: interact with the environment, collect multiple trajectories, and at time step t, according to the action strategy π θ (a|s) select action a t , collect status s t 、Action a t and reward r t , and save the data of the trajectory;

[0028] S323, calculate advantage function: use time difference method to estimate advantage function A t :

[0029] A t =δ t +γδ t+1 +γ 2 δ t+2 +…;

[0030] Among them, δ t is the TD error, specifically δ t =r t +γV φ (s t+1 )-V φ (s t ), V φ (s t ) is the value function of the current state, γ is the discount factor;

[0031] S324, constructing an objective function: including a policy update objective, a value function update objective, and an entropy regularization term;

[0032] S325, update parameters: update the strategy parameters θ and value function parameters φ through stochastic gradient descent using the Adam optimizer;

[0033] S326, repeat training: repeat steps S322-S325 until the strategy converges or the maximum number of training steps is reached.

[0034] Preferably, in step S324, the strategy updates the target, as shown below:

[0035] L CLIP (θ) = E t [min(r t (θ)·A t ,clip(r t(θ),1-τ,1+τ)·A t )];

[0036] Among them, r t (θ) is the probability ratio, which indicates the probability ratio of the new and old strategies in a certain state action pair. t is the advantage function, which is used to measure the quality of an action relative to the current strategy; ∈ is the clipping threshold, which controls the update amplitude of the new and old strategies; clip(·) is the pruning function, which changes r t (θ) is clipped to the interval [1-τ,1+τ];

[0037] The value function update target is optimized using mean square error, as follows:

[0038]

[0039] Among them, R t is the cumulative return starting from time t;

[0040] The entropy regularization term is as follows:

[0041]

[0042] Where β is the entropy regularization coefficient.

[0043] Therefore, the present invention adopts the above-mentioned running speed safety control method based on the reinforcement learning hybrid control strategy, and the beneficial effects are as follows:

[0044] The present invention combines the precision of control theory with the learning-oriented adaptive mechanism in reinforcement learning, designs a hybrid control strategy with fractional-order PID as the main controller and reinforcement learning PPO algorithm as the auxiliary controller, innovatively integrates the determinism of fractional-order PID control with the adaptability of reinforcement learning, inherits the inherent robustness and stability characteristics of fractional-order PID controller, and realizes the instant update of control strategy through reinforcement learning PPO algorithm, effectively enhances the ability of high-speed train speed controller to suppress uncertainty and interference in strong wind environment, greatly improves the speed control accuracy of high-speed train, realizes the accurate tracking of target running speed by high-speed train, and achieves the purpose of safe operation of high-speed train.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a framework diagram of a hybrid control strategy of an embodiment of a running speed safety control method based on a reinforcement learning hybrid control strategy of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0048] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0049] Embodiment 1

[0050] A running speed safety control method based on a reinforcement learning hybrid control strategy comprises the following steps:

[0051] S1. Construct a high-speed train operation process model.

[0052] High-speed trains are highly nonlinear, strongly coupled and complex objects. To enable high-speed trains to accurately track a given target running speed, the train running process must first be dynamically analyzed and mathematically described.

[0053] The dynamic equation is as follows:

[0054]

[0055] Where, v is the train speed, m is the acceleration coefficient, F is the resultant force on the train, n is the traction or braking force of the train, and is the system control input, f r is the basic resistance of the train, a0 is the rolling mechanical force coefficient of the train, a1 is the other mechanical resistance coefficient of the train, and a2 is the external air resistance coefficient of the train.

[0056] a0, a1 and a2 are all resistance coefficients during the operation of the train. a0, a1 and a2 are closely related to the train operation environment. Especially in strong wind environments, it is difficult to obtain their precise values, so they are highly uncertain.

[0057] S2. Construct a reward function for a hybrid control strategy based on reinforcement learning and fractional-order PID.

[0058] The reward function defines the immediate feedback that the agent receives after taking an action in the environment. The role of the reward function is to guide the agent on how to learn and optimize the behavior strategy.

[0059] The reward function is as follows:

[0060]

[0061] Among them, e t represents the control error at time t, c is a constant reward and is set to 100. When the control error e t When the absolute value of is less than 0.001, a constant reward c is added.

[0062] The design of this reward function enables the high-speed train to have the ability to resist disturbances and suppress uncertainties in strong wind environments, thereby improving the control accuracy of the high-speed train's running speed.

[0063] S3. Construct a hybrid control strategy based on reinforcement learning and fractional-order PID to jointly generate the final control signal through superposition processing.

[0064] like Figure 1 As shown in the figure, the hybrid control strategy adopts a main-auxiliary structure. In this strategy, the fractional-order PID is used as the main controller to provide the initial control baseline to ensure the stable operation and tracking efficiency of the system. The design of this main controller is based on control system theory and is used to adjust the system output to meet the preset reference trajectory. The reinforcement learning proximal policy optimization algorithm (PPO algorithm) assumes the function of auxiliary learning controller. The algorithm is designed according to the Actor-Critic architecture and optimizes the control strategy through interactive learning, thereby improving the performance of the main controller. In the Actor-Critic architecture, the Actor network is responsible for proposing control strategies, while the Critic network is responsible for evaluating the effects of these strategies, thereby promoting iterative updates and improvements of control strategies.

[0065] S31. In the intelligent hybrid control strategy, the main controller fractional-order PID is designed based on the definition of Grünwald-Letnikov (GL) fractional-order calculus. The definition of Grünwald-Letnikov fractional-order calculus is as follows:

[0066]

[0067] Where α is the fractional order, usually α>0 means differentiation, α<0 means integration; h means the time step; is the generalized binomial coefficient, specifically

[0068] Based on the definition of Grunwald-Letnikoff fractional-order calculus, the transfer function of the fractional-order PID is as follows:

[0069]

[0070] Among them, K p is the proportional gain, K d is the differential gain, K i is the integral gain, s is the Laplace operator, λ is the integral order, usually 0≤λ≤1, and μ is the differential order, usually 0≤μ≤1.

[0071] S32. In the hybrid intelligent control strategy, the reinforcement learning PPO algorithm plays the role of assisting the learning controller; the PPO algorithm is based on the Trust Region Policy Optimization (TRPO) algorithm, and uses a truncation method to construct the objective function to ensure that the difference between the new strategy and the old strategy is controlled within a certain range to improve the stability of model training. The PPO algorithm is an Actor-Critic method, and the Actor network is responsible for generating specific action strategies π θ , determines what action the agent takes in the current state a t ; The Critic network is responsible for evaluating the Actor's strategic performance and providing the direction and size of optimization; the optimization goal is to find the optimal neural network parameter θ * To obtain the maximum return, the objective function formula is as follows:

[0072] L CLIP (θ) = E t [min(r t (θ)·A t ,clip(r t (θ),1-τ,1+τ)·A t )];

[0073] Among them, r t (θ) is the probability ratio, which indicates the probability ratio of the new and old strategies in a certain state-action pair; A t is the advantage function, which is used to measure the quality of an action relative to the current strategy; ∈ is the clipping threshold, which controls the update amplitude of the new and old strategies; clip(·) is the pruning function that changes r t (θ) is clipped to the interval [1-τ,1+τ].

[0074] The detailed steps of the PPO algorithm are as follows:

[0075] S321, Initialization: Initialize the strategy parameters θ and value function parameters φ, as well as initialize the environment and experience storage.

[0076] S322, Collect trajectories: interact with the environment, collect multiple trajectories, and at time step t, according to the action strategy π θ (a|s) select action a t , collect status s t 、Action a t and reward r t , and save the data of the trajectory.

[0077] S323, calculate advantage function: use time difference method to estimate advantage function A t :

[0078] At =δ t +γδ t+1 +γ 2 δ t+2 +…;

[0079] Among them, δ t is the TD error, specifically δ t =r t +γV φ (s t+1 )-V φ (s t ), V φ (s t ) is the value function of the current state, and γ is the discount factor that controls the importance of future rewards.

[0080] S324. Construct objective function: The objective function consists of three parts: policy update objective, value function update objective and entropy regularization term.

[0081] The policy update targets are as follows:

[0082] L CLIP (θ) = E t [min(r t (θ)·A t ,clip(r t (θ),1-τ,1+τ)·A t )];

[0083] Among them, r t (θ) is the probability ratio, which indicates the probability ratio of the new and old strategies in a certain state-action pair; A t is the advantage function, which is used to measure the quality of an action relative to the current strategy; ∈ is the clipping threshold, which controls the update amplitude of the new and old strategies; clip(·) is the pruning function that changes r t (θ) is clipped to the interval [1-τ,1+τ].

[0084] The value function update goal is to optimize the value function using the mean square error (MSE), as follows:

[0085]

[0086] Among them, R t is the cumulative return starting from time t.

[0087] The entropy regularization term is used to increase the exploratory nature of the strategy, as shown below:

[0088]

[0089] Where β is the entropy regularization coefficient.

[0090] S325. Update parameters: Update the strategy parameters θ and value function parameters φ through stochastic gradient descent using the Adam optimizer.

[0091] S326, repeat training: repeat steps S322-S325 until the strategy converges or the maximum number of training steps is reached.

[0092] S33, the outputs of the main controller and the auxiliary learning controller are superimposed and processed to jointly generate the final control signal. This combination aims to make the high-speed train more robust to environmental disturbances and parameter uncertainties in strong wind environments by integrating traditional control methods with advanced machine learning technology, thereby optimizing the overall control performance and stability.

[0093] Therefore, the present invention adopts the above-mentioned running speed safety control method based on reinforcement learning hybrid control strategy to solve the problems of poor anti-disturbance ability and slow system response speed in the prior art, realize the accurate tracking of the target running speed of the high-speed train, and achieve the purpose of safe operation of the high-speed train.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A running speed safety control method based on reinforcement learning hybrid control strategy, characterized in that: The following steps are involved: S1. Construct a high-speed train operation process model; S2. Construct a reward function for a hybrid control strategy based on reinforcement learning and fractional-order PID. S3. Construct a hybrid control strategy based on reinforcement learning and fractional-order PID to jointly generate the final control signal through superposition processing.

2. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 1 is characterized in that: In step S1, the high-speed train operation process is dynamically analyzed and mathematically described, and the dynamic equation is as follows: Where v is the train speed, m is the acceleration coefficient, F is the resultant force on the train, n is the traction or braking force of the train, and f r is the basic resistance of the train, a0 is the rolling mechanical force coefficient of the train, a1 is the other mechanical resistance coefficient of the train, and a2 is the external air resistance coefficient of the train.

3. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 1 is characterized in that: In step S2, the reward function is as follows: Among them, e t represents the control error at time t, c is a constant reward, and when the control error e t When the absolute value of is less than 0.001, a constant reward c is added.

4. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 1 is characterized in that: S3 is specifically: S31, designing the transfer function C(s) of the fractional-order PID of the main controller based on the definition of Grunwald-Letnikov fractional-order calculus; S32, constructing the objective function of the auxiliary learning controller based on the truncation method of the reinforcement learning PPO algorithm, and updating the strategy parameter θ and the value function parameter φ until the strategy converges or reaches the maximum number of training steps; S33, superimposing the outputs of the main controller and the auxiliary learning controller to jointly generate a final control signal.

5. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 4 is characterized in that: In step S31, the definition of Grunwald-Letnikov fractional calculus is as follows: Where α is the fractional order, α>0 means differentiation, α<0 means integration; h means the time step; is the generalized binomial coefficient, specifically Based on the definition of Grunwald-Letnikoff fractional-order calculus, the transfer function of the fractional-order PID is as follows: Among them, K p is the proportional gain, K d is the differential gain, K i is the integral gain, s is the Laplace operator, λ is the integral order, usually 0≤λ≤1, and μ is the differential order, usually 0≤μ≤1.

6. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 5 is characterized in that: S32 is specifically: S321, initialization: initializing the strategy parameters θ and value function parameters φ, as well as initializing the environment and experience storage; S322, Collect trajectories: interact with the environment, collect multiple trajectories, and at time step t, according to the action strategy π θ (a|s) select action a t , collect status s t 、Action a t and reward r t , and save the data of the trajectory; S323, calculate advantage function: use time difference method to estimate advantage function A t : A t =d t +gd t+1 +g 2 d t+2 +…; Among them, δ t is the TD error, specifically δ t =r t +γV φ (s t+1 )-V φ (s t ), V φ (s t ) is the value function of the current state, γ is the discount factor; S324, constructing an objective function: including a policy update objective, a value function update objective, and an entropy regularization term; S325, update parameters: update the strategy parameters θ and value function parameters φ through stochastic gradient descent using the Adam optimizer; S326, repeat training: repeat steps S322-S325 until the strategy converges or the maximum number of training steps is reached.

7. The running speed safety control method based on reinforcement learning hybrid control strategy according to claim 6 is characterized in that: In step S324, the strategy updates the target, as shown below: L CLIP (θ)=E t [min(r t (θ)·A t ,clip(r t (θ),1-τ,1+τ)·A t )]; Among them, r t (θ) is the probability ratio, which indicates the probability ratio of the new and old strategies in a certain state action pair. t is the advantage function, which is used to measure the quality of an action relative to the current strategy; ∈ is the clipping threshold, which controls the update amplitude of the new and old strategies; clip(·) is the pruning function, which changes r t (θ) is clipped to the interval [1-τ,1+τ]; The value function update target is optimized using mean square error, as follows: Among them, R t is the cumulative return starting from time t; The entropy regularization term is as follows: Where β is the entropy regularization coefficient.

Images