Automatic train operation control method, device, equipment, storage medium and product

Abstract

The application provides a train automatic operation control method, device, equipment, storage medium and product, the method comprises the following steps: inputting the current train operation state into a strategy network to obtain a control action probability distribution; inputting the control action probability distribution into an MPC controller to predict future train operation states and output data; inputting the future train operation states and output data into a value function network to obtain cumulative return estimation values of each actionable action under the current state; and optimizing the control action probability distribution based on the cumulative return estimation values and a control target to obtain an optimized control action probability distribution, so as to adjust the control action of the train and realize train operation control. The strategy network and the value function network are integrated into the MPC controller, the learned control strategy can be applied to the real-time operation of the train, the optimal train operation effect is realized, and the train operation comfort, control accuracy and robustness are improved.

Description

Technical Field

[0001] This invention relates to the field of train control technology, and in particular to a method, apparatus, equipment, storage medium, and product for automatic train operation control. Background Technology

[0002] ATO (Automatic Train Operation) system is an important component of urban rail transit signaling system, mainly used to realize automatic train control. Existing automatic train control methods mainly include the following two types: (1) Traditional PID (Proportional-Integral-Derivative) control algorithm: using PID control algorithm, by setting target speed or position, the control command is adjusted according to real-time error; (2) Rule-based control method: using predefined rule base (such as segmented speed curve) to control train operation, and adjusting the control strategy through empirical parameters.

[0003] However, the existing control methods have the following drawbacks: (1) poor comfort: large vibrations and noise are easily generated during train acceleration / braking; (2) insufficient control effect: the control strategy cannot be dynamically adjusted in complex scenarios, resulting in large stopping errors; (3) sensitive to uncertain factors, control commands are prone to deviate from expectations, and have poor robustness.

[0004] Therefore, how to automatically control train operation to improve train running comfort, control accuracy, and robustness is an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method, apparatus, equipment, storage medium, and product for automatic train operation control, which addresses the shortcomings of existing automatic train operation control schemes, such as poor comfort, insufficient control effect, and poor robustness, to achieve optimal train operation performance and improve train operation comfort, control accuracy, and robustness.

[0006] This invention provides a method for automatic train operation control, the method comprising: The current train operation status is input into the policy network of the trained reinforcement learning model to obtain the probability distribution of control actions output by the policy network. The probability distribution of the control action is input into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, the future train operation status and output data used to characterize the future train performance indicators are predicted. The future train operating state and the output data are input into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operating state, as output by the value function network. Based on the cumulative reward estimate of each action and the control objective, the probability distribution of the control action is optimized to obtain the optimized probability distribution of the control action. Based on the optimized control action probability distribution, the train's control actions are adjusted to achieve train operation control.

[0007] According to the automatic train operation control method provided by the present invention, the reinforcement learning model is trained in the following manner: Obtain a reference trajectory; the reference trajectory includes the train's operating status, control actions, and rewards at multiple time steps; Calculate the true cumulative return of the reference trajectory at the current time step; Calculate the estimated return value of the state-action value function; Calculate the first gradient of the current policy network and the second gradient of the current value function network; Based on the first gradient and the estimated reward value, the parameters of the current policy network are updated, and based on the actual cumulative reward, the estimated reward value, and the second gradient, the parameters of the current value function network are updated until the optimization objective is achieved.

[0008] According to a train automatic operation control method provided by the present invention, the optimization objective is to maximize the expected value of a first objective function; the optimization objective is expressed by the following formula: ; in, It represents the logarithm of the probability of choosing action a in state s; This represents the state-action value function after performing action a in state s; Let the first objective function be... Indicating in strategy Below, make To maximize the expected value; These are adjustable parameters of the policy network; This indicates the optimization objective.

[0009] According to the present invention, an automatic train operation control method is provided, wherein the train operation simulation model includes at least a train dynamics model, a track model, and a train-track interaction model; the step of inputting the control action probability distribution and the train operation simulation model into a model prediction (MPC) controller to predict the future train operation state and output data for characterizing future train performance indicators includes: Based on the probability distribution of the control action, the control action with the highest probability is determined, and the traction force corresponding to the control action with the highest probability is determined. Based on the aforementioned traction force and train dynamics model, predict the future train operating status; Based on the future train operation status and the train operation simulation model, predict the output data used to characterize the future train performance indicators.

[0010] According to a train automatic operation control method provided by the present invention, the step of optimizing the control action probability distribution based on the cumulative reward estimate of each action and the control objective to obtain an optimized control action probability distribution includes: Obtain the second objective function of the MPC controller; the second objective function is designed based on the cumulative reward estimate of each action and the control objective, the control objective including an action smoothing term and a reference action tracking term; The control action probability distribution is optimized to minimize the second objective function, resulting in the optimized control action probability distribution.

[0011] According to a train automatic operation control method provided by the present invention, the current train operation state is obtained in the following manner: Acquire the current train speed, current train position, real-time acceleration, and running timestamp collected in real time by the on-board sensors; Based on the current train speed, the current train position, the real-time acceleration, and the running timestamp, a multi-dimensional state vector is constructed and used as the current train running state.

[0012] The present invention also provides a train automatic operation control device, comprising: The strategy output module is used to input the current train operation status into the strategy network of the trained reinforcement learning model to obtain the control action probability distribution output by the strategy network. The prediction module is used to input the probability distribution of the control action into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, predict the future train operation status and output data used to characterize the future train performance indicators. The reward estimation module is used to input the future train operation state and the output data into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operation state output by the value function network. The optimization module is used to optimize the probability distribution of the control actions based on the cumulative reward estimate of each action and the control objective, so as to obtain the optimized probability distribution of the control actions. The control adjustment module is used to adjust the train's control actions based on the optimized control action probability distribution to achieve train operation control.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the automatic train operation control method described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic train operation control method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the automatic train operation control method as described above.

[0016] The train automatic operation control method, device, equipment, storage medium, and product provided by this invention learn the optimal control strategy by combining a policy network and a value function network, which is beneficial to achieving a more robust control effect. By predicting the future train operating state and output through the MPC controller and adaptively adjusting and optimizing the control strategy, it is beneficial to improve the robustness and stability of the train automatic operation control. By integrating the policy network and the value function network into the MPC controller, the learned control strategy can be applied to the real-time operation of the train to achieve the optimal train operation effect, thereby improving the train operation comfort, control accuracy, and robustness. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the automatic train operation control method provided in the embodiments of the present invention.

[0019] Figure 2 This is a schematic diagram of a reinforcement learning model based on the Actor-Critic algorithm provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the model predictive control train process provided in an embodiment of the present invention.

[0021] Figure 4This is the second flowchart of the automatic train operation control method provided in this embodiment of the invention.

[0022] Figure 5 This is a schematic diagram of the vehicle control results using a traditional PID algorithm.

[0023] Figure 6 This is a schematic diagram of the vehicle control results provided by the reinforcement learning Actor-Critic algorithm in an embodiment of the present invention.

[0024] Figure 7 This is a schematic diagram of the automatic train operation control device provided in an embodiment of the present invention.

[0025] Figure 8 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] In the description of embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0028] Existing automatic train control methods mainly include the following two types: (1) Traditional PID control algorithm: The PID control algorithm is used to adjust the control command (e.g., acceleration and braking level) according to the real-time error by setting the target speed or position. The technical means of the PID control algorithm is to dynamically adjust the PID parameters according to the speed error and to trigger the braking or acceleration command through a fixed threshold. (2) Rule-based control method: The train operation is controlled by a predefined rule base (e.g., segmented speed curves), and the acceleration / braking strategy is adjusted by empirical parameters. The technical means of the rule-based control method is to divide the operation into multiple control stages according to the operating section and to use a fixed acceleration or braking force in each control stage.

[0029] However, the existing control methods have the following drawbacks: (1) poor comfort: traditional PID control algorithms and rule-based control methods use fixed acceleration thresholds, which will lead to frequent and large-amplitude level switching during acceleration / braking, resulting in large vibrations and noise; (2) insufficient control effect: the control strategy cannot be dynamically adjusted in complex scenarios, resulting in large parking errors (i.e., average error > 0.5m); (3) sensitive to uncertain factors (such as slope changes, track friction coefficient fluctuations, etc.), and control commands are prone to deviate from expectations (i.e., variance > 10%).

[0030] To address the aforementioned issues, this invention integrates the policy network and value function network into the MPC controller, enabling the learned control strategy to be applied to the real-time operation of the train, achieving optimal train operation results, thereby improving train operation comfort, control accuracy, and robustness.

[0031] Figure 1 This is one of the flowcharts illustrating the automatic train operation control method provided in this embodiment of the invention. (Refer to...) Figure 1 This invention provides a method for automatic train operation control, which may specifically include the following steps: Step 101: Input the current train operation status into the policy network of the trained reinforcement learning model to obtain the control action probability distribution output by the policy network.

[0032] It should be noted that the executing entity of the train automatic operation control method provided in this embodiment of the invention can be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This embodiment of the invention does not specifically limit the specific implementation of these devices. The following embodiments of the invention describe the executing entity as a train controller.

[0033] In this embodiment of the invention, a real-time control system can be used to apply the control strategy learned by the reinforcement learning model to actual train operation, thereby achieving optimal control performance. This embodiment of the invention performs automatic train operation control based on the prediction results of the reinforcement learning model, applying the learned control strategy to the real-time operation of the train and continuously adjusting the control actions to make the train's operating state as close as possible to the expected state, thus improving the train's control performance and achieving more robust control and higher operating efficiency.

[0034] In some embodiments, train operation data can be collected in real time by on-board sensors. Based on the train operation data collected by various on-board sensors, a multi-dimensional state vector can be constructed to obtain the current train operation state (i.e., the current actual state of the train).

[0035] In some embodiments, the Actor-Critic algorithm can be used to pre-train the control policy to obtain a trained reinforcement learning model. The policy network (i.e., the Actor network) can be used to output control actions, and the value function network (i.e., the Critic network) can be used to evaluate state values ​​and guide the policy network updates.

[0036] In some embodiments, the control action can be an acceleration value or a braking level.

[0037] In some embodiments, the probability distribution of control actions output by the policy network can refer to the probability distribution of multiple feasible control actions predicted by the policy network based on the current train operating state.

[0038] Step 102: Input the probability distribution of the control action into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, predict the future train operation status and output data used to characterize the future train performance indicators.

[0039] In this embodiment of the invention, Model Predictive Control (MPC) is a model-based control method that optimizes control strategies to achieve optimal control performance by predicting future system states and outputs. In train control, the MPC controller can be used to predict the future operating state and output of the train and optimize control actions to achieve optimal train operation control.

[0040] In this embodiment of the invention, when applying the learned control strategy to the real-time operation of the train, it is necessary to integrate the strategy network and the value function network into the MPC controller. Specifically, the output of the strategy network can be used as the control action of the MPC controller, that is, the probability distribution of the strategy network output can be interpreted as the probability distribution of the control action; and the output of the value function network can be used as the state estimate of the MPC controller, that is, the cumulative reward estimate of the value function network output can be used as the state of the train.

[0041] In some embodiments, the train operation simulation model and control objectives can be pre-input into the MPC controller. After inputting the output of the policy network (i.e., the control action probability distribution) into the MPC controller, the action with the highest probability can be determined based on the control action probability distribution output by the policy network, and the action with the highest probability can be executed in the train operation simulation model. After the action is executed, the future train operation state and output are obtained.

[0042] The predicted future train status can include future train position, speed, acceleration, and other train operation information.

[0043] The output data predicted by the train operation simulation model can be data used to characterize future train performance indicators, such as comfort and energy consumption.

[0044] Step 103: Input the future train operation state and the output data into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operation state output by the value function network.

[0045] In some embodiments, the value function network can be used to evaluate state value. Based on predicted future train operating states and output data, the output of the value function network in the trained reinforcement learning model can be calculated; that is, the value function network calculates an estimate of the cumulative reward for each possible action taken in the current train operating state.

[0046] Step 104: Based on the cumulative reward estimate of each action and the control objective, optimize the control action probability distribution to obtain the optimized control action probability distribution.

[0047] In some embodiments, the objective function of the MPC controller can be designed based on the output of the value function network and the control objective, thereby enabling rolling optimization of the control strategy through the MPC controller. The control strategy can be optimized based on the cumulative reward estimates of each action that can be taken under the current train operating state, as well as the control objective, which are output in real time by the value function network, to minimize the objective function and obtain the optimized control strategy.

[0048] Among them, the optimized control strategy can refer to optimizing the probability distribution of control actions output by the strategy network, so as to make the train's operating state as close as possible to the expected state.

[0049] Step 105: Based on the optimized control action probability distribution, adjust the train's control actions to achieve train operation control.

[0050] In some embodiments, the control action with the highest probability can be determined based on the optimized control action probability distribution (i.e., the optimized control strategy), and the braking level corresponding to the control action with the highest probability can be determined, thereby outputting the corresponding braking level to control the train operation, which can achieve the optimal train operation effect.

[0051] In some embodiments, during actual train operation, the train's control actions can be continuously adjusted so that the actual operating state of the train is as close as possible to the expected state.

[0052] This invention combines a policy network and a value function network to learn the optimal control strategy, which is beneficial for achieving more robust control effects. By predicting the future train operating state and output through the MPC controller and adaptively adjusting and optimizing the control strategy, it is beneficial for improving the robustness and stability of automatic train operation control. By integrating the policy network and the value function network into the MPC controller, the learned control strategy can be applied to the real-time operation of the train to achieve the optimal train operation effect and improve train operation comfort, control accuracy and robustness.

[0053] In an optional embodiment, the current train operating status can be obtained through the following steps: Step S11: Obtain the current train speed, current train position, real-time acceleration, and running timestamp collected in real time by the on-board sensors; Step S12: Based on the current train speed, the current train position, the real-time acceleration, and the running timestamp, construct a multi-dimensional state vector and use it as the current train running state.

[0054] State representation refers to converting train operation status information into a form that can be processed by a computer. This invention can employ a state representation method based on sensor data, whereby the train's operation status is represented as a vector, containing multi-dimensional train status information such as speed, position, acceleration, and time.

[0055] In some embodiments, the on-board sensors may include speed sensors, position encoders, accelerometers, clock modules, etc. Multidimensional train status information can be collected in real time using the on-board sensors, thereby constructing a multidimensional state vector.

[0056] As an example, a four-dimensional state vector can be constructed. By representing the train's operating state as a four-dimensional state vector, it can be conveniently input into a deep reinforcement learning model for learning and decision-making. Here, v is the current train speed (in m / s), x is the remaining distance from the target stopping point (i.e., the current train position) (in m), a is the real-time acceleration (in m / s²), t is the running timestamp (in s), and T represents the vector transpose operation.

[0057] In some embodiments, additional train state information, such as train load, ambient temperature, and track gradient, can be added to the four-dimensional state information as needed to form a high-dimensional state vector, thereby improving the accuracy and robustness of the reinforcement learning model. Through continuous learning and optimization, the algorithm can gradually improve the train control effect, thereby achieving more robust control and higher operating efficiency.

[0058] In one alternative embodiment, the reinforcement learning model can be trained through the following steps: Step S21, obtain the reference trajectory; the reference trajectory includes the train running status, control actions and rewards at multiple time steps; Step S22: Calculate the true cumulative return of the reference trajectory at the current time step; Step S23: Calculate the estimated return value of the state-action value function; Step S24: Calculate the first gradient of the current policy network and the second gradient of the current value function network; Step S25: Update the parameters of the current policy network based on the first gradient and the estimated reward value, and update the parameters of the current value function network based on the actual cumulative reward, the estimated reward value, and the second gradient, until the optimization objective is achieved.

[0059] Reinforcement learning models can learn the optimal control strategy based on the train's operating state and control actions. This invention employs a deep reinforcement learning-based model, using the train's operating state as input and learning the optimal control actions through a deep neural network. Specifically, this invention uses an Actor-Critic-based algorithm, combining a policy network and a value function network to achieve more robust control performance.

[0060] Figure 2 This is a schematic diagram of a reinforcement learning model based on the Actor-Critic algorithm provided in an embodiment of the present invention. (Refer to...) Figure 2In this embodiment of the invention, the Actor-Critic-based algorithm is a method that combines policy evaluation and policy improvement. Policy evaluation is performed through a value function network, while policy improvement is performed through a policy network. The Actor-Critic algorithm can include two neural networks: one is a policy network (Actor), which outputs control actions, and the other is a value function network (Critic), which evaluates the value of the current state.

[0061] In some embodiments, to achieve the optimization objective of the Actor-Critic-based algorithm, a reinforcement learning model needs to be trained. First, the current policy network can be used. Generate a reference trajectory: (1); Where T is the length of the trajectory, s is the train's running state, a is the control action, and r is the reward. In the current state... Next, actions can be taken based on the current strategy. and receive a reward Then, it transitions to the next state. This process continues until the mission is completed.

[0062] Next, the reference trajectory at the current time step t can be calculated. True cumulative returns: (2); in, It is a discount factor, and its value range is... This is used to control the importance of future rewards; is the reward for the i-th segment of the trajectory; T is the length of the reference trajectory; It represents the actual cumulative return at the current time step t.

[0063] Then, the state-action value function can be calculated. Estimated return value ,in These are the parameters of the value function network, and the policy gradient (i.e., the first gradient) of the current policy network. .

[0064] Finally, the gradient ascent method can be used to update the parameters of the policy network and the value function network to maximize the expected value of the objective function, thereby completing model training when the optimization objective is achieved.

[0065] The update formula for the policy network can be: (3); in, It is the learning rate, and its range is... This affects the step size of parameter updates; It is the policy gradient of the current policy network; It is the current trajectory. The true cumulative return; These are the parameters of the current policy network. These are the parameters of the updated policy network (i.e., the parameters of the policy network at the next time step).

[0066] The update formula for the value function network can be: (4); in, It is the learning rate; This refers to the TD (Temporal Difference Error), which can be the difference between the actual cumulative return and the estimated return. Representing the value function network in state and actions The gradient below; These are the parameters of the current value function network. These are the parameters of the updated value function network (i.e., the parameters of the value function network at the next time step).

[0067] This invention continuously updates the parameters of the value function network using TD errors, enabling the policy network and the value function network to cooperate. The output of the value function network is closer to the true cumulative reward, thereby improving the performance and stability of the reinforcement learning model.

[0068] In an alternative embodiment, the optimization objective may be to maximize the expected value of the first objective function.

[0069] In some embodiments, it is assumed that the current train operating status is If the output of the policy network is 'a' and the output of the value function network is 'V(s)', then the optimization objective of the Actor-Critic algorithm can be to maximize the state-action value function (in this case, the state-action value function is the first objective function), that is: (5); in, These are adjustable parameters of the policy network; It is the output of the policy network; This represents the state-action value function after performing action a in state s; Indicating in strategy Under the following conditions, the state-action value function The expected value.

[0070] In some embodiments, the objective function formula (5) can be rewritten as the following formula (6) according to the policy gradient theorem: (6); in, It represents the logarithm of the probability of choosing action a in state s; This represents the state-action value function after performing action a in state s; Let the first objective function be... Indicating in strategy Below, make Maximize the expected value; These are adjustable parameters of the policy network; This indicates the optimization objective.

[0071] In an optional embodiment, the train operation simulation model includes at least a train dynamics model, a track model, and a train-track interaction model; the step of inputting the control action probability distribution and the train operation simulation model into the model prediction MPC controller to predict the future train operation state and output data for characterizing future train performance indicators may specifically include: Step S31: Based on the probability distribution of the control action, determine the control action with the highest probability, and determine the traction force corresponding to the control action with the highest probability; Step S32: Based on the traction force and train dynamics model, predict the future train operating status; Step S33: Based on the future train operation status and the train operation simulation model, predict the output data used to characterize the future train performance indicators.

[0072] In some embodiments, a train operation simulation model and control objectives can be input into the MPC controller to predict future train states and outputs, such as future train position, speed, and acceleration.

[0073] In some embodiments, it is assumed that a calculation is performed every Δt (e.g., 0.1 seconds) within the prediction time domain N (e.g., the next 10 seconds). The dynamic equations can be as follows: (7); Among them, traction force F 牵引 The control action is determined by the action probability distribution output by the Actor network (that is, the control action with the highest probability in the action probability distribution output by the Actor network is determined as the control action used to determine the traction force); by using the current train speed v t By inputting the data into a train operation simulation model to simulate train operation, the train resistance F can be obtained. 阻力 (v t); m is the weight of the train; a t+1 The predicted train acceleration for the next time step.

[0074] In some embodiments, the MPC controller can adjust A0(a0,v0,x0), A1(a1,v1,x1), ..., A in the rolling optimization. N-1 (a N-1 ,v N-1 ,x N-1 The sequence is used to minimize the objective function. The train speed at the next time step. and the train's location at the next time step The formula for calculating can be as follows: (8); (9); Therefore, in the predicted sequence, the acceleration sequence is: The velocity sequence is The position sequence is .

[0075] By applying a set of control action sequences u t u t+1 ,...,u t+N-1 The possible state trajectories s t+1 ,s t+2 ,...,s t+N .

[0076] In an optional embodiment, optimizing the control action probability distribution based on the cumulative reward estimate of each action and the control objective to obtain an optimized control action probability distribution may specifically include: Step S41: Obtain the second objective function of the MPC controller; the second objective function is designed based on the cumulative reward estimate of each action and the control objective, the control objective including an action smoothing term and a reference action tracking term; Step S42: Optimize the control action probability distribution to minimize the second objective function, thereby obtaining the optimized control action probability distribution.

[0077] In some embodiments, the objective function (i.e., the second objective function) of the MPC controller aims to optimize real-time control. The formula for the objective function of the MPC controller can be as follows: (10); in, It minimizes the rate of change of control actions, which can ensure smooth changes in traction / braking force, reduce vibration and shock during train operation, and improve passenger comfort (0.5m / s³ can be the comfort threshold, and a rate of change jerk < 0.5m / s³ indicates that the train operation control meets the comfort requirements).

[0078] in, S is the smoothness weight (for each dimension of acceleration / braking force); It is the control action at step t+k. It is the control action at step t+k-1, where T is the transpose operation.

[0079] in, It is a reference tracking term that makes the control action close to the reference action suggested by the policy network, ensuring that the MPC controller does not deviate from the learned optimal policy.

[0080] in, R is the reference tracking weight, α is the reference tracking coefficient, and N is the prediction time domain (approximately 10-20 seconds). It is the control action at step t+k. This is the reference action at step t+k, where T is the transpose operation. Based on the cumulative reward estimates of each action output by the value function network, it can help select the action that maximizes long-term reward as the reference action in the reference tracking term.

[0081] In some embodiments, control action constraints can be set. For example, u can be set. min ≤u t+k ≤u max , ≤Δu max ; where u min For the maximum braking acceleration (m / s²), u max For the maximum traction acceleration (m / s²), Δu max This is the limit for the rate of change of acceleration (m / s³).

[0082] As an example, the maximum braking acceleration can be -20 m / s², the maximum traction acceleration can be 20 m / s², and the acceleration change rate limit can be 0.5 m / s³.

[0083] This invention utilizes the objective function of the MPC controller for control strategy optimization. A smoothing term can be designed into the objective function to reduce the abruptness of level switching, thus improving operational comfort. By designing a reference tracking term, the control action is made close to the reference action suggested by the policy network, ensuring that the MPC controller does not deviate from the learned optimal strategy. This maintains stable control performance in uncertain environments, prevents control commands from deviating from expectations, and improves robustness. Furthermore, by continuously and dynamically adjusting the control strategy through the MPC controller, stopping errors can be reduced, thereby improving control accuracy and performance.

[0084] The process of model predictive control of trains is illustrated below through an example.

[0085] Figure 3 This is a schematic diagram of the model predictive control train process provided in an embodiment of the present invention. (Refer to...) Figure 3 In some embodiments, when applying the learned control strategy to the real-time operation of the train, it is necessary to integrate the policy network and value function network into the MPC controller. This can be done using the following steps: Step 1: Use the output of the policy network as the control action of the MPC controller, that is, interpret the probability distribution of the policy network output as the probability distribution of the control action.

[0086] Step 2: Use the output of the value function network as the state estimate of the MPC controller, that is, use the cumulative reward estimate of the value function network output as the state of the train.

[0087] Step 3: Input the train's operation model and control objectives into the MPC controller. The predictive model will then predict the future train status and output, including future train position, speed, and acceleration.

[0088] Step 4: Based on the predicted train state and output, calculate the output of the value function network, that is, calculate the estimated cumulative reward of taking each action in the current state.

[0089] Step 5: Based on the output of the value function network and the control objective, design the objective function of the MPC controller, including minimizing the train's energy consumption and minimizing the train's running time.

[0090] Step 6: Optimize the control strategy to minimize the objective function. For example, optimize the probability distribution of control actions so that the train's operating state is as close as possible to the expected state.

[0091] Step 7: Adjust the train's control actions according to the optimized control strategy to achieve the best train operation effect.

[0092] Step 8: Repeat Steps 3-7 continuously to achieve real-time train control.

[0093] In this way, the learned control strategy can be applied to the real-time operation of the train to achieve optimal train operation results. At the same time, the MPC controller can adaptively adjust the control strategy according to the real-time status and output of the train, improving the robustness and stability of the control.

[0094] Figure 4 This is the second schematic flowchart of the automatic train operation control method provided in this embodiment of the invention. (Refer to...) Figure 4 In some embodiments, the train automatic operation control process based on the fusion of deep reinforcement learning and model predictive control (MPC) can be as follows: First, multi-dimensional sensor data is collected, and a multi-dimensional state vector s is constructed. t Candidate action a is generated through a policy network. t The MPC controller is used to perform rolling optimization of the control strategy, predict the train's running state in the next N steps, calculate the objective function J, and optimize and adjust the action sequence to obtain the optimized control action a. t `, and execute the corresponding control commands to update the next train status s t+1 This allows for continuous optimization and updates to achieve real-time train operation control.

[0095] To verify the effectiveness of the ATO control algorithm based on a deep reinforcement learning model provided in this embodiment of the invention, an experiment can be conducted in a train simulation environment. The experimental steps are as follows: Step 1: Establish a train simulation environment, including the train's dynamics model, track model, and train-track interaction model.

[0096] Step 2: Design the train's operation tasks, including minimizing the rate of change of control actions and minimizing the train's running time.

[0097] Step 3: Train the Actor-Critic model to learn the optimal control strategy so that the train can obtain the maximum cumulative reward during operation.

[0098] Step 4: In the train simulation environment, use the trained Actor-Critic model to control the train's operation and record data such as the train's status, control actions, and cumulative rewards.

[0099] Step 5: Repeat Step 4 to collect enough data to calculate indicators such as train comfort and average running time, and evaluate the effectiveness of the proposed deep reinforcement learning-based ATO control algorithm.

[0100] Step 6: Compare the proposed ATO control algorithm based on deep reinforcement learning with the traditional PID control algorithm to evaluate its advantages and disadvantages.

[0101] Step 7: Perform statistical analysis on the experimental results to verify the effectiveness of the proposed ATO control algorithm based on deep reinforcement learning.

[0102] Figure 5 This is a schematic diagram of the vehicle control results using a traditional PID algorithm. Figure 6 This is a schematic diagram illustrating the vehicle control results of the reinforcement learning Actor-Critic algorithm provided in an embodiment of the present invention. (Refer to...) Figure 5-6 In a train simulation environment, the ATO control algorithm based on deep reinforcement learning and the traditional PID control algorithm provided in this embodiment of the invention are used to control the train, and data such as the train's state, control actions (levels), and cumulative rewards are recorded. Figure 5 and Figure 6 It can be seen that, under the same ATO speed, the ATO control algorithm based on deep reinforcement learning provides more stable ATO levels, smaller variance, and better robustness compared to the traditional PID control algorithm during train operation, achieving better results in terms of both operational comfort and control performance. This indicates that the proposed ATO control algorithm based on deep reinforcement learning has good performance and application prospects in train control.

[0103] The embodiments of the present invention have the following advantages: (1) Significantly improved comfort: Through strategy optimization of reinforcement learning and rolling adjustment of MPC, the fluctuation variance of control commands (acceleration / braking level) is reduced by 50%. The number of rapid acceleration / braking is reduced by 60%, which can significantly reduce the vibration and noise of the carriage and achieve the highest rating of international comfort standards. (2) Optimized and improved control accuracy: The target stopping point error is reduced from 0.5 meters in the traditional method to 0.2 meters. (3) Strong robustness in complex environments: Under complex scenarios (such as slope change ±5%), the control stability is still maintained above 95%.

[0104] In summary, through experiments conducted in a train simulation environment, the experimental results verified the effectiveness of the deep reinforcement learning-based ATO control algorithm proposed in this embodiment of the invention, and compared it with the traditional PID control algorithm, further demonstrating the superiority of the algorithm.

[0105] The automatic train operation control device provided by the present invention is described below. The automatic train operation control device described below can be referred to in correspondence with the automatic train operation control method described above.

[0106] Figure 7 This is a schematic diagram of the train automatic operation control device provided in an embodiment of the present invention. (Refer to...) Figure 7This invention provides a train automatic operation control device, which may specifically include the following modules: The strategy output module 710 is used to input the current train running state into the strategy network of the trained reinforcement learning model to obtain the control action probability distribution output by the strategy network. Prediction module 720 is used to input the probability distribution of the control action into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, predict the future train operation status and output data for characterizing the future train performance indicators. The reward estimation module 730 is used to input the future train operation state and the output data into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operation state output by the value function network. The optimization module 740 is used to optimize the probability distribution of the control actions based on the cumulative reward estimate of each action and the control objective, so as to obtain the optimized probability distribution of the control actions. The control adjustment module 750 is used to adjust the train's control actions based on the optimized control action probability distribution to achieve train operation control.

[0107] This invention combines a policy network and a value function network to learn the optimal control strategy, which is beneficial for achieving more robust control effects. By predicting the future train operating state and output through the MPC controller and adaptively adjusting and optimizing the control strategy, it is beneficial for improving the robustness and stability of automatic train operation control. By integrating the policy network and the value function network into the MPC controller, the learned control strategy can be applied to the real-time operation of the train to achieve the optimal train operation effect, thereby improving train operation comfort, control accuracy and robustness.

[0108] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communications bus 840. The processor 810 can call logic instructions in the memory 830 to execute a train automatic operation control method. This method includes: inputting the current train operating state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model prediction (MPC) controller, and based on a train operation simulation model in the MPC controller, predicting future train operating states and output data characterizing future train performance indicators; inputting the future train operating state and the output data into a value function network in a trained reinforcement learning model to obtain a cumulative reward estimate for each action taken in the current train operating state, output by the value function network; optimizing the control action probability distribution based on the cumulative reward estimate and the control objective to obtain an optimized control action probability distribution; and adjusting the train's control actions based on the optimized control action probability distribution to achieve train operation control.

[0109] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the automatic train operation control method provided by the above methods. The method includes: inputting the current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model prediction (MPC) controller, and based on the train operation simulation model in the MPC controller, predicting the future train operation state and output data for characterizing future train performance indicators; inputting the future train operation state and the output data into a value function network in a trained reinforcement learning model to obtain a cumulative reward estimate of each action that can be taken in the current train operation state, output by the value function network; optimizing the control action probability distribution based on the cumulative reward estimate of each action and the control objective to obtain an optimized control action probability distribution; and adjusting the train's control actions based on the optimized control action probability distribution to achieve train operation control.

[0111] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the automatic train operation control method provided by the above methods. The method includes: inputting the current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model prediction (MPC) controller, and based on a train operation simulation model in the MPC controller, predicting the future train operation state and output data characterizing future train performance indicators; inputting the future train operation state and the output data into a value function network in a trained reinforcement learning model to obtain a cumulative reward estimate of each action that can be taken in the current train operation state, output by the value function network; optimizing the control action probability distribution based on the cumulative reward estimate of each action and the control objective to obtain an optimized control action probability distribution; and adjusting the train's control actions based on the optimized control action probability distribution to achieve train operation control.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic train operation control, characterized in that, The method includes: The current train operation status is input into the policy network of the trained reinforcement learning model to obtain the probability distribution of control actions output by the policy network. The probability distribution of the control action is input into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, the future train operation status and output data used to characterize the future train performance indicators are predicted. The future train operating state and the output data are input into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operating state, as output by the value function network. Based on the cumulative reward estimate of each action and the control objective, the probability distribution of the control action is optimized to obtain the optimized probability distribution of the control action. Based on the optimized control action probability distribution, the train's control actions are adjusted to achieve train operation control; The reinforcement learning model is trained in the following way: Obtain a reference trajectory; the reference trajectory includes the train's operating status, control actions, and rewards at multiple time steps; Calculate the true cumulative return of the reference trajectory at the current time step; Calculate the estimated return value of the state-action value function; Calculate the first gradient of the current policy network and the second gradient of the current value function network; Based on the first gradient and the estimated reward value, the parameters of the current policy network are updated, and based on the actual cumulative reward, the estimated reward value and the second gradient, the parameters of the current value function network are updated until the optimization objective is achieved; The optimization objective is to maximize the expected value of the first objective function; the optimization objective is expressed by the following formula: ； in, It represents the logarithm of the probability of choosing action a in state s; This represents the state-action value function after performing action a in state s; Let the first objective function be... Indicating in strategy Below, make Maximize the expected value; These are adjustable parameters of the policy network; This indicates the optimization objective.

2. The automatic train operation control method according to claim 1, characterized in that, The train operation simulation model includes at least a train dynamics model, a track model, and a train-track interaction model; the step of inputting the control action probability distribution and the train operation simulation model into the Model Prediction (MPC) controller to predict the future train operation state and output data used to characterize future train performance indicators includes: Based on the probability distribution of the control action, the control action with the highest probability is determined, and the traction force corresponding to the control action with the highest probability is determined. Based on the aforementioned traction force and train dynamics model, predict the future train operating status; Based on the future train operation status and the train operation simulation model, predict the output data used to characterize the future train performance indicators.

3. The automatic train operation control method according to claim 1, characterized in that, The optimization of the control action probability distribution based on the cumulative reward estimate of each action and the control objective yields an optimized control action probability distribution, including: Obtain the second objective function of the MPC controller; the second objective function is designed based on the cumulative reward estimate of each action and the control objective, the control objective including an action smoothing term and a reference action tracking term; The control action probability distribution is optimized to minimize the second objective function, resulting in the optimized control action probability distribution.

4. The automatic train operation control method according to claim 1, characterized in that, The current train operating status is obtained through the following methods: Acquire the current train speed, current train position, real-time acceleration, and running timestamp collected in real time by the on-board sensors; Based on the current train speed, the current train position, the real-time acceleration, and the running timestamp, a multi-dimensional state vector is constructed and used as the current train running state.

5. A train automatic operation control device, characterized in that, include: The strategy output module is used to input the current train operation status into the strategy network of the trained reinforcement learning model to obtain the probability distribution of control actions output by the strategy network. The prediction module is used to input the probability distribution of the control action into the model prediction MPC controller, and based on the train operation simulation model in the MPC controller, predict the future train operation status and output data used to characterize the future train performance indicators. The reward estimation module is used to input the future train operation state and the output data into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operation state output by the value function network. An optimization module is used to optimize the probability distribution of the control actions based on the cumulative reward estimate of each action and the control objective, so as to obtain an optimized probability distribution of the control actions. The control adjustment module is used to adjust the train's control actions based on the optimized control action probability distribution to achieve train operation control. The reinforcement learning model is trained in the following way: Obtain a reference trajectory; the reference trajectory includes the train's operating status, control actions, and rewards at multiple time steps; Calculate the true cumulative return of the reference trajectory at the current time step; Calculate the estimated return value of the state-action value function; Calculate the first gradient of the current policy network and the second gradient of the current value function network; Based on the first gradient and the estimated reward value, the parameters of the current policy network are updated, and based on the actual cumulative reward, the estimated reward value and the second gradient, the parameters of the current value function network are updated until the optimization objective is achieved; The optimization objective is to maximize the expected value of the first objective function; the optimization objective is expressed by the following formula: ； in, It represents the logarithm of the probability of choosing action a in state s; This represents the state-action value function after performing action a in state s; Let the first objective function be... Indicating in strategy Below, make Maximize the expected value; These are adjustable parameters of the policy network; This indicates the optimization objective.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the automatic train operation control method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the automatic train operation control method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the automatic train operation control method as described in any one of claims 1 to 4.

Images