Train automatic operation control method, device, equipment, storage medium and product

Abstract

The invention provides an automatic train operation control method and device, equipment, a storage medium and a product, and the method comprises the steps: inputting a current train operation state into a strategy network, and obtaining control action probability distribution; the control action probability distribution is input into an MPC controller, and the future train running state and output data are predicted; inputting the future train operation state and the output data into the value function network to obtain an accumulated return estimated value of each feasible action adopted in the current state; and optimizing the control action probability distribution based on the accumulative return estimated value and the control target to obtain the optimized control action probability distribution so as to adjust the control action of the train and realize train operation control. According to the method, the strategy network and the value function network are integrated into the MPC controller, the learned control strategy can be applied to real-time operation of the train, the optimal train operation effect is achieved, and the train operation comfort, the control precision and the robustness are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of train control technology, and particularly relates to a train automatic operation control method, device, equipment, storage medium and product. BACKGROUND

[0002] The ATO (Automatic Train Operation) system is an important part of the urban rail transit signal system, and is mainly used for realizing automatic driving control of the train. The existing train automatic driving control method mainly includes the following two kinds: (1) a traditional PID (Proportional-Integral-Derivative) control algorithm: a PID control algorithm is adopted, and a target speed or position is set, and a control instruction is adjusted according to a real-time error; (2) a rule-based control method: a pre-defined rule library (such as a segmented speed curve) is used to control train operation, and an experience parameter is used to adjust a control strategy.

[0003] However, the existing control method has the following disadvantages: (1) poor comfort: in the process of train acceleration / braking, large vibration and noise are easily generated; (2) insufficient control effect: the control strategy cannot be dynamically adjusted in a complex scene, resulting in large parking error; (3) sensitive to uncertainty factors, and the control instruction is easy to deviate from the expectation, and the robustness is poor.

[0004] Therefore, how to control the automatic operation of the train to improve the train operation comfort, control accuracy and robustness is a problem to be solved at present. SUMMARY

[0005] The present application provides a train automatic operation control method, device, equipment, storage medium and product, to solve the defects of poor comfort, insufficient control effect and poor robustness of the existing train automatic operation control scheme, realize the optimal train operation effect, and improve the train operation comfort, control accuracy and robustness.

[0006] The present application provides a train automatic operation control method, which comprises: inputting a current train operation state into a policy network in a trained reinforcement learning model, to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model predictive MPC controller, and predicting a future train operation state and output data for representing future train performance indicators based on a train operation simulation model in the MPC controller; The future train operating state and the output data are input into the value function network in the trained reinforcement learning model to obtain the cumulative reward estimate of each possible action taken under the current train operating state, as output by the value function network. Based on the cumulative reward estimate of each action and the control objective, the probability distribution of the control action is optimized to obtain the optimized probability distribution of the control action. Based on the optimized control action probability distribution, the train's control actions are adjusted to achieve train operation control.

[0007] According to the automatic train operation control method provided by the present invention, the reinforcement learning model is trained in the following manner: Obtain a reference trajectory; the reference trajectory includes the train's operating status, control actions, and rewards at multiple time steps; Calculate the true cumulative return of the reference trajectory at the current time step; Calculate the estimated return value of the state-action value function; Calculate the first gradient of the current policy network and the second gradient of the current value function network; Based on the first gradient and the estimated reward value, the parameters of the current policy network are updated, and based on the actual cumulative reward, the estimated reward value, and the second gradient, the parameters of the current value function network are updated until the optimization objective is achieved.

[0008] According to a train automatic operation control method provided by the present invention, the optimization objective is to maximize the expected value of a first objective function; the optimization objective is expressed by the following formula: ; in, It represents the logarithm of the probability of choosing action a in state s; This represents the state-action value function after performing action a in state s; Let the first objective function be... Indicating in strategy Below, make To maximize the expected value; These are adjustable parameters of the policy network; This indicates the optimization objective.

[0009] According to the present invention, an automatic train operation control method is provided, wherein the train operation simulation model includes at least a train dynamics model, a track model, and a train-track interaction model; the step of inputting the control action probability distribution and the train operation simulation model into a model prediction (MPC) controller to predict the future train operation state and output data for characterizing future train performance indicators includes: determine a control action with the maximum probability based on the control action probability distribution, and determine a traction force corresponding to the control action with the maximum probability; predict a future train operation state based on the traction force and a train dynamics model; predict output data for representing a future train performance index based on the future train operation state and the train operation simulation model.

[0010] According to the train automatic operation control method provided by the application, the control action probability distribution is optimized based on the cumulative return estimation value of each action and a control target, and an optimized control action probability distribution is obtained, which comprises: obtain a second target function of the MPC controller; the second target function is designed based on the cumulative return estimation value of each action and a control target, and the control target comprises an action smoothing term and a reference action tracking term; optimize the control action probability distribution to minimize the second target function, and obtain an optimized control action probability distribution.

[0011] According to the train automatic operation control method provided by the application, the current train operation state is obtained by the following method: obtain the current train speed, the current train position, the real-time acceleration and the operation time stamp collected by the on-board sensor in real time; construct a multi-dimensional state vector based on the current train speed, the current train position, the real-time acceleration and the operation time stamp, and use it as the current train operation state.

[0012] The application also provides a train automatic operation control device, which comprises: a policy output module, which is used for inputting the current train operation state into a policy network in a trained reinforcement learning model, and obtaining a control action probability distribution output by the policy network; a prediction module, which is used for inputting the control action probability distribution into a model prediction MPC controller, predicting a future train operation state and output data for representing a future train performance index based on a train operation simulation model in the MPC controller; a return estimation module, which is used for inputting the future train operation state and the output data into a value function network in the trained reinforcement learning model, and obtaining a cumulative return estimation value of each action under the current train operation state output by the value function network; an optimization module, which is used for optimizing the control action probability distribution based on the cumulative return estimation value of each action and a control target, and obtaining an optimized control action probability distribution; The control adjustment module is configured to adjust the control action of the train based on the optimized control action probability distribution, so as to realize train operation control.

[0013] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the train automatic operation control method according to any one of the above when executing the computer program.

[0014] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the train automatic operation control method according to any one of the above.

[0015] The application further provides a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the train automatic operation control method according to any one of the above.

[0016] The train automatic operation control method, device, equipment, storage medium and product provided by the application learn the optimal control strategy by combining the policy network and the value function network, which is conducive to realizing more robust control effect; the future train operation state and output are predicted by the MPC controller, and the optimal control strategy is adaptively adjusted, which is conducive to improving the robustness and stability of the train automatic operation control; by integrating the policy network and the value function network into the MPC controller, the learned control strategy can be applied to the real-time operation of the train, and the optimal train operation effect is realized, so that the train operation comfort, control precision and robustness are improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0018] Figure 1 is one of the flowcharts of the train automatic operation control method provided by the embodiments of the application.

[0019] Figure 2 is a schematic diagram of the reinforcement learning model based on the Actor-Critic algorithm provided by the embodiments of the application.

[0020] Figure 3 is a flowchart of the model predictive control train provided by the embodiments of the application.

[0021] Figure 4Figure 2 is a flowchart of a train automatic operation control method according to an embodiment of the present application.

[0022] Figure 5 Figure 3 is a schematic diagram of a result of a conventional PID algorithm for train control.

[0023] Figure 6 Figure 4 is a schematic diagram of a result of a reinforcement learning Actor-Critic algorithm for train control according to an embodiment of the present application.

[0024] Figure 7 Figure 5 is a schematic diagram of a train automatic operation control device according to an embodiment of the present application.

[0025] Figure 8 Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0027] In the description of the embodiments of the present application, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art according to specific circumstances.

[0028] The existing train automatic driving control method mainly includes the following two kinds: (1) Conventional PID control algorithm: PID control algorithm is adopted, target speed or position is set, and control instructions (such as acceleration, brake level) are adjusted according to real-time error. The technical means of PID control algorithm is to dynamically adjust PID parameters according to speed error, and to trigger brake or acceleration instructions through fixed threshold. (2) Rule-based control method: A pre-defined rule base (such as a segmented speed curve) is used to control train operation, and acceleration / braking strategy is adjusted through experience parameters. The technical means of rule-based control method is to divide multiple control stages according to running interval, and to use fixed acceleration or braking force in each control stage.

[0029] However, the existing control method has the following disadvantages: (1) poor comfort: the traditional PID control algorithm and the rule-based control method use a fixed acceleration threshold, which can cause frequent and large step switching during acceleration / braking, resulting in large vibration and noise; (2) insufficient control effect: the control strategy cannot be dynamically adjusted in complex scenarios, resulting in large parking errors (i.e., average error > 0.5 m); (3) sensitive to uncertain factors (e.g., slope changes, track friction coefficient fluctuations, etc.), control instructions are prone to deviate from expectations (i.e., variance > 10%).

[0030] To solve the above problems, the embodiment of the present application integrates the policy network and the value function network into the MPC controller, which can apply the learned control strategy to the real-time operation of the train, and achieve the optimal train operation effect, thereby improving the train operation comfort, control accuracy and robustness.

[0031] Figure 1 is one of the flowcharts of the train automatic operation control method provided by the embodiment of the present application. Referring to Figure 1 The embodiment of the present application provides a train automatic operation control method, which can specifically include the following steps: Step 101, input the current train running state into the policy network of the trained reinforcement learning model to obtain the control action probability distribution output by the policy network.

[0032] It should be noted that the execution subject of the train automatic operation control method provided by the embodiment of the present application can be an electronic device, a component in the electronic device, an integrated circuit or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Illustratively, the mobile electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc., and the non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not limited by the embodiment of the present application. The embodiment of the present application will be described below with the train controller as the execution subject.

[0033] In the embodiments of the present application, the control strategy learned by the reinforcement learning model can be applied to actual train operation through the real-time control system to achieve optimal control effect. In the embodiments of the present application, the train automatic operation control is performed based on the prediction result of the reinforcement learning model, that is, the learned control strategy is applied to the real-time operation of the train, and the control action is continuously adjusted so that the running state of the train is as close as possible to the expected state, thereby improving the control effect of the train and achieving more robust control effect and higher operation efficiency.

[0034] In some embodiments, train operation data can be collected in real time by on-board sensors, and a multi-dimensional state vector can be constructed based on the train operation data collected by various on-board sensors to obtain the current train operation state (i.e., the actual state of the train).

[0035] In some embodiments, the Actor-Critic algorithm can be used to pre-train the control strategy to obtain the trained reinforcement learning model. The policy network (i.e., the Actor network) can be used to output the control action, and the value function network (i.e., the Critic network) can be used to evaluate the state value and guide the policy network to update.

[0036] In some embodiments, the control action can be an acceleration value or a brake level.

[0037] In some embodiments, the control action probability distribution output by the policy network can refer to the probability distribution of a plurality of feasible control actions predicted by the policy network according to the current train operation state.

[0038] In step 102, the control action probability distribution is input into a model prediction MPC controller, and based on a train operation simulation model in the MPC controller, future train operation states and output data for representing future train performance indicators are predicted.

[0039] In the embodiments of the present application, model predictive control (MPC) is a model-based control method that can optimize the control strategy to achieve optimal control effect by predicting the future system state and output. In train control, the MPC controller can be used to predict the future running state and output of the train, and optimize the control action to achieve optimal train operation control effect.

[0040] In the embodiments of the present application, when the learned control strategy is applied to the real-time operation of the train, the policy network and the value function network need to be integrated into the MPC controller. Specifically, the output of the policy network can be taken as the control action of the MPC controller, that is, the probability distribution output by the policy network is interpreted as the probability distribution of the control action; and the output of the value function network can be taken as the state estimation of the MPC controller, that is, the cumulative return estimation value output by the value function network is taken as the state of the train.

[0041] In some embodiments, the train operation simulation model and the control target can be input into the MPC controller in advance. After the output of the policy network (i.e., the control action probability distribution) is input into the MPC controller, the most probable actionable action can be determined based on the control action probability distribution output by the policy network, and the most probable actionable action is executed in the train operation simulation model, and the future train operation state and output are obtained after the action is executed.

[0042] The predicted future train state can include future train position, speed, acceleration, and other train operation information.

[0043] The output data predicted by the train operation simulation model can be data for representing future train performance indicators, such as comfort, energy consumption, and the like.

[0044] Step 103: input the future train operation state and the output data into the value function network in the trained reinforcement learning model to obtain the cumulative return estimation values of each actionable action under the current train operation state output by the value function network.

[0045] In some embodiments, the value function network can be used to evaluate the state value. The output of the value function network in the trained reinforcement learning model can be calculated according to the predicted future train operation state and the output data, that is, the cumulative return estimation values of each actionable action under the current train operation state are calculated by the value function network.

[0046] Step 104: based on the cumulative return estimation values of each actionable action and the control target, the control action probability distribution is optimized to obtain an optimized control action probability distribution.

[0047] In some embodiments, the objective function of the MPC controller can be designed according to the output of the value function network and the control target, so that the rolling optimization of the control strategy is realized by the MPC controller. Based on the cumulative return estimation values of each actionable action under the current train operation state output by the value function network in real time and the control target, the control strategy is optimized so that the objective function is minimized, and an optimized control strategy is obtained.

[0048] The optimization control strategy can refer to an optimization of a control action probability distribution output by a strategy network, so that the running state of the train is as close as possible to the expected state.

[0049] At step 105, a control action of the train is adjusted based on the optimized control action probability distribution, so as to realize train running control.

[0050] In some embodiments, the control action with the maximum probability can be determined according to the optimized control action probability distribution (i.e., the optimized control strategy), and the brake level corresponding to the control action with the maximum probability is determined, so as to output the corresponding brake level to control the train running, and the optimal train running effect can be realized.

[0051] In some embodiments, during the actual running of the train, the control action of the train can be continuously adjusted, so that the actual running state of the train is as close as possible to the expected state.

[0052] The embodiments of the present application learn the optimal control strategy by combining the strategy network and the value function network, which is beneficial to realize a more robust control effect; the future train running state and output are predicted by the MPC controller, and the optimized control strategy is adaptively adjusted, which is beneficial to improve the robustness and stability of the train automatic running control; by integrating the strategy network and the value function network into the MPC controller, the learned control strategy can be applied to the real-time running of the train, the optimal train running effect is realized, and the train running comfort, control accuracy and robustness are improved.

[0053] In an alternative embodiment, the current train running state can be obtained by the following steps: At step S11, a current train speed, a current train position, a real-time acceleration and a running timestamp collected by an on-board sensor in real time are acquired. At step S12, a multi-dimensional state vector is constructed based on the current train speed, the current train position, the real-time acceleration and the running timestamp, and is taken as the current train running state.

[0054] The state representation refers to the conversion of train running state information into a form that can be processed by a computer. The embodiments of the present application can adopt a state representation method based on sensor data, i.e., the running state of the train is represented as a vector, and the state vector contains multi-dimensional train state information such as speed, position, acceleration, time, etc.

[0055] In some embodiments, the on-board sensor can include a speed sensor, a position encoder, an accelerometer, a clock module, etc. Multi-dimensional train state information can be collected in real time by the on-board sensor, so as to construct a multi-dimensional state vector.

[0056] As an example, a four-dimensional state vector can be constructed By representing the running state of the train as a four-dimensional state vector, the train can be conveniently input into a deep reinforcement learning model for learning and decision-making. Wherein, v is the current train speed (unit: m / s), x is the remaining distance of the train from the target stopping point (i.e. the current train position) (unit: m), a is the real-time acceleration (unit: m / s2), t is the running timestamp (unit: s), and T represents the transposition operation of the vector.

[0057] In some embodiments, other train state information such as train load, environmental temperature, track slope, etc. can also be added according to actual needs on the basis of the four-dimensional state information to form a high-dimensional state vector, thereby improving the accuracy and robustness of the reinforcement learning model. Through continuous learning and optimization, the algorithm can gradually improve the control effect of the train, thereby achieving more robust control effect and higher running efficiency.

[0058] In an alternative embodiment, the reinforcement learning model can be trained by the following steps: Step S21, obtaining a reference trajectory; the reference trajectory includes train running states, control actions and rewards of multiple time steps; Step S22, calculating the real cumulative return of the reference trajectory at the current time step; Step S23, calculating the estimated return value of the state-action value function; Step S24, calculating the first gradient of the current policy network and the second gradient of the current value function network; Step S25, updating the parameters of the current policy network based on the first gradient and the estimated return value, and updating the parameters of the current value function network based on the real cumulative return, the estimated return value and the second gradient, until the optimization goal is reached.

[0059] The reinforcement learning model can learn the optimal control strategy according to the running state and control action of the train. The embodiment of the present application adopts a model based on deep reinforcement learning, i.e. taking the running state of the train as input and learning the optimal control action through a deep neural network. Specifically, the embodiment of the present application adopts an algorithm based on Actor-Critic, i.e. combining policy network and value function network, thereby achieving more robust control effect.

[0060] Figure 2 is a schematic diagram of the reinforcement learning model based on the Actor-Critic algorithm provided by the embodiment of the present application. Referring to Figure 2In the embodiments of the present application, the Actor-Critic based algorithm is a method combining policy evaluation and policy improvement, in which the policy evaluation is performed by a value function network, and the policy improvement is performed by a policy network. The Actor-Critic algorithm can include two neural networks: one is a policy network (Actor) for outputting control actions, and the other is a value function network (Critic) for evaluating the value of the current state.

[0061] In some embodiments, in order to achieve the optimization goal of the Actor-Critic based algorithm, it is necessary to train the reinforcement learning model. First, the current policy network A reference trajectory is generated: (1) ; Where T is the length of the trajectory, s is the train running state, a is the control action, and r is the reward. At the current state , the action can be taken according to the current policy, and the reward is obtained, and then the next state is transferred, and this is continuously carried out until the task is completed.

[0062] Next, the true cumulative return of the reference trajectory at the current time step t can be calculated: (2) ; Where is a discount factor, and its value range is , which is used to control the importance of future rewards; is the reward of the i-th trajectory; T is the length of the reference trajectory; is the true cumulative return at the current time step t.

[0063] Then, the estimated return value of the state-action value function can be calculated , where is the parameter of the value function network, and the policy gradient (i.e. the first gradient) of the current policy network is calculated.

[0064] Finally, the parameters of the policy network and the value function network can be updated using the gradient ascent method, so that the expected value of the objective function is maximized, thereby completing the model training when the optimization goal is achieved.

[0065] The update formula of the policy network can be: (3) ; Where It is the learning rate, and its range is... This affects the step size of parameter updates; It is the policy gradient of the current policy network; It is the current trajectory. The true cumulative return; These are the parameters of the current policy network. These are the parameters of the updated policy network (i.e., the parameters of the policy network at the next time step).

[0066] The update formula for the value function network can be: (4); in, It is the learning rate; This refers to the TD (Temporal Difference Error), which can be the difference between the actual cumulative return and the estimated return. Representing the value function network in state and actions The gradient below; These are the parameters of the current value function network. These are the parameters of the updated value function network (i.e., the parameters of the value function network at the next time step).

[0067] This invention continuously updates the parameters of the value function network using TD errors, enabling the policy network and the value function network to cooperate. The output of the value function network is closer to the true cumulative reward, thereby improving the performance and stability of the reinforcement learning model.

[0068] In an alternative embodiment, the optimization objective may be to maximize the expected value of the first objective function.

[0069] In some embodiments, it is assumed that the current train operating status is If the output of the policy network is 'a' and the output of the value function network is 'V(s)', then the optimization objective of the Actor-Critic algorithm can be to maximize the state-action value function (in this case, the state-action value function is the first objective function), that is: (5); in, These are adjustable parameters of the policy network; It is the output of the policy network; This represents the state-action value function after performing action a in state s; Indicating in strategy Under the following conditions, the state-action value function The expected value.

[0070] In some embodiments, the objective function formula (5) can be rewritten into the following formula (6) according to the policy gradient theorem: (6); wherein, denotes the logarithm of the probability of selecting action a in state s; denotes the state-action value function after performing action a in state s; is the first objective function, denotes maximizing the expected value of under the policy ; is the adjustable parameter of the policy network; denotes the optimization target.

[0071] In an optional embodiment, the train operation simulation model at least includes a train dynamics model, a track model and a train-track interaction model; the inputting of the control action probability distribution and the train operation simulation model into the model predictive MPC controller to predict the future train operation state and the output data for characterizing the future train performance index can specifically include: Step S31, determining the control action with the maximum probability based on the control action probability distribution, and determining the traction force corresponding to the control action with the maximum probability; Step S32, predicting the future train operation state based on the traction force and the train dynamics model; Step S33, predicting the output data for characterizing the future train performance index based on the future train operation state and the train operation simulation model.

[0072] In some embodiments, the train operation simulation model and the control target can be input into the MPC controller to predict the future train state and the output, to predict the future train position, speed, acceleration, etc.

[0073] In some embodiments, it is assumed that within the prediction time domain N (for example, 10s in the future), a step calculation is performed every Δt (for example, 0.1s). The dynamics equation can be as follows: (7); wherein, the traction force F 牵引 is determined by the control action, which can come from the action probability distribution output by the Actor network (i.e., the control action with the maximum probability in the action probability distribution output by the Actor network is determined as the control action for determining the traction force); by inputting the current train speed v t into the train operation simulation model for train simulation operation, the train resistance F 阻力 (v t); m is the train weight; a t+1 is the predicted next time step train acceleration.

[0074] In some embodiments, the MPC controller can adjust the A0(a0, v0, x0), A1(a1, v1, x1), …, A N-1 (a N-1 , v N-1 , x N-1 ) sequence to minimize the objective function. The train speed at the next time step and the train position at the next time step can be calculated as follows: (8). (9). Therefore, in the predicted sequence, the acceleration sequence is , the speed sequence is , and the position sequence is .

[0075] By applying a sequence of control actions u t , u t+1 ,..., u t+N-1 , a state trajectory s t+1 , s t+2 ,..., s t+N can be generated.

[0076] In an optional embodiment, the control action probability distribution is optimized based on the cumulative return estimate of each actionable action and the control target to obtain an optimized control action probability distribution, which can specifically include: Step S41, a second objective function of the MPC controller is obtained; the second objective function is designed based on the cumulative return estimate of each actionable action and a control target, and the control target includes an action smoothing term and a reference action tracking term; Step S42, the control action probability distribution is optimized to minimize the second objective function to obtain an optimized control action probability distribution.

[0077] In some embodiments, the objective function (i.e., the second objective function) of the MPC controller aims to control the optimization target in real time. The formula of the objective function of the MPC controller can be as follows: (10). wherein, is the jerk, which is the minimum rate of change of control action, can guarantee the smooth change of traction / braking force, reduce the jitter and impact in train operation, and improve passenger comfort (0.5 m / s³ can be the comfort threshold, and the jerk <0.5 m / s³ can indicate that the train operation control meets the comfort requirement).

[0078] wherein, S is the smoothness weight (each dimension of acceleration / braking force); is the control action at the t+k step, is the control action at the t+k-1 step, and T is the transpose operation.

[0079] wherein, is the reference tracking term, which makes the control action close to the reference action suggested by the policy network, and can guarantee that the MPC controller does not deviate from the learned optimal policy.

[0080] wherein, R is the reference tracking weight, and a is the reference tracking coefficient, and N is the prediction horizon (about 10-20 seconds), is the control action at the t+k step, is the reference action at the t+k step, and T is the transpose operation. Based on the cumulative return estimates of each actionable action output by the value function network, the reference action that can maximize the long-term return can be selected as the reference action in the reference tracking term.

[0081] In some embodiments, control action constraints can be set. For example, u min ≤ u t+k ≤ u max , Δu max ; wherein u min is the maximum braking acceleration (m / s²), u max is the maximum traction acceleration (m / s²), and Δu max is the acceleration rate limit (m / s³).

[0082] As an example, the maximum braking acceleration can be -20 m / s², the maximum traction acceleration can be 20 m / s², and the acceleration rate limit can be 0.5 m / s³.

[0083] The embodiment of the present application utilizes the target function of the MPC controller to optimize the control strategy, can design the action smoothing term in the target function, reduces the mutation of level switching through the smooth control action, is favorable for improving the operation comfort; through the design of the reference tracking term, the control action is close to the reference action recommended by the strategy network, ensures that the MPC controller does not deviate from the learned optimal strategy, thereby maintaining stable control effect in the uncertain environment, avoiding the deviation of the control instruction from the expectation, and improving the robustness; through the dynamic adjustment of the control strategy by the MPC controller, the parking error can be reduced, thereby improving the control precision and improving the control effect.

[0084] The process of the model predictive control train is described below through an embodiment.

[0085] Figure 3 is the process schematic diagram of the model predictive control train provided by the embodiment of the present application. Refer to Figure 3 In some embodiments, when the learned control strategy is applied to the real-time operation of the train, the strategy network and the value function network need to be integrated into the MPC controller. The following steps can be adopted: Step 1, the output of the strategy network is taken as the control action of the MPC controller, that is, the probability distribution output by the strategy network is interpreted as the probability distribution of the control action.

[0086] Step 2, the output of the value function network is taken as the state estimation of the MPC controller, that is, the cumulative return estimation value output by the value function network is taken as the state of the train.

[0087] Step 3, the running model and control target of the train are transmitted into the MPC controller, the future train state and output are predicted through the prediction model, and the future train position, speed, acceleration, etc. are predicted.

[0088] Step 4, according to the predicted train state and output, the output of the value function network is calculated, that is, the estimated value of the cumulative return of each actionable action under the current state is calculated.

[0089] Step 5, according to the output of the value function network and the control target, the target function of the MPC controller is designed, including minimizing the energy consumption of the train and minimizing the running time of the train, etc.

[0090] Step 6, the control strategy is optimized to minimize the target function, for example, the control action probability distribution is optimized to make the running state of the train as close as possible to the expected state.

[0091] Step 7, according to the optimized control strategy, the control action of the train is adjusted to achieve the optimal train running effect.

[0092] Step 8: Repeat Steps 3-7 continuously to achieve real-time train control.

[0093] In this way, the learned control strategy can be applied to the real-time operation of the train to achieve optimal train operation results. At the same time, the MPC controller can adaptively adjust the control strategy according to the real-time status and output of the train, improving the robustness and stability of the control.

[0094] Figure 4 This is the second schematic flowchart of the automatic train operation control method provided in this embodiment of the invention. (Refer to...) Figure 4 In some embodiments, the train automatic operation control process based on the fusion of deep reinforcement learning and model predictive control (MPC) can be as follows: First, multi-dimensional sensor data is collected, and a multi-dimensional state vector s is constructed. t Candidate action a is generated through a policy network. t The MPC controller is used to perform rolling optimization of the control strategy, predict the train's running state in the next N steps, calculate the objective function J, and optimize and adjust the action sequence to obtain the optimized control action a. t `, and execute the corresponding control commands to update the next train status s t+1 This allows for continuous optimization and updates to achieve real-time train operation control.

[0095] To verify the effectiveness of the ATO control algorithm based on a deep reinforcement learning model provided in this embodiment of the invention, an experiment can be conducted in a train simulation environment. The experimental steps are as follows: Step 1: Establish a train simulation environment, including the train's dynamics model, track model, and train-track interaction model.

[0096] Step 2: Design the train's operation tasks, including minimizing the rate of change of control actions and minimizing the train's running time.

[0097] Step 3: Train the Actor-Critic model to learn the optimal control strategy so that the train can obtain the maximum cumulative reward during operation.

[0098] Step 4: In the train simulation environment, use the trained Actor-Critic model to control the train's operation and record data such as the train's status, control actions, and cumulative rewards.

[0099] Step 5: Repeat Step 4 to collect enough data to calculate indicators such as train comfort and average running time, and evaluate the effectiveness of the proposed deep reinforcement learning-based ATO control algorithm.

[0100] Step 6, compared with the traditional PID control algorithm, evaluate the pros and cons of the proposed ATO control algorithm based on deep reinforcement learning.

[0101] Step 7, statistical analysis of experimental results, verify the effectiveness of the proposed ATO control algorithm based on deep reinforcement learning.

[0102] Figure 5 is a schematic diagram of the result of the traditional PID algorithm vehicle control. Figure 6 is a schematic diagram of the result of the vehicle control of the reinforcement learning Actor-Critic algorithm provided by the embodiment of the present application. Refer to Figures 5-6 In the train simulation environment, the train is controlled using the ATO control algorithm based on deep reinforcement learning provided by the embodiment of the present application and the traditional PID control algorithm, and the state of the train, control action (level) and cumulative reward data are recorded. Based on Figure 5 and Figure 6 It can be seen that, under the same ATO speed, the ATO control algorithm based on deep reinforcement learning is more stable in giving ATO level in train operation than the traditional PID control algorithm, with smaller variance and better robustness, and better results in running comfort index and control effect index. This shows that the proposed ATO control algorithm based on deep reinforcement learning has good effect and application prospect in train control.

[0103] The embodiment of the present application has the following advantages: (1) Comfort is significantly improved: through the strategy optimization of reinforcement learning and the rolling adjustment of MPC, the fluctuation variance of control command (acceleration / braking level) is reduced by 50%. The number of sudden acceleration / braking is reduced by 60%, which can greatly reduce the car vibration and noise, and achieve the highest rating of international comfort standard. (2) Control accuracy is optimized and improved: the target stopping point error is reduced from 0.5 meters of the traditional method to 0.2 meters. (3) Robustness in complex environment: under complex scenarios (such as slope change ±5%), the control stability still maintains above 95%.

[0104] In summary, through experiments in the train simulation environment, the effectiveness of the ATO control algorithm based on deep reinforcement learning proposed by the embodiment of the present application is verified, and compared with the traditional PID control algorithm, the superiority of the algorithm is further proved.

[0105] The train automatic operation control device provided by the present application is described below, and the train automatic operation control device described below can be correspondingly referred to the train automatic operation control method described above.

[0106] Figure 7 is a structural schematic diagram of the train automatic operation control device provided by the embodiment of the present application. Refer to Figure 7The embodiment of the present application provides a train automatic operation control device, and the device can specifically include the following modules. The policy output module 710 is configured to input a current train operation state into a policy network in the trained reinforcement learning model, and obtain a control action probability distribution output by the policy network. The prediction module 720 is configured to input the control action probability distribution into a model prediction MPC controller, and predict a future train operation state and output data for representing a future train performance index based on a train operation simulation model in the MPC controller. The reward estimation module 730 is configured to input the future train operation state and the output data into a value function network in the trained reinforcement learning model, and obtain cumulative reward estimation values of each actionable action in the current train operation state output by the value function network. The optimization module 740 is configured to optimize the control action probability distribution based on the cumulative reward estimation values of the each actionable action and a control target, and obtain an optimized control action probability distribution. The control adjustment module 750 is configured to adjust a control action of the train based on the optimized control action probability distribution, so as to realize train operation control.

[0107] The embodiment of the present application combines the policy network and the value function network to learn the optimal control strategy, which is beneficial to realize more robust control effect; the MPC controller is used to predict the future train operation state and the output, so as to adaptively adjust and optimize the control strategy, which is beneficial to improve the robustness and stability of the train automatic operation control; the policy network and the value function network are integrated into the MPC controller, so that the learned control strategy can be applied to the real-time operation of the train, the optimal train operation effect is realized, and the train operation comfort, the control precision and the robustness are improved.

[0108] Figure 8 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 8As shown, the electronic device can include a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 complete mutual communication through the communications bus 840. The processor 810 can invoke a logic instruction in the memory 830 to execute a train automatic operation control method, which includes: inputting a current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model predictive MPC controller, predicting future train operation states and output data for representing future train performance indicators based on a train operation simulation model in the MPC controller; inputting the future train operation states and the output data into a value function network in the trained reinforcement learning model to obtain cumulative return estimation values of each actionable action under the current train operation state output by the value function network; optimizing the control action probability distribution based on the cumulative return estimation values of each actionable action and a control target to obtain an optimized control action probability distribution; and adjusting a control action of the train based on the optimized control action probability distribution to realize train operation control.

[0109] In addition, the logic instruction in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0110] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the train automatic operation control method provided by the above method, which comprises: inputting a current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model predictive MPC controller, and predicting a future train operation state and output data for representing a future train performance index based on a train operation simulation model in the MPC controller; inputting the future train operation state and the output data into a value function network in the trained reinforcement learning model to obtain cumulative return estimation values of each actionable action taken under the current train operation state output by the value function network; optimizing the control action probability distribution based on the cumulative return estimation values of each actionable action and a control target to obtain an optimized control action probability distribution; and adjusting a control action of the train based on the optimized control action probability distribution to achieve train operation control.

[0111] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program can be executed by a processor to implement the train automatic operation control method provided by the above method, which comprises: inputting a current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model predictive MPC controller, and predicting a future train operation state and output data for representing a future train performance index based on a train operation simulation model in the MPC controller; inputting the future train operation state and the output data into a value function network in the trained reinforcement learning model to obtain cumulative return estimation values of each actionable action taken under the current train operation state output by the value function network; optimizing the control action probability distribution based on the cumulative return estimation values of each actionable action and a control target to obtain an optimized control action probability distribution; and adjusting a control action of the train based on the optimized control action probability distribution to achieve train operation control.

[0112] The device embodiments described above are only schematic, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement it without creative labor.

[0113] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and necessary general hardware platforms through the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. A train automatic operation control method characterized by comprising: The method comprises: inputting a current train operation state into a policy network in a trained reinforcement learning model to obtain a control action probability distribution output by the policy network; inputting the control action probability distribution into a model predictive MPC controller, and predicting future train operation states and output data for representing future train performance indicators based on a train operation simulation model in the MPC controller; inputting the future train operation states and the output data into a value function network in the trained reinforcement learning model to obtain cumulative return estimation values of each actionable action under the current train operation state output by the value function network; optimizing the control action probability distribution based on the cumulative return estimation values of each actionable action and a control target to obtain an optimized control action probability distribution; adjusting a control action of the train based on the optimized control action probability distribution to realize train operation control.

2. The automatic train operation control method according to claim 1, characterized by, The reinforcement learning model is trained in the following manner: obtaining a reference trajectory; the reference trajectory comprises train operation states, control actions and rewards of multiple time steps; calculating a real cumulative return of the reference trajectory at a current time step; calculating an estimated return value of a state-action value function; calculating a first gradient of a current policy network and a second gradient of a current value function network; updating parameters of the current policy network based on the first gradient and the estimated return value, and updating parameters of the current value function network based on the real cumulative return, the estimated return value and the second gradient, until an optimization target is reached.

3. The automatic train operation control method according to claim 2, characterized by, The optimization target is to maximize the expected value of a first target function; the optimization target is represented by the following formula: ； wherein, represents a log of a probability of selecting action a in state s; represents a state-action value function after performing action a in state s; is a first objective function, represents an expected value of under a policy ; is an adjustable parameter of the policy network; represents an optimization target.

4. The automatic train operation control method according to claim 1, characterized by, The train operation simulation model at least comprises a train dynamics model, a track model and a train-track interaction model; the inputting of the control action probability distribution and the train operation simulation model into the model predictive MPC controller to predict future train operation states and output data for representing future train performance indicators comprises: determining a control action with the maximum probability based on the control action probability distribution, and determining a traction force corresponding to the control action with the maximum probability; predicting future train operation states based on the traction force and the train dynamics model; predicting output data for representing future train performance indicators based on the future train operation states and the train operation simulation model.

5. The automatic train operation control method according to claim 1, characterized by, The optimization of the control action probability distribution based on the cumulative return estimation values of each actionable action and the control target to obtain the optimized control action probability distribution comprises: obtaining a second target function of the MPC controller; the second target function is designed based on the cumulative return estimation values of each actionable action and a control target, and the control target comprises an action smoothing term and a reference action tracking term; optimizing the control action probability distribution to minimize the second target function to obtain the optimized control action probability distribution.

6. The automatic train operation control method according to claim 1, characterized by, The current train operation state is obtained in the following manner: acquire a current train speed, a current train position, a real-time acceleration and a running time stamp collected by a vehicle-mounted sensor in real time; construct a multi-dimensional state vector based on the current train speed, the current train position, the real-time acceleration and the running time stamp, and take the multi-dimensional state vector as a current train running state.

7. A train automatic operation control device characterized by comprising: comprise: a policy output module configured to input the current train running state into a policy network in a trained reinforcement learning model, and obtain a control action probability distribution output by the policy network; a prediction module configured to input the control action probability distribution into a model prediction MPC controller, and predict a future train running state and output data for representing a future train performance index based on a train running simulation model in the MPC controller; a reward estimation module configured to input the future train running state and the output data into a value function network in the trained reinforcement learning model, and obtain cumulative reward estimation values of each actionable action taken under the current train running state output by the value function network; an optimization module configured to optimize the control action probability distribution based on the cumulative reward estimation values of each actionable action and a control target, and obtain an optimized control action probability distribution; a control adjustment module configured to adjust a control action of the train based on the optimized control action probability distribution, so as to realize train running control.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the train automatic running control method in any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the train automatic running control method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the train automatic running control method in any one of claims 1 to 6.

Images