A Train Operation Scheduling Optimization Method Based on Deep Reinforcement Learning

Abstract

This invention provides a train operation scheduling optimization method based on deep reinforcement learning, comprising the following steps: establishing a system model; deep reinforcement of the Markov decision process; results and analysis. This invention addresses the impact of the uncertainty of unforeseen events on train operation, which can cause time delays, resulting in serious social impacts and significant economic losses. Based on train operation characteristics, a train dynamics model and a Markov model of the train operation scheduling system are established. Considering minimizing total train delay time and energy consumption, a train operation scheduling strategy based on a deep deterministic policy gradient algorithm is proposed. By adding noise to the training actions, the generalization ability and robustness of the method under uncertain environments are improved. The scheduling strategy proposed in this invention effectively reduces total train delay time and lowers energy consumption.

Description

Technical Field

[0001] This invention relates to train operation scheduling optimization, specifically to a train operation scheduling optimization method based on deep reinforcement learning. Background Technology

[0002] With the rapid development of China's high-speed rail network, the dispatching and command of high-speed rail has become extremely important. The high-speed rail network is expanding in scale and becoming more complex, exhibiting increasingly pronounced characteristics such as strong coupling, rapid evolution, and multiple constraints. Furthermore, the unpredictability of unforeseen events such as severe weather and equipment failures can lead to delays for single or multiple trains, causing serious social impact and significant economic losses. Therefore, research on real-time train dispatching optimization has crucial practical significance.

[0003] As train delays spread across the railway network, they can disrupt the operation of numerous trains. Therefore, dynamic scheduling of train schedules is necessary to better coordinate train operations, minimize delays, and reduce the scope of impact, thereby achieving high-speed rail train operation that is fast, safe, punctual, and comfortable for passengers. Research on dynamic scheduling of high-speed rail trains can be broadly categorized into three methods: simulation methods, operations research methods, and intelligent algorithms. Simulation methods model the research object, providing a detailed description of complex systems. To address the stability of optimal scheduling schemes under stochastic traffic disturbances, a framework combining railway optimization by means of alternative graphs (ROMA) with an environment for the design and simulation of railway network (EGTRAIN) has been proposed. Jia Zhilong et al. used their established simulation platform to simulate train flow adjustments under stochastic conditions. A train operation simulation model based on component combination and considering bidirectional operation and random disturbances during train operation can quickly generate a simulation framework corresponding to the train operation environment and rules, based on the required train operation plan and related road network data. The model's effectiveness was also confirmed in the final simulation verification. However, this simulation method has limitations. The quality of the model and the data significantly affect the simulation results, and it is difficult to simulate complex systems, particularly those with many logical decision steps, and cannot perform complex analyses. Operations research methods, on the other hand, typically establish a mathematical model first and then use a suitable algorithm to solve scheduling problems, achieving higher accuracy. Luan et al. used mixed-integer nonlinear programming (MINLP) and mixed-integer linear programming (MILP) methods to study the integration problem of real-time traffic management and train control, further considering minimizing both train delays and energy consumption during interval operation as optimization objectives to generate operation diagrams and speed curves. LAN et al. studied a branch-pricing-based solution algorithm with the goal of minimizing total arrival delays, verifying its superiority through comparison with the GUROBI solver. The dynamic scheduling optimization problem of high-speed trains is a large-scale combinatorial optimization problem with high complexity. Using operations research methods can lead to low search efficiency. Intelligent algorithms, however, can handle complex system models and effectively utilize the powerful computing capabilities of computers. To address the impact of unforeseen events, a particle swarm optimization algorithm for adjusting policy control parameters is proposed and compared with conventional genetic algorithms (GA) and particle swarm optimization (PSO) algorithms.Fan et al. employed ant colony optimization and genetic algorithms, and their results showed that the delay costs of these two algorithms were reduced by 30% and 28% respectively compared to the first-in-first-out (FIFO) method. Xu optimized train scheduling by improving the firefly algorithm (IFA), which can control the adverse effects of train delays to a very small extent while ensuring safe train operation.

[0004] The aforementioned studies primarily employ three methods for train operation scheduling optimization: common simulation methods, operations research methods, and intelligent algorithms. However, simulation methods, while providing intuitive models based on certain assumptions, struggle with large-scale problems, and their results often deviate significantly from reality. Operations research methods lack real-time performance and adaptability, failing to meet the needs of dynamic train adjustments. Intelligent algorithms are prone to getting trapped in local optima, and under multiple constraints, obtaining better solutions requires a significant increase in iterations, leading to excessive computation time. Deep reinforcement learning, combining the perceptual capabilities of deep learning with the decision-making abilities of reinforcement learning, can handle complex scheduling problems involving high-dimensional variables. Using reinforcement learning to solve combinatorial optimization problems, with appropriate reward function guidance, can achieve better solutions than manual scheduling. Therefore, this paper addresses the impact of unforeseen events on delays in high-speed railway trains. Combining the characteristics of continuous interaction and trial and error between agents and the environment in deep reinforcement learning, a dynamic scheduling model for high-speed railway trains is established with the goal of minimizing the total delay time of each train and the total energy consumption of the trains. A dual experience replay pool is introduced, a reward-based priority experience replay pool is adopted, and an improved DDPG method is used to optimize the scheduling strategy. Summary of the Invention

[0005] This invention provides a train operation scheduling optimization method based on deep reinforcement learning to solve the problems in the prior art.

[0006] The technical problem solved by this invention is achieved by the following technical solution:

[0007] This invention provides a train operation scheduling optimization method based on deep reinforcement learning, comprising the following steps:

[0008] Step 1: Establish a system model

[0009] 1.1 Train Operation Control System: The train braking curve is determined based on the target distance, target speed, and the train's own performance, without setting speed levels for each block section; by integrating with the train dispatching system, optimized train dispatching is achieved.

[0010] 1.2 Dynamic Model Formula: Determine the dynamic model of train operation and optimize train scheduling;

[0011] 1.3 Establishing a Markov Model for Train Operation Scheduling System

[0012] Step Two: Deepening the Markov Decision Process

[0013] A deep neural network (DNN) is used to approximate the reinforcement learning function, making it applicable to train operation scheduling problems in continuous state and action space. The deep deterministic policy gradient (DDPG) algorithm based on the actor-criticism framework is selected. Then, the optimal policy function is estimated by the deep neural network, and the train operation policy model based on MDP is solved to adapt to the uncertainty in the train operation process.

[0014] Step 3: Results and Analysis

[0015] To verify the effectiveness of the optimized scheduling strategy of the train operation scheduling system.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] 1. This invention studies a real-time scheduling strategy for high-speed train operation. Addressing the issue that the uncertainty of unforeseen events can cause time delays in train operations, resulting in severe social impacts and significant economic losses, a train dynamics model and a Markov model for the train operation scheduling system are established based on train operation characteristics. Considering minimizing total train delay time and energy consumption, a train operation scheduling strategy based on a deep deterministic policy gradient algorithm is proposed. By introducing noise into the training actions, the generalization ability and robustness of the method under uncertain environments are improved. The scheduling strategy proposed in this invention effectively reduces total train delay time and lowers energy consumption.

[0018] 2. High-speed railways, as the backbone of the national comprehensive transportation system, have experienced rapid and vigorous development over the past decade. This rapid development has also led to increased network complexity and wider distribution, placing higher demands on dynamic scheduling. The uncertainty of unforeseen events can cause train delays, which can even propagate along the network, resulting in widespread train arrival and departure delays, causing serious social impact and significant economic losses. Current manual scheduling methods for this problem lack foresight and specificity, making it difficult to quickly adjust for affected trains. To address these issues, this paper establishes a dynamic scheduling model for high-speed railways with the objective of minimizing the total delay time of all trains and the total energy consumption of all trains. The DDPG algorithm from deep reinforcement learning is used to solve the model. To accelerate the convergence speed of the algorithm network, a dual experience replay pool is introduced. A reward-based priority experience replay pool stores multiple consecutive samples as a basic unit, while an additional experience replay pool uses a resampling optimization mechanism to store samples with higher learning value. Finally, experiments verified the effectiveness of the improved DDPG algorithm for dynamic scheduling of high-speed rail, providing a good basis for high-speed rail dispatchers to make optimization decisions. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is the target distance-speed control curve in an embodiment of the present invention;

[0021] Figure 2 This invention presents a train operation scheduling framework based on the DDPG algorithm.

[0022] Figure 3 This is the multi-step priority experience replay pool structure of the present invention;

[0023] Figure 4 This is the reward curve for the training process in an embodiment of the present invention;

[0024] Figure 5 This is a train operation diagram according to an embodiment of the present invention;

[0025] Figure 6 This is a speed-distance curve according to an embodiment of the present invention. Detailed Implementation

[0026] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below with reference to specific illustrations.

[0027] Embodiments of the present invention provide a technical solution: a train operation scheduling optimization method based on deep reinforcement learning, comprising the following steps:

[0028] I. System Model

[0029] 1.1 Train Operation Control System

[0030] The train control system is responsible for ensuring the safe and rapid operation of trains. Its main functions are to control train intervals and speed. Currently, China's high-speed rail uses a target distance speed control curve system, employing a continuous single-stage braking speed control method. Figure 1 As shown, the train braking curve is determined based on the target distance, target speed, and the train's own performance, without setting a speed level for each block section.

[0031] The train can achieve real-time speed control based on the target distance speed control curve, enabling overspeed protection and automatic control of train deceleration, mitigation, and acceleration. By integrating with the train dispatching system, optimized train scheduling can be achieved.

[0032] 1.2 Dynamic Model Formula

[0033] To optimize train scheduling, we first need to know the dynamic model of train operation. The dynamic equations of a train can be expressed by the Lomonossoff equations:

[0034]

[0035] In the formula, M train Indicates train mass; s tr Indicates the train's location; t represents the train's travel time; F tr Indicates traction or braking force; R vr F represents mechanical resistance; gr This indicates additional resistance. Where:

[0036]

[0037] In the formula, v represents the train speed; μ f μ represents the train traction coefficient. b b[v(t)] represents the braking force coefficient of the train; b[v(t)] represents the braking force at the current speed v(t) of the train; g(s) tr) represents the gradient force; r(v) represents the basic drag at the current velocity, which can be expressed by the Davis equation containing constants B, C, and D as follows:

[0038] r(v) = B + Cv + Dv 2 (3)

[0039] Based on the total delay time T of all the above trains,

[0040]

[0041] In the formula, t a (ij), t d (ij) represent the actual arrival and departure times of train i at station j, respectively. Let a be the planned arrival and departure times of train i at station j. i (t) represents the acceleration of train i at time t, v i (t) represents the speed of train i at time t, s j Let t be the distance between station j and the starting station. i * The total planned running time for train i.

[0042] The train's total energy consumption E is

[0043]

[0044] Where, f[v i [t] represents the speed of train i as v. i The traction force at (t).

[0045] Based on the above, the objective function J, which includes the sum of the delay times of all trains and the total energy consumption of the trains, is:

[0046] J = ω t F t T+ω e F e E (6)

[0047] Where ω t ω e F represents the weighting coefficient. t F represents the cost per second of latency. e This indicates the unit energy cost per kilowatt-hour.

[0048] 1.3 Markov Model of Train Operation Scheduling System

[0049] Reinforcement learning is a machine learning method in which an intelligent agent takes certain actions to maximize cumulative rewards during continuous interaction with an unknown environment. It can be represented by a Markov Decision Process (MDP). Specifically, it can be represented by a quintuple.<s,a,p,r,γ> Let , where s represents all states, a represents all actions the agent can take, p represents the state transition probability, r represents the immediate reward for a specific state and action, and γ represents the return discount. A Markov model for train operation scheduling optimization is established with the objective of minimizing the sum of train delay times and the total energy consumption of the trains.

[0050] 6) State: s t ={t,s tr ,s j The observed train operation scheduling system status includes time, train position, distance from the originating station to the current station, and current traction force.

[0051] 7) Action: a t ={v i (t), a i The action performed is the agent's reaction to the environment after observing the state of the environment at the current time t.

[0052] 8) Reward: The reward function serves as a guide for the agent in formulating its strategy; it represents the desired control objective. Considering the optimal action guiding the train operation scheduling system to perform rational scheduling, the reward function is set as the system's objective function J.

[0053] r(s t ,a t )=ω t F t T(s t ,a t )+ω e F e E(s t ,a t (7)

[0054] Comfort affects ride quality and safety. When the train's acceleration changes beyond a certain value δ, an additional reward function r2 is provided to meet passengers' comfort needs.

[0055]

[0056] 9) Policy: In Markov decision processes, the policy is defined as the probability of a system transitioning from an observed state to an action state. The policy is defined as follows:

[0057] π(a|s)=P[a t =a∣s t =s] s∈S,a∈A (9)

[0058] 10) Reward: State-action value function Q π (s,a) represents the expected cumulative reward after performing action a in state s. The state-action value function Bellman equation is expressed as:

[0059] Qπ(s t ,a t )=Eπ[r t +γQπ(s t+1 ,a t+1 )∣s t ,a t (10)

[0060] The Bellman equation shows that the action value of the current state depends only on the current reward / penalty value and the action value of the next state, and can be solved iteratively. Solving for the optimal policy of the system is equivalent to solving for the optimal state-action value function, i.e.:

[0061] π * =argmaxQ π (s,a) (11)

[0062] The corresponding Bellman equation is:

[0063]

[0064] II. Deep Reinforcement Learning for Markov Decision Processes

[0065] In deep reinforcement learning, reinforcement learning is used to define the problem and optimization objective, while deep learning is used to solve for the policy function or value function. Traditional reinforcement learning methods perform well in small-scale discrete space problems. However, when dealing with continuous state variables, the number of discrete states increases exponentially with the increase of space dimension. This leads to dimensionality explosion and ineffective learning. In analyzing the train operation scheduling problem studied in this paper, since the variables in the state space are all continuous quantities, traditional reinforcement learning methods often cannot solve it effectively. To address this issue, we use deep neural networks (DNNs) to approximate the reinforcement learning function, making it applicable to train operation scheduling problems in continuous state and action spaces. We chose the Deep Deterministic Policy Gradient (DDPG) algorithm based on the actor-criticism framework. Then, we estimate the optimal policy function through deep neural networks and solve the MDP-based train operation policy model to adapt to the uncertainties in the train operation process.

[0066] 2.1 Train Operation Scheduling Framework Based on ddpg

[0067] A train operation scheduling framework is established based on the DDPG (Deep deterministic policy gradient) algorithm, such as... Figure 2 As shown. For the DDPG algorithm, the input to the target network is a 4-dimensional state vector, and the output is a 2-dimensional action vector. The input to the main network is the state vector s. t and action vector a t The output is the action value function Q(s). t ,a t During the learning process, the DDPG algorithm employs the experience replay mechanism found in deep Q-networks, creating a replay buffer by storing the agent's experience at each time interval. When training the agent, a small batch of experience samples is randomly extracted from the replay buffer each time, and the network parameters are updated according to gradient rules.

[0068] Historical data is used as the environmental state to train the DDPG algorithm network offline. Then, the parameters of the trained DDPG algorithm are fixed to solve the train operation scheduling problem. For the train operation scheduling system, when the scheduling task occurs in each cycle, it is based on the current system state s. t The agent uses a pre-trained DDPG algorithm target network and main network to select scheduling actions. The agent takes an action to enter the next environmental state and receives a reward. Then, system state information for time period t+1 is collected as new samples to derive the decision for that time period. In this way, the agent obtains a dynamic scheduling strategy.

[0069] 2.2 Dual-priority experience replay pool for reward guidance

[0070] The most crucial technique in the algorithm is experience replay, i.e., setting up an experience pool. The transition samples (s, a, r, s') obtained from the agent's interaction with the environment at each time step are stored in the replay memory unit. Then, some samples are randomly selected for training to update the parameters. The priority of the experience replay pool is determined by the TD-error, i.e., the difference between the target Q-value and the estimated Q-value, to evaluate the value of a sample. The larger the TD-error, the higher the priority p of the sample. The sampling probability P(i) of sample i is:

[0071]

[0072] Where p(i) = |δ i |+ε represents the priority of sample i, δ i This represents the sample TD error, used to adjust the degree of importance; a value of 0 indicates uniform sampling.

[0073] If the TD error of the sampled data is large, it means that the prediction accuracy of the agent has significant room for improvement, making this sample worthwhile for training the neural network. In the train operation scheduling system, the size of the reward obtained by the system is positively correlated with the quality of the actions taken. Therefore, rewards can be used to guide action selection. The proposed reward-guided priority experience replay mechanism is as follows:

[0074]

[0075] in The higher the reward value, the greater the priority of the action. When the reward is less than 0, the importance of the selected action is reduced. ε is the offset.

[0076] Traditional algorithms use only one experience pool to store data for updating network parameters. This paper uses two experience replay pools, B1 and B2. B1 is a multi-step priority experience replay pool, and B2 is a single-step priority experience replay pool. B1 stores samples indiscriminately, while B2 uses a resampling mechanism to store samples with probability P(i). During training, the network iteratively updates parameters after each round, using samples from B1. Once the model achieves good results, the parameter updates are changed to single-step iterative updates within each round, using samples from B2. By using better training samples and increasing the number of iterations, network performance is improved.

[0077] Multi-step priority experience replay pool B1, such as Figure 3 As shown, samples are stored in the experience pool without discrimination, and the priority of queue q is updated according to P(i).

[0078] 2.3 Specific steps of the ddp-g algorithm

[0079] The DDPG algorithm uses two independent networks to approximate the critic and actor functions, each with its own target network parameters θ. Q' and θ π' , where Q' and π' are the target Q value and the target policy, respectively.

[0080] In a value network, the loss function Loss(θ) is used. Q Optimize the parameters:

[0081]

[0082] in

[0083] y t =r t +γQ'(s t+1 ,π'(s t+1 |θ π' )|θ Q' (16)

[0084] y t The target action value, r, is obtained through the target value network. t In state s t The instantaneous reward obtained after taking an action, where γ is the discount factor, Q(s) t ,a t |θ Q The loss function (Loss(θ)) is the action value evaluated through an online value network after an action is taken. Q ) for θ Q The gradient is

[0085]

[0086] The network parameter values ​​are updated using gradient descent along the direction of decrease of the loss function.

[0087]

[0088] Where μ Q The learning rate of the value network.

[0089] In policy networks, The policy network is updated using the policy gradient method to represent the gradient of the action execution direction.

[0090]

[0091] Then, update the policy network parameters θ according to the deterministic policy gradient. π :

[0092]

[0093] Where is μ π Policy network learning rate.

[0094] The parameter update methods for the target value network and the target policy network are as follows:

[0095] θ Q '←τθ Q +(1-τ)θ Q '(twenty one)

[0096] θ π '←τθ π +(1-τ)θ π '(twenty two)

[0097] Where τ is the soft update coefficient By adjusting τ to control the update speed of the neural network, the target network can slowly track the online network, thereby improving the stability of learning.

[0098] III. Results and Analysis

[0099] In this section, to verify the effectiveness of the proposed train operation scheduling system optimization strategy based on the improved DDPG algorithm, this paper takes the Beijing-Shanghai High-Speed ​​Railway as a prototype for analysis and conducts a reasonable feasibility analysis on the solution results.

[0100] 3.1 Algorithm Verification and Analysis

[0101] To verify the feasibility of the algorithm's scheduling, this paper uses the Beijing-Shanghai High-Speed ​​Railway as a prototype for analysis. The Beijing-Shanghai High-Speed ​​Railway starts at Beijing South Station in the north and ends at Shanghai Hongqiao Station in the south. Specific route information is shown in Table 1. It passes through 21 stations, which divide the Beijing-Shanghai Railway line into 22 sections. The Beijing-Shanghai High-Speed ​​Railway operates more than 100 trains per day. The experiment will select 100 high-speed trains actually operating on the Beijing-Shanghai line for scheduling optimization.

[0102] Table 1 Basic Operating Data of Beijing-Shanghai High-Speed ​​Railway

[0103]

[0104] The optimization of the automatic train speed curve is determined by the train's structural parameters and the line it runs on. The basic parameters and calculation formulas of a certain type of CRH high-speed train selected in this paper are as follows.

[0105] Traction force calculation formula (v unit: km / h):

[0106]

[0107] Basic resistance calculation formula:

[0108] r(v) = 8.63 + 0.7295v + 0.0012v 2 (twenty four)

[0109] The above method was used to train a deep reinforcement learning model for a train operation scheduling system. During agent training, the network parameters of DDPG were repeatedly adjusted until the maximum reward was obtained. The algorithm uses two hidden layers in both the main and target networks, each with 100 neurons, and the activation function for the hidden layers is ReLU. For DDPG, the reward discount factor γ was 0.98, the batch size was 128, and the experience pool size was set to 2000. The learning rate for both the main and target networks was 0.0001, and the soft update coefficient was 0.001. After continuous training, the agent's training results are as follows: Figure 4 As shown.

[0110] like Figure 4The graph shows the reward curve during agent training. Continuous testing of the algorithm reveals that, as the agent is initially unfamiliar with the environment, the reward value is relatively small in the early stages. As training continues, the agent interacts with the environment to gain experience and eventually converges, proving the algorithm's feasibility.

[0111] 3.2 Scheduling Result Analysis

[0112] To demonstrate the optimization process of train scheduling, a simulation of the operation and scheduling of 100 trains was conducted. At 13:29 on a certain year, the driver of train G138 (Shanghai Hongqiao-Beijing South) on the Beijing-Shanghai High-Speed ​​Railway reported that car 07, running on the northbound track between Changzhou North Station and Danyang North Station at kilometer marker 1123, was affected by the automatic lowering of the pantograph. The train was forced to stop between Danyang North Station and Zhenjiang South Station, resuming normal speed at 14:59, causing varying degrees of delays to subsequent trains. Based on this, the operation and scheduling results of three randomly selected trains were analyzed. For example... Figure 5 The diagram shows a comparison between the original and current train schedules after the selected trains were rescheduled. A pantograph malfunction caused an emergency stop for train G138, resulting in a delay for G138 and a subsequent 35-minute delay for train G140. Currently, most railway scheduling relies on manual dispatching, which is inefficient due to the dispatcher's experience. This paper uses the DDPG algorithm to optimize train scheduling to address unexpected train delays. As shown in the diagram, the total delay time for trains G138 and G140 was reduced from one hour to 43 minutes after the rescheduling, reducing delay losses and improving on-time performance.

[0113] Because trains must simultaneously meet the requirements of punctuality and low energy consumption, train speed is controlled during dispatching, such as... Figure 6 As shown, this is the speed and distance control curve of train G138 after the lifting of the traffic restrictions. In order to meet the punctuality requirements as much as possible, the train runs at the fastest speed while meeting the minimum energy consumption. Compared with the original planned speed, the energy consumption is reduced by 22.81%.

[0114] IV. Conclusion

[0115] This invention studies a real-time scheduling strategy for high-speed train operation. Addressing the issue that the uncertainty of unforeseen events can cause time delays, resulting in severe social impacts and significant economic losses, a train dynamics model and a Markov model for the train operation scheduling system are established based on train operation characteristics. Considering minimizing total train delay time and energy consumption, a train operation scheduling strategy based on a deep deterministic policy gradient algorithm is proposed. By introducing noise into the training actions, the generalization ability and robustness of the method under uncertain environments are improved. The proposed scheduling strategy effectively reduces total train delay time and lowers energy consumption.

[0116] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A train operation scheduling optimization method based on deep reinforcement learning, comprising the following steps: Step 1: Establish a system model 1.1 Train Operation Control System: The train braking curve is determined based on the target distance, target speed, and the train's own performance, without setting speed levels for each block section; by integrating with the train dispatching system, optimized train dispatching is achieved. 1.2 Dynamic Model Formula: Determine the dynamic model of train operation and optimize train scheduling; Step 1.2 includes: To optimize train scheduling, we first need to know the dynamic model of train operation. The dynamic equations of a train can be expressed by the Lomonossoff equations: In the formula, M train Indicates train mass; s tr Indicates the train's location; t represents the train's travel time; F tr Indicates traction or braking force; R vr F represents mechanical resistance; gr Indicates additional resistance; in: In the formula, v represents the train speed; μ f μ represents the train traction coefficient. b b[v(t)] represents the braking force coefficient of the train; b[v(t)] represents the braking force at the current speed v(t) of the train; g(s) tr ) represents the gradient force; r(v) represents the basic drag at the current velocity, which can be expressed by the Davis equation containing constants B, C, and D as follows: r(v)=B+Cv+Dv 2 (3) Based on the total delay time T of all the above trains, In the formula, t a (ij), t d (ij) represent the actual arrival and departure times of train i at station j, respectively. Let a be the planned arrival and departure times of train i at station j; i (t) represents the acceleration of train i at time t, v i (t) represents the speed of train i at time t, s j Let t be the distance between station j and the starting station. i * The total planned running time for train i; The train's total energy consumption E is Among them, f[v i [t] represents the speed of train i as v. i The traction force at (t); Based on the above, the objective function J, which includes the sum of the delay times of all trains and the total energy consumption of the trains, is: J=ω t F t T+ω e F e E (6) Where ω t ω e F represents the weighting coefficient. t F represents the cost per second of latency. e This indicates the unit energy cost per kilowatt-hour; 1.3 Establishing a Markov Model for Train Operation Scheduling System Step Two: Deepening the Markov Decision Process A deep neural network (DNN) is used to approximate the reinforcement learning function, making it applicable to train operation scheduling problems in continuous state and action space. The deep deterministic policy gradient (DDPG) algorithm based on the actor-criticism framework is selected. Then, the optimal policy function is estimated by the deep neural network, and the train operation policy model based on MDP is solved to adapt to the uncertainty in the train operation process. Step 3: Results and Analysis To verify the effectiveness of the optimized scheduling strategy of the train operation scheduling system.

2. The train operation scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, Step 1.3 includes: Reinforcement learning can be represented by a Markov Decision Process (MDP); specifically, it can be represented by a quintuple.<s,a,p,r,γ> Let $\mathbf{ ... 1) State: s t ={t,s tr ,s j The observed train operation scheduling system status includes time, train position, distance from the originating station to the current station, and current traction force. 2) Action: a t ={v i (t), a i The action performed is the agent's reaction to the environment after observing its state at the current time t; 3) Reward: The reward function serves as a guide for the agent in formulating its strategy; it represents the desired control objective. Considering the optimal action guiding the train operation scheduling system to perform reasonable scheduling, the reward function is set as the system's objective function J. r(s t ,a t )=ω t F t T(s t ,a t )+ω e F e E(s t ,a t ) (7) Comfort affects ride quality and safety. When the train's acceleration changes beyond a certain value δ, an additional reward function r2 is provided to meet passengers' comfort needs. 4) Policy: In Markov decision processes, the policy is defined as the probability of a system transitioning from an observed state to an action state. The policy is defined as follows: π(a∣s)=P[a t =a∣s t =s] s∈S,a∈A (9) 5) Reward: State-action value function Q π (s,a) represents the expected cumulative reward after performing action a in state s. The state-action value function Bellman equation is expressed as: Q π (s t ,a t )=E π [r t +γQ π (s t+1 ,a t+1 )∣s t ,a t ] (10) The Bellman equation shows that the action value of the current state depends only on the current reward / penalty value and the action value of the next state, and can be solved iteratively; solving for the optimal policy of the system is equivalent to solving for the optimal state-action value function, that is: p * =argmaxQ π (s,a) (11) The corresponding Bellman equation is:

3. The train operation scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that, Step two includes: 2.1 Train Operation Scheduling Framework Based on ddpg A train operation scheduling framework is established based on the DDPG (Deep deterministic policy gradient) algorithm. For the DDPG algorithm, the input of the target network is a 4-dimensional state vector, and the output is a 2-dimensional action vector; the input of the main network is the state vector s. t and action vector a t The output is the action value function Q(s) t ,a t During the learning process, the DDPG algorithm adopts the experience replay mechanism in deep Q networks, which stores the agent's experience at each time period to form a replay buffer. When training the agent, a small batch of experience samples is randomly extracted from the replay buffer each time, and the network parameters are updated according to the gradient rule. Using historical data as the environmental state, the DDPG algorithm network is trained offline; then, the parameters of the trained DDPG algorithm are fixed to solve the train operation scheduling problem. For a train operation scheduling system, when a scheduling task is performed in each cycle, it is based on the current system state s. t The agent selects scheduling actions using the trained DDPG algorithm target network and main network; the agent takes action to enter the next environmental state and receives a reward; then, the system state information of time period t+1 is collected as a new sample to derive the decision for that time period; in this way, the agent obtains a dynamic scheduling strategy. 2.2 Dual-priority experience replay pool for reward guidance The most crucial technique in the algorithm is experience replay, which involves setting up an experience pool; The transition samples (s, a, r, s') obtained from the agent's interaction with the environment at each time step are stored in the replay memory unit. Then, some samples are randomly selected for training to update the parameters. The priority experience replay pool evaluates the value of a sample by the TD-error, which is the difference between the target Q-value and the estimated Q-value. The larger the TD-error, the higher the priority p of the sample. The sampling probability P(i) of sample i is: Where p(i) = |δ i |+ε represents the priority of sample i, δ i This is the sample TD error, used to adjust the importance level; a value of 0 indicates uniform sampling. If the TD error of the sampled data is large, it means that the prediction accuracy of the agent has a lot of room for improvement, and this sample is worth training the neural network on. In the train operation scheduling system, the size of the reward obtained by the system is positively correlated with the quality of the actions taken. Therefore, rewards can be used to guide the selection of actions, and a priority experience playback mechanism based on reward guidance is proposed. in The higher the reward value, the greater the priority of the action. When the reward is less than 0, the importance of the selected action is reduced. ε is the offset. Traditional algorithms use only one experience pool to store data and update network parameters. This paper uses two experience replay pools, B1 and B2; B1 is a multi-step priority experience replay pool, and B2 is a single-step priority experience replay pool. B1 stores samples indiscriminately, while B2 uses a resampling mechanism to store samples with probability P(i). During training, the network iteratively updates parameters after each round, with samples taken from B1. After the model achieves good results, the parameter updates are changed to single-step iterative updates within each round, with samples taken from B2. By using better training samples and increasing the number of iterations, the network performance is improved. The samples are stored in the experience pool without discrimination, and the priority of the q queue is updated according to P(i); 2.3 Specific steps of the ddp-g algorithm The DDPG algorithm uses two independent networks to approximate the critic and actor functions, each with its own target network parameters θ. Q' and θ π' , where Q' and π' are the target Q value and the target policy, respectively; In a value network, the loss function Loss(θ) is used. Q Optimize the parameters: in y t The target action value, r, is obtained through the target value network. t In state s t The instantaneous reward obtained after taking an action, where γ is the discount factor, Q(s) t ,a t |θ Q The loss function Loss(θ) is the action value evaluated through an online value network after an action is taken. Q ) for θ Q The gradient is The network parameter values ​​are updated using gradient descent along the direction of decrease of the loss function. Where μ Q The learning rate of the value network; In policy networks, The policy network is updated using the policy gradient method to represent the gradient of the action execution direction. Then, update the policy network parameters θ according to the deterministic policy gradient. π : Where is μ π Policy network learning rate; The parameter update method for the target value network and the target policy network is as follows: i Q' ←tth Q +(1-τ)θ Q' (21) i π' ←tth π +(1-τ)θ π' (22) Wherein, τ is the soft update coefficient (τ << 1). By adjusting τ, the update speed of the neural network is controlled, so that the target network slowly tracks the online network, thereby improving the stability of learning.

Images