Train operation adjusting method and device based on reinforcement learning and electronic equipment

Abstract

The invention relates to a train operation adjustment method and device based on reinforcement learning and electronic equipment, and the method comprises the steps: obtaining a first train planned operation scheme; establishing a state set and an action set; the state set comprises the delay value of each train at each station; the action set comprises the stopping duration of each train at each station and the running speed of each train in each running interval; configuring a target return function according to the delay value of each train at each station, the stop duration and the first planned stop duration of each train at each station, and the operation duration and the first planned operation duration of each train in each operation interval; training a reinforcement learning model according to the state set, the action set and the target return function; using the trained reinforcement learning model to obtain a second train planned operation scheme; according to the method, the operation scheme of the train group on the same train operation line can be integrally adjusted, the overall delay time of the train is shortened, and large-scale adjustment of the operation scheme of the train group can be avoided.

Description

Technical Field

[0001] The present application relates to the field of rail transit, and in particular to a train operation adjustment method, device and electronic equipment based on reinforcement learning. Background Art

[0002] With the rapid development of high-speed railways, the scale of high-speed railway networks has expanded, and the network structure has become more complex. In addition, the impact of emergencies such as bad weather and equipment failures may cause train delays, which may cause huge social impacts and economic losses in serious cases. Train operation adjustment is an important part of railway traffic dispatching and command work. By dynamically adjusting the train operation plan, it is possible to coordinate each train to resume orderly operation as soon as possible, reduce delay time, and reduce the scope of impact. Therefore, the study of train operation adjustment has very important practical significance. The complexity of the train operation environment determines that the train operation adjustment problem is a large-scale complex combinatorial optimization problem. Reinforcement learning can learn the optimal strategy through interaction with the environment, can perform autonomous learning and gradual optimization in a complex environment, and is suitable for decision-making problems in complex environments. Therefore, it is often used in train operation adjustment problems. However, the existing train operation adjustment methods based on reinforcement learning have the disadvantages of high computational complexity and difficulty in overall adjustment of train operation plans. Summary of the invention

[0003] In view of this, the present application proposes a train operation adjustment method, device and electronic equipment based on reinforcement learning, which can make overall adjustments to the operation plans of multiple trains on the same train operation line, reduce the overall train delay time, and ensure that the adjusted train operation plan will not be too different from the original train operation plan, thereby avoiding large-scale adjustments to the operation plans of the train group.

[0004] According to one aspect of the present application, a train operation adjustment method based on reinforcement learning is provided, including: obtaining a first train planned operation plan for multiple trains on the same train operation line; the train operation line includes multiple stations; the first train planned operation plan includes a first planned arrival time of each of the multiple trains at each station in the train operation line, a first planned stop time of each train at each station, and a first planned operation time of each train in each operation section in the train operation line; the operation section represents the section between two adjacent stations in the train operation line; establishing a train operation state set and a train operation action set; the train operation state set includes the delay value of each train at each station; the train operation action set includes the stop time of each train at each station and the travel speed of each train in each operation section; the delay value represents the difference between the arrival time of each train at each station and the planned arrival time of each train at the The difference between the first planned arrival times of each station; configuring a target reward function according to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each operating interval and the first planned running time; the running time of each train in each operating interval is determined according to the distance of each operating interval and the travel speed of each train in each operating interval; according to the train running state set, the train running action set and the target reward function, the reinforcement learning model is trained based on the deep deterministic policy gradient DDPG algorithm to obtain a trained reinforcement learning model; using the trained reinforcement learning model, a second train planned operation plan is obtained; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station and the second planned running time of each train in each operating interval.

[0005] In one possible implementation, if the first planned stop time of the first train at the first station is not 0, then the stop time of the first train at the first station in the train operation action set is also not 0, and the stop time of the first train at the first station is greater than or equal to the minimum preset stop time; wherein, the first train is any train among the multiple trains; and the first station is any station among the multiple stations.

[0006] In a possible implementation, in the train operation action set, the running speed of the first train in each operating interval is less than or equal to the maximum preset running speed of the first train, and the running speed of the first train in the first operating interval is less than or equal to the maximum allowed running speed of the first operating interval; the first operating interval is any operating interval among the operating intervals.

[0007] In a possible implementation, the target reward function includes a first reward function, a second reward function and a third reward function; the value of the target reward function is obtained by weighted summing the values ​​of the first reward function, the second reward function and the third reward function; wherein the value of the first reward function is calculated according to the difference between the delay value of each train at the second station and the delay value of each train at the third station; the value of the second reward function is calculated according to the difference between the delay value of the departure station of each train among the multiple stations and the delay value of the terminal station of each train among the multiple stations; the value of the third reward function is calculated according to the difference between the first planned stop time of each train at the first station and the stop time of each train at the first station, and the difference between the first planned running time of each train in the first operating interval and the running time of each train in the first operating interval; the first station is any station among the multiple stations; the second station is any station among the multiple stations except the terminal station; the third station is the next station adjacent to the second station among the multiple stations; the first operating interval is any operating interval among the various operating intervals.

[0008] In a possible implementation, the reinforcement learning model includes a current actor Actor network, a target Actor network, a current value Critic network and a target Critic network; the train route includes M+1 stations; the reinforcement learning model is trained based on the deep deterministic policy gradient DDPG algorithm according to the train running state set, the train running action set and the target reward function to obtain a trained reinforcement learning model, including: inputting the kth train running state into the current Actor network to obtain the action difference corresponding to the kth train running state; the kth train running state is the action difference of each train in the train running state set. The delay value of the train at the kth station in the train operation line; the action difference corresponding to the kth train operation state includes the difference between the stop time of each train at the kth station and the first planned stop time of each train at the kth station, and the difference between the running time of each train in the k+1th running section and the first planned running time of each train in the k+1th running section; the k+1th running section is the running section between the kth station and the k+1th station in the train operation line; 0≤k≤M; according to the action difference corresponding to the kth train operation state, the action corresponding to the kth train operation state is obtained; the kth train operation state is The corresponding action includes the stop time of each train at the k-th station and the travel speed of each train in the k+1-th operation section selected from the train operation action set; the k-th train operation state and the action corresponding to the k-th train operation state are input into the current Critic network to obtain the current reward value; the current reward value is calculated according to the target reward function; the parameters of the current Actor network are updated according to the current reward value until the first preset training condition is met, and the updating of the parameters of the current Actor network is stopped to obtain the trained current Actor network; the k+1-th train operation state is input into the target Ac tor network, obtain the action difference corresponding to the k+1th train running state; obtain the action corresponding to the k+1th train running state according to the action difference corresponding to the k+1th train running state; input the k+1th train running state and the action corresponding to the k+1th train running state into the target Critic network to obtain the target reward value; the target reward value is calculated according to the target reward function; the parameters of the current Critic network are updated according to the current reward value and the target reward value until the second preset training condition is met, and the parameters of the current Critic network are stopped to obtain the trained current Critic network;Using the parameters of the trained current Actor network as the parameters of the target Actor network, and using the parameters of the trained current Critic network as the parameters of the target Critic network, a trained target Actor network and a trained target Critic network are obtained; and obtaining a trained reinforcement learning model according to the trained current Actor network, the trained current Critic network, the trained target Actor network and the trained target Critic network. ;

[0009] In a possible implementation, the current Actor network and the target Actor network include a hidden layer and an output layer; the k-th train running state is input into the current Actor network to obtain the action difference corresponding to the k-th train running state, including: inputting the k-th train running state into the hidden layer of the current Actor network and applying the linear rectifier ReLU activation function to obtain a first eigenvalue; inputting the first eigenvalue into the output layer of the current Actor network and applying the hyperbolic tangent tanh activation function and then multiplying by a preset coefficient to obtain the difference between the running time of each train in the k+1-th running section and the first planned running time of each train in the k+1-th running section; inputting the first eigenvalue into the output layer of the current Actor network and applying the tanh activation function to obtain a second eigenvalue; dividing the second eigenvalue by 2 and applying the ReLU activation function to obtain the stop time of each train at the k-th station and the difference between the stop time of each train at the k-th station and the first planned running time of each train at the k+1-th running section. The difference in the first planned stop time of the train at the k-th station; the inputting the k+1-th train running state into the target Actor network to obtain the action difference corresponding to the k+1-th train running state, including: inputting the k+1-th train running state into the hidden layer of the target Actor network and applying the ReLU activation function to obtain the third eigenvalue; inputting the third eigenvalue into the output layer of the target Actor network and applying the tanh activation function and then multiplying by the preset coefficient to obtain the difference between the running time of each train in the k+2-th running interval and the first planned running time of each train in the k+2-th running interval; inputting the third eigenvalue into the output layer of the target Actor network and applying the tanh activation function to obtain the fourth eigenvalue; dividing the fourth eigenvalue by 2 and applying the ReLU activation function to obtain the difference between the stop time of each train at the k+1-th station and the first planned stop time of each train at the k+1-th station.

[0010] In a possible implementation, the use of the trained reinforcement learning model to obtain a second train planned operation plan includes: obtaining an initial delay set; the initial delay set includes the delay values ​​of the stations that the trains have arrived at on the train operation route; determining the second planned stop duration of the trains at the stations and the second planned operation duration of the trains in the operation intervals according to the initial delay set, the train operation action set and the trained reinforcement learning model; determining the second planned arrival time of the trains at the stations according to the initial delay set, the second planned stop duration of the trains at the stations and the second planned operation duration of the trains in the operation intervals.

[0011] According to another aspect of the present application, a train operation adjustment device based on reinforcement learning is provided, comprising: an acquisition module, used to acquire a first train planned operation plan for multiple trains on the same train operation line; the train operation line includes multiple stations; the first train planned operation plan includes a first planned arrival time of each of the multiple trains at each station in the train operation line, a first planned stop time of each train at each station, and a first planned operation time of each train in each operation section in the train operation line; the operation section represents the section between two adjacent stations in the train operation line; a state set and action set establishment module, used to establish a train operation state set and a train operation action set; the train operation state set includes the delay value of each train at each station; the train operation action set includes the stop time of each train at each station and the travel speed of each train in each operation section; the delay value represents the difference between the time when each train arrives at each station and the time when each train plans to arrive at each station a reward function configuration module, configured with a target reward function according to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each operating interval and the first planned running time; the running time of each train in each operating interval is determined according to the distance of each operating interval and the travel speed of each train in each operating interval; a training module, trained on a reinforcement learning model based on a deep deterministic policy gradient DDPG algorithm according to the train running state set, the train running action set and the target reward function, to obtain a trained reinforcement learning model; an operation plan adjustment module, used to obtain a second train planned operation plan using the trained reinforcement learning model; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station and the second planned running time of each train in each operating interval.

[0012] According to another aspect of the present application, an electronic device is provided, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to implement the above-mentioned reinforcement learning-based train operation adjustment method when executing the instructions stored in the memory.

[0013] According to another aspect of the present application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the above-mentioned reinforcement learning-based train operation adjustment method.

[0014] According to another aspect of the present application, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned reinforcement learning-based train operation adjustment method.

[0015] The reinforcement learning-based train operation adjustment method of the present application establishes a state set according to the delay value of the train at each station, establishes an action set according to the stop time of the train at each station and the speed of the train in each operation interval, sets a reward function according to the delay value of the train and the gap between the original train operation plan and the action adopted by the train in the action set, and trains the reinforcement learning model based on the DDPG algorithm. The adjusted train operation plan is obtained through the trained reinforcement learning model, and the operation plan of the train group on the same train operation line can be adjusted as a whole to reduce the overall delay time of the train, and it can be ensured that the adjusted train operation plan will not be too different from the original train operation plan, avoiding large-scale adjustments to the operation plan of the train group.

[0016] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0018] Figure 1 A schematic diagram showing the process of reinforcement learning.

[0019] Figure 2 A flowchart of a train operation adjustment method based on reinforcement learning according to an embodiment of the present application is shown.

[0020] Figure 3 A schematic diagram showing the input and output process of the neural network in the traditional DDPG algorithm.

[0021] Figure 4 A structural schematic diagram of a train operation adjustment device based on reinforcement learning according to an embodiment of the present application is shown.

[0022] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0024] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0025] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0026] In the process of reinforcement learning, the agent obtains a certain state by interacting with the environment, and decides the action to be taken based on the obtained state and the environment; after taking the action, the agent updates its own state by interacting with the environment, and obtains corresponding rewards through the change of state and the action taken. In the learning process, the agent repeats the above process continuously to learn the corresponding decision-making method.

[0027] The train operation adjustment method based on reinforcement learning mainly achieves continuous learning through the interaction between the intelligent agent and the environment to optimize the selection strategy. Figure 1 A schematic diagram of the reinforcement learning process is shown as follows: Figure 1 As shown in the figure, the agent selects the train operation adjustment plan according to the initial selection strategy, makes adjustments according to this plan, interacts with the environment, and then gets rewards. According to the rewards, the agent's strategy selection method is updated, and then the strategy selection step is returned. After step-by-step iterative optimization, a convergent strategy selection method (that is, the optimal train operation adjustment plan) can be finally obtained, thereby obtaining a trained reinforcement learning model.

[0028] A very important step in the process of reinforcement learning is to establish a reinforcement learning environment for the agent to interact. The reinforcement learning environment generally includes the following aspects: state set, action set, state transfer matrix, and reward function.

[0029] The state set is used to describe the basic problem. In the train operation adjustment method based on reinforcement learning, the continuous time is usually discretized to model the train operation process as a Markov process. The state set is used to describe the specific operation of the train in such a discrete time series. Different ways of looking at the problem will lead to obvious differences in the description of the state set. For example, there is a related study that mainly adjusts the arrival and departure sequence of trains in a certain station. Therefore, the pick-up time and departure time of L trains in station s are used as a state set to describe the operation status of the train. This state set description is limited to the consideration of a certain station. For train modeling, this state set description is more accurate and convenient for train operation planning. However, when the same state set description is applied to more stations and multiple stations are considered as a whole, the description space of the state set will be very large. Therefore, such a state set description greatly limits the possibility of adjusting the train operation network as a whole, making it difficult to adjust the train operation plan on the entire train operation line as a whole.

[0030] The action set is the action selected from one state to the next. The most important thing to pay attention to in the action set is the rationality of the selected action. Some related studies have chosen to select actions by updating the scheme. This method is biased towards the search algorithm. Basically, the adjustment strategy (i.e., action) is selected by permutation and combination, and then the feasibility of the selected adjustment strategy is judged and screened when interacting with the environment. However, such an action set is factorial in size, which will generate a huge search space, thereby greatly increasing the computational complexity.

[0031] The state transition matrix is ​​used to describe the state jump. When the train is not at the terminal station, the train will definitely jump when it takes action a at station s. By defining the probability of jumping to state s+1 after taking action a in state s, the Markov process before the train reaches the terminal station can be fully described.

[0032] The reward function needs to be set according to actual needs. In related studies, the total delay time of the train to the terminal station is generally used as the reward function. For example, some related studies take the negative value of the total delay time as the reward function. However, this method of setting the reward function only considers the impact of the total delay time of the train on the entire operation line, and does not consider the impact of the delay time of the train at each station on each station, and does not consider the gap between the adjusted train operation plan and the original train operation plan, which may lead to a large difference between the adjusted train operation plan and the original train operation plan.

[0033] The existing train operation adjustment method based on reinforcement learning has the disadvantages of high computational complexity, difficulty in overall adjustment of the train operation plan, and the possibility of significant differences between the adjusted train operation plan and the original train operation plan. In order to solve the above problems, this application proposes a train operation adjustment method based on reinforcement learning, which optimizes the establishment of state sets, action sets and reward functions, and the corresponding state transfer part is determined by the previously established state sets, action sets and reward functions.

[0034] Train operation adjustment is mainly to change the operation of the train by giving different train operation plans, and finally achieve train delay recovery or to be close to the original train operation plan to a certain extent. The research object of the train operation adjustment problem should not be limited to the adjustment of a certain train, but the operation of a group of trains on a certain operation line, and the overall adjustment of the train group should be made to restore the overall delay of these trains to the greatest extent. Therefore, this application describes the train operation adjustment problem as: for N trains on a train operation line, the original train operation plan is {P1, P2, …, PN}, and the initial train delay is {Delay1, Delay2, …, DelayN}, where Pi represents the original operation plan of train i on the entire operation line, and Delayi represents the delay of train i at the first few stations, 1≤i≤N. It is necessary to train a reinforcement learning model through reinforcement learning methods. The reinforcement learning model can give a new train operation plan {P1', P2', ..., PN'} according to the original train operation plan and the initial delay situation, where Pi' represents the new operation plan of train i on the entire operation line; the new operation plan can restore the overall delay situation of the train group as much as possible compared with the original operation plan, and there will not be too much difference between the new operation plan and the original operation plan.

[0035] In establishing the state set, the most important part to be considered in adjusting the train operation plan is the train delay situation, so the train delay situation can be used as the train operation state.

[0036] In establishing the action set, constraints can be set for the train adjustment strategy (i.e., action) according to the actual train operation situation to ensure the rationality of the action. The search subspace (i.e., action set) of the adjustment strategy can be constructed according to the constraints to be selected, thereby optimizing the selection of the adjustment strategy. The main constraints to be considered may include: (1) station capacity: the number of trains stopping at the station cannot exceed the maximum number of trains that can be accommodated; (2) departure time interval: due to the number of arrival and departure lines, trains traveling in the same direction in the station generally cannot depart at the same time; (3) train stop time: for trains stopping, sufficient time is required for passengers to get on and off, and for trains passing through, they cannot occupy the passing track at the same time; (4) necessary travel time: the maximum travel speed of the train and the distance between stations will limit the minimum time required for the train to travel between stations. This time cannot be adjusted to dynamically make up for the delay time.

[0037] In the setting of the reward function, for the problem of train operation adjustment, the ultimate goal is to restore the train delay, so the setting of the reward function needs to focus on the train delay; at the same time, the train operation adjustment method of the present application also hopes that the obtained train operation adjustment plan will not have too large a gap with the original operation plan; therefore, the reward function can be calculated based on two aspects, one is the train delay, and the other is the gap between the train operation adjustment plan and the original train operation plan. When setting the reward function according to the train delay, considering that there are multiple stations in the train operation line, two methods can be considered for the reward function. One is to use a temporary reward function, and the reward value is calculated once every time the train arrives at a station, that is, to give immediate feedback; the other is to use the overall reward function, and the reward value is calculated together when the train reaches the terminal station, and the total delay time of the train is settled and fed back; by setting these two different reward functions and the proportion of each reward function, the reinforcement learning model can balance its choice of the optimal adjustment plan - one is to hope that the final total delay time of the train is minimized, and the other is to hope that all delayed trains have the least impact on a station. At the same time, setting a reward function according to the gap between the train operation adjustment plan and the original train operation plan can avoid large-scale adjustments to the operation plan of the train group.

[0038] Based on the above ideas, this application proposes a train operation adjustment method based on reinforcement learning, which can make overall adjustments to the operation plans of multiple trains on the same train operation line, reduce the calculation complexity, reduce the overall train delay time, and ensure that the adjusted train operation plan will not be too different from the original train operation plan, thereby avoiding large-scale adjustments to the operation plans of the train group.

[0039] Figure 2A flow chart of a train operation adjustment method based on reinforcement learning according to an embodiment of the present application is shown, and the method is used to adjust the operation plans of multiple trains on the same train operation line, such as Figure 2 As shown, the method may include:

[0040] S201. Obtain a first train planned operation plan for multiple trains on the same train running line; the train running line includes multiple stations; the first train planned operation plan includes a first planned arrival time of each of the multiple trains at each station in the train running line, a first planned stop time of each train at each station, and a first planned running time of each train in each running section in the train running line; the running section represents the section between two adjacent stations in the train running line.

[0041] The first train planned operation plan is the original train planned operation plan. Exemplarily, the first train planned operation plan may include the time when each train in the train operation line was originally planned to arrive at each station (i.e., the first planned arrival time), the time when each train was originally planned to stop at each station (i.e., the first planned stop time), the time when each train was originally planned to depart from each station (which can be called the first planned departure time), the time when each train was originally planned to run in each operation interval (i.e., the first planned operation time), and the travel speed of each train in each operation interval (which can be called the first planned interval travel speed) and other information.

[0042] S202. Establish a train operation status set and a train operation action set; the train operation status set includes the delay value of each train at each station; the train operation action set includes the stop time of each train at each station and the travel speed of each train in each operation interval; the delay value represents the difference between the arrival time of each train at each station and the first planned arrival time of each train at each station.

[0043] Assume that there are N trains and M+1 stations on the train route, where the kth station indicates the kth station that the train arrives at after departing from the starting station, 0≤k≤M, the 0th station indicates the starting station, and the Mth station indicates the terminal station.

[0044] For the establishment of the train operation state set (corresponding to the state set in the reinforcement learning environment), the train operation adjustment method of the embodiment of the present application considers the delay adjustment of the train group on the same operation line, so the defined train operation state set should be the set of delay conditions of all trains on the train operation line. In actual situations, the prediction of train delays is made station by station according to the train operation line, and when establishing the train operation state set, the delay conditions of each train at each station are also used as the train operation state.

[0045] For example, the train running state set can be recorded as S = {S0, S2, ..., S M},S k represents the kth state, S k ={d1,d2,…,d N}; where d i It represents the delay value of the i-th train at the k-th station (i.e., the difference between the time when the train arrives at the k-th station and the first planned arrival time of the train at the k-th station. If the train arrives at the k-th station later than the first planned arrival time, the delay value is positive; if the train arrives at the k-th station earlier than the first planned arrival time, the delay value is negative), 1≤i≤N.

[0046] Since the train running status defined in the embodiment of the present application needs to consider the train delay values ​​of all trains at a certain station, the train running statuses existing in the train running status set are not limited. For example, an upper delay limit value and a lower delay limit value may be defined, so that all delay values ​​in the train running status set are within the range from the lower delay limit value to the upper delay limit value, and the delay values ​​may be discretized, for example, the delay values ​​may be integer minutes.

[0047] For the establishment of the train operation action set (corresponding to the action set in the reinforcement learning environment), the embodiment of the present application is set to be in state S k The two actions of the train from the kth station to the k+1th station include the train's stop time at the kth station and the train's running speed in the operating section between the kth station and the k+1th station.

[0048] For example, the train operation action set can be recorded as Act = {Act1, Act2, ..., Act M}, Act k+1 Act represents the action taken by the train in the kth state (i.e., the action taken by the train between the kth station and the k+1th station). k+1 = {V1, P1; V2, P2; ...; V N ,P N}; where V irepresents the speed of the i-th train in the operating section between the k-th station and the k+1-th station, P i It represents the duration of the stop of the i-th train at the k-th station.

[0049] In order to ensure the rationality of the actions in the train operation action set, constraints can be set for the train operation action set. For the stop time of each train in the train operation action set at each station, the stop time can be constrained based on the stop and pass behaviors of each train at each station in the original train operation plan; for the travel speed of each train in the train operation action set in each operation interval, the travel speed can be constrained based on the maximum travel speed of each train and the speed limit of each operation interval.

[0050] Exemplarily, if the first planned stop time of the first train at the first station is not 0, then the stop time of the first train at the first station in the train operation action set is also not 0, and the stop time of the first train at the first station is greater than or equal to the minimum preset stop time; wherein the first train is any train among the multiple trains; and the first station is any station among the multiple stations.

[0051] As an example, if the first planned stop time of the i-th train at the k-th station is not 0 (i.e., the i-th train is scheduled to stop at the k-th station in the original train operation plan), then the stop time of the i-th train at the k-th station in the train operation action set is also not 0, and the stop time is not less than the specified minimum preset stop time P min , namely Act k+1 P i Not 0 and P i ≥P min .P min It can be set according to actual needs. If the first planned stop time of the i-th train at the k-th station is 0 (that is, in the original train plan, the i-th train is planned to pass through the k-th station), then the stop time of the i-th train at the k-th station in the train operation action set can be 0 or non-0. In this way, if the train was originally planned to stop at a certain station, then in the given train operation action set, the behavior of the train at this station is also a stop; and if the train was originally planned to pass through a certain station, then in the given train operation action set, the behavior of the train at this station can be either a pass or a stop; this can ensure that the originally planned stop behavior of the train is not affected, thereby ensuring the rationality of the stop time of the train at each station in the train operation action set.

[0052] Exemplarily, in the train operation action set, the running speed of the first train in each operating interval is less than or equal to the maximum preset running speed of the first train, and the running speed of the first train in the first operating interval is less than or equal to the maximum allowable running speed of the first operating interval; the first operating interval is any operating interval among the operating intervals.

[0053] As an example, if the maximum speed of the i-th train is set to V max1 , then the speed of the i-th train in each running section in the train running action set cannot be higher than V max1 , and if the maximum permissible speed of the operating section between the kth station and the k+1th station is V max2 , then the speed of the i-th train in the running section between the k-th station and the k+1-th station in the train running action set cannot be higher than V max2 , namely Act k+1 V i ≤V max1 And V i ≤V max2 This can ensure the rationality and feasibility of the train's running speed in each running section in the train running action set.

[0054] In this way, constraining the train operation actions in the process of establishing the train operation action set can ensure the rationality and feasibility of the selectable train operation actions and reduce the calculation complexity in the subsequent process of selecting the train operation actions.

[0055] S203, configuring a target reward function according to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each operating interval and the first planned running time; the running time of each train in each operating interval is determined according to the distance of each operating interval and the traveling speed of each train in each operating interval.

[0056] The reward function can be configured based on the train's delay value and the gap between the original train operation plan and the actions taken in the train operation action set.

[0057] The reward function configured according to the train delay value needs to be determined according to the specific delay value. Since the train delay value will change as the train runs, the reward function determined by the train delay value can be calculated from two aspects. On the one hand, when the train arrives at each station, a corresponding delay value will appear. According to the corresponding delay value obtained by each action of the train, a reward value can be calculated. This reward value will be given to the reinforcement learning model every time an action is taken, that is, an immediate reward; on the other hand, after the train completes the running line, a total delay value will be generated. According to the total delay value, another reward value can be calculated. This reward value will be given to the reinforcement learning model after the train completes the running line. By weighted combination of these two reward values, the reinforcement learning model can eventually learn the relevant decisions.

[0058] For the reward function configured according to the gap between the original train operation plan and the actions adopted in the train operation action set, it is necessary to give the reinforcement learning model after the train has completed the operation route. After the train has completed the operation route, the set of actions taken by the train constitutes the new train operation plan. According to the gap between the new train operation plan and the original train operation plan (that is, the adjustment range of the planned operation plan of the entire train group), a reward value can be calculated and given to the reinforcement learning model, thereby guiding the reinforcement learning model to learn the adjustment range of the train operation plan.

[0059] In a possible implementation, the target reward function may include a first reward function, a second reward function, and a third reward function.

[0060] Exemplarily, the value of the first reward function can be calculated based on the difference between the delay value of each train at the second station and the delay value of each train at the third station; the second station is any station among the multiple stations except the terminal station; the third station is the next station adjacent to the second station among the multiple stations.

[0061] As an example, assuming that there are N trains on the train running line, the calculation formula of the first reward function R1 can be as follows:

[0062]

[0063] Among them, Delay k-1,i Delay represents the delay value of the i-th train in the k-1th state (that is, the delay value of the i-th train at the k-1th station). k,i Indicates the delay value of the i-th train in the k-th state (i.e., the delay value of the i-th train at the k-th station). k-1,i - Delay k,i) represents the reduction of the delay value of the i-th train at the k-th station relative to the delay value at the k-1-th station. If (Delay k-1,i - Delay k,i ) is a positive value, it means that the delay value of the i-th train at the k-th station is reduced relative to the delay value at the previous station. k-1,i - Delay k,i ) is a negative value, which means that the delay value of the i-th train at the k-th station increases relative to the delay value at the previous station. In the train operation adjustment problem defined in this application, we expect the delay value of the train at each station to gradually decrease, so we expect the value of R1 to be positive and the larger the better.

[0064] In this way, when the train arrives at each station, the reduction of the train's delay value at that station relative to the delay value at the previous station is added to calculate the first reward function value. This can give instant feedback to the reinforcement learning model every time the train takes an action, so that the reinforcement learning model can learn in the direction of gradually reducing the train's delay value at each station during the process of selecting actions.

[0065] Exemplarily, the value of the second reward function may be calculated based on the difference between the delay value of the departure station of each train among the multiple stations and the delay value of the terminal station of each train among the multiple stations.

[0066] As an example, assuming that there are N trains and M+1 stations on the train running line, the calculation formula of the second reward function R2 can be as follows:

[0067]

[0068] Among them, Delay 0,i Delay represents the initial delay value of the i-th train at the departure station (i.e., the 0th station). M,i Indicates the delay value of the i-th train at the terminal station (i.e. the M-th station). 0,i - Delay M,i ) represents the total delay value of the i-th train after running the entire line. If (Delay 0,i - Delay M,i ) is a positive value, it means that the total delay value of the i-th train on the entire running line is reduced. 0,i - Delay M,i ) is a negative value, which means that the total delay value of the i-th train on the entire running line increases. In the train operation adjustment problem defined in this application, we expect the total delay value of the train to decrease, so we expect the value of R2 to be positive and the larger the better.

[0069] In this way, by calculating the total delay value of the train as the second reward function value after the train has traveled the entire route, the reinforcement learning model can learn in the direction of reducing the total delay value of the train in the process of selecting actions.

[0070] Exemplarily, the value of the third reward function can be calculated based on the difference between the first planned stop time of each train at the first station and the stop time of each train at the first station, and the difference between the first planned running time of each train in the first operating interval and the running time of each train in the first operating interval; the first station is any station among the multiple stations; the first operating interval is any operating interval among the operating intervals.

[0071] As an example, assuming that there are N trains on the train running line, the calculation formula of the third reward function R3 can be as follows:

[0072]

[0073] Among them, Δ k,i,s represents the difference between the first planned stop time of the i-th train at the k-th station and the stop time of the i-th train at the k-th station in the action set after the i-th train has traveled the entire route, Δ k,i,r It represents the difference between the first planned running time of the i-th train between the k-th station and the k+1-th station and the running time between the k-th station and the k+1-th station in the set of actions taken by the i-th train after running the entire running line. The running time between the k-th station and the k+1-th station can be calculated by dividing the distance between the k-th station and the k+1-th station by the running speed taken by the i-th train between the k-th station and the k+1-th station. In the train operation adjustment problem defined in this application, we expect that the adjusted train operation plan will not be too different from the original train operation plan, so we expect the absolute value of R3 to be as small as possible.

[0074] Exemplarily, the value of the target reward function may be obtained by weighted summing the values ​​of the first reward function, the second reward function, and the third reward function. The value R of the target reward function may be calculated according to the following formula:

[0075] R=a1·R1+a2·R2+a3·R3 (4)

[0076] Wherein, a1, a2, and a3 are positive numbers. a1, a2, and a3 can be set by those skilled in the art according to actual needs. For the train operation adjustment problem defined in this application, we expect that the larger the value of R, the better.

[0077] As an example, a1, a2, and a3 may all be 1, that is, R=R1+R2+R3.

[0078] In the configuration of the reward function in the embodiment of the present application, the values ​​of the first reward function and the second reward function are calculated according to the delay value of the train. The reward values ​​of these two parts can ensure that the reinforcement learning model learns in the direction of reducing the train delay value in the process of selecting actions, so that the adjusted train operation plan can achieve train delay recovery; the value of the third reward function is calculated according to the gap between the original train operation plan and the actions adopted in the train operation action set. On the one hand, this part of the reward value can ensure that the action selection strategy learned by the reinforcement learning model will not be adjusted significantly, and on the other hand, it can avoid the adjusted train operation plan causing most trains to arrive at the station early.

[0079] S204. According to the train running state set, the train running action set and the target reward function, the reinforcement learning model is trained based on the deep deterministic policy gradient DDPG algorithm to obtain a trained reinforcement learning model.

[0080] The Deep Deterministic Policy Gradient (DDPG) algorithm is a hybrid algorithm based on the Deep Q-Network (DQN) and the Policy Gradient (PG). Based on the basic idea of ​​the Q-learning algorithm in the reinforcement learning algorithm, the DDPG algorithm uses the structure of a neural network to replace the Q-value table in the Q-learning algorithm to implement the selection of the optimization strategy and the update of the Q-value table after receiving the reward. The neural network structure in the DDPG algorithm includes the following four neural networks: the current actor network (Actor network), the target actor network, the current value network (Critic network) and the target critic network. Among them, the current actor network and the target actor network have the same structure, the input is the state variable, and the output is the action selected under the input state; the current critic network and the target critic network have the same structure, the input is the state variable and the action taken, and the output is the Q value (i.e., the reward value) obtained after taking the action under the state.

[0081] Figure 3 The schematic diagram of the input and output process of the neural network in the traditional DDPG algorithm is shown as follows: Figure 3As shown in the figure, the input of the current Actor network is the current state S, and the output is the action a selected in the current state S; the input of the target Actor network is the next state S', and the output is the action a' selected in the next state S'; this process replaces the process of selecting actions according to the relevant values ​​in the Q value table in the Q-learning algorithm. The input of the current Critic network is the current state S and action a, and the output is the current reward value Q(S,a) obtained after taking action a in the current state S; the input of the target Critic network is the next state S' and action a', and the output is the target reward value Q'(S',a') obtained after taking action a' in the next state S'; this process replaces the calculation process of the Q value in the Q-learning algorithm.

[0082] The reinforcement learning method based on the DDPG algorithm requires first using the current Actor network and the reinforcement learning environment to interact to generate a certain amount of samples, and then update the neural network parameters. In the entire reinforcement learning process, sample generation and parameter update are the two most important processes.

[0083] As for the method of generating samples, the current state S is input into the current Actor network to obtain the action a selected under the current state S. The current state S and action a are input into the reinforcement learning environment. After interacting with the reinforcement learning environment, the next state S' and the corresponding reward value r after taking action a under the current state S can be obtained. These four variables are combined into a quaternary group (S, a, r, S') as samples and put into the experience pool. The above process is repeated continuously, and the corresponding next state variables and reward values ​​are obtained by taking different actions under different states, thereby completing the sample collection process. The parameters of the neural network can be updated later through the quaternary group samples generated above.

[0084] As for the updating method of neural network parameters, firstly, the current Actor network and the current Critic network are sampled and updated, and then the current Actor network parameters are used to update the target Actor network parameters (i.e., the current Actor network parameters are used as the target Actor network parameters) after a period of time, and the target Critic network parameters are updated using the current Critic network parameters (i.e., the current Critic network parameters are used as the target Critic network parameters). The updating process of the current Critic network parameters is as follows: take a sample (S, a, r, S') from the experience pool, input the current state S and action a into the current Critic network to obtain the current reward value Q(S, a), input the next state S' into the target Actor network to obtain the next action a', input the next state S' and the next action a' into the target Critic network to obtain the target reward value Q'(S', a'), and then use the gradient descent method to update the current Critic network parameters according to Q(S, a) and Q'(S', a'), so that the Q(S, a) output by the current Critic network is as close as possible to the label Q'(S', a'). The update process of the current Actor network parameters is as follows: use the current Actor network to obtain the action a under the current state S, use the current Crtic network to obtain the current reward value Q(S,a), and continuously update the parameters of the current Actor network according to Q(S,a) so that the output Q(S,a) can be maximized.

[0085] For the train operation adjustment problem defined in this application, the state set in the reinforcement learning environment needs to consider the delay values ​​of all trains at a certain station, so the train operation states in the state set are not finite. In contrast, the traditional Q-learning reinforcement learning method requires a finite state set, each state corresponds to the value in the Q value table, and the corresponding action under state S is selected accordingly. In actual application scenarios, although the train delay value has been discretized, there are still an infinite number of states, so a neural network is used instead of the Q value table to describe and process the delay states of various trains. Similarly, this application uses the train operation adjustment scheme (i.e., the train's stop time at each station and the train's travel speed in each operating interval) as an action for reinforcement learning, and there are also infinite problems, so the neural network structure is also used to make decisions on the action selection process. Although in the DQN reinforcement learning method, a neural network is also used to replace the Q value table, its neural network structure will output the Q value corresponding to the finite action set according to the input state S, so it is not applicable in the application scenarios considered in this application. To sum up, the reinforcement learning application scenario in this application is relatively complex, and it is difficult to use some traditional reinforcement learning algorithms. The DDPG algorithm can effectively describe and process the application scenario of reinforcement learning in this application, solving the problem of large state sets and action sets established in this application.

[0086] The following describes the process of training the reinforcement learning model of the embodiment of the present application based on the DDPG algorithm.

[0087] In a possible implementation, the reinforcement learning model includes a current Actor network, a target Actor network, a current Critic network, and a target Critic network; the train running line includes M+1 stations; the reinforcement learning model is trained based on the deep deterministic policy gradient DDPG algorithm according to the train running state set, the train running action set, and the target reward function to obtain the trained reinforcement learning model, which may include:

[0088] (1) Inputting the k-th train running state into the current Actor network, obtaining the action difference corresponding to the k-th train running state; the k-th train running state is the delay value of each train in the train running state set at the k-th station in the train running line; the action difference corresponding to the k-th train running state includes the difference between the stop time of each train at the k-th station and the first planned stop time of each train at the k-th station, and the difference between the running time of each train in the k+1-th running section and the first planned running time of each train in the k+1-th running section; the k+1-th running section is the running section between the k-th station and the k+1-th station in the train running line; 0≤k≤M.

[0089] (2) According to the action difference corresponding to the k-th train running state, the action corresponding to the k-th train running state is obtained; the action corresponding to the k-th train running state includes the stop time of each train at the k-th station and the running speed of each train in the k+1-th running section selected from the train running action set.

[0090] Assuming that the first planned stop duration of the i-th train at the k-th station obtained from the first train planned operation plan is p1, and the difference between the stop duration of the i-th train at the k-th station output by the current Actor network and the first planned stop duration of the i-th train at the k-th station is Δ1, then the stop duration of the i-th train at the k-th station selected from the train operation action set can be calculated to be p2=p1+Δ1.

[0091] Assume that the running time of the i-th train in the k+1 running interval obtained from the first train plan is t1, the difference between the running time of the i-th train in the k+1 running interval output by the current Actor network and the first planned running time of the i-th train in the k+1 running interval is Δ2, and the distance of the k+1 running interval is s, then the running time of the i-th train in the k+1 running interval can be calculated as t2=t1+Δ2, and the speed v of the i-th train in the k+1 running interval selected from the train running action set is i =s / t2.

[0092] (3) The k-th train running state and the action corresponding to the k-th train running state are input into the current Critic network to obtain a current reward value; the current reward value is calculated according to the target reward function.

[0093] The current reward value r can be calculated according to the above formula (4).

[0094] (4) updating the parameters of the current Actor network according to the current reward value until a first preset training condition is met, stopping updating the parameters of the current Actor network, and obtaining a trained current Actor network.

[0095] You can stop updating when the current reward value reaches the maximum value to get the trained current Actor network.

[0096] (5) Inputting the k+1th train running state into the target Actor network to obtain the action difference corresponding to the k+1th train running state.

[0097] The k+1th train operation status is the delay value of each train in the train operation status set at the k+1th station in the train operation line; the action difference corresponding to the k+1th train operation status includes the difference between the stop time of each train at the k+1th station and the first planned stop time of each train at the k+1th station, and the difference between the running time of each train in the k+2th operating section (i.e. the running section between the k+1th station and the k+2th station in the train operation line) and the first planned running time of each train in the k+2th operating section.

[0098] (6) Obtaining the action corresponding to the k+1th train running state according to the action difference corresponding to the k+1th train running state.

[0099] The specific process of obtaining the action corresponding to the k+1th train running state according to the action difference corresponding to the k+1th train running state can refer to the above step (2), which will not be repeated here.

[0100] (7) Inputting the k+1th train operation state and the action corresponding to the k+1th train operation state into the target Critic network to obtain a target reward value; the target reward value is calculated according to the target reward function.

[0101] The target return value r' can be calculated according to the above formula (4).

[0102] It should be noted that, in the process of calculating the value of the target reward function, the values ​​of R2 and R3 need to be calculated after the train has completed the entire route. Therefore, when calculating the reward value corresponding to the action taken under each train operating state, it is necessary to simulate the process of the train traveling the entire route starting from the initial operating state of the train at the departure station to obtain the corresponding reward value.

[0103] (8) Updating the parameters of the current critic network according to the current reward value and the target reward value until a second preset training condition is met, stopping updating the parameters of the current critic network, and obtaining a trained current critic network.

[0104] The gradient descent method can be used to update the parameters of the current Critic network. When the current reward value is closest to the target reward value, the update is stopped to obtain the trained current Critic network.

[0105] (9) Using the parameters of the trained current Actor network as the parameters of the target Actor network, and using the parameters of the trained current Critic network as the parameters of the target Critic network, to obtain a trained target Actor network and a trained target Critic network.

[0106] (10) Obtain a trained reinforcement learning model based on the trained current Actor network, the trained current Critic network, the trained target Actor network, and the trained target Critic network.

[0107] The setting of the neural network in the reinforcement learning model of the embodiment of the present application is adjusted relative to the traditional DDPG algorithm. The output of the Actor network (including the current Actor network and the target Actor network) in the traditional DDPG algorithm is the action taken in state s, while in the reinforcement learning model of the embodiment of the present application, the output of the Actor network is adjusted to the difference between the action taken in state s and the action originally planned in state s, and the actual action taken in state s is subsequently calculated based on the action difference. The output of the Actor network can reflect the gap between the adjusted train plan operation plan and the original train plan operation plan, making it easier to calculate the reward value corresponding to the adjustment range of the train operation plan.

[0108] Exemplarily, the current Actor network and the target Actor network may include a hidden layer and an output layer. There may be two hidden layers, and the hidden layer may be composed of a linear layer and a linear rectification function (ReLU). The output layer may be one layer, and the output layer may use a hyperbolic tangent (tanh) activation function and a ReLU activation function.

[0109] Exemplarily, after the running status of the kth train is input into the hidden layer of the current Actor network and the ReLU activation function is applied, the first eigenvalue can be obtained; the first eigenvalue is input into the output layer of the current Actor network, and the tanh activation function is applied in the output layer and then multiplied by the preset coefficient, the difference between the running time of each train in the k+1th running interval and the first planned running time of each train in the k+1th running interval can be obtained; the first eigenvalue is input into the output layer of the current Actor network and the tanh activation function is applied in the output layer, the second eigenvalue can be obtained; the second eigenvalue is divided by 2 and then the ReLU activation function is applied (if there are N second eigenvalues, each second eigenvalue is divided by 2 and then the ReLU activation function is applied respectively), the difference between the stop time of each train at the kth station and the first planned stop time of each train at the kth station can be obtained.

[0110] Exemplarily, the operating status of the k+1th train is input into the hidden layer of the target Actor network and the ReLU activation function is applied to obtain the third eigenvalue; the third eigenvalue is input into the output layer of the target Actor network and the tanh activation function is applied and then multiplied by the preset coefficient to obtain the difference between the running time of each train in the k+2th running interval and the first planned running time of each train in the k+2th running interval; the third eigenvalue is input into the output layer of the target Actor network and the tanh activation function is applied to obtain the fourth eigenvalue; the fourth eigenvalue is divided by 2 and then the ReLU activation function is applied (if there are N fourth eigenvalues, each fourth eigenvalue is divided by 2 and then the ReLU activation function is applied respectively), to obtain the difference between the stop time of each train at the k+1th station and the first planned stop time of each train at the k+1th station.

[0111] In actual train operation, in order to get on and off the train, the train needs to have the most basic stop time at the corresponding station. The stop time originally planned can be considered as the shortest stop time. In the reinforcement learning model of the embodiment of the present application, after the output layer of the Actor network applies the tanh activation function to the eigenvalues ​​output by the hidden layer, half of the output eigenvalue is input into the ReLU activation function to obtain the difference between the stop time taken and the stop time originally planned. This part will not produce a negative output after passing through the ReLU activation function, which can ensure that the output action will not appear in the action where the stop time taken is less than the stop time originally planned, thereby ensuring the rationality of the stop time of the train at each station in the adjusted train plan operation plan. In the reinforcement learning model of the embodiment of the present application, after the output layer of the Actor network applies the tanh activation function to the eigenvalues ​​output by the hidden layer, it is multiplied by a preset coefficient as the difference between the interval operation time taken and the interval operation time originally planned. Through such processing, the adjusted train plan operation plan can be generated on the basis of the original train plan operation plan, and the existing knowledge can be effectively used to generate the adjusted train plan operation plan.

[0112] For example, the current critic network and the target critic network may be composed of two linear layers and a ReLU activation function. The input of the current critic network and the target critic network may be passed through two linear layers and then a ReLU activation function may be used to obtain an output result.

[0113] S205. Utilize the trained reinforcement learning model to obtain a second train planned operation plan; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station, and the second planned operation time of each train in each operation section.

[0114] In a possible implementation, the using the trained reinforcement learning model to obtain the second train planned operation plan may include:

[0115] (1) Obtaining an initial delay set; the initial delay set includes the delay values ​​of the stations that the trains have arrived at on the train route.

[0116] If a train has arrived at n stations on the train running route, the initial delay set may include the delay values ​​of the train at the first n stations, that is, the initial train set may represent the running status of the first n trains.

[0117] Exemplarily, the initial delay set may include the delay value of each train at the departure station, and the initial delay set may represent the initial train operation status.

[0118] (2) Determine, based on the initial delay set, the train operation action set and the trained reinforcement learning model, the second planned stop duration of each train at each station and the second planned operation duration of each train in each operation section.

[0119] The initial delay set and the train operation action set can be input into the trained reinforcement learning model. The trained reinforcement learning model has learned the optimal selection strategy. Through the trained reinforcement learning model, the optimal stop time of each train at each subsequent station except the station that has been departed (i.e., the second planned stop time) and the optimal travel speed of each train in each operating section that has not been traveled (which can be called the second planned section travel speed) can be selected from the train operation action set. According to the second planned section travel speed of the train in the operating section that has not been traveled and the distance of the operating section, the second planned operation time of the train in the operating section can be calculated.

[0120] (3) Determine the second planned arrival time of each train at each station based on the initial delay set, the second planned stop time of each train at each station, and the second planned running time of each train in each running section.

[0121] According to the delay value of each train at the arrived stations and the first planned arrival time, the actual arrival time of each train at these stations can be obtained; according to the actual arrival time of each train at these stations, the second planned stop time of each train at subsequent stations except the station it has left, and the second planned operating time of each subsequent operating section, the departure time of each train from the subsequent stations except the station it has left (which can be called the second planned departure time) and the arrival time at the subsequent stations (i.e., the second planned arrival time) can be obtained, so that the second train planned operation plan can be obtained.

[0122] The reinforcement learning-based train operation adjustment method of the embodiment of the present application establishes a state set according to the delay value of the train at each station, establishes an action set according to the stop time of the train at each station and the speed of the train in each operation interval, sets a reward function according to the delay value of the train and the difference between the original train operation plan and the action adopted by the train in the action set, and trains the reinforcement learning model based on the DDPG algorithm. The adjusted train operation plan is obtained by the trained reinforcement learning model, and the operation plan of the train group on the same train operation line can be adjusted as a whole to reduce the overall delay time of the train, and it can be ensured that the adjusted train operation plan will not be too different from the original train operation plan, thereby avoiding large-scale adjustments to the operation plan of the train group.

[0123] In one embodiment, the parameters of the reinforcement learning model can be set as follows: the number of Actor network layers (i.e., network depth) can be 3 layers (2 hidden layers and 1 output layer), the number of neurons in the two hidden layers of the Actor network (i.e., network width) can be set to 256, the hidden layer of the Actor network uses the ReLU activation function, the output layer uses the tanh activation function and the ReLU activation function, and the output of the Actor network is a vector of the size of the number of trains*2; the number of Critic network layers can be 2 layers (2 linear layers), the number of neurons in the two linear layers of the Critic network can be set to 256, the Critic network uses the ReLU activation function, and the output of the Critic network is the reward value; the learning rate can be set to 0.001, the future reward decay rate can be set to 0.99, the number of training samples per batch can be set to 64, and the maximum capacity of the experience pool samples can be set to 1000. The reinforcement learning model can be trained using the train operation data of the high-speed railway line running section from Guangzhou South Station to Chibi North Station in the Wuhan-Guangzhou High-speed Railway Line. The train operation line includes 15 stations (Guangzhou South, Guangzhou North, Qingyuan, Yingde West, Shaoguan, Lechang East, Binzhou West, Leiyang West, Hengyang East, Hengshan West, Zhuzhou West, Changsha South Wuguan, Miluo East, Yueyang East, Chibi North) and 14 train operation sections (the travel distance of each train operation section can be obtained by query). The train operation data can include the train number of each train, the train departure date, the name of the station where the train arrives, the time when the train arrives at each station, the time when the train is scheduled to arrive at each station, the time when the train departs from each station, the planned departure time of the train at each station, the track number when the train enters the station, and the delay time (i.e., the difference between the actual time when the train arrives at the station and the planned arrival time). Table 1 shows some examples of train operation data.

[0124] Table 1

[0125]

[0126] Except for some larger stations on this route, it is generally less affected by cross-line trains. The simulation is a single line, so only trains departing from Guangzhou South Station to Chibi North Station are considered. Trains entering this line in the middle and trains leaving through a station in the middle are not considered. The operation plans of 30 trains at the first 7 stations are used as adjustment objects, and the training rounds are set to 50 rounds. The operation of the trains is sampled 100 times in each round. The reinforcement learning model is trained, and finally the trained reinforcement learning model and the train operation adjustment plan are obtained. Table 2 shows some examples of the generated train operation adjustment plan. Among them, Stop i It represents the planned stop time of the train at the i-th station after adjustment, Run iIt indicates the planned running time of the adjusted train in the i-th operating section (i.e. the operating section between the i-th station and the i+1-th station), in minutes.

[0127] Table 2

[0128]

[0129]

[0130] On the whole, the new train operation plan can play a role in delay recovery for the train's delay value. Some obviously unreasonable adjustment plans (for example, the planned running time of train 6 in the seventh running interval in Table 2 is 3 minutes, and the running time is less than the minimum running time of the running interval) can be removed. After removing the unreasonable train operation plan, the total train operation recovery time is 109 minutes, and each train recovers 4 minutes on average in the running line of 7 stations. In order to verify the adjustment effect of the new train operation plan on the actual operation of the train, the embodiment of the present application calculates and compares the actual running time of the new train operation plan with the actual running time according to the original train operation plan. Table 3 shows the delay recovery of the first 10 trains, in minutes.

[0131] Table 3

[0132] Train number Actual running time of the new plan Originally planned actual running time Delayed recovery time 1 42 67.5 25.5 2 46 67 21 3 56.5 72.5 16 4 72.5 71 -1.5 5 48.5 68.5 20 6 70.5 67 -3.5 7 73 76 3 8 78 73.5 -4.5 9 56 73.5 17.5 10 49.5 69 19.5

[0133] According to the comparison results of the actual running time of the new plan and the actual running time of the original plan, it can be seen that the reinforcement learning model trained in the embodiment of the present application has basic train delay adjustment capabilities, can reduce the overall train delay time, and can play a role in delay recovery for most trains.

[0134] The embodiment of the present application also proposes a train operation adjustment device based on reinforcement learning.

[0135] Figure 4 A structural diagram of a train operation adjustment device based on reinforcement learning according to an embodiment of the present application is shown. Figure 4As shown, the device may include: an acquisition module 401, used to acquire the first train planned operation plan of multiple trains on the same train running line; the train running line includes multiple stations; the first train planned operation plan includes the first planned arrival time of each train in the multiple trains to arrive at each station in the train running line, the first planned stop time of each train at each station, and the first planned operation time of each train in each operation section in the train running line; the operation section represents the section between two adjacent stations in the train running line; a state set and action set establishment module 402, used to establish a train running state set and a train running action set; the train running state set includes the delay value of each train at each station; the train running action set includes the stop time of each train at each station and the travel speed of each train in each operation section; the delay value represents the difference between the time when each train arrives at each station and the first planned arrival time when each train arrives at each station; The reward function configuration module 403 is used to configure the target reward function according to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each operating interval and the first planned running time; the running time of each train in each operating interval is determined according to the distance of each operating interval and the travel speed of each train in each operating interval; the training module 404 is used to train the reinforcement learning model based on the deep deterministic policy gradient DDPG algorithm according to the train running state set, the train running action set and the target reward function to obtain a trained reinforcement learning model; the operation plan adjustment module 405 is used to use the trained reinforcement learning model to obtain a second train planned operation plan; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station and the second planned running time of each train in each operating interval.

[0136] In one possible implementation, if the first planned stop time of the first train at the first station is not 0, then the stop time of the first train at the first station in the train operation action set is also not 0, and the stop time of the first train at the first station is greater than or equal to the minimum preset stop time; wherein, the first train is any train among the multiple trains; and the first station is any station among the multiple stations.

[0137] In a possible implementation, in the train operation action set, the running speed of the first train in each operating interval is less than or equal to the maximum preset running speed of the first train, and the running speed of the first train in the first operating interval is less than or equal to the maximum allowed running speed of the first operating interval; the first operating interval is any operating interval among the operating intervals.

[0138] In a possible implementation, the target reward function includes a first reward function, a second reward function and a third reward function; the value of the target reward function is obtained by weighted summing the values ​​of the first reward function, the second reward function and the third reward function; wherein the value of the first reward function is calculated according to the difference between the delay value of each train at the second station and the delay value of each train at the third station; the value of the second reward function is calculated according to the difference between the delay value of the departure station of each train among the multiple stations and the delay value of the terminal station of each train among the multiple stations; the value of the third reward function is calculated according to the difference between the first planned stop time of each train at the first station and the stop time of each train at the first station, and the difference between the first planned running time of each train in the first operating interval and the running time of each train in the first operating interval; the first station is any station among the multiple stations; the second station is any station among the multiple stations except the terminal station; the third station is the next station adjacent to the second station among the multiple stations; the first operating interval is any operating interval among the various operating intervals.

[0139] In a possible implementation, the reinforcement learning model includes a current Actor network, a target Actor network, a current Critic network and a target Critic network; the train route includes M+1 stations; the training module 404 is further used to: input the k-th train operation state into the current Actor network to obtain the action difference corresponding to the k-th train operation state; the k-th train operation state is the delay value of each train in the train operation state set at the k-th station in the train operation route; the action difference corresponding to the k-th train operation state includes the stop time of each train at the k-th station and the first planned stop of each train at the k-th station. The difference between the stop time of each train in the k+1th operation section and the first planned operation time of each train in the k+1th operation section; the k+1th operation section is the operation section between the kth station and the k+1th station in the train operation line; 0≤k≤M; according to the action difference corresponding to the kth train operation state, the action corresponding to the kth train operation state is obtained; the action corresponding to the kth train operation state includes the stop time of each train at the kth station selected from the train operation action set and the travel speed of each train in the k+1th operation section; the kth train operation state and the kth train operation state are combined. The action corresponding to the k+1th train running state is input into the current Critic network to obtain the current reward value; the current reward value is calculated according to the target reward function; the parameters of the current Actor network are updated according to the current reward value until the first preset training condition is met, and the parameters of the current Actor network are stopped to obtain the trained current Actor network; the k+1th train running state is input into the target Actor network to obtain the action difference corresponding to the k+1th train running state; according to the action difference corresponding to the k+1th train running state, the action corresponding to the k+1th train running state is obtained; the k+1th train running state and the k+1th train running state are compared. The action corresponding to the running state is input into the target Critic network to obtain the target reward value; the target reward value is calculated according to the target reward function; the parameters of the current Critic network are updated according to the current reward value and the target reward value until the second preset training condition is met, and the updating of the parameters of the current Critic network is stopped to obtain the trained current Critic network; the parameters of the trained current Actor network are used as the parameters of the target Actor network, and the parameters of the trained current Critic network are used as the parameters of the target Critic network to obtain the trained target Actor network and the trained target Critic network;According to the trained current Actor network, the trained current Critic network, the trained target Actor network and the trained target Critic network, a trained reinforcement learning model is obtained. ;

[0140] In one possible implementation, the current Actor network and the target Actor network include a hidden layer and an output layer; the training module 404 is further used to: input the running state of the kth train into the hidden layer of the current Actor network and apply the linear rectifier ReLU activation function to obtain a first eigenvalue; input the first eigenvalue into the output layer of the current Actor network and apply the tanh activation function and then multiply by a preset coefficient to obtain the difference between the running time of each train in the k+1th running interval and the first planned running time of each train in the k+1th running interval; input the first eigenvalue into the output layer of the current Actor network and apply the hyperbolic tangent tanh activation function to obtain a second eigenvalue; divide the second eigenvalue by 2 and apply the ReLU activation function to obtain the stop time of each train at the kth station The running state of the k+1th train is input into the hidden layer of the target Actor network and the ReLU activation function is applied to obtain the third eigenvalue; the third eigenvalue is input into the output layer of the target Actor network and the tanh activation function is applied and then multiplied by the preset coefficient to obtain the running time of each train in the k+2th running interval and the first planned running time of each train in the k+2th running interval; the third eigenvalue is input into the output layer of the target Actor network and the tanh activation function is applied to obtain the fourth eigenvalue; the fourth eigenvalue is divided by 2 and the ReLU activation function is applied to obtain the difference between the stop time of each train at the k+1th station and the first planned stop time of each train at the k+1th station.

[0141] In a possible implementation, the operation plan adjustment module 405 is also used to: obtain an initial delay set; the initial delay set includes the delay values ​​of the stations that each train has arrived at on the train operation route; determine the second planned stop time of each train at each station and the second planned running time of each train in each operation interval according to the initial delay set, the train operation action set and the trained reinforcement learning model; determine the second planned arrival time of each train at each station according to the initial delay set, the second planned stop time of each train at each station and the second planned running time of each train in each operation interval.

[0142] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0143] The reinforcement learning-based train operation adjustment device of the embodiment of the present application can make overall adjustments to the operation plans of the train groups on the same train operation line, reduce the overall train delay time, and ensure that the adjusted train operation plan will not differ too much from the original train operation plan, thereby avoiding large-scale adjustments to the operation plans of the train groups.

[0144] The embodiment of the present application also proposes a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned train operation adjustment method based on reinforcement learning is implemented. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0145] An embodiment of the present application also proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned reinforcement learning-based train operation adjustment method when executing the instructions stored in the memory.

[0146] An embodiment of the present application also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned reinforcement learning-based train operation adjustment method.

[0147] Figure 5 1 is a block diagram of an electronic device 1900 according to an embodiment of the present application. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-mentioned train operation adjustment method based on reinforcement learning.

[0148] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0149] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above-mentioned reinforcement learning-based train operation adjustment method.

[0150] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0151] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0152] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0153] The computer program instructions for performing the operation of the present application can be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of a computer-readable program instruction to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute a computer-readable program instruction, thereby realizing various aspects of the present application.

[0154] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0157] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a dedicated hardware-based system that performs the function or action of the specification, or can be realized by a combination of special-purpose hardware and computer instructions.

[0158] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A train operation adjustment method based on reinforcement learning, It is characterized in that include: Obtaining the first train operation plan of multiple trains on the same train operation line; The train operation route includes a plurality of stations; The first train operation plan includes a first planned arrival time of each train among the multiple trains at each station in the train operation line, a first planned stop time of each train at each station, and a first planned operation time of each train in each operation section in the train operation line; the operation section refers to the section between two adjacent stations in the train operation line; Establishing a train operation state set and a train operation action set; the train operation state set includes the delay value of each train at each station; the train operation action set includes the stop time of each train at each station and the travel speed of each train in each operation interval; the delay value represents the difference between the time when each train arrives at each station and the first planned arrival time of each train at each station; According to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each operation section and the first planned running time, the target reward function is configured; the running time of each train in each operation section is determined according to the distance of each operation section and the travel speed of each train in each operation section; According to the train running state set, the train running action set and the target reward function, the reinforcement learning model is trained based on the deep deterministic policy gradient DDPG algorithm to obtain a trained reinforcement learning model; The trained reinforcement learning model is used to obtain a second train planned operation plan; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station, and the second planned operation time of each train in each operation interval.

2. The method according to claim 1, It is characterized in that If the first planned stop time of the first train at the first station is not 0, then the stop time of the first train at the first station in the train operation action set is also not 0, and the stop time of the first train at the first station is greater than or equal to the minimum preset stop time; wherein, the first train is any train among the multiple trains; and the first station is any station among the multiple stations.

3. The method according to claim 2, It is characterized in that In the train operation action set, the running speed of the first train in each operation section is less than or equal to the maximum preset running speed of the first train, and the running speed of the first train in the first operation section is less than or equal to the maximum allowed running speed of the first operation section; The first operating interval is any operating interval among the operating intervals.

4. The method according to claim 1, It is characterized in that The target reward function includes a first reward function, a second reward function and a third reward function; the value of the target reward function is obtained by weighted summing the value of the first reward function, the value of the second reward function and the value of the third reward function; The value of the first reward function is calculated according to the difference between the delay value of each train at the second station and the delay value of each train at the third station; The value of the second reward function is calculated according to the difference between the delay value of the departure station of each train among the multiple stations and the delay value of the terminal station of each train among the multiple stations; The value of the third reward function is calculated according to the difference between the first planned stop time of each train at the first station and the stop time of each train at the first station, and the difference between the first planned running time of each train in the first running section and the running time of each train in the first running section; The first station is any station among the multiple stations; the second station is any station among the multiple stations except the terminal station; the third station is the next station adjacent to the second station among the multiple stations; and the first operating section is any operating section among the operating sections.

5. The method according to claim 4, It is characterized in that The reinforcement learning model includes a current actor Actor network, a target Actor network, a current value Critic network and a target Critic network; the train operation line includes M+1 stations; The step of training the reinforcement learning model based on the deep deterministic policy gradient (DDPG) algorithm according to the train running state set, the train running action set and the target reward function to obtain a trained reinforcement learning model includes: Input the k-th train running state into the current Actor network to obtain the action difference corresponding to the k-th train running state; the k-th train running state is the delay value of each train in the train running state set at the k-th station in the train running line; the action difference corresponding to the k-th train running state includes the difference between the stop time of each train at the k-th station and the first planned stop time of each train at the k-th station, and the difference between the running time of each train in the k+1-th running section and the first planned running time of each train in the k+1-th running section; the k+1-th running section is the running section between the k-th station and the k+1-th station in the train running line; 0≤k≤M; According to the action difference corresponding to the k-th train running state, an action corresponding to the k-th train running state is obtained; the action corresponding to the k-th train running state includes the stop time of each train at the k-th station and the travel speed of each train in the k+1-th running section selected from the train running action set; Inputting the k-th train running state and the action corresponding to the k-th train running state into the current Critic network to obtain a current reward value; the current reward value is calculated according to the target reward function; Updating the parameters of the current Actor network according to the current reward value until a first preset training condition is met, stopping updating the parameters of the current Actor network, and obtaining a trained current Actor network; Inputting the k+1th train running state into the target Actor network to obtain the action difference corresponding to the k+1th train running state; Obtaining the action corresponding to the k+1th train running state according to the action difference corresponding to the k+1th train running state; Inputting the k+1th train running state and the action corresponding to the k+1th train running state into the target Critic network to obtain a target reward value; the target reward value is calculated according to the target reward function; Updating the parameters of the current Critic network according to the current reward value and the target reward value until a second preset training condition is met, stopping updating the parameters of the current Critic network, and obtaining a trained current Critic network; Using the parameters of the trained current Actor network as the parameters of the target Actor network, and using the parameters of the trained current Critic network as the parameters of the target Critic network, to obtain a trained target Actor network and a trained target Critic network; A trained reinforcement learning model is obtained according to the trained current Actor network, the trained current Critic network, the trained target Actor network and the trained target Critic network.

6. The method according to claim 5, It is characterized in that The current Actor network and the target Actor network include a hidden layer and an output layer; The step of inputting the k-th train running state into the current Actor network to obtain the action difference corresponding to the k-th train running state includes: Inputting the k-th train running state into the hidden layer of the current Actor network and applying a linear rectification ReLU activation function to obtain a first eigenvalue; Input the first eigenvalue to the output layer of the current Actor network and apply the hyperbolic tangent tanh activation function and then multiply by a preset coefficient to obtain the difference between the running time of each train in the k+1th running section and the first planned running time of each train in the k+1th running section; Input the first eigenvalue to the output layer of the current Actor network and apply the tanh activation function to obtain the second eigenvalue; divide the second eigenvalue by 2 and apply the ReLU activation function to obtain the difference between the stop time of each train at the k-th station and the first planned stop time of each train at the k-th station; The step of inputting the k+1th train running state into the target Actor network to obtain the action difference corresponding to the k+1th train running state includes: Inputting the k+1th train running state into the hidden layer of the target Actor network and applying a ReLU activation function to obtain a third eigenvalue; Input the third eigenvalue to the output layer of the target Actor network and apply the tanh activation function and then multiply by the preset coefficient to obtain the difference between the running time of each train in the k+2th running section and the first planned running time of each train in the k+2th running section; The third eigenvalue is input into the output layer of the target Actor network and the tanh activation function is applied to obtain the fourth eigenvalue; the fourth eigenvalue is divided by 2 and the ReLU activation function is applied to obtain the difference between the stop time of each train at the k+1th station and the first planned stop time of each train at the k+1th station.

7. The method according to claim 5 or 6, It is characterized in that The method of using the trained reinforcement learning model to obtain a second train operation plan includes: Acquire an initial delay set; the initial delay set includes delay values ​​of stations that each train has arrived at on the train running route; Determine, according to the initial delay set, the train operation action set and the trained reinforcement learning model, the second planned stop duration of each train at each station and the second planned operation duration of each train at each operation section; The second planned arrival time of each train at each station is determined based on the initial delay set, the second planned stop time of each train at each station and the second planned running time of each train in each running section.

8. A train operation adjustment device based on reinforcement learning, It is characterized in that include: An acquisition module, used for acquiring a first train operation plan of multiple trains on the same train operation line; The train operation route includes a plurality of stations; The first train planned operation plan includes a first planned arrival time of each train among the multiple trains at each station in the train operation line, a first planned stop time of each train at each station, and a first planned operation time of each train in each operation section of the train operation line; The running section refers to the section between two adjacent stations in the train running line; A state set and action set establishment module, used to establish a train operation state set and a train operation action set; the train operation state set includes the delay value of each train at each station; the train operation action set includes the stop time of each train at each station and the travel speed of each train in each operation interval; the delay value represents the difference between the time when each train arrives at each station and the first planned arrival time of each train to arrive at each station; A reward function configuration module, configured to configure a target reward function according to the delay value of each train at each station, the stop time of each train at each station and the first planned stop time, the running time of each train in each running section and the first planned running time; the running time of each train in each running section is determined according to the distance of each running section and the travel speed of each train in each running section; A training module, used for training the reinforcement learning model based on the deep deterministic policy gradient DDPG algorithm according to the train running state set, the train running action set and the target reward function, to obtain a trained reinforcement learning model; An operation plan adjustment module is used to use the trained reinforcement learning model to obtain a second train planned operation plan; the second train planned operation plan includes the second planned arrival time of each train at each station, the second planned stop time of each train at each station and the second planned operation time of each train in each operation interval.

9. An electronic device, It is characterized in that include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 7 when executing the instructions stored in the memory.

10. A non-volatile computer-readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images