Method and system for adjusting operation of ordinary-speed railway train based on reinforcement learning

Abstract

The invention provides a general-speed railway train operation adjustment method and system based on reinforcement learning, belongs to the technical field of railway dispatching commanding, and aims at minimizing the total delay of a train at each station and minimizing the stop times of a freight train by considering the dynamism and randomness in the train operation process. Constructing a normal-speed railway train operation adjustment model; and solving the normal-speed train operation adjustment model in two stages based on a reinforcement learning DQN solving algorithm to realize normal-speed train operation adjustment. According to the method, the uncertainty of time division when the freight train runs in the interval is considered, and the priority relationship between the freight train and the passenger train is preferentially considered during adjustment, so that the delay of the passenger train is reduced as much as possible. And the intelligent agent explores and realizes a better scheduling scheme by applying a reinforcement learning technology and combining actual scheduling measures on site.

Description

Technical Field

[0001] The present invention relates to the technical field of railway dispatching and command, and in particular to a conventional railway train operation adjustment method and system based on reinforcement learning. Background Art

[0002] The ability of the railway dispatching and command system to deal with various risks and accidents has always been the focus of the railway operation department. Due to bad weather, temporary speed limits, natural disasters and even train accidents, the original operation plan will be disrupted, so it is necessary to formulate a train operation adjustment plan in real time. However, in the actual operation of the railway, the dispatching and commanding personnel can only adjust the train operation diagram based on their own historical experience. This method is prone to large errors, low stability and strong subjectivity in the adjustment results. On the lines where passenger and freight trains are mixed, passenger trains are usually given priority in dispatching and adjustment because the operation level of passenger trains is higher than that of freight trains. When the arrival and departure time of passenger trains is affected, the arrival and departure time of freight trains is often greatly delayed. At the same time, due to the large uncertainty of the interval operation time of freight trains in each section, the operation of freight trains poses a potential threat to the safety of passenger trains. Therefore, how to effectively coordinate the operation plans of passenger and freight trains in a complex and changing operation environment to ensure the safety and efficiency of train operation is a major challenge facing the railway dispatching and command system.

[0003] The train operation adjustment problem or dynamic operation adjustment is an NP-complete problem. With the development of artificial intelligence, machine learning, reinforcement learning and other technologies have been widely used in railway transportation, but there are few related studies in the field of conventional railway operation. In particular, there are few studies on the adjustment of conventional railway train operation using reinforcement learning. In summary, there is an urgent need for a method that can provide decision support for train dispatchers to formulate conventional railway train operation adjustment plans in real time. Summary of the invention

[0004] The object of the present invention is to provide a conventional railway train operation adjustment method and system based on reinforcement learning to solve at least one technical problem existing in the above-mentioned background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a conventional railway train operation adjustment method based on reinforcement learning, comprising:

[0007] Considering the dynamics and randomness of train operation, a conventional railway train operation adjustment model is constructed with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains.

[0008] Based on the reinforcement learning DQN solution algorithm, the conventional railway train operation adjustment model is solved in two stages to realize the conventional train operation adjustment. First, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted. Then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

[0009] Furthermore, the conventional railway train dispatching adjustment model is as follows:

[0010] The objective function consists of two parts: the total delay time of the train and the number of stops of the freight train at each station:

[0011]

[0012] Where: f is the train, F is the set of all trains, s is the station, is the decision variable, representing the adjusted train departure time, is the scheduled train departure time, fh is a freight train, It is a 0-1 decision variable, representing whether the freight train stops at each station.

[0013] Furthermore, the DQN algorithm includes: during the train operation adjustment process, first obtaining the current network state s t , and according to the ε-greedy strategy, select an action a from all feasible actions t , guide the train into the new state s t+1 , and give a reward value r that reflects the quality of the new state t ; Then, the agent makes decisions again based on the new state and reward value, and continues to do so over time until all trains arrive at their final destination.

[0014] Furthermore, the intelligent agent includes a target network and an evaluation network, which uses an experience pool to store train status information, specific decision information and corresponding reward information before and after the decision is made in the train scheduling process; a batch of status information is randomly extracted from the experience pool and input into the target network and the evaluation network respectively; the evaluation network executes strategy iteration according to the two decisions, and the difference between the two decisions constitutes a loss value; the loss value is minimized by a neural network optimizer, thereby training the network parameters of the evaluation network; after a certain time interval, the parameters of the evaluation network are synchronized to the target network.

[0015] Furthermore, the ε-greedy strategy is as follows:

[0016]

[0017] In the learning process, a random number rand∈(0,1) is generated. If the random number is greater than 1-ε, then t In the state, a random action is selected. In other cases, a random action is selected in s t Select an action with the largest Q value in the state; where A * (s t ) means in s t The possible action with the largest Q value in the state, A(s t ) means in s t The set of all possible actions in a state.

[0018] Furthermore, the Q value update formula is as follows:

[0019] Q(s t ,a t θ t )←Q(s t ,a t θ t )+α[r t +γmax(s t+1 ,a t+1 θ - )-Q(s t ,a t θ t )]

[0020] In the formula, Q(s t ,a t θ t ) is the current state s t and action a t The corresponding Q value, θ t is the parameter of the neural network for calculating the Q value, α is the learning rate, which controls the step size of the Q value update; r t is in state s t Take action a t The immediate reward after the reward is obtained; γ is the discount factor, which indicates the discount rate of future rewards; max(s t+1 ,a t+1 θ - ) is in the next state s t+1 Next, choose the best action a t+1 The corresponding maximum Q value.

[0021] In a second aspect, the present invention provides a conventional railway train operation adjustment system based on reinforcement learning, comprising:

[0022] The construction module is used to consider the dynamics and randomness of train operation, and to build a conventional railway train operation adjustment model with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains;

[0023] The solution module is used to solve the conventional railway train operation adjustment model in two stages based on the reinforcement learning DQN solution algorithm to realize the conventional train operation adjustment; first, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted; then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

[0024] In a third aspect, the present invention provides a non-transitory computer-readable storage medium, which is used to store computer instructions. When the computer instructions are executed by a processor, the conventional railway train operation adjustment method based on reinforcement learning as described in the first aspect is implemented.

[0025] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the conventional railway train operation adjustment method based on reinforcement learning as described in the first aspect.

[0026] In a fifth aspect, the present invention provides an electronic device, comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the conventional railway train operation adjustment method based on reinforcement learning as described in the first aspect.

[0027] The beneficial effects of the present invention are as follows: the uncertainty of the running time of freight trains in the section is taken into account, and the priority relationship between freight trains and passenger trains is given priority during adjustment to minimize the delay of passenger trains. And by using reinforcement learning technology and combining the actual dispatching measures on site, the intelligent agent can explore and implement a better dispatching solution.

[0028] Additional advantages of the present invention will be more clearly given in the following description or learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1This is a flow chart of the DQN algorithm of the conventional train operation adjustment method based on reinforcement learning described in an embodiment of the present invention.

[0031] Figure 2 This is a detailed framework diagram of the conventional train operation adjustment method based on reinforcement learning described in an embodiment of the present invention.

[0032] Figure 3 This is a learning framework diagram of the reinforcement learning described in an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of encoding and sorting all location resources on a line according to stations and sections as described in an embodiment of the present invention.

[0034] Figure 5 This is a schematic diagram of state switching according to an embodiment of the present invention.

[0035] Figure 6 This is a train operation diagram solved according to the DQN algorithm described in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below by the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0037] It should be understood by those skilled in the art that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0038] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with that in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0039] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or groups thereof.

[0040] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. Different embodiments or examples described in this specification and features of different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

[0041] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings, and the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0042] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0043] Example 1

[0044] In this embodiment 1, a conventional railway train operation adjustment system based on reinforcement learning is first provided, including: a construction module, which is used to consider the dynamics and randomness of the train operation process, and to construct a conventional railway train operation adjustment model with the goal of minimizing the total delay of the train at each station and the number of stops of the freight train; a solution module, which is used to solve the conventional railway train operation adjustment model in two stages based on the reinforcement learning DQN solution algorithm to achieve conventional train operation adjustment; wherein, firstly, according to the DQN algorithm, the running time and the use of arrival and departure lines of passenger trains are adjusted; then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and the use of arrival and departure lines of the freight train are adjusted.

[0045] like Figures 1 to 5As shown, in this embodiment, the above system is used to implement a conventional railway train operation adjustment method based on reinforcement learning, including: considering the dynamics and randomness of the train operation process, with the goal of minimizing the total delay of the train at each station and the minimum number of stops for freight trains, constructing a conventional railway train operation adjustment model; solving the conventional railway train operation adjustment model in two stages based on the reinforcement learning DQN solution algorithm to achieve conventional train operation adjustment; wherein, first, according to the DQN algorithm, the running time and arrival and departure line application of the passenger train are adjusted; then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line application of the freight train are adjusted. Specifically, considering the dynamics and randomness of the train operation process, the conventional railway train operation adjustment model is constructed with the minimum total delay of the train at each station and the minimum number of stops for freight trains, and the conventional train operation adjustment is achieved in two stages based on the reinforcement learning DQN solution algorithm. Among them, according to the DQN algorithm, the running time of passenger trains and the use of arrival and departure lines are adjusted; the adjustment results of passenger train operation are fixed, and the interval running time of freight trains is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line of freight trains are adjusted. Finally, the code is written in Python to solve, and the final train operation plan is determined according to the solution results of the DQN algorithm.

[0046] The conventional railway train dispatch adjustment model is as follows:

[0047] The objective function consists of two parts: the total delay time of the train and the number of stops of the freight train at each station:

[0048] Where: f is the train, F is the set of all trains, s is the station, is the decision variable, representing the adjusted train departure time, is the scheduled train departure time, fh is a freight train, It is a 0-1 decision variable representing whether the freight train stops at a station on its route.

[0049] The constraints of the train operation adjustment model are as follows:

[0050] ①For passenger trains, the departure time cannot be earlier than the scheduled departure time.

[0051]

[0052] Where fk is a passenger train, s is a station, and S is the set of all stations.

[0053] ② Minimum stop time constraint: For all trains stopping at stations, the actual stop time cannot be less than the minimum stop time to ensure that there is enough time to complete various operations.

[0054]

[0055] In the formula, is the minimum stop time of train f at station s, If train f stops at station s, it is 1, and if it does not stop, it is 0, which is a known constant.

[0056] ③ Section running time constraint: The section running time of train f in section e cannot be less than the minimum section running time.

[0057]

[0058] In the formula, γe is the starting station of interval e, λe is the terminal station of interval e, is the minimum interval running time.

[0059] ⑤Consistency of train arrival sequence and arrival time variables:

[0060]

[0061] In the formula, is a 0-1 parameter, which is 1 when train f arrives at station s before train f', and 0 otherwise. f It is the collection of stations on the route of train f.

[0062] ⑥Train departure and arrival tracking interval constraints:

[0063]

[0064] In the formula, is the departure interval time, is the arrival interval time, is the running order of the train in the section, which is a 0-1 parameter, E f is the set of intervals on the running path of train f.

[0065] ⑦ Relevant constraints for the use of the departure line:

[0066] Unique constraint on train track selection: If the train stops at the current station, it will choose a track that meets the boarding and alighting conditions to stop at, otherwise it will choose the mainline track to pass.

[0067]

[0068] In the formula, is the arrival and departure line set of train f at station s, is the mainline set of train f at station s, k s is the track set of station s, It is a 0-1 variable, which is 1 if train f selects track k, and 0 otherwise.

[0069] Track occupancy interval constraint: When two trains choose the same track k, the following train must wait a certain period of time after the preceding train leaves before being allowed to enter the track.

[0070]

[0071] In the formula, τ k is the minimum occupation interval time of track k.

[0072] Track quantity constraint: At the same time, the number of tracks occupied by station s cannot exceed the track capacity.

[0073]

[0074] In the formula, c s is the track capacity of station s.

[0075] The consideration of the uncertainty of the interval running time is based on the statistical analysis of the historical data of the running time of freight trains in each interval, establishing a corresponding probability fitting function to describe the data distribution, and selecting the value with the largest density in the probability distribution as the representative value, that is, as the interval running time that the freight train should meet in each interval.

[0076] like Figure 3 As shown, the DQN algorithm is as follows: During the train operation adjustment process, first obtain the current network state s t , and according to the ε-greedy strategy, select an action a from all feasible actions t , guide the train into the new state s t+1 , and give a reward value r that reflects the quality of the new state t The agent then makes decisions and selects actions again based on the new state and reward value, until all trains arrive at the final destination of the dispatching section over time. Through the continuous interaction between the agent and the train operating environment, the agent gradually learns the best value function by updating the Q value, maximizes the long-term return, and finally obtains a better dispatching strategy.

[0077] (1) Environment

[0078] Based on the established mathematical model, an environment is designed for the reinforcement learning agent to interact with. The purpose of establishing an interactive environment is to allow the agent to continuously try and find the best strategy to learn and get higher rewards. Before establishing the environment, the following assumptions are made:

[0079] ① Each train is regarded as a particle, and the train is located on the station track or the inter-station block section, and will not occupy both the station track and the inter-station block section at the same time.

[0080] ② Passenger trains are not allowed to depart early, but freight trains can depart at the station a certain time in advance to effectively compensate for the delays caused by passenger trains.

[0081] ③ Each train has only two actions: moving forward and stopping. The train travels at a constant speed, and the process of speeding up and decelerating the train is ignored.

[0082] The state space, action space and reward function are set by the constructed scheduling adjustment mathematical model.

[0083] (2) State: Assuming that time is discrete, the agent obtains observations from the environment and takes actions once every minute. If the scheduling time range is T = [t 1 ,t 2 ], it can be discretized as T = {t 1 ,t 1 +1,…,t 1 +i,…,t 2}. All location resources on the line are coded and sorted according to stations and sections, such as Figure 4 Based on this, the state is defined as the resources currently located by each train per minute. For the case of N trains, the state vector s t =[h 1 ,h 2 ,…,h n ], which indicates the resources occupied by N trains at time t. The resources occupied by the first train are h 1 , and the same can be applied to subsequent trains. Figure 5 Shown is from s t Status t+1 State switching, related location resources occupied by the train.

[0084] (3) Definition of actions and ε-greedy strategy: In the train operation adjustment problem, when a train is delayed, it is necessary to select an appropriate action based on the real-time train position and the network resource occupancy status. For each train, at the current moment, the actions that can be selected are forward (1) or stop (0). Therefore, when a i =1 means that train i moves to the next resource block. i = 0 means that train i will continue to stay on the current resource block. For the case of N trains, the action vector can be defined as an N-dimensional vector a = [a 1 ,a 2 ,…,a i ,…,a n ], indicating the actions taken by each train. The number of possible actions in the N-dimensional action space is 2 N, as the scale of the problem increases, its size grows exponentially. When the number of trains is large, the feasible action space will be very large, resulting in a sharp increase in search time. To this end, an action feasibility check is added, each generated action is substituted into the constraints of the model for verification, and the reasonable actions that meet the conditions are saved for selection at the next moment.

[0085] In this embodiment, the ε-greedy strategy is used for action selection. The advantage of adopting this strategy is that it can not only ensure that the agent moves in a better direction, but also retain a certain degree of exploration, thereby avoiding falling into a local optimal solution. Through this balance, the agent can continue to discover new potential paths in the process of searching for the optimal solution, thereby improving the efficiency and effect of the overall scheduling.

[0086]

[0087] In the formula, episode is the number of iterations, and max_episode is the maximum number of iterations in the learning process. This design is to ensure that as the number of training times increases, ε gradually decreases and approaches zero, tending to randomly search for actions in the early stage, and use learning experience in the later stage to select the action with the largest Q value.

[0088] The ε-greedy strategy is as follows:

[0089]

[0090] In the learning process, a random number rand∈(0,1) is generated. If the random number is greater than 1-ε, then t In the state, a random action is selected. In other cases, a random action is selected in s t Select an action with the largest Q value in the state. * (s t ) means in s t The possible action with the largest Q value in the state, A(s t ) means in s t The set of all possible actions in a state.

[0091] (4) Reward: The objective function of the model is Then the reward is defined as the inverse of the objective function.

[0092]

[0093] The agent's reward increases as the total delay time decreases and the number of stops decreases. When the agent reaches the final state, it can get a reward. In addition, some penalty mechanisms need to be designed to promote the convergence of the agent. For example, when the iteration time t i Update to time threshold Tmax And when all trains fail to reach the terminal, the training process is interrupted and the reward is negative. Therefore, the reward at each moment is further rewritten as:

[0094]

[0095] T max =(T 2 -T 1 )+T set

[0096] In the formula, r t is the reward function of the train, T max is the time threshold, T 2 -T 1 is the time span between the arrival time of the first train and the arrival time of the last train, T set is the time constant of the increase.

[0097] Q-value update method: The scheduling agent contains an evaluation network and a target network. The two networks have the same structure but different initial network parameters. The agent uses the experience pool to store the train status information, specific decision information, and corresponding reward information before and after the decision in the train scheduling process. A batch of status information is randomly extracted from the experience pool and input into the target network and the evaluation network respectively. The two networks make different decisions based on the input information. The evaluation network performs policy iteration according to the two decisions, and the difference between the two decisions constitutes the loss value. The network parameters of the evaluation network are trained by minimizing the loss value through the neural network optimizer. After a certain time interval, the parameters of the evaluation network are synchronized to the target network. In this way, the agent continues to learn. When the loss value is reduced to a certain threshold, the learning process is completed, and finally the parameters of the evaluation network are used to provide a decision basis for discrete train scheduling.

[0098] The Q value update formula is as follows:

[0099] Q(s t ,a t θ t )←Q(s t ,a t θ t )+α[r t +γmax(s t+1 ,a t+1 θ - )-Q(s t ,a t θ t )]

[0100] In the formula, Q(s t ,a t θt ) is the current state s t and action a t The corresponding Q value is calculated by a neural network. The structure of this neural network is determined by the parameter θ t α is the learning rate, which controls the step size of Q value update. t is in state s t Take action a t The immediate reward after γ is the discount factor, which indicates the discount rate of future rewards. max(s t+1 ,a t+1 θ - ) is in the next state s t+1 Next, choose the best action a t+1 The maximum Q value corresponding to this value is determined by the target network (i.e., the parameter is θ - The y-values ​​of the y-values ​​are estimated by using a neural network.

[0101] The loss calculation formula is as follows:

[0102] Loss(θ t )=[r t +γmax(s t+1 ,a t+1 θ - )-Q(s t ,a t θ t )] 2 .

[0103] In this embodiment, a specific operation adjustment plan is taken as an example to illustrate the method. First, by collecting data such as the planned arrival and departure times of passenger and freight trains on conventional lines (on a certain day), a train operation adjustment model is established, and then the DQN algorithm is used to design and solve the model to output the train operation adjustment plan.

[0104] In this embodiment, the train data planned to run on a certain day are selected to generate a train operation plan, including 6 stations Tianshui, Nanhechuan, Sanyangchuan, Huagang Town, Xinyang Town, and Gangu. The disturbance is set to K543 train departure time at Tianshui Station delayed by 30 minutes, K1309 train departure time at Tianshui Station delayed by 30 minutes, Z151 train delay at Tianshui Station 30 minutes, about the departure and arrival time interval of the train I 追 ,I 通 ,I 到 ,I 发 ,I 通发 ,I 到通 ,I 发通The interval operation time and the minimum interval operation time of the train obtained according to the statistical analysis results are shown in Table 1. The track information of the station is shown in Table 2.

[0105] Table 1 Interval running time

[0106]

[0107] Table 2 Station track information

[0108]

[0109] The train operation diagram is solved according to the DQN algorithm. The solution is as follows Figure 6 The comparison between the planned running time and the actual running time is shown in Table 3. After checking and verification, the scheduling plan meets all the constraints in the mathematical model. The final result shows that all trains are delayed by a total of 838 minutes. Freight train No. 10609 makes an additional stop at Weinan Town Station.

[0110] Table 3 Train operation diagram data

[0111]

[0112]

[0113]

[0114]

[0115] Example 2

[0116] This embodiment 2 provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the conventional railway train operation adjustment method based on reinforcement learning as described above is implemented. The method includes:

[0117] Considering the dynamics and randomness of train operation, a conventional railway train operation adjustment model is constructed with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains.

[0118] Based on the reinforcement learning DQN solution algorithm, the conventional railway train operation adjustment model is solved in two stages to realize the conventional train operation adjustment. First, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted. Then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

[0119] Example 3

[0120] This embodiment 3 provides a computer device, including a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, the processor calls the program instructions to execute the conventional railway train operation adjustment method based on reinforcement learning as described above, the method comprising:

[0121] Considering the dynamics and randomness of train operation, a conventional railway train operation adjustment model is constructed with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains.

[0122] Based on the reinforcement learning DQN solution algorithm, the conventional railway train operation adjustment model is solved in two stages to realize the conventional train operation adjustment. First, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted. Then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

[0123] Example 4

[0124] This embodiment 4 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes instructions for implementing the conventional railway train operation adjustment method based on reinforcement learning as described above, the method comprising:

[0125] Considering the dynamics and randomness of train operation, a conventional railway train operation adjustment model is constructed with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains.

[0126] Based on the reinforcement learning DQN solution algorithm, the conventional railway train operation adjustment model is solved in two stages to realize the conventional train operation adjustment. First, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted. Then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

[0127] In summary, the embodiment of the present invention provides a method for adjusting the operation of conventional railway trains based on reinforcement learning, and establishes a conventional railway train operation adjustment model with the minimum total delay of trains at each station and the minimum number of stops of freight trains at each station. Based on the priority relationship between passenger trains and freight trains, the model solution is divided into two stages: passenger train adjustment and freight train adjustment. The first stage adjusts the arrival and departure time and the use of arrival and departure lines of passenger trains according to the train delay situation. The second stage adjusts the arrival and departure time and the use of arrival and departure lines of freight trains based on the adjustment results of passenger trains and considering the uncertainty of the interval running time of freight trains. According to the model, a simulation environment for interacting with intelligent agents is designed. The DQN algorithm in reinforcement learning is used to solve each stage. At the same time, in order to avoid the dimensionality disaster, the feasibility of the intelligent agent's actions is tested according to the model constraints, which reduces the number of trial and error of reinforcement learning. Finally, the effectiveness of the DQN algorithm for dynamic scheduling of conventional railways is verified through specific simulation examples. This method realizes dynamic adjustment of conventional railway train operation plans for disturbance scenarios of train delays, and is expected to provide an effective and powerful solution to the conventional railway scheduling problem in complex networks, thereby significantly improving the operational efficiency of conventional railways.

[0128] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative work on the basis of the technical solution disclosed in the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for adjusting the operation of conventional railway trains based on reinforcement learning, characterized in that: include: Considering the dynamics and randomness of train operation, a conventional railway train operation adjustment model is constructed with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains. Based on the reinforcement learning DQN solution algorithm, the conventional railway train operation adjustment model is solved in two stages to realize the conventional train operation adjustment. First, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted. Then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

2. The conventional railway train operation adjustment method based on reinforcement learning according to claim 1 is characterized in that: The conventional railway train dispatching adjustment model is as follows: The objective function consists of two parts: the total delay time of the train and the number of stops of the freight train at each station: Where: f is the train, F is the set of all trains, s is the station, is the decision variable, representing the adjusted train departure time, is the scheduled train departure time, fh is a freight train, It is a 0-1 decision variable, representing whether the freight train stops at each station.

3. The conventional railway train operation adjustment method based on reinforcement learning according to claim 1 is characterized in that: The DQN algorithm includes: during the train operation adjustment process, first obtain the current network state s t , and according to the ε-greedy strategy, select an action a from all feasible actions t , guide the train into the new state s t+1 , and give a reward value r that reflects the quality of the new state t ; Then, the agent makes decisions again based on the new state and reward value, and over time, all trains arrive at the final destination of the scheduling section.

4. The conventional railway train operation adjustment method based on reinforcement learning according to claim 3 is characterized in that: The intelligent agent includes a target network and an evaluation network. It uses an experience pool to store train status information, specific decision information, and corresponding reward information before and after decisions are made during train scheduling. A batch of status information is randomly extracted from the experience pool and input into the target network and the evaluation network respectively. The evaluation network iterates the strategy execution according to the two decisions, and the difference between the two decisions constitutes the loss value. The loss value is minimized by the neural network optimizer to train the network parameters of the evaluation network. After a certain time interval, the parameters of the evaluation network are synchronized to the target network.

5. The conventional railway train operation adjustment method based on reinforcement learning according to claim 3 is characterized in that: The ε-greedy strategy is as follows: In the learning process, a random number rand∈(0,1) is generated. If the random number is greater than 1-ε, then t In the state, a random action is selected. In other cases, a random action is selected in s t Select an action with the largest Q value in the state; where A * (s t ) means in s t The possible action with the largest Q value in the state, A(s t ) means in s t The set of all possible actions in a state.

6. The conventional railway train operation adjustment method based on reinforcement learning according to claim 3 is characterized in that: The Q value update formula is as follows: Q(s t ,a t ;θ t )←Q(s t ,a t ;θ t )+α[r t +γmax(s t+1 ,a t+1 ;θ - )-Q(s t ,a t ;θ t )] In the formula, Q(s t ,a t θ t ) is the current state s t and action a t The corresponding Q value, θ t is the parameter of the neural network for calculating the Q value, α is the learning rate, which controls the step size of the Q value update; r t is in state s t Take action a t The immediate reward after the reward is obtained; γ is the discount factor, which indicates the discount rate of future rewards; max(s t+1 ,a t+1 θ - ) is in the next state s t+1 Next, choose the best action a t+1 The corresponding maximum Q value.

7. A conventional railway train operation adjustment system based on reinforcement learning, characterized in that: include: The construction module is used to consider the dynamics and randomness of train operation, and to construct a conventional railway train operation adjustment model with the goal of minimizing the total delay of trains at each station and the number of stops of freight trains; The solution module is used to solve the conventional railway train operation adjustment model in two stages based on the reinforcement learning DQN solution algorithm to realize the conventional train operation adjustment; first, according to the DQN algorithm, the running time and arrival and departure line usage of the passenger train are adjusted; then, the passenger train operation adjustment result is fixed, and the interval running time of the freight train is obtained according to the statistical analysis of historical data, and the running time and arrival and departure line usage of the freight train are adjusted.

8. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the conventional railway train operation adjustment method based on reinforcement learning as described in any one of claims 1-6 is implemented.

9. A computer device, characterized in that: It includes a memory and a processor, the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the conventional railway train operation adjustment method based on reinforcement learning as described in any one of claims 1-6.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the conventional railway train operation adjustment method based on reinforcement learning as described in any one of claims 1-6.

Images