A High-Speed ​​Rail Timetable Optimization and Rescheduling Method Based on Deep Reinforcement Learning

Abstract

This application provides a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning, relating to the fields of high-speed rail scheduling and resource optimization technology, aiming to achieve efficient and accurate optimization and rescheduling of high-speed train timetables. The method includes: constructing a policy network based on a Markov decision process model for high-speed rail scheduling, wherein the policy network learns the action decision policies of the Markov decision process model for high-speed rail scheduling; inputting the current station state into the policy network to obtain the action decision of the departure order of all trains at the current station; simulating the train operation environment based on the departure order of all trains at the current station using a train state transition model to determine the arrival and departure times of all trains at the current station that satisfy the train operation safety constraints; and iteratively calculating the arrival and departure times of all trains at each station along the route according to the above steps to obtain the train timetable.

Description

Technical Field

[0001] This application relates to the field of high-speed rail scheduling and resource optimization technology, and in particular to a high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning. Background Technology

[0002] High-speed rail, with its convenience, speed, and punctuality, has become the preferred mode of transportation for passengers. Train timetables connect various transportation activities within the high-speed rail system, playing a fundamental and leading role in train operation organization. During the planning phase, dispatchers need to develop efficient train timetables based on given stopping plans, determining the arrival and departure times of trains at each station and minimizing the total travel time for all trains. This process is called Train Timetable Optimization (TTO). Under normal circumstances, high-speed rail operates according to the planned timetable. However, unavoidable disruptions caused by unforeseen events such as severe weather and equipment failures frequently occur, leading to delays for single or multiple trains. Train Timetable Rescheduling (TTR) aims to quickly restore train order through reordering, retiring, and rerouting strategies to reduce total delay time. Therefore, the rational development of train timetables and minimizing deviations in actual operation are crucial for ensuring the safe operation of high-speed rail and meeting passenger needs.

[0003] Currently, high-speed rail dispatching employs a decentralized autonomous dispatching centralized system, which solves automation issues such as dispatching instruction issuance and remote signal control. However, optimizing and rescheduling train timetables still relies primarily on manual methods, heavily depending on dispatchers' experience and judgment. This results in insufficient real-time performance and poor performance on large-scale problems. Furthermore, the number of decision variables in high-speed rail train dispatching exhibits a polynomial relationship with the problem size, making real-time optimization and timetable rearrangement difficult, especially in real-world large-scale problem scenarios. Therefore, providing an efficient dispatching framework for TTO and TTR problems is of significant value, not only reducing the workload of dispatchers but also improving the operational efficiency of the railway system. Summary of the Invention

[0004] In view of the above problems, this application provides a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning, so as to overcome the above problems or at least partially solve them.

[0005] A first aspect of this application discloses a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning, the method comprising:

[0006] Based on the Markov decision process model for high-speed rail scheduling, a policy network is constructed. The policy network learns the action decision policies of the Markov decision process model for high-speed rail scheduling. The policy network includes an input embedding module, an encoder, and a decoder. The action decision policies represent the scheduling strategies of high-speed trains. The Markov decision process model for high-speed rail scheduling is constructed with the goal of minimizing the total train travel time or minimizing the total train delay time.

[0007] The current station status is input into the policy network to obtain the action decision of the departure order of all trains at the current station. The current station status includes station features that characterize station capacity and station location, as well as train features that characterize train operation information.

[0008] Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival and departure times of all trains at the current station that meet the train operation safety constraints are determined.

[0009] Following the steps above, the arrival and departure times of all trains at each station along the route are calculated iteratively to obtain the train timetable.

[0010] Optionally, based on the Markov decision process model for high-speed rail scheduling, a policy network is constructed, including:

[0011] Based on the Markov decision process model of high-speed rail scheduling, an action decision strategy for policy network learning is designed.

[0012] With the goal of learning the action decision-making strategy, the policy network is trained using a deep reinforcement learning algorithm. After the training termination condition is met, the trained policy network is obtained.

[0013] Optionally, with the goal of learning the action decision-making strategy, the policy network is trained using a deep reinforcement learning algorithm, including:

[0014] Based on the current policy network parameters and the target station status, the policy network generates the departure sequence of all trains at the target station.

[0015] The reward value and the state of the next target station are calculated based on the departure sequence of all trains at the target station using the train state transition model.

[0016] The target station status, the departure sequence of all trains at the target station, and the reward value are used as a scheduling trajectory. By repeating the above steps until the destination station, a complete scheduling trajectory is obtained.

[0017] The gradient is calculated based on the complete scheduling trajectory, and the policy network parameters of the policy network are updated.

[0018] Optionally, the current station state is input into the policy network to obtain the action decision of the departure sequence of all trains at the current station, including:

[0019] The current station status is input into the input embedding module to obtain a station embedding containing global information of the current station and all subsequent stations, and a node embedding containing all train information.

[0020] The encoder extracts the dependencies between trains by using an attention mechanism on the node embeddings to obtain an encoded train embedding containing global information of all trains.

[0021] The decoder decodes the train embedding and the station embedding to obtain the departure order of all trains at the current station.

[0022] Optionally, the current station status includes station features and train features, the train features include train dynamic features and train static features; the input embedding module includes a first long short-term memory module, a second long short-term memory module, and a linear projection module;

[0023] The current station status is input into the input embedding module to obtain a station embedding containing global information about the current station and all subsequent stations, and a node embedding containing information about all trains, including:

[0024] The station features are input into the first long short-term memory module to obtain a station embedding that includes global information of the current station and all subsequent stations.

[0025] The train static features are input into the second long short-term memory module to obtain a static node embedding containing all train static information.

[0026] The train dynamic features are input into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding.

[0027] Based on the static node embedding and the dynamic node embedding, a node embedding containing all train information is obtained.

[0028] Optionally, the encoder includes multiple attention coding layers, each of which includes a multi-head attention layer and a fully connected multilayer perceptual feedforward layer;

[0029] The encoder extracts dependencies between trains from the node embeddings using an attention mechanism, resulting in an encoded train embedding containing global information about all trains, including:

[0030] The node embedding is input into the multiple attention coding layers for multi-layer attention coding to obtain a coded train embedding containing global information of all trains.

[0031] Optionally, the decoder decodes the train embedding and the station embedding to obtain the departure order of all trains at the current station, including:

[0032] At each decision time step, a context embedding is determined based on the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, wherein the sequence embedding represents a partially determined departure sequence, and the aggregate embedding of the coded train embedding represents the mean of the coded train embedding.

[0033] Based on the context embedding and the coded train embedding, the next departing train is determined from the selectable trains that meet the constraints according to the strategy probability distribution; the strategy probability distribution represents the probability distribution of each train being selected as the next departing train; the selectable trains that meet the constraints include: trains that have not been selected, and the N trains whose arrival times at the current station are ranked first.

[0034] Repeat the above steps multiple times until all trains leave the current station, thus obtaining the departure order of all trains at the current station.

[0035] Optionally, based on the context embedding and the coded train embedding, the next departing train is determined from the selectable trains that meet the constraints according to the policy probability distribution, including:

[0036] The context embedding and the coded train embedding are input into the first decoding layer for attention interaction to obtain a new context embedding; wherein, the first decoding layer determines the key vector and value vector of attention interaction based on the coded train embedding, and determines the query vector of attention interaction based on the context embedding;

[0037] The new context embedding and the coded train embedding are input into the second decoding layer to obtain attention weights, which represent the weight of each train being selected as the next departing train.

[0038] Based on the attention weights, a strategy probability distribution is obtained, and the next departing train is determined based on the strategy probability distribution.

[0039] Optionally, based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station to determine the arrival and departure times of all trains at the current station that satisfy the train operation safety constraints, including:

[0040] Based on the aforementioned traffic safety constraints and the departure times of all trains at the previous station, determine the initial arrival times of all trains at the current station.

[0041] For each target train, determine the track occupancy status of the current station at the target time. The target time is the initial arrival time of the target train minus the given shortest departure-arrival interval. The target train is any train selected sequentially from the departure order of all trains.

[0042] If there are no available tracks, the arrival time of the target train at the current station is updated based on the track occupancy status, and the arrival times of all subsequent trains that departed later than the target train at the previous station are also updated at the current station.

[0043] If there are available tracks when the track occupancy status is such that an available track is randomly assigned to the target train, and the arrival time of the target train at the current station is the initial arrival time;

[0044] Based on the train safety constraints and the arrival time at the current station, the readiness time of the target train is determined, and the departure time of the target train at the current station is determined based on the readiness time. The readiness time represents the time when the train has completed the stopping operation and is ready to leave the station.

[0045] Optionally, the Markov decision process model for high-speed rail scheduling is constructed in the following manner:

[0046] The station state, composed of train characteristics and station characteristics, is used as the state of the Markov decision process model for high-speed rail scheduling.

[0047] The departure order of all trains at the current station is taken as the action of the Markov decision process model for high-speed rail scheduling.

[0048] The process of transitioning from the current station state to the next station state based on the departure sequence of all trains at the station is used as the state transition of the Markov decision process model for high-speed rail scheduling.

[0049] In the case of initial optimization of train timetables, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station with the objective of minimizing the total train travel time; or, in the case of rescheduling train timetables, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station with the objective of minimizing the total train delay time.

[0050] Optionally, given the initial optimized train timetable, with the objective of minimizing the total train travel time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station, including:

[0051] Under the constraint of driving safety, with the goal of minimizing the total train travel time, a first objective function is constructed based on the train travel time between two stations. The train travel time between two stations includes: the minimum stopping time at the station, the minimum running time in the section, and the buffer time.

[0052] The negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model for high-speed rail scheduling.

[0053] Optionally, in the case of rescheduling train timetables, with the objective of minimizing total train delay time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure sequence of all trains at the station, including:

[0054] Under the constraint of train operation safety, with the goal of minimizing the total train delay time, a second objective function is constructed based on the planned arrival time of the train at the station and the planned departure time of the train at the station, which are the times obtained from the initial optimized train timetable.

[0055] The negative objective function corresponding to the second objective function is used as the reward value of the Markov decision process model for high-speed rail scheduling.

[0056] Optionally, the driving safety constraints include:

[0057] Operating time constraints, which are used to ensure that the train's stopping time at each station and the operating time in the section are not less than the minimum time limit;

[0058] Overtaking constraint, which restricts a train from overtaking within the section between stations;

[0059] Time interval constraint, which is used to ensure that the time interval between adjacent trains is not less than the safe time interval, which includes the station arrival time interval, the station departure time interval, and the section tracking time interval;

[0060] Station capacity constraint, which is used to constrain that each track of a station can only be allocated to a single train at the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train is not less than the minimum departure-arrival time interval to satisfy train safety.

[0061] Arrival and departure time constraints are used to address the problem of rescheduling train timetables. Under normal circumstances, trains will run according to the train timetable; or under normal circumstances, trains are only allowed to arrive earlier than scheduled, and are not allowed to depart earlier than the original departure time in the train timetable.

[0062] A second aspect of this application discloses a high-speed rail train timetable optimization and rescheduling device based on deep reinforcement learning, the device comprising:

[0063] A construction module is used to build a policy network based on the Markov decision process model of high-speed rail scheduling. The policy network learns the action decision policies of the Markov decision process model of high-speed rail scheduling. The policy network includes an input embedding module, an encoder and a decoder. The action decision policies represent the scheduling policies of high-speed rail trains. The Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total train travel time or minimizing the total train delay time.

[0064] The input module is used to input the current station status into the policy network to obtain the action decision as the departure sequence of all trains at the current station. The current station status includes station features that characterize station capacity and station location, as well as train features that characterize train operation information.

[0065] The determination module is used to determine the arrival and departure times of all trains at the current station that meet the train operation safety constraints, based on the train state transition model and the train operation environment simulation according to the departure sequence of all trains at the current station.

[0066] The iterative module is used to iteratively calculate the arrival and departure times of all trains at each station on the route according to the above steps, and obtain the train timetable.

[0067] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning described in the first aspect of this application.

[0068] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning described in the first aspect of this application.

[0069] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning described in the first aspect of this application.

[0070] The embodiments of this application have the following advantages:

[0071] In this embodiment, the high-speed rail timetable optimization and rescheduling problem is modeled as a two-stage real-time decision-making model. The first stage is a Markov decision process model for high-speed rail scheduling, and a policy network is constructed based on this model. This allows the policy network to learn the action decision-making strategies of the high-speed rail scheduling Markov decision process model. Since the high-speed rail scheduling Markov decision process model is constructed with the objective of minimizing the total train travel time or minimizing the total train delay time, the policy network can make action decisions based on the current station state, resulting in a departure sequence for all trains at the current station that satisfies the objective of minimizing the total train travel time or minimizing the total train delay time. The second stage is a train state transition model. A heuristic algorithm is used to simulate the train operating environment based on the departure sequence of all trains at the current station, determining the arrival and departure times of all trains at the current station that satisfy the train operation safety constraints. This ensures that trains can run safely and normally while satisfying the objective of minimizing the total train travel time or minimizing the total train delay time. Then, the above two stages are iterated station by station to calculate the arrival and departure times of all trains at each station on the route, resulting in the train timetable. In this way, by breaking down each decision step of high-speed rail timetable optimization and rescheduling into two stages, scheduling problems under different operating plans and delay scenarios can be handled in real time. The train departure sequence and precise arrival and departure times at each station are calculated sequentially. Therefore, in real-world large-scale problem scenarios, this method can optimize the initial timetable and provide the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction. Attached Figure Description

[0072] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a schematic diagram of high-speed train dispatching provided in an embodiment of this application;

[0074] Figure 2 This is a flowchart illustrating the steps of a high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning, as provided in an embodiment of this application.

[0075] Figure 3 This is an architecture diagram of a policy network provided in an embodiment of this application;

[0076] Figure 4 This is a schematic diagram of the overall framework of a general high-speed rail scheduling model provided in an embodiment of this application;

[0077] Figure 5 This is a schematic diagram of the structure of a high-speed rail train timetable optimization and rescheduling device based on deep reinforcement learning provided in an embodiment of this application;

[0078] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0079] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0080] Similar to many scheduling problems, TTO and TTR problems are classified as NP-hard problems because they are inherently mixed-integer programming problems with complex constraints and a large number of integer variables, which increase with the size of the high-speed rail system (including the number of trains and stations). In terms of modeling, their decision variables and constraints are essentially the same, with the main difference lying in the objective function and parameters. This commonality suggests that these two problems can be solved within a unified framework.

[0081] In related technologies, numerous solutions have been proposed for two types of train scheduling problems, which can be categorized into four types: operations research methods, heuristic methods, manually designed rules, and machine learning methods. The first type typically uses different mathematical programming models to model the high-speed rail scheduling problem, such as mixed-integer linear programming models designed for TTO and TTR. Building upon this, to improve solution speed, researchers have proposed algorithms such as branch and bound, Lagrange relaxation, and column generation to decompose complex problems. However, mathematical methods require significant computational time to obtain accurate solutions for large-scale problems, making them difficult to apply to actual high-speed rail lines. To address this issue, the second type of method designs heuristic algorithms based on swarm intelligence and iterative optimization, such as genetic algorithms, hybrid particle swarm optimization, and adaptive large-scale neighborhood search, which have shorter response times compared to the first type of method. However, heuristic algorithms generally suffer from poor transferability, requiring iterative solutions for different operational scenarios, resulting in a still computationally intensive problem.

[0082] While the two methods mentioned above have achieved good results, in practice, dispatchers often use the third type of manually formulated fixed rules to meet real-time and safety requirements. Popular scheduling rules include First-Come-First-Served (FCFS) and First-Scheduled-First-Served (FSFS) strategies, but these do not yield optimal timetables, hindering the full realization of the high-speed rail system's potential. Therefore, designing a scheduling strategy with low time cost and high performance has become a key challenge in the field of high-speed rail scheduling. In recent years, artificial intelligence technology has been increasingly applied to the field of intelligent transportation and has proven to outperform traditional methods in terms of solution quality and computational efficiency. Therefore, the fourth type of learning-based methods, which can automatically learn advanced heuristics, especially deep reinforcement learning, has become a new focus for solving high-speed rail scheduling problems.

[0083] To overcome the shortcomings of existing scheduling strategies in terms of solution quality and real-time response, this application provides a high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning, which solves the problems of initial timetable optimization and rescheduling. On the one hand, initial timetable optimization lays the foundation for the smooth operation of trains according to the planned scheduling scheme under ideal conditions. On the other hand, timetable rescheduling ensures the adaptability and resilience of transportation services in the face of actual emergencies. This method not only optimizes the initial timetable but also has the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0084] like Figure 1 As shown, the high-speed rail train scheduling in this embodiment considers the initial optimization and rescheduling problem of a single-direction (up or down) train timetable on a passenger dedicated line, assuming that multiple trains of different speed levels operate on the line. Initial timetable optimization involves creating a preliminary timetable before travel, focusing on optimizing operational objectives under known train and line constraints to minimize travel time or maximize resource utilization. Its purpose is to create a feasible and practical timetable based on a specified number of trains, train types, and stop plans for each train. During daily operation, timetable rescheduling dynamically adjusts the timetable to address unforeseen disruptions during travel, restoring the original timetable or minimizing deviations. Its purpose is to adjust train arrival and departure times, departure sequences, and track occupancy online based on the planned timetable, avoiding network-wide delays.

[0085] The following description, in conjunction with the accompanying drawings, illustrates the high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning, according to an embodiment of this application.

[0086] To better understand the technical solution of this application, the variables involved in the embodiments of this application are defined. The embodiments of this application consider a two-way, double-track high-speed railway line, with trains traveling in the up and down directions running on their respective tracks, and assume that multiple trains of different speed levels are allowed. Therefore, the embodiments of this application focus on solving the initial optimization and rescheduling problem of train timetables in one direction (up or down). For the above two types of scheduling problems, the following parameters are defined: the set of stations indexed by the label k. , ; by label or Indexed train set , ; by label Indexed stations orbit set , ,in Indicates station The total number of tracks. The ultimate goal of both the initial optimization and rescheduling problems is to provide an efficient train timetable; therefore, the decision variables are the same, including: trains. Arrival at the station Time ;train Leaving the station Time Departure sequence indicator variable If the train On the train Before leaving the station The value is 1 if the value is 1, otherwise it is 0; Track occupancy indicator variable If the train At the station Occupy track If it is 1, then it is 1; otherwise, it is 0.

[0087] Reference Figure 2 As shown, Figure 2 This is a flowchart illustrating the steps of a high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning, as provided in an embodiment of this application. Figure 2 As shown, the high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning may include steps S210 to S240:

[0088] Step S210: Based on the Markov decision process model for high-speed rail scheduling, a policy network is constructed. The policy network learns the action decision policies of the Markov decision process model for high-speed rail scheduling. The policy network includes an input embedding module, an encoder, and a decoder. The action decision policies represent the scheduling strategies of high-speed trains. The Markov decision process model for high-speed rail scheduling is constructed with the goal of minimizing the total train travel time or minimizing the total train delay time.

[0089] Among them, the Markov decision process model for high-speed rail scheduling is to construct the high-speed rail scheduling process as a Markov decision process model. The construction method of the high-speed rail scheduling Markov decision process model is different for different train scheduling problems. Specifically, under the condition of initial optimization of train timetable, the high-speed rail scheduling process is constructed as a high-speed rail scheduling Markov decision process model with the goal of minimizing the total train travel time. Under the condition of rescheduling train timetable, the high-speed rail scheduling process is constructed as a high-speed rail scheduling Markov decision process model with the goal of minimizing the total train delay time.

[0090] The policy network is an attention-based policy network that adopts an encoder-decoder framework. It is built on the Markov decision process model of high-speed rail scheduling so that the policy network can learn the action policy of the Markov decision process model of high-speed rail scheduling (i.e., learn the scheduling policy of high-speed trains).

[0091] Step S220: Input the current station status into the policy network to obtain the action decision of the departure sequence of all trains at the current station. The current station status includes station features representing station capacity and station location, and train features representing train operation information.

[0092] The operating route is a single-direction passenger dedicated line, either the downhill or uphill route. This route includes multiple stations, and multiple trains operate on it. The current station status refers to the station and train characteristics of the current station and subsequent stations. Station characteristics include station location and capacity. Train characteristics include dynamic and static characteristics. Static characteristics (fixed characteristics) refer to fixed train operation information, such as the shortest travel time between sections, the shortest stop time at stations, station buffer time, and planned departure time. Dynamic characteristics (variable characteristics) refer to dynamic operation information that changes according to each decision step, i.e., the ready time (when the train has completed its stop operation and is ready to leave the station).

[0093] Understandably, the current station status will differ depending on the specific train scheduling problem. For example, regarding train static characteristics, in the case of initial optimization of train timetables (i.e., initial optimization problem), train static characteristics include the shortest station dwell time and the shortest travel time between sections. In the case of rescheduling train timetables (rescheduling problem), train static characteristics include the station buffer time and the planned departure time.

[0094] The policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling. Therefore, the policy network can make action decisions based on the current station status, give the departure order of all trains at the current station, and the departure order of all trains at the current station satisfies the objective of minimizing the total train travel time or minimizing the total train delay time.

[0095] Specifically, the policy network includes an input embedding module, an encoder, and a decoder. Station features and train features are input into the policy network. The input embedding module generates a station embedding containing global information of the current station and all subsequent stations, as well as a node embedding containing information of all trains. The encoder further processes the station embedding using an attention mechanism to extract complex dependencies between trains and obtain an encoded train embedding containing global information of all trains. The decoder performs decoding at multiple decision time steps based on the encoded train embedding and the station embedding to obtain the departure order of all trains at the current station.

[0096] At each decision time step, the coded train embedding and context embedding are used as inputs. Based on the partially determined departure sequence (included in the context embedding), the next departing train is sampled from the available trains that meet the constraints according to the policy probability distribution. This process continues until all trains have left the current station to obtain the complete departure sequence. The context embedding includes the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding. The sequence embedding represents the departure sequence that has been partially determined at the current decision step, and the aggregate embedding of the coded train embedding represents the mean of the coded train embeddings.

[0097] Step S230: Based on the train state transition model, simulate the train operation environment according to the departure sequence of all trains at the current station, and determine the arrival and departure times of all trains at the current station that meet the train operation safety constraints.

[0098] Among them, the train state transition model can use heuristic algorithms to simulate the train operation environment. Based on the train state transition model, the arrival time and departure time of all trains at the current station can be determined according to the departure order of all trains at the current station. That is, for the current station, based on the train state transition model, under the condition of meeting the train operation safety constraints, the arrival time of each train at the current station and the departure time (departure time from the current station) of each train at the current station can be determined.

[0099] Step S240: Following the above steps, iteratively calculate the arrival and departure times of all trains at each station on the route to obtain the train timetable.

[0100] In this embodiment, each decision step of the high-speed rail timetable optimization and rescheduling, namely the departure sequence of all trains at each station and the arrival and departure times of all trains, is broken down into two stages for iterative calculation according to the station. The departure sequence of all trains at each station on the route and the precise arrival and departure times are calculated sequentially until a complete train timetable is obtained.

[0101] It is understandable that, under the condition of initial optimization of train timetable, the resulting train timetable is the initial optimized train timetable, while under the condition of rescheduling train timetable, the resulting train timetable is the rescheduling train timetable that dynamically adjusts the initial optimized train timetable in response to interference.

[0102] The technical solution of this application's embodiments models the high-speed rail timetable optimization and rescheduling problem as a two-stage real-time decision-making model. The first stage is a Markov decision process model for high-speed rail scheduling, and a policy network is constructed based on this model. This allows the policy network to learn the action decision-making strategies of the high-speed rail scheduling Markov decision process model. Since the high-speed rail scheduling Markov decision process model is constructed with the objective of minimizing the total train travel time or minimizing the total train delay time, the policy network can make action decisions based on the current station state, resulting in a departure sequence for all trains at the current station that satisfies the objective of minimizing the total train travel time or minimizing the total train delay time. The second stage is a train state transition model. A heuristic algorithm is used to simulate the train operating environment based on the departure sequence of all trains at the current station, determining the arrival and departure times of all trains at the current station that satisfy the train operation safety constraints. This ensures that trains can operate safely and normally while satisfying the objective of minimizing the total train travel time or minimizing the total train delay time. Then, following the above two stages iteratively by station, the arrival and departure times of all trains at each station on the operating route are calculated to obtain the train timetable. In this way, by breaking down each decision step of high-speed rail timetable optimization and rescheduling into two stages, scheduling problems under different operating plans and delay scenarios can be handled in real time. The train departure sequence and precise arrival and departure times at each station are calculated sequentially. Therefore, in real-world large-scale problem scenarios, this method can optimize the initial timetable and provide the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0103] In this embodiment of the application, high-speed train operation is subject to strict safety constraints. To ensure the feasibility of the obtained train timetable, the departure sequence of all trains at the preceding station and the arrival and departure times of all trains at the current station must both meet the operational safety constraints. Specifically, the operational safety constraints include: running time constraints, overtaking constraints, time interval constraints, station capacity constraints, arrival and departure time constraints, and decision variable constraints. These operational safety constraints are explained below.

[0104] (1) First type of constraint: running time constraint, which is used to constrain the train's stopping time at each station and the running time in the section to be no less than the minimum time limit.

[0105] For each station, the shortest dwell time depends on the number of passengers boarding and alighting and the station's infrastructure. For each section, the shortest travel time depends on train information and track conditions. Therefore, the travel time constraint can be expressed as:

[0106]

[0107]

[0108] in, Indicates train At the station Shortest stop time, Indicates train At the station and Shortest running time between (i.e., interval) ).

[0109] (2) The second type of constraint: overtaking constraint, which is used to restrict the train from overtaking in the section between stations.

[0110] In other words, on a two-way high-speed rail line, trains traveling in the same direction have only one main track in each section, so overtaking is not allowed within the section. This constraint can be indirectly represented by the next constraint (the third constraint).

[0111] (3) The third type of constraint: time interval constraint, which is used to constrain the time interval between adjacent trains to be no less than the safe time interval, which includes the station arrival time interval, the station departure time interval, and the section tracking time interval.

[0112] To ensure safe train operation, a safe distance (safe time interval) needs to be maintained between adjacent trains. Assuming all trains travel at a constant speed within the section, the tracking time interval can be considered only when trains enter and leave the station. Therefore, the time interval constraint can be expressed as:

[0113]

[0114]

[0115] in, For a sufficiently large number, This indicates the shortest interval between two adjacent trains arriving at the same station. This represents the shortest interval between two adjacent trains departing from the same station. This represents the shortest time interval between two adjacent trains operating within a section. Therefore, , .

[0116] (4) Fourth constraint: station capacity constraint, which is used to constrain that each track of the station can only be allocated to a single train in the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train is not less than the minimum departure-arrival time interval to meet train safety.

[0117] At each station, a train needs to be assigned a track to stop or pass through, and each track can only be occupied by a single train at a time. If two trains occupy the same track within the station, the time interval between the departure of the preceding train and the arrival of the following train should not be less than the minimum departure-arrival time interval. Therefore, the station capacity constraint can be expressed as:

[0118]

[0119]

[0120] in, This indicates the shortest departure-arrival interval between two adjacent trains occupying the same track.

[0121] (5) Fifth constraint: arrival and departure time constraint, which is used to address the problem of rescheduling train timetables and to constrain trains to run according to the train timetable in the absence of disturbance; or to constrain trains to only be allowed to arrive earlier than the original departure time in the train timetable in the presence of disturbance.

[0122] In other words, before the disturbance occurs, the trains operate strictly according to the train timetable. After the disturbance occurs, the trains are only allowed to arrive earlier than scheduled, and are not allowed to depart earlier than the original time, otherwise many passengers will miss the train.

[0123]

[0124] in, Indicates train At the station The initial delay time, Indicates the time when the disturbance occurred. This indicates the scheduled time for train X to arrive at station X. This represents the planned time for train 𝑖 to leave station 𝑘. The planned time is the timetable obtained from the initial optimization problem.

[0125] (6) The sixth type of constraint: decision variable constraint, which defines the range of values ​​for continuous decision variables and 0-1 integer decision variables. Decision variable constraints can be expressed as:

[0126]

[0127]

[0128] It should be noted that since the rescheduling is based on the planned timetable (train timetable) obtained from the initial optimization, the initial optimization problem is considered first. High-speed train operation has strict safety constraints. To ensure the feasibility of the obtained train timetable, the operational safety constraints satisfied by the initial optimization include the first, second, third, fourth, and sixth constraints. For the rescheduling problem, due to the occurrence of sudden interference, the first, second, third, fourth, fifth, and sixth constraints must be satisfied.

[0129] The following sections provide a detailed explanation of the high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning in this application, specifically: Section 1.1, Section 1.2, Section 1.3, Section 1.4, Section 1.5, Section 1.6, Section 1.7, Section 1.8, Section 1.9, Section 1.0, Section 1.1.1, Section 1.2, Section 1.3, Section 1.4 ...9, Section 1.0, Section 1.1.2, Section 1.3, Section 1.4, Section 1.5, Section 1.6, Section 1.7, Section 1.8, Section 1.9, Section 1.0,

[0130] 1.1 Method for constructing a Markov decision process model for high-speed rail scheduling:

[0131] In conjunction with the above embodiments, in one embodiment, this application also provides a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning. In this method, the Markov decision process model for high-speed train scheduling is constructed as follows, including steps A1 to A4:

[0132] Step A1: The station state, composed of train features and station features, is used as the state of the Markov decision process model for high-speed rail scheduling.

[0133] In this embodiment of the application, station features include station location and station capacity, and station features can be represented as follows: ,in, Indicates the location of the station. Indicates station capacity.

[0134] Station characteristics are divided into train dynamic characteristics and train static characteristics. The train static characteristics include train... The following parameters: interval Shortest travel time, stations Shortest stop time, station Buffer time and station The planned departure time. Specifically, for the initial optimization and rescheduling problems, the static characteristics of the train can be expressed as follows:

[0135]

[0136] in, To represent the static characteristics of the train, for the initial optimization problem, the static characteristics of the train include the shortest stopping time at stations. (i.e., train) At the station (shortest stop time) and shortest running time between stations (i.e., train) At the station and The shortest running time between, i.e., the interval Regarding the rescheduling problem, the static characteristics of trains include the station's buffer time. (i.e., train) At the station (buffer time) and scheduled departure time (i.e., train) Leaving the station (Planned time).

[0137] In response to the problem of rescheduling, trains At the station buffer time The calculation formula is expressed as:

[0138]

[0139] Where M represents the total number of stations, Indicates train Arrival at the station The planned time, Indicates train Leaving the station The planned time, Indicates train At the station and The shortest running time between.

[0140] The dynamic characteristics of a train change with each decision-making step, and are defined as the ready time. This indicates that the train has completed its stop operation and is ready to leave the station. The readiness time is expressed as:

[0141]

[0142] in, Indicates train At the station The initial delay time.

[0143] Therefore, the current decision-making steps The status of the current station (station status) can be represented as: ,in Representing all trains, This represents the distance from the current station to the destination station. Only the characteristics of the current station and subsequent stations are considered here, as the parameter effects of previous stations are already reflected in the train's variable characteristics.

[0144] Step A2: Take the departure order of all trains at the current station as the action of the Markov decision process model for high-speed rail scheduling.

[0145] In this embodiment of the application, based on the current station status The Markov decision process model for high-speed rail scheduling gives the departure sequence (i.e., actions) of all trains at the current station. Therefore, the action It can be expressed as: ,in It refers to the station The first The train number.

[0146] Step A3: The process of transitioning from the current station state to the next station state based on the departure sequence of all trains at the station serves as the state transition of the Markov decision process model for the high-speed rail scheduling.

[0147] In this embodiment, state transition refers to the simulation of the train's operating environment, which can be described as based on a given action. From the current station status Switch to the next station status The process.

[0148] Step A4: With the initial optimized train timetable, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station, with the goal of minimizing the total train travel time; or, with the train timetable rescheduled, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station, with the goal of minimizing the total train delay time.

[0149] In this embodiment of the application, the reward values ​​for different optimization problems (i.e., initial optimization of train timetable and rescheduling of train timetable) are calculated in different ways. The negative objective function corresponding to the objective function of the optimization problem can be used as the reward value. The reward values ​​for the two optimization problems are explained below.

[0150] (1) Under the condition of initial optimization of train timetable, with the goal of minimizing the total train travel time, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station, including: under the condition of satisfying the traffic safety constraints, with the goal of minimizing the total train travel time, a first objective function is constructed based on the travel time of the train between two stations, the travel time of the train between two stations includes: the minimum stopping time at the station, the minimum running time in the section, and the buffer time; the negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model of high-speed rail scheduling.

[0151] For the initial optimization of train timetables (initial optimization problem), the goal is to minimize the total train travel time while meeting operational safety constraints. That is, the first objective function:

[0152]

[0153] in, and These represent the total number of stations and the total number of trains, respectively. The arrival time of trains at the originating station is defined as... .train The travel time between two stations consists of three parts: the minimum stopping time at each station, the minimum running time within the section, and the buffer time. Specifically, the train... The travel time between two stations is expressed as:

[0154]

[0155] Among them, train At the station Shortest stopping time and train At the station and Shortest running time between All are given constants; Indicates train At the station The buffer time for the corresponding interval is the additional travel time allocated beyond the minimum travel time required to meet safety constraints. Therefore, minimizing the total travel time is equivalent to minimizing the total buffer time. The first objective function can be reformulated as:

[0156]

[0157] Therefore, under the initial optimization of the train timetable, the train at the station In order The reward value obtained from the Markov decision process model of high-speed rail scheduling is issued. It can be represented as:

[0158]

[0159] (2) In the case of rescheduling train timetables, with the goal of minimizing the total train delay time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station. This includes: under the constraint of satisfying train safety, with the goal of minimizing the total train delay time, constructing a second objective function based on the planned arrival time of trains at the station and the planned departure time of trains from the station, wherein the planned arrival time of trains at the station and the planned departure time of trains from the station are the times obtained from the initial optimization of the train timetable; and using the negative objective function corresponding to the second objective function as the reward value of the Markov decision process model for high-speed rail scheduling.

[0160] Regarding the rescheduling of train timetables (the rescheduling problem), it is necessary to restore the railway system's operational order as soon as possible and minimize the total train delay time. That is, the second objective function can be expressed as:

[0161]

[0162] in, Indicates train Arrival at the station The planned time, Indicates train Leaving the station The planned time is the timetable obtained from the initial optimization problem.

[0163] Therefore, in the case of rescheduling train timetables, the trains at the station In order The reward value obtained from the Markov decision process model of high-speed rail scheduling is issued. It can be represented as:

[0164]

[0165] Using the technical solution of this application embodiment, the constructed Markov decision process model for high-speed rail scheduling is applicable to the current station. The policy network can adjust the current station status based on the current station status. Generate the departure sequence of all trains Then, the train operating environment performs actions. Later received a reward And move to the next decision station. Thus, this process is repeated iteratively until a complete train timetable is generated.

[0166] 1.2 Methods for constructing policy networks:

[0167] In conjunction with the above embodiments, in one implementation, this application also provides a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning. In this method, step S210, "constructing a policy network based on a Markov decision process model for high-speed train scheduling," specifically includes:

[0168] Based on the Markov decision process model of high-speed rail scheduling, an action decision strategy for learning by the policy network is designed; with the goal of learning the action decision strategy, the policy network is trained by a deep reinforcement learning algorithm, and the trained policy network is obtained after the training termination condition is met.

[0169] In this embodiment, constructing the policy network is a process based on a deep reinforcement learning algorithm, involving continuous updating of the policy network parameters. By updating the policy network parameters, the policy network learns the action decision-making strategies of the Markov decision process model for high-speed rail scheduling. The trained policy network can then be directly applied as a scheduling strategy.

[0170] Policy networks can be based on their own policy network parameters Give the current station status. Next, select an action. The probability of (i.e., the departure order of all trains at the current station) :

[0171]

[0172] Specifically, with the goal of learning the action decision-making strategy, the policy network is trained using a deep reinforcement learning algorithm, including steps D1 to D4:

[0173] Step D1: Based on the current policy network parameters and the target station status, the policy network generates the departure sequence of all trains at the target station.

[0174] Step D2: Calculate the reward value and the state of the next target station based on the departure sequence of all trains at the target station using the train state transition model.

[0175] Step D3: Take the target station status, the departure sequence of all trains at the target station, and the reward value as a scheduling trajectory, and repeat the above steps until the destination station to obtain the complete scheduling trajectory.

[0176] Step D4: Calculate the gradient based on the complete scheduling trajectory and update the policy network parameters of the policy network.

[0177] In this embodiment, the above-described strategy is used for distribution. The policy network, based on its current parameters and the target station's state, samples the departure sequence of all trains at the target station. Then, using a train state transition model, it calculates the reward value based on this departure sequence. and the status of the next target station Finally, by repeating the above steps until the final station, the complete dispatch trajectory can be obtained:

[0178]

[0179] in, The dispatch trajectory for the first station includes the station status of the first station, the departure sequence of all trains, and the reward value. The scheduling trajectory for the k-th station consists of the station status of the k-th station, the departure order of all trains, and the reward value. The scheduling trajectory for the (M-1)th station includes the station status of the (M-1)th station, the departure sequence of all trains, and the reward value.

[0180] The specific process of calculating the gradient based on the complete scheduling trajectory and updating the policy network parameters of the policy network is as follows:

[0181] Each decision station in the complete scheduling trajectory Define long-term discount returns :

[0182]

[0183] in, It is a discount factor, and its value range is ( .

[0184] Considering the efficiency and stability of the training process, the REINFORCE algorithm based on baseline greed in reinforcement learning is used to train the policy network, defining gradients with baseline. for:

[0185]

[0186] Among them, baseline Defined as obtaining a deterministic long-term reward by greedily selecting the action with the highest probability using a baseline network. In this way, the state can be estimated. The difficulty of the corresponding high-speed rail scheduling example is reduced, the gradient variance is decreased, and the convergence speed is improved. (Baseline network) With policy network The structure is identical, and the parameters are fixed. At the end of each training epoch, the current training policy (through greedy decoding) is compared with the baseline policy using a paired t-test. If the improvement is significant, the parameters of the baseline network are replaced. Thus, after the training termination condition is met (e.g., the preset number of training iterations is met), the trained policy network is obtained.

[0187] For example, the training process of the policy network can be implemented using the baseline-based greedy REINFORCE algorithm in Table 1.

[0188] Table 1 shows the REINFORCE algorithm derived based on baseline greed:

[0189]

[0190] By adopting the technical solution of this application embodiment, after the decision model is trained offline, the trained policy network can be used for online high-speed rail scheduling. That is, the policy network can select the train with the highest probability to depart at each decoding decision step, thus meeting the real-time requirements.

[0191] 1.3 A method for generating train departure sequence based on policy networks:

[0192] In conjunction with the above embodiments, in one embodiment, this application also provides a method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning. In this method, step S220, "inputting the current station state into the policy network to obtain the action decision as the departure order of all trains at the current station," specifically includes steps C1 to C3:

[0193] Step C1: Input the current station status into the input embedding module to obtain the station embedding containing global information of the current station and all subsequent stations, and the node embedding containing all train information.

[0194] The current station status includes station features and train features, and the train features include train dynamic features and train static features; the input embedding module includes a first long short-term memory module, a second long short-term memory module, and a linear projection module.

[0195] Specifically, the current station status is input into the input embedding module to obtain a station embedding containing global information of the current station and all subsequent stations, and a node embedding containing all train information. This includes: inputting the station features into the first long short-term memory module to obtain a station embedding containing global information of the current station and all subsequent stations; inputting the train static features into the second long short-term memory module to obtain a static node embedding containing all train static information; inputting the train dynamic features into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding; and obtaining a node embedding containing all train information based on the static node embedding and the dynamic node embedding.

[0196] In this embodiment, considering that the length of each station state is different, a Long Short-Term Memory (LSTM) network (i.e., the first LSTM module) is selected. Second Long Short-Term Memory Module The system processes train and station features. Specifically, station embedding is obtained from the terminal station through the first long short-term memory module. to the current station in sequence The system transmits global station information. For node embedding, both static and dynamic train features need to be considered. The second long short-term memory module processes the static train features of each train node, i.e., it obtains information from the terminal station... to the current station in sequence The system transmits global static information for each train; simultaneously, it uses a linear projection module. The train dynamic features are linearly projected to the same embedding dimension as the static node embedding dimension.

[0197] For example, station embedding and node embedding It can be represented as:

[0198]

[0199]

[0200] Wherein, LP represents a parameter that can be learned. and The linear projection layer (linear projection module). , This represents the static characteristics of train i from its current station to its final destination. This represents the dynamic characteristics of train i at station k. This indicates the station characteristics from the current station to the final station.

[0201] Node embeddings obtained through the input embedding module Further processing will be performed on the encoder, station embedding It will be used directly as the input to the decoder.

[0202] Step C2: The encoder extracts the dependencies between trains from the node embeddings using an attention mechanism to obtain an encoded train embedding containing global information about all trains.

[0203] In this embodiment, the dependencies between trains are extracted from the node embeddings using an attention mechanism. Considering that existing deep reinforcement learning methods for high-speed rail scheduling typically ignore the relationships between different trains, resulting in a lack of foresight in action decisions and the attainment of only locally optimal strategies, this embodiment, referencing the Transformer architecture, utilizes a self-attention mechanism to complete information exchange between train nodes. By aggregating information from all nodes, each train node gains a holistic perspective on the scheduling problem, rather than focusing solely on its local information.

[0204] Specifically, the encoder includes multiple attention coding layers, each of which includes a multi-head attention layer and a fully connected multi-layer perceptron (MLP) feedforward layer; in addition, residual connections and batch normalization (BN) layers are added to each multi-head attention (MHA) layer and the fully connected multi-layer perceptron (MLP) feedforward layer.

[0205] Specifically, the encoder extracts the dependencies between trains by using an attention mechanism to obtain an encoded train embedding containing global information of all trains. This includes: inputting the node embedding into the multiple attention encoding layers for multi-layer attention encoding to obtain an encoded train embedding containing global information of all trains.

[0206] For example, a multi-head attention layer and a fully connected multilayer perceptual feedforward layer can be represented as:

[0207]

[0208]

[0209] The attention encoding layer consists of L layers. Indicates the batch normalization layer. This indicates that multi-head attention computation is performed on the node embedding of train i. Indicates multi-head attention layer The train i at the station Node embedding, Indicates multi-head attention layer Exported trains At the station Node embedding, Represents a multi-layer sensing feedforward layer This indicates that it is composed of attention encoding layers. The given train At the station The node embedding, the final train node embedding go through The result is obtained after the attention encoding layer and will be used as input in the decoder.

[0210] Step C3: The decoder decodes the train embedding and the station embedding to obtain the departure sequence of all trains at the current station.

[0211] In this embodiment, the decoder performs decoding based on the encoded train embedding and station embedding in a step-by-step decoding process. The decoder selects the next departing train based on the train embedding and station embedding output by the first two modules, as well as the currently determined partial departure sequence, thereby gradually obtaining the departure order of all trains.

[0212] Specifically, the decoder decodes the train embedding and the station embedding to obtain the departure order of all trains at the current station, including:

[0213] Step C3-1: At each decision time step, determine the context embedding based on the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, where the sequence embedding represents the partially determined departure sequence and the aggregate embedding of the coded train embedding represents the mean of the coded train embedding.

[0214] Step C3-2: Based on the context embedding and the coded train embedding, determine the next departing train from the selectable trains that meet the constraints according to the strategy probability distribution; the strategy probability distribution represents the probability distribution of each train being selected as the next departing train; the selectable trains that meet the constraints include: trains that have not been selected, and the N trains with the highest arrival times at the current station.

[0215] Step C3-3: Repeat the above steps for multiple decision time steps until all trains leave the current station, and obtain the departure sequence of all trains at the current station.

[0216] In this embodiment, at each decision time step, the encoded train embedding and context embedding are used as inputs. Then, based on the partially determined departure order (including in the context embedding), the next departing train is sampled from the optional trains that meet the constraints according to the policy probability distribution. The complete departure order can be obtained until all trains leave the current station.

[0217] To improve decoding efficiency, a context embedding is used for decoding. It can be represented as:

[0218]

[0219] in, This represents the aggregate embedding of the coded train embedding, i.e., the mean of the coded train embedding; Indicates station embedding; This represents a sequence embedding, where the departure sequence is partially determined at the current decision time step t. .

[0220] Sequence embedding It can be based on Long Short-Term Memory networks (i.e., the third long short-term memory module) yields the sequence embedding. It can be represented as:

[0221]

[0222] in, The sequence embedding represents the decision time step t-1. This indicates the coded train embedding corresponding to the departure train that has been determined in decision time step t-1.

[0223] After obtaining the context embedding, step C3-2 is executed, which determines the next departing train from the selectable trains that meet the constraints based on the policy probability distribution, based on the context embedding and the coded train embedding.

[0224] In this embodiment, a train masking rule is designed to address the operational safety constraints of high-speed rail scheduling, ensuring the feasibility of the departure sequence. Specifically, according to the train masking rule, only trains meeting the constraints can be selected as the next departing train. The train masking rule includes: 1) already departed trains are masked; 2) only trains arriving before the arrival time are allowed to depart. 1) Select the next departing train (excluding trains that have already left the station), and mask other trains; 2) For the train timetable rescheduling problem, before the interference occurs, trains run according to the original timetable, the departure order is unique, and other trains are masked. Based on the above train masking rules, the selectable trains that meet the constraints include: trains that have not been selected, and the N trains with the highest arrival times at the current station.

[0225] Specifically, based on the context embedding and the coded train embedding, determining the next departing train from the selectable trains that meet the constraints according to the policy probability distribution includes: inputting the context embedding and the coded train embedding into a first decoding layer for attention interaction to obtain a new context embedding; wherein, the first decoding layer determines the key vector and value vector of the attention interaction based on the coded train embedding, and determines the query vector of the attention interaction based on the context embedding; inputting the new context embedding and the coded train embedding into a second decoding layer to obtain attention weights, the attention weights representing the weight of each train being selected as the next departing train; obtaining the policy probability distribution based on the attention weights, and determining the next departing train based on the policy probability distribution.

[0226] In this embodiment, the first decoding layer is a one-to-many multi-head attention layer, and the second decoding layer is a one-to-many single-head attention layer. For example, the new context embedding... It can be expressed as:

[0227]

[0228] in, This indicates the first decoding layer, which is a one-to-many multi-head attention layer.

[0229] Attention weight :

[0230]

[0231] in, This indicates that the context node is at the station. The query vector, Indicates train node At the station The key vector, constant C, and function tanh will clip the result to... Within the range.

[0232] Based on the attention weights, obtaining the policy probability distribution means treating the attention weights as log-likelihoods and using the SoftMax function to calculate the policy probability distribution. It can be expressed as:

[0233]

[0234] Thus, the agent, according to the policy probability distribution Randomly select the next departing train and repeat steps C3-1 and C3-2 until all trains have departed, thus obtaining the departure order of all trains at the current station.

[0235] For example, the framework of a policy network is as follows: Figure 3 As shown, the policy network includes an input embedding module, an encoder, and a decoder. The input embedding module includes a first long short-term memory module (LSM). ), Second Long Short-Term Memory Module ( ) and linear projection module ( For a given set of station and train features, the input embedding module can generate a station embedding containing global information about the current station and all subsequent stations, and a node embedding containing information about all trains. The encoder includes multiple attention coding layers ( The node embeddings are input into the multiple attention encoding layers for multi-layer attention encoding, resulting in an encoded train embedding containing global information about all trains. The decoder includes a first decoding layer (One to many). ), second decoding layer (One to many) The decoder, at each decision time step, determines the context embedding based on the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding. The context embedding and the coded train embedding are then input into the first decoding layer for attention interaction to obtain a new context embedding. The new context embedding and the coded train embedding are then input into the second decoding layer to obtain attention weights. These attention weights are then treated as log-likelihoods, and the SoftMax function is used to calculate the policy probability distribution. The next departing train is determined based on this policy probability distribution. By repeating this process multiple decision time steps, the departure order of all trains at the current station is obtained.

[0236] 1.4 Determining train timetables based on train state transition model:

[0237] In conjunction with the above embodiments, in one embodiment, this application also provides a high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning. In this method, the train state transition model can use heuristic algorithms to simulate the train operating environment, providing the accurate arrival and departure times of each train at the current station, and naturally determining track occupancy. Specifically, step S220 above, "based on the train state transition model, simulating the train operating environment according to the departure sequence of all trains at the current station, and determining the arrival and departure times of all trains at the current station that satisfy traffic safety constraints," specifically includes steps B1 to B5:

[0238] Step B1: Determine the initial arrival times of all trains at the current station based on the aforementioned train safety constraints and the departure times of all trains at the previous station.

[0239] The initial arrival times of all trains at the current station can be calculated iteratively based on the departure times of all trains at the previous station. For the current station k, the initial arrival times of all trains at the current station can be expressed as:

[0240]

[0241] in, Indicates train At the station and The shortest running time between the intervals, Indicates train Leaving the station Time (i.e., train) (departure time at the previous station) Indicates train At the station The train that left the previous train Arrival at the station The time.

[0242] Step B2: For each target train, determine the track occupancy status of the current station at the target time. The target time is the initial arrival time of the target train minus the given shortest departure-arrival interval. The target train is any train selected sequentially from the departure order of all trains.

[0243] The target time can be expressed as: , For train Arrival Station The initial time, This is the shortest departure-arrival interval between two adjacent trains occupying the same track. It involves determining the track occupancy at the current station at the target time, i.e., calculating the track occupancy at the current station at the target time. Number of occupied tracks .

[0244] Step B3: If there are no empty tracks in the track occupancy status, update the arrival time of the target train at the current station according to the track occupancy status, and update the arrival times of all subsequent trains that departed later than the target train at the previous station at the current station.

[0245] Specifically, "no empty tracks" means the current station is at the target time. Number of occupied tracks greater than or equal to the station Total number of orbits At this point, the arrival time of the target train at the current station needs to be updated based on the track occupancy status. This can be done using Formula 2:

[0246]

[0247] Among them, train For the first Listed in target time Later from the station The train ID of the departing train, therefore at the station Assigned to train Tracks and trains same.

[0248] Furthermore, to avoid new conflicts, the arrival times of subsequent trains need to be modified one by one, and the arrival times of subsequent trains need to be able to be updated according to Formula 3:

[0249]

[0250] Among them, train Indicates at the station All those who arrived later than the train The train is departing. Indicates train At the station Departure order The position in the middle.

[0251] Step B4: If there are available tracks in the track occupancy status, randomly assign an available track to the target train, and the arrival time of the target train at the current station is the initial arrival time.

[0252] Step B5: Determine the readiness time of the target train based on the train safety constraints and the arrival time at the current station, and determine the departure time of the target train at the current station based on the readiness time. The readiness time represents the time when the train has completed the stopping operation and is ready to leave the station.

[0253] After arriving at the current station, the target train will wait for departure at the current station. (Ready time...) Using Formula 1 above, if the train safety constraints are met, the target train will depart immediately, thus obtaining the actual departure time:

[0254]

[0255] Among them, train It is a train At the station Regarding the rescheduling issue, if the preceding train departs... ,but .

[0256] At this point, based on the departure time of the target train at the current station, the initial arrival time of all trains at the next station (i.e., the earliest arrival time of the trains at the next station) can be determined. The initial arrival times of all trains at the next station can be expressed as:

[0257]

[0258] Regarding the rescheduling problem, if ,but .

[0259] Thus, based on the departure sequence of all trains at the current station, the train state transition model can be used to simulate the train operation environment using heuristic algorithms. That is, by following steps B1 to B5 above, the accurate arrival and departure times of each train can be determined in sequence, and the track occupancy status can be determined naturally.

[0260] For example, the specific implementation algorithms for steps B1 to B5 above can be implemented using the train state transition heuristic algorithm in Table 2.

[0261] Table 2. Heuristic Algorithm for Train State Transition:

[0262]

[0263] For example, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the overall framework of a general high-speed rail scheduling model provided in this application embodiment. Specifically, the high-speed rail train timetable optimization and rescheduling problem is modeled as a two-stage real-time decision-making model. The first stage is a Markov decision process model for high-speed rail scheduling, and the second stage is a train state transition model. In the first stage, the Markov decision process model for high-speed rail scheduling, a policy network is used to learn the action decision-making strategy of the Markov decision process model for high-speed rail scheduling. The policy network can generate the departure sequence of all trains based on the current state, and the train operating environment executes actions. Later received a reward And move to the next decision station. In the second-stage train state transition model, the train operation environment can be simulated based on the departure sequence of all trains at the current station, determining the arrival and departure times of all trains at the current station that satisfy operational safety constraints. This process is iterated until a complete train timetable is generated. This method decomposes each decision step of high-speed rail timetable optimization and rescheduling into two stages, enabling real-time handling of scheduling problems under different operating plans and delay scenarios. It sequentially calculates the train departure sequence and precise arrival and departure times at each station. Therefore, in real-world large-scale problem scenarios, this method can optimize the initial timetable and provides the flexibility needed to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0264] Furthermore, to better illustrate the performance of the high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning in this application embodiment, the Beijing-Shanghai High-Speed ​​Railway Passenger Dedicated Line with 23 stations is used as an example to verify the effectiveness of this method (high-speed rail general scheduling model) in solving the initial optimization and rescheduling problems. This line allows trains of various speed levels to run together, with maximum speeds of 350 km / h, 300 km / h, and 250 km / h. Basic line data is shown in Table 3.

[0265] Table 3. Basic data of the Beijing-Shanghai High-Speed ​​Railway:

[0266]

[0267] To address the initial optimization problem, the maximum train operating speed is randomly selected from three levels, which determines the shortest travel time between sections. Taking into account the train's stop time, each train will randomly select a station. Whether to stop; if so, the duration will be randomly set. Regarding rescheduling issues, latency scenarios primarily focus on minimizing interference and delay time. Values ​​were randomly selected within a 5-60 minute timeframe, assuming the time of interference occurred. To delay the earliest arrival time of the train. In addition, other parameters are set as follows according to the actual situation: the shortest arrival or departure interval is 4 minutes, the shortest tracking interval is 4 minutes, and the shortest departure-arrival interval is 2 minutes. Table 4 shows the other required hyperparameters.

[0268] Table 4. Hyperparameters required for the general high-speed rail scheduling model:

[0269]

[0270] The high-speed rail general scheduling model proposed in this application is applicable to both initial optimization and rescheduling problems. Therefore, the initial optimization scheduling model USM is trained first. O And its performance is evaluated, and then USM is used. O The model outputs a planned timetable, which is then used as input to reschedule instances to further train the rescheduling model USM. R .

[0271] (1) Results of the initial optimization experiment of the train timetable:

[0272] In the experiment, the size of the problem instance is defined as... ,in For the number of stations, Let be the number of trains. The training case requires scheduling 20 trains across 13 stations (i.e., Beijing South-Bengbu South dispatching section and Xuzhou East-Shanghai Hongqiao dispatching section) to complete the planned timetable, denoted as . The comparison method uses a manually designed First Come First Served (FCFS) strategy, which determines the departure order based on arrival time. Additionally, the commercial solver Gurobi provides a benchmark for the exact algorithm; however, due to its high computational cost, the calculation terminates when the optimal difference is less than 5% or the computation time reaches 300 seconds.

[0273] To verify the generalization and scalability of the method, the scheduling model was first trained on a small-scale problem. Then, the trained policy network was transferred to solve large-scale initial optimization problems, including 13 stations and 50 trains, 13 stations and 80 trains, and 23 stations (i.e., the complete Beijing-Shanghai High-Speed ​​Railway line) and 50 trains. Greedy decoding was used to evaluate the method's performance, and the results are shown in Table 5, including response time and scheduling scheme quality on 100 test instances.

[0274] Table 5. Initial optimization results of train timetables in various case studies:

[0275]

[0276] For the target value, USM O The model consistently outperforms the FCFS scheduling rule across different problem sizes. The Gurobi solver only achieves better results on small-scale problems; however, as the scale increases, it fails to generate feasible scheduling schemes for large-scale instances within the time limit. Regarding runtime, both FCFS and the method proposed in this application can provide planned timetables for problem instances of all sizes within 0.1 seconds. Although FCFS consumes the least time, the significant difference from the optimal value is unacceptable. Furthermore, Gurobi, due to its exact solution characteristic, exhibits an exponential relationship between solution time and the number of trains and stations, resulting in poor performance on large-scale problems.

[0277] (2) Results of the train timetable rescheduling experiment

[0278] With the help of trained USM O The model can obtain a large number of scheduled timetables as training data for the train rescheduling problem, and then... The USMR rescheduling model was trained on a case-scale dataset. In addition to FCFS and the Gurobi solver, the comparison method also introduced the First Schedule First Served (FSFS) strategy. This method also uses a manually designed scheduling rule, assigning the departure order according to the planned timetable. Table 6 shows the trained USMR on 100 instances. R The effect of the model.

[0279] Table 6. Results of train timetable rescheduling in various case studies:

[0280]

[0281] The results show that the policy network trained on small-scale instances can effectively generalize to large-scale real-world problems. The method proposed in this application significantly outperforms the two scheduling rules in terms of optimality gap, especially for large-scale instances. USM R The scheduling scheme generated by the model consistently deviated from the target value of the Gurobi baseline by less than 8%, and even in some cases... A better timetable was achieved for the problem size. If the solution time of Gurobi is limited to within 20 seconds to meet the requirements of real-time response, then USM... R The model's effectiveness is more evident. Compared to scheduling rules, the strategy learned by this model can improve performance metrics by 60% to 90%. Furthermore, similar to the initial optimization experiment results, the learning-based model also shows a significant advantage in runtime compared to exact solution methods, meeting the real-time requirements of high-speed rail rescheduling.

[0282] This application also provides a high-speed rail train timetable optimization and rescheduling device based on deep reinforcement learning, referring to... Figure 5 As shown, Figure 5 This is a schematic diagram of a high-speed rail train timetable optimization and rescheduling device based on deep reinforcement learning, provided in an embodiment of this application. The device includes:

[0283] The construction module 510 is used to construct a policy network based on the Markov decision process model of high-speed rail scheduling. The policy network learns the action decision policy of the Markov decision process model of high-speed rail scheduling. The policy network includes an input embedding module, an encoder and a decoder. The action decision policy represents the scheduling policy of high-speed rail trains. The Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total train travel time or minimizing the total train delay time.

[0284] The input module 520 is used to input the current station status into the strategy network to obtain the action decision as the departure sequence of all trains at the current station. The current station status includes station features that characterize station capacity and station location, as well as train features that characterize train operation information.

[0285] The determination module 530 is used to determine the arrival and departure times of all trains at the current station that meet the traffic safety constraints, based on the train state transition model and the train operation environment simulation according to the departure sequence of all trains at the current station.

[0286] The iteration module 540 is used to iteratively calculate the arrival and departure times of all trains at each station on the route according to the above steps, and obtain the train timetable.

[0287] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps of the high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning described in the embodiments of this application.

[0288] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the high-speed rail train timetable optimization and rescheduling method based on deep reinforcement learning described in this application.

[0289] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of high-speed rail timetable optimization and rescheduling based on deep reinforcement learning as described in this application.

[0290] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0291] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0292] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0293] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0294] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0295] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0296] The above provides a detailed description of a high-speed rail timetable optimization and rescheduling method based on deep reinforcement learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for optimizing and rescheduling high-speed train timetables based on deep reinforcement learning, characterized in that, include: Based on the Markov decision process model for high-speed rail scheduling, a policy network is constructed. The policy network learns the action decision policies of the Markov decision process model for high-speed rail scheduling. The policy network includes an input embedding module, an encoder, and a decoder. The action decision policies represent the scheduling strategies of high-speed trains. The Markov decision process model for high-speed rail scheduling is constructed with the goal of minimizing the total train travel time or minimizing the total train delay time. The current station status is input into the policy network to obtain the action decision of the departure order of all trains at the current station. The current station status includes station features that characterize station capacity and station location, as well as train features that characterize train operation information. Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival and departure times of all trains at the current station that meet the train operation safety constraints are determined. Following the steps above, the arrival and departure times of all trains at each station along the route are calculated iteratively to obtain the train timetable.

2. The method according to claim 1, characterized in that, Based on the Markov decision process model for high-speed rail scheduling, a policy network is constructed, including: Based on the Markov decision process model of high-speed rail scheduling, an action decision strategy for policy network learning is designed. With the goal of learning the action decision-making strategy, the policy network is trained using a deep reinforcement learning algorithm. After the training termination condition is met, the trained policy network is obtained.

3. The method according to claim 2, characterized in that, With the goal of learning the action decision-making strategy, the policy network is trained using a deep reinforcement learning algorithm, including: Based on the current policy network parameters and the target station status, the policy network generates the departure sequence of all trains at the target station. The reward value and the state of the next target station are calculated based on the departure sequence of all trains at the target station using the train state transition model. The target station status, the departure sequence of all trains at the target station, and the reward value are used as a scheduling trajectory. By repeating the above steps until the destination station, a complete scheduling trajectory is obtained. The gradient is calculated based on the complete scheduling trajectory, and the policy network parameters of the policy network are updated.

4. The method according to claim 1, characterized in that, The current station status is input into the policy network to obtain the action decision of the departure order of all trains at the current station, including: The current station status is input into the input embedding module to obtain a station embedding containing global information of the current station and all subsequent stations, and a node embedding containing all train information. The encoder extracts the dependencies between trains by using an attention mechanism on the node embeddings to obtain an encoded train embedding containing global information of all trains. The decoder decodes the train embedding and the station embedding to obtain the departure order of all trains at the current station.

5. The method according to claim 4, characterized in that, The current station status includes station features and train features, and the train features include train dynamic features and train static features; The input embedding module includes a first long short-term memory module, a second long short-term memory module, and a linear projection module; The current station status is input into the input embedding module to obtain a station embedding containing global information about the current station and all subsequent stations, and a node embedding containing information about all trains, including: The station features are input into the first long short-term memory module to obtain a station embedding that includes global information of the current station and all subsequent stations. The train static features are input into the second long short-term memory module to obtain a static node embedding containing all train static information. The train dynamic features are input into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding. Based on the static node embedding and the dynamic node embedding, a node embedding containing all train information is obtained.

6. The method according to claim 4, characterized in that, The encoder includes multiple attention coding layers, each of which includes a multi-head attention layer and a fully connected multi-layer perceptual feedforward layer. The encoder extracts dependencies between trains from the node embeddings using an attention mechanism, resulting in an encoded train embedding containing global information about all trains, including: The node embedding is input into the multiple attention coding layers for multi-layer attention coding to obtain a coded train embedding containing global information of all trains.

7. The method according to claim 4, characterized in that, The decoder decodes the train embedding and the station embedding based on the encoded train embedding to obtain the departure order of all trains at the current station, including: At each decision time step, a context embedding is determined based on the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, wherein the sequence embedding represents a partially determined departure sequence, and the aggregate embedding of the coded train embedding represents the mean of the coded train embedding. Based on the context embedding and the coded train embedding, the next departing train is determined from the selectable trains that meet the constraints according to the strategy probability distribution; the strategy probability distribution represents the probability distribution of each train being selected as the next departing train; the selectable trains that meet the constraints include: trains that have not been selected, and the N trains whose arrival times at the current station are ranked first. Repeat the above steps multiple times until all trains leave the current station, thus obtaining the departure order of all trains at the current station.

8. The method according to claim 7, characterized in that, Based on the context embedding and the coded train embedding, the next departing train is determined from the selectable trains that meet the constraints according to the policy probability distribution, including: The context embedding and the coded train embedding are input into the first decoding layer for attention interaction to obtain a new context embedding; wherein, the first decoding layer determines the key vector and value vector of attention interaction based on the coded train embedding, and determines the query vector of attention interaction based on the context embedding; The new context embedding and the coded train embedding are input into the second decoding layer to obtain attention weights, which represent the weight of each train being selected as the next departing train. Based on the attention weights, a strategy probability distribution is obtained, and the next departing train is determined based on the strategy probability distribution.

9. The method according to claim 1, characterized in that, Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station to determine the arrival and departure times of all trains at the current station that satisfy the train operation safety constraints, including: Based on the aforementioned traffic safety constraints and the departure times of all trains at the previous station, determine the initial arrival times of all trains at the current station. For each target train, determine the track occupancy status of the current station at the target time. The target time is the initial arrival time of the target train minus the given shortest departure-arrival interval. The target train is any train selected sequentially from the departure order of all trains. If there are no available tracks, the arrival time of the target train at the current station is updated based on the track occupancy status, and the arrival times of all subsequent trains that departed later than the target train at the previous station are also updated at the current station. If there are available tracks when the track occupancy status is such that an available track is randomly assigned to the target train, and the arrival time of the target train at the current station is the initial arrival time; Based on the train safety constraints and the arrival time at the current station, the readiness time of the target train is determined, and the departure time of the target train at the current station is determined based on the readiness time. The readiness time represents the time when the train has completed the stopping operation and is ready to leave the station.

10. The method according to claim 1, characterized in that, The Markov decision process model for high-speed rail scheduling is constructed in the following manner: The station state, composed of train characteristics and station characteristics, is used as the state of the Markov decision process model for high-speed rail scheduling. The departure order of all trains at the current station is taken as the action of the Markov decision process model for high-speed rail scheduling. The process of transitioning from the current station state to the next station state based on the departure sequence of all trains at the station is used as the state transition of the Markov decision process model for high-speed rail scheduling. In the case of initial optimization of train timetables, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station with the objective of minimizing the total train travel time; or, in the case of rescheduling train timetables, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station with the objective of minimizing the total train delay time.

11. The method according to claim 10, characterized in that, Given an initially optimized train timetable, and with the objective of minimizing total train travel time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station, including: Under the constraint of driving safety, with the goal of minimizing the total train travel time, a first objective function is constructed based on the train travel time between two stations. The train travel time between two stations includes: the minimum stopping time at the station, the minimum running time in the section, and the buffer time. The negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model for high-speed rail scheduling.

12. The method according to claim 10, characterized in that, In the case of rescheduling train timetables, with the objective of minimizing total train delay time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated based on the departure order of all trains at the station, including: Under the constraint of train operation safety, with the goal of minimizing the total train delay time, a second objective function is constructed based on the planned arrival time of the train at the station and the planned departure time of the train at the station, which are the times obtained from the initial optimized train timetable. The negative objective function corresponding to the second objective function is used as the reward value of the Markov decision process model for high-speed rail scheduling.

13. The method according to any one of claims 1-12, characterized in that, The driving safety constraints include: Operating time constraints, which are used to ensure that the train's stopping time at each station and the operating time in the section are not less than the minimum time limit; Overtaking constraint, which restricts a train from overtaking within the section between stations; Time interval constraint, which is used to ensure that the time interval between adjacent trains is not less than the safe time interval, which includes the station arrival time interval, the station departure time interval, and the section tracking time interval; Station capacity constraint, which is used to constrain that each track of a station can only be allocated to a single train at the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train is not less than the minimum departure-arrival time interval to satisfy train safety. Arrival and departure time constraints are used to address the problem of rescheduling train timetables. Under normal circumstances, trains will run according to the train timetable; or under normal circumstances, trains are only allowed to arrive earlier than scheduled, and are not allowed to depart earlier than the original departure time in the train timetable.

Images