High-speed train timetable optimization and rescheduling method based on deep reinforcement learning

Abstract

The invention provides a high-speed train timetable optimization and rescheduling method based on deep reinforcement learning, relates to the technical field of high-speed train scheduling and resource optimization, and aims to realize efficient and accurate high-speed train timetable optimization and rescheduling. Comprising the steps that a strategy network is constructed based on a Markov decision process model of high-speed rail scheduling, and the strategy network learns an action decision strategy of the Markov decision process model of the high-speed rail scheduling; inputting the current station state into the strategy network to obtain an action decision as a departure sequence of all trains at the current station; based on a train state transition model, simulating a train operation environment according to a departure sequence of all trains in the current station, and determining arrival time and departure time of all trains in the current station, which meet traffic safety constraints; according to the above steps, the arrival time and departure time of all trains in each station on the running route are iteratively calculated in sequence, and a train timetable is obtained.

Description

Technical Field

[0001] The present application relates to the technical field of high-speed rail scheduling and resource optimization, and in particular to a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning. Background Art

[0002] High-speed rail has become the first choice for passengers due to its convenience, speed and punctuality. The train timetable connects various transportation activities in the high-speed rail system and plays a fundamental leading role in the operation organization process. In the planning stage, the dispatcher needs to develop an efficient train timetable based on the given parking plan, determine the arrival and departure time of the train at each station, and minimize the total travel time of all trains. This process is called Train Timetable Optimization (TTO). Under normal circumstances, high-speed rail actually operates according to the planned timetable. However, many unavoidable interferences caused by emergencies such as bad weather and equipment failure often occur, resulting in delays of single or multiple trains. Train Timetable Rescheduling (TTR) aims to quickly restore the order of traffic through re-sequencing, retiming and re-routing scheduling strategies to reduce the total delay time. Therefore, it is of great significance to reasonably compile the train timetable and minimize the deviation in actual operation to ensure the safe operation of high-speed rail and meet the needs of passengers.

[0003] At present, high-speed rail dispatching adopts a decentralized autonomous dispatching centralized system, which solves the automation problems such as dispatching instructions and remote signal control. However, how to optimize and reschedule train schedules is still mainly manual, heavily relying on the experience and judgment of dispatchers, resulting in insufficient real-time performance and poor performance on large-scale problems; and the number of decision variables in high-speed rail train dispatching is polynomially related to the problem scale, making it difficult to optimize and reschedule train schedules in real time, especially in actual large-scale problem scenarios. Therefore, it is of great value to provide an efficient dispatching framework for TTO and TTR problems, which can not only reduce the workload of dispatchers, but also improve the operating efficiency of the railway system. Summary of the invention

[0004] In view of the above problems, an embodiment of the present application provides a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning is disclosed, the method comprising: Based on the Markov decision process model of high-speed rail scheduling, a strategy network is constructed, wherein the strategy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling, wherein the strategy network includes an input embedding module, an encoder and a decoder, wherein the action decision strategy represents the scheduling strategy of the high-speed rail train, and the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train; Inputting the current station state into the strategy network, obtaining an action decision as the departure sequence of all trains at the current station, wherein the current station state includes station characteristics representing station capacity and station location, and train characteristics representing train operation information; Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival time and departure time of all trains at the current station that meet the driving safety constraints are determined; According to the above steps, the arrival time and departure time of all trains at each station on the route are iteratively calculated in sequence to obtain the train timetable.

[0006] Optionally, a policy network is constructed based on the Markov decision process model of high-speed rail scheduling, including: According to the Markov decision process model of high-speed rail dispatching, an action decision strategy for strategy network learning is designed; With the goal of learning the action decision strategy, the strategy network is trained through a deep reinforcement learning algorithm, and after the training end condition is met, a trained strategy network is obtained.

[0007] Optionally, with the goal of learning the action decision strategy, the strategy network is trained by a deep reinforcement learning algorithm, including: The policy network generates the departure sequence of all trains at the target station based on the current policy network parameters and the target station status; Calculate the reward value and the next target station state according to the departure order of all trains at the target station through the train state transition model; The state of the target station, the departure order of all trains at the target station and the reward value are taken as a scheduling trajectory, and the complete scheduling trajectory is obtained by repeating the above steps until the destination station; A gradient is calculated according to the complete scheduling trajectory, and a policy network parameter of the policy network is updated.

[0008] Optionally, the current station state is input into the policy network, and the action decision is obtained as the departure order of all trains at the current station, including: Input the current station state into the input embedding module to obtain a station embedding including global information of the current station and all subsequent stations and a node embedding including information of all trains; The encoder extracts dependencies between trains from the node embeddings according to the attention mechanism to obtain an encoded train embedding containing global information of all trains; The decoder decodes the coded train embedding and the station embedding to obtain the departure order of all trains at the current station.

[0009] Optionally, the current station state includes station features and train features, and the train features include train dynamic features and train static features; the input embedding module includes a first long short-term memory module, a second long short-term memory module and a linear projection module; The current station state is input into the input embedding module to obtain a station embedding including global information of the current station and all subsequent stations and a node embedding including information of all trains, including: Inputting the station features into the first long short-term memory module to obtain a station embedding including global information of the current station and all subsequent stations; Inputting the train static features into the second long short-term memory module to obtain a static node embedding containing all train static information; Inputting the train dynamic features into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding; According to the static node embedding and the dynamic node embedding, a node embedding containing all train information is obtained.

[0010] Optionally, the encoder comprises a plurality of attention encoding layers, each of which comprises a multi-head attention layer and a fully connected multi-layer perceptual feed-forward layer; The encoder extracts dependencies between trains from the node embedding according to the attention mechanism to obtain an encoded train embedding containing global information of all trains, including: The node embedding is input into the multiple attention encoding layers for multi-layer attention encoding to obtain an encoded train embedding containing global information of all trains.

[0011] Optionally, decoding by the decoder according to the coded train embedding and the station embedding to obtain the departure order of all trains at the current station includes: At each decision time step, a context embedding is determined according to the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, wherein the sequence embedding represents a partially determined departure sequence, and the aggregate embedding of the coded train embedding represents a mean value of the coded train embedding; Based on the context embedding and the encoded train embedding, the next departing train is determined from the optional trains that meet the constraint conditions according to the strategy probability distribution; the strategy probability distribution represents the probability distribution of each train being selected as the next departing train; the optional trains that meet the constraint conditions include: trains that are not selected and N trains with the arrival time at the current station ranked first; According to the above steps, multiple decision time steps are repeated until all trains leave the current station, and the departure order of all trains at the current station is obtained.

[0012] Optionally, based on the context embedding and the coded train embedding, determining the next departing train from the optional trains satisfying the constraint condition according to a strategic probability distribution includes: Input the context embedding and the encoding train embedding into the first decoding layer for attention interaction to obtain a new context embedding; wherein the first decoding layer determines a key vector and a value vector of the attention interaction according to the encoding train embedding, and determines a query vector of the attention interaction according to the context embedding; Input the new context embedding and the encoded train embedding into the second decoding layer to obtain attention weights, where the attention weights represent the weights of each train being selected as the next train to leave the station; According to the attention weight, a strategy probability distribution is obtained, and the next departing train is determined according to the strategy probability distribution.

[0013] Optionally, based on the train state transition model, simulating the train operation environment according to the departure sequence of all trains at the current station, and determining the arrival time and departure time of all trains at the current station that meet the driving safety constraints, includes: Determine the initial arrival time of all trains at the current station according to the driving safety constraint and the departure time of all trains at the previous station; For each target train, determine the track occupancy of the current station at the target time, the target time being the initial arrival time of the target train minus the given shortest departure-arrival interval time, and the target train is any train selected in sequence according to the departure order of all trains; When the track occupancy condition is that there is no idle track, the arrival time of the target train at the current station is updated according to the track occupancy condition, and the arrival time of all subsequent trains at the previous station that depart later than the target train at the current station is updated; When the track occupancy condition is that there is an idle track, a free track is randomly allocated to the target train, and the arrival time of the target train at the current station is the initial arrival time; The ready time of the target train is determined according to the driving safety constraint and the arrival time at the current station, and the departure time of the target train at the current station is determined according to the ready time, wherein the ready time represents the time when the train has completed the stop operation and is ready to leave the station.

[0014] Optionally, the Markov decision process model for high-speed rail scheduling is constructed in the following manner: The station state composed of train characteristics and station characteristics is used as the state of the Markov decision process model of the high-speed rail scheduling; The departure order of all trains at the current station is used as the action of the Markov decision process model of the high-speed rail scheduling; The process of switching from the current station state to the next station state based on the departure sequence of all trains at the station is used as the state transition of the Markov decision process model of the high-speed rail scheduling; In the case of the initial optimization of the train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train travel time; or, in the case of the re-scheduling of the train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train delay time.

[0015] Optionally, in the case of initially optimizing the train schedule, with the goal of minimizing the total train travel time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated according to the departure sequence of all trains at the station, including: Under the condition of satisfying driving safety constraints, with the goal of minimizing the total travel time of the train, a first objective function is constructed according to the travel time of the train between two stations, wherein the travel time of the train between the two stations includes: the minimum stop time at the station, the minimum running time in the section and the buffer time; The negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model of the high-speed rail scheduling.

[0016] Optionally, in the case of rescheduling the train schedule, with the goal of minimizing the total train delay time, the reward value of the Markov decision process model for high-speed rail scheduling is calculated according to the departure sequence of all trains at the station, including: Under the condition of satisfying the driving safety constraint, with the goal of minimizing the total train delay time, a second objective function is constructed according to the planned time for the train to arrive at the station and the planned time for the train to leave the station, wherein the planned time for the train to arrive at the station and the planned time for the train to leave the station are the times obtained by the initial optimization of the train schedule; The negative objective function corresponding to the second objective function is used as the reward value of the Markov decision process model of the high-speed rail scheduling.

[0017] Optionally, the driving safety constraints include: Running time constraints, which are used to constrain the train's stop time at each station and the running time of the section to not be less than a minimum time limit; Overtaking constraint, which is used to restrict the train from overtaking in the section between stations; A time interval constraint, wherein the time interval constraint is used to constrain the time interval between adjacent trains to be no less than a safety time interval, wherein the safety time interval includes a station arrival time interval, a station departure time interval, and a section tracking time interval; Station capacity constraints, where each track of the station can only be allocated to a single train in the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train occupying the same track is not less than the minimum departure-arrival time interval that meets train safety; Arrival and departure time constraints, the arrival and departure times are used to address the problem of rescheduling train timetables, constraining that trains run according to the train timetable when no disturbance occurs; or to constrain that in the event of a disturbance, trains are only allowed to arrive early and are not allowed to depart earlier than the original departure time in the train timetable.

[0018] In a second aspect of the embodiments of the present application, a high-speed rail train schedule optimization and rescheduling device based on deep reinforcement learning is disclosed, the device comprising: A construction module is used to construct a policy network based on a Markov decision process model of high-speed rail scheduling, wherein the policy network learns an action decision strategy of the Markov decision process model of high-speed rail scheduling, wherein the policy network includes an input embedding module, an encoder, and a decoder, wherein the action decision strategy represents a scheduling strategy of a high-speed rail train, and wherein the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of a train or minimizing the total delay time of a train; An input module, used for inputting the current station state into the policy network, and obtaining the action decision as the departure sequence of all trains at the current station, wherein the current station state includes station characteristics representing station capacity and station location, and train characteristics representing train operation information; A determination module, for simulating a train operation environment according to the departure sequence of all trains at the current station based on a train state transfer model, and determining the arrival time and departure time of all trains at the current station that meet the driving safety constraints; The iteration module is used to iteratively calculate the arrival time and departure time of all trains at each station on the running route according to the above steps to obtain the train timetable.

[0019] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning described in the first aspect of the embodiment of the present application are implemented.

[0020] In a fourth aspect of an embodiment of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning described in the first aspect of the embodiment of the present application are implemented.

[0021] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning described in the first aspect of the embodiment of the present application.

[0022] The embodiments of the present application include the following advantages: In the embodiment of the present application, the high-speed rail train schedule optimization and rescheduling problem is modeled as a two-stage real-time decision model. The first stage is the Markov decision process model of high-speed rail scheduling, and a policy network is constructed based on the Markov decision process model of high-speed rail scheduling, so that the policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling. Since the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train, the policy network can make action decisions according to the current station state, and the departure order of all trains in the current station obtained meets the goal of minimizing the total travel time of the train or minimizing the total delay time of the train. The second stage is the train state transition model, which uses a heuristic algorithm to simulate the train operation environment according to the departure order of all trains in the current station, and determines the arrival time and departure time of all trains in the current station that meet the driving safety constraints, ensuring that the train can travel safely and normally under the condition of meeting the goal of minimizing the total travel time of the train or minimizing the total delay time of the train. Then, according to the above two stages, it is iterated by station, and the arrival time and departure time of all trains at each station on the running route are calculated to obtain the train schedule. In this way, by decomposing each decision step of high-speed rail train schedule optimization and rescheduling into two stages, it is possible to handle scheduling problems under different operation plans and delay scenarios in real time, and calculate the train departure sequence and precise arrival and departure times at each station in turn. Therefore, in actual large-scale problem scenarios, this method can optimize the initial timetable and have the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 This is a high-speed train scheduling schematic diagram provided in an embodiment of the present application; Figure 2 It is a flowchart of the steps of a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning provided in an embodiment of the present application; Figure 3 This is an architecture diagram of a policy network provided in an embodiment of the present application; Figure 4 It is a schematic diagram of the overall framework of a high-speed rail general scheduling model provided in an embodiment of the present application; Figure 5It is a structural schematic diagram of a high-speed railway train schedule optimization and rescheduling device based on deep reinforcement learning provided in an embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Similar to many scheduling problems, both TTO and TTR problems are classified as NP-hard because of their inherent nature as mixed integer programming problems with complex constraints and a large number of integer variables, which increase with the scale of the HSR system (including the number of trains and stations). In terms of modeling, their decision variables and constraints are basically the same, while the main difference lies in the objective function and parameters. This commonality suggests that both problems can be solved within a unified framework.

[0027] In the related art, a large number of solutions have been proposed for the two train scheduling problems, which can be divided into four categories: operations research methods, heuristic methods, manually designed rules and machine learning methods. The first category usually uses different mathematical programming models to model high-speed rail scheduling problems, such as mixed integer linear programming models designed for TTO and TTR. On this basis, in order to improve the solution speed, relevant scholars have proposed algorithms such as branch and bound, Lagrangian relaxation, and column generation to decompose complex problems. However, mathematical methods require a lot of computing time to obtain accurate solutions under large-scale problems, so it is difficult to apply them to actual high-speed rail lines. In order to solve the above problems, the second method is to design heuristic algorithms based on swarm intelligence and iterative optimization, such as genetic algorithms, hybrid particle swarm algorithms, and adaptive large-scale neighborhood searches, which have shorter response times than the first method. However, heuristic algorithms generally have the defect of poor transferability, and they need to be iterated again when solving different operating scenarios, and the amount of calculation is still large.

[0028] Although both of the above two types of methods have achieved good results, in practice, dispatchers often use the third type of manually formulated fixed rules to meet real-time and safety requirements. Popular dispatching rules include First-Come-First-Served (FCFS) and First-Scheduled-First-Served (FSFS) strategies, which cannot obtain a schedule with better performance, hindering the full realization of the potential of high-speed railway systems. Therefore, how to design a dispatching strategy with low time cost and good performance has become a key challenge in the current high-speed rail dispatching field. In recent years, artificial intelligence technology has been increasingly applied to the field of intelligent transportation, and has been proven to be superior to traditional methods in terms of solution quality and computational efficiency. Therefore, the fourth type of learning-based methods that can automatically learn advanced heuristic rules, especially deep reinforcement learning, has become a new focus for solving the current high-speed rail dispatching problem.

[0029] In order to overcome the shortcomings of existing scheduling strategies in terms of solution quality and real-time response, the embodiment of the present application provides a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning, which is used to solve the problems of initial optimization and rescheduling of train schedules. On the one hand, the initial optimization of the train schedule lays the foundation for the smooth operation of trains according to the planned scheduling scheme under ideal conditions. On the other hand, the rescheduling of the train schedule ensures the adaptability and resilience of transportation services when facing actual emergencies. This method can not only optimize the initial timetable, but also has the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0030] like Figure 1 As shown, the high-speed train scheduling of the embodiment of the present application considers the initial optimization and rescheduling of the train schedule in a single direction (upward or downward) on a passenger dedicated line, and assumes that trains of multiple speed levels are running on the line. The initial optimization of the train schedule is to make a preliminary formulation of the train schedule before traveling, focusing on optimizing the operation objectives under the constraints of known trains and lines, minimizing travel time or maximizing resource utilization. Its purpose is to create a feasible and practical schedule based on the specified number of trains, train types, stop plans for each train, etc. In the daily operation process, the train schedule rescheduling dynamically adjusts the train schedule for unforeseen interference during the trip to restore the original schedule or minimize deviations. Its purpose is to make online adjustments to the train arrival and departure times, departure order, track occupancy, etc. on the basis of the planned schedule to avoid delays in the entire network.

[0031] The high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning in an embodiment of the present application is explained below in conjunction with the accompanying drawings.

[0032] In order to better understand the technical solution of the present application, the variables involved in the embodiments of the present application are defined. The embodiments of the present application consider a bidirectional double-track high-speed railway line, where up and down trains run on the lines in their respective directions, and it is assumed that trains of various speed levels are allowed to run. Therefore, the embodiments of the present application focus on solving the initial optimization and rescheduling problems of train schedules in a single direction (up or down), and define the following parameters for the above two types of scheduling problems: the set of stations indexed by label k , ; By the label or Indexed train set , ; By the label Station index Track collection , ,in Indicates the station The ultimate goal of both the initial optimization and rescheduling problems is to provide an efficient train schedule, so the decision variables are the same, including: train Arrival at the station Time ;train Leaving the station Time ; Departure order indicator variable If the train On the train Before leaving the station 1 if it is, 0 otherwise; track occupancy indicator variable If the train At the station Occupied track 1 if yes, 0 otherwise.

[0033] Reference Figure 2 As shown, Figure 2 This is a flowchart of the steps of a high-speed train schedule optimization and rescheduling method based on deep reinforcement learning provided in an embodiment of the present application. Figure 2 As shown, the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning may include steps S210 to S240: Step S210: Based on the Markov decision process model of high-speed rail scheduling, a strategy network is constructed, wherein the strategy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling, and the strategy network includes an input embedding module, an encoder and a decoder. The action decision strategy represents the scheduling strategy of the high-speed rail train, and the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train.

[0034] Among them, the Markov decision process model of high-speed rail scheduling is to construct the high-speed rail scheduling process as a Markov decision process model. For different train scheduling problems, the construction method of the Markov decision process model of high-speed rail scheduling is different. Specifically, in the case of initial optimization of the train timetable, the high-speed rail scheduling process is constructed as a Markov decision process model of high-speed rail scheduling with the goal of minimizing the total travel time of the train. In the case of re-scheduling the train timetable, the high-speed rail scheduling process is constructed as a Markov decision process model of high-speed rail scheduling with the goal of minimizing the total delay time of the train.

[0035] The policy network is an attention-based policy network that adopts an encoder-decoder framework. The policy network is constructed based on the Markov decision process model of high-speed rail scheduling so that the policy network learns the action strategy of the Markov decision process model of high-speed rail scheduling (that is, learns the scheduling strategy of high-speed rail trains).

[0036] Step S220: Input the current station state into the strategy network to obtain the action decision as the departure order of all trains at the current station, wherein the current station state includes station characteristics representing station capacity and station location, and train characteristics representing train operation information.

[0037] Among them, the running route is a single direction on a passenger dedicated line, that is, a down direction route or an up direction route on the passenger dedicated line, including multiple stations on the running route, and multiple trains run on the running route. The current station status refers to the station characteristics and train characteristics of the current station and subsequent stations. The station characteristics include station location and station capacity. The train characteristics include train dynamic characteristics and train static characteristics. The train static characteristics (fixed characteristics) refer to the fixed running information of the train. The fixed running information can be parameters such as the shortest running time of the section, the shortest stop time of the station, the buffer time of the station, and the planned departure time of the station; the train dynamic characteristics (variable characteristics) refer to the dynamic running information that changes according to each decision step, that is, the ready time (the train has completed the stop operation and is ready to leave the station).

[0038] It is understandable that the current station status will be different for different train scheduling problems. For example, for the static characteristics of the train, in the case of the initial optimization of the train schedule (i.e., the initial optimization problem), the static characteristics of the train include the shortest stop time at the station and the shortest running time between intervals. In the case of the rescheduling of the train schedule (rescheduling problem), the static characteristics of the train include the buffer time of the station and the planned departure time.

[0039] The policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling. Therefore, the policy network can make action decisions based on the current station status and give the departure order of all trains at the current station. The departure order of all trains at the current station meets the goal of minimizing the total travel time of the train or minimizing the total delay time of the train.

[0040] Specifically, the policy network includes an input embedding module, an encoder and a decoder. Station features and train features are input into the policy network. The input embedding module generates a station embedding containing global information of the current station and all subsequent stations, and a node embedding containing information of all trains. The encoder further processes the station embedding using an attention mechanism, extracts complex dependencies between trains, and obtains an encoded train embedding containing global information of all trains. The decoder decodes multiple decision time steps based on the encoded train embedding and station embedding to obtain the departure order of all trains at the current station.

[0041] In each decision time step, the coded train embedding and context embedding are used as inputs. According to the partially determined departure sequence (included in the context embedding), the next departing train is sampled from the optional trains that meet the constraints according to the strategy probability distribution until all trains leave the current station to obtain the complete departure sequence. The context embedding includes the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding. The sequence embedding represents the departure sequence that has been partially determined in the current decision step, and the aggregate embedding of the coded train embedding represents the mean of the coded train embedding.

[0042] Step S230: Based on the train state transfer model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival time and departure time of all trains at the current station that meet the driving safety constraints are determined.

[0043] Among them, the train state transfer model can use heuristic algorithms to simulate the train operation environment. Based on the train state transfer model, the arrival time and departure time of all trains at the current station are determined according to the departure order of all trains at the current station. That is, for the current station, based on the train state transfer model, while meeting the driving safety constraints, the arrival time of each train at the current station and the departure time at the current station (the time to leave the current station) can be determined.

[0044] Step S240: According to the above steps, the arrival time and departure time of all trains at each station on the running route are iteratively calculated in sequence to obtain the train timetable.

[0045] In the embodiment of the present application, each decision step of the high-speed rail train schedule optimization and rescheduling, that is, the departure order of all trains at each station, and the arrival time and departure time of all trains, is broken down into two stages and iterative calculations are performed according to the stations, and the departure order of all trains at each station on the operating route, as well as the precise arrival time and departure time are calculated in turn until a complete train schedule is obtained.

[0046] It can be understood that in the case of the initial optimized train schedule, the obtained train schedule is the initial optimized train schedule, and in the case of the rescheduled train schedule, the obtained train schedule is the rescheduled train schedule that dynamically adjusts the initial optimized train schedule according to interference conditions.

[0047] The technical solution of the embodiment of the present application is adopted to model the high-speed rail train schedule optimization and rescheduling problem as a two-stage real-time decision model. The first stage is the Markov decision process model of high-speed rail scheduling, and a policy network is constructed based on the Markov decision process model of high-speed rail scheduling, so that the policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling. Since the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train, the policy network can make action decisions according to the current station state, and the departure order of all trains in the current station obtained meets the goal of minimizing the total travel time of the train or minimizing the total delay time of the train. The second stage is the train state transition model, which uses a heuristic algorithm to simulate the train operation environment according to the departure order of all trains in the current station, and determines the arrival time and departure time of all trains in the current station that meet the driving safety constraints, ensuring that the train can travel safely and normally under the condition of meeting the goal of minimizing the total travel time of the train or minimizing the total delay time of the train. Then, according to the above two stages, it is iterated by station, and the arrival time and departure time of all trains at each station on the running route are calculated to obtain the train schedule. In this way, by decomposing each decision step of high-speed rail train schedule optimization and rescheduling into two stages, it is possible to handle scheduling problems under different operation plans and delay scenarios in real time, and calculate the train departure sequence and precise arrival and departure times at each station in turn. Therefore, in actual large-scale problem scenarios, this method can optimize the initial timetable and have the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0048] In the embodiment of the present application, the operation of high-speed trains has strict safety constraints. In order to ensure that the obtained train schedule is feasible, the driving safety constraints need to be met when determining the departure order of all trains at the previous station and the arrival time and departure time of all trains at the current station. Specifically, the driving safety constraints include: running time constraints, overtaking constraints, time interval constraints, station capacity constraints, arrival and departure time constraints, and decision variable constraints. These driving safety constraints are described below.

[0049] (1) The first constraint: running time constraint, which is used to constrain the train’s stop time at each station and the running time in the section to not be less than the minimum time limit.

[0050] For each station, the shortest stop time depends on the number of passengers getting on and off and the station infrastructure. For each section, the shortest running time depends on the train information and line conditions. Therefore, the running time constraint can be expressed as:

[0051]

[0052] in, Indicates train At the station Minimum stop time, Indicates train At the station and The shortest running time between ).

[0053] (2) The second constraint: overtaking constraint, which is used to prevent a train from overtaking in the section between stations.

[0054] That is to say, in a bidirectional high-speed rail line, trains traveling in the same direction have only one main track in each section, so trains are not allowed to overtake in the section. This constraint can be indirectly expressed by the next constraint (the third constraint).

[0055] (3) The third constraint: time interval constraint, which is used to constrain the time interval between adjacent trains to be no less than the safety time interval, which includes the station arrival time interval, station departure time interval and section tracking time interval.

[0056] In order to ensure driving safety, it is necessary to maintain a safe distance (safe time interval) between adjacent trains. Assuming that all trains run at a uniform speed in the section, the tracking time interval can be considered only when the train enters and leaves the station. Then the time interval constraint can be expressed as:

[0057]

[0058] in, For a sufficiently large number, It indicates the shortest interval time between two adjacent trains arriving at the same station. It indicates the shortest interval between two adjacent trains departing from the same station. It represents the shortest tracking time interval between two adjacent trains running in the interval. , .

[0059] (4) The fourth constraint: station capacity constraint. The station capacity constraint is used to constrain each track of the station to be allocated to only one train in the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train occupying the same track is not less than the minimum departure-arrival time interval that meets train safety.

[0060] At each station, a train needs to be assigned a track to stop or pass through, and each track can only be occupied by one train at a time. If two trains occupy the same track in the station, the time interval between the departure of the previous train and the arrival of the next train should not be less than the minimum departure-arrival time interval. Therefore, the station capacity constraint can be expressed as:

[0061]

[0062] in, It represents the shortest departure-arrival interval time between two adjacent trains occupying the same track.

[0063] (5) The fifth constraint: arrival and departure time constraint. The arrival and departure time constraint is used to address the problem of rescheduling train timetables. It constrains that trains run according to the train timetable when no disturbance occurs; or it constrains that when a disturbance occurs, trains are only allowed to arrive early and are not allowed to depart earlier than the original departure time in the train timetable.

[0064] That is to say, before the disturbance occurs, the trains run strictly according to the train schedule. After the disturbance occurs, the trains are only allowed to arrive early and are not allowed to depart earlier than the scheduled time, otherwise many passengers will miss the train.

[0065]

[0066] in, Indicates train At the station The initial delay time, represents the time when the disturbance occurs, represents the scheduled time for train 𝑖 to arrive at station 𝑘, It represents the planned time for train 𝑖 to leave station 𝑘, and the planned time is the planned schedule obtained by the initial optimization problem.

[0067] (6) The sixth constraint: decision variable constraint, which defines the value range of continuous decision variables and 0-1 integer decision variables. The decision variable constraint can be expressed as:

[0068]

[0069] It should be noted that since the rescheduling is based on the planned timetable (train timetable) obtained by the initial optimization, the initial optimization problem is considered first. The operation of high-speed trains has strict safety constraints. In order to ensure the feasibility of the obtained train timetable, the driving safety constraints satisfied by the initial optimization include the first constraint, the second constraint, the third constraint, the fourth constraint, and the sixth constraint. For the rescheduling problem, due to sudden interference, the above-mentioned first constraint, second constraint, third constraint, fourth constraint, fifth constraint, and sixth constraint need to be satisfied.

[0070] The following describes in detail the high-speed rail schedule optimization and rescheduling method based on deep reinforcement learning in the embodiments of the present application through Section 1.1, the Markov decision process model construction method for high-speed rail scheduling, Section 1.2, the policy network construction method, Section 1.3, the method for generating train departure sequence based on the policy network, and Section 1.4, the train state transition model based on the train to determine the train schedule.

[0071] 1.1 Method for constructing Markov decision process model for high-speed rail dispatching: In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning. In this method, the Markov decision process model of high-speed rail scheduling is constructed in the following manner, including steps A1 to A4: Step A1: The station state composed of train characteristics and station characteristics is used as the state of the Markov decision process model of the high-speed rail scheduling.

[0072] In the embodiment of the present application, the station characteristics include station location and station capacity. The station characteristics can be expressed as ,in, Indicates the station location. Indicates the station capacity.

[0073] Station characteristics are divided into train dynamic characteristics and train static characteristics. The train static characteristics include train The following parameters: interval Shortest running time, station The shortest stop time, station Buffer time and station Specifically, for the initial optimization and rescheduling problems, the static characteristics of the train can be expressed as:

[0074] in, Represents the static characteristics of the train. For the initial optimization problem, the static characteristics of the train include the shortest stop time at the station (i.e. train At the station The shortest stop time) and the shortest running time of the section (i.e. train At the station and The shortest running time between ), for the rescheduling problem, the static characteristics of the train include the buffer time of the station (i.e. train At the station buffer time) and planned departure time (i.e. train Leaving the station scheduled time).

[0075] For the rescheduling problem, train At the station Buffer time The calculation formula is expressed as:

[0076] Where M represents the total number of stations. Indicates train Arrival at the station The planned time, Indicates train Leaving the station The planned time, Indicates train At the station and The shortest running time between.

[0077] The dynamic characteristics of the train change with each decision step, defined as the readiness time , indicating that the train has completed the stop operation and is ready to leave the station. The ready time is expressed as:

[0078] in, Indicates train At the station The initial delay time.

[0079] Therefore, the current decision step The state of (current station) (station state) can be expressed as ,in Represents all trains, Represents the current station to the terminal station. Here, only the characteristics of the current station and subsequent stations are considered, because the parameter influence of the previous station has been reflected in the train variable characteristics.

[0080] Step A2: The departure sequence of all trains at the current station is used as the action of the Markov decision process model of the high-speed rail scheduling.

[0081] In the embodiment of the present application, based on the current station status The Markov decision process model of high-speed rail scheduling gives the departure order of all trains at the current station (i.e., action ). Therefore, the action It can be expressed as: ,in From the station Issued The train number of the train.

[0082] Step A3: The process of switching from the current station state to the next station state based on the departure sequence of all trains at the station is used as the state transfer of the Markov decision process model of the high-speed rail scheduling.

[0083] In the embodiment of the present application, the state transfer refers to simulating the train operation environment, which can be described as based on a given action. , from the current station status Transition to next station state process.

[0084] Step A4: In the case of the initial optimization of the train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train travel time; or, in the case of the re-scheduled train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train delay time.

[0085] In the embodiment of the present application, the reward values ​​corresponding to different optimization problems (i.e., the initial optimized train schedule and the re-scheduled train schedule) are calculated differently. The negative objective function corresponding to the objective function of the optimization problem can be used as the reward value. The reward values ​​corresponding to the two optimization problems are explained below.

[0086] (1) In the case of the initial optimization of the train schedule, the reward value of the Markov decision process model of the high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total travel time of the train, including: under the condition of satisfying the driving safety constraint, the first objective function is constructed according to the travel time of the train between two stations with the goal of minimizing the total travel time of the train, and the travel time of the train between two stations includes: the minimum stop time at the station, the minimum running time in the section and the buffer time; the negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model of the high-speed rail scheduling.

[0087] For the initial optimization of the train schedule (initial optimization problem), it is necessary to minimize the total train travel time while satisfying the driving safety constraints. , that is, the first objective function:

[0088] in, and are the total number of stations and the total number of trains respectively. The train arrival time at the starting station is defined as .train The travel time between two stations consists of three parts: the minimum stop time at the station, the minimum running time in the section, and the buffer time. The travel time between two stations is expressed as:

[0089] Among them, train At the station Minimum stop time and train At the station and Minimum running time between are all given constants; Indicates train At the station and the buffer time of the corresponding interval is the additional travel time allocated beyond the minimum running time required to meet the safety constraint. Therefore, minimizing the total travel time is equivalent to minimizing the total buffer time , the first objective function can be reformulated as:

[0090] Therefore, in the case of the initial optimized train schedule, all trains are at the station In order The reward value of the Markov decision process model for high-speed rail scheduling is issued It can be expressed as:

[0091] (2) In the case of rescheduling the train schedule, with the goal of minimizing the total train delay time, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station, including: under the condition of satisfying the driving safety constraint, with the goal of minimizing the total train delay time, a second objective function is constructed according to the planned time for the train to arrive at the station and the planned time for the train to leave the station, the planned time for the train to arrive at the station and the planned time for the train to leave the station being the time obtained by the initial optimized train schedule; and the negative objective function corresponding to the second objective function is used as the reward value of the Markov decision process model of high-speed rail scheduling.

[0092] For the rescheduling of train schedules (rescheduling problem), it is necessary to restore the order of the railway system as soon as possible and minimize the total delay time of trains. , that is, the second objective function can be expressed as:

[0093] in, Indicates train Arrival at the station The planned time, Indicates train Leaving the station The planned time is the planned schedule obtained from the initial optimization problem.

[0094] Therefore, in the case of rescheduling the train schedule, the trains are In order The reward value of the Markov decision process model for high-speed rail scheduling is issued It can be expressed as:

[0095] The Markov decision process model for high-speed rail scheduling constructed by adopting the technical solution of the embodiment of the present application is , the strategy network can be based on the current station status Generate the departure order of all trains , then, the train operation environment performs actions Afterwards get reward , and transfer to the next decision station In this way, the process is iterated until a complete train timetable is generated.

[0096] 1.2 Method of constructing policy network: In combination with the above embodiments, in one implementation, the present application embodiment further provides a method for optimizing and rescheduling a high-speed rail train schedule based on deep reinforcement learning. In this method, the "constructing a strategy network based on a Markov decision process model for high-speed rail scheduling" in the above step S210 specifically includes: According to the Markov decision process model of high-speed rail dispatching, an action decision strategy for policy network learning is designed; with the goal of learning the action decision strategy, the policy network is trained through a deep reinforcement learning algorithm, and after the training end conditions are met, a trained policy network is obtained.

[0097] In the embodiment of the present application, constructing a policy network is a process of continuously updating policy network parameters based on a deep reinforcement learning algorithm. By updating the policy network parameters, the policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling, and the trained policy network can be used as a scheduling strategy and applied directly.

[0098] The policy network can be based on its own policy network parameters , given at the current station status Next, select Action (i.e. the departure order of all trains at the current station) :

[0099] Specifically, with the goal of learning the action decision strategy, the strategy network is trained by a deep reinforcement learning algorithm, including steps D1 to D4: Step D1: The policy network generates the departure sequence of all trains at the target station based on the current policy network parameters and the target station status.

[0100] Step D2: Calculate the reward value and the next target station state according to the departure order of all trains at the target station through the train state transition model.

[0101] Step D3: The target station status, the departure order of all trains at the target station and the reward value are taken as a scheduling trajectory, and the complete scheduling trajectory is obtained by repeating the above steps until the destination station.

[0102] Step D4: Calculate the gradient according to the complete scheduling trajectory, and update the policy network parameters of the policy network.

[0103] In this application embodiment, the above strategy distribution is used , the strategy network samples the departure order of all trains at the target current station based on the current strategy network parameters and the target station state. Then, the reward value is calculated based on the departure order of all trains at the target station through the train state transition model. and the next target station status Finally, by repeating the above steps until the terminal station, the complete scheduling trajectory can be obtained:

[0104] in, As the dispatch trajectory of the first station, they are the station status of the first station, the departure order of all trains and the reward value; As the dispatch trajectory of the k-th station, they are the station status of the k-th station, the departure order of all trains, and the reward value; As the scheduling trajectory of the M-1th station, they are the station status of the M-1th station, the departure order of all trains and the reward value.

[0105] The gradient is calculated according to the complete scheduling trajectory, and the policy network parameters of the policy network are updated. The specific process is: Each decision station in the complete scheduling trajectory , defining the long-term discounted return :

[0106] in, is the discount factor, and its value range is ( .

[0107] Considering the efficiency and stability of the training process, the REINFORCE algorithm based on baseline greed in reinforcement learning is used to train the policy network, and the gradient with baseline is defined for:

[0108] Among them, the baseline It is defined as using the baseline network to greedily select the action with the highest probability, thus obtaining a certain long-term reward. In this way, the state The difficulty of the corresponding high-speed rail scheduling example, reducing the gradient variance, improving the convergence speed, baseline network and Strategy Network The structure is the same and the parameters are fixed. At the end of each training round, the current training strategy (via greedy decoding) is paired with the baseline strategy for a T-test. If the improvement is significant, the parameters of the baseline network are replaced. In this way, after the training end condition is met (for example, the preset number of training times is met), a trained strategy network is obtained.

[0109] For example, the training process of the policy network can be implemented by the REINFORCE algorithm based on the baseline greedy induction in Table 1.

[0110] Table 1 REINFORCE algorithm based on baseline greedy:

[0111] By adopting the technical solution of the embodiment of the present application, after completing the offline training of the decision model, the trained strategy network can be used for online high-speed rail scheduling, that is, the strategy network can select the train with the highest probability to send out in each decoding decision step, meeting the real-time requirements.

[0112] 1.3 Method for generating train departure sequence based on strategy network: In combination with the above embodiments, in one implementation, the embodiment of the present application also provides a high-speed train schedule optimization and rescheduling method based on deep reinforcement learning. In this method, the above step S220 "inputting the current station state into the policy network to obtain the action decision as the departure order of all trains at the current station" specifically includes steps C1 to C3: Step C1: Input the current station state into the input embedding module to obtain a station embedding containing global information of the current station and all subsequent stations and a node embedding containing information of all trains.

[0113] Among them, the current station state includes station characteristics and train characteristics, and the train characteristics include train dynamic characteristics and train static characteristics; the input embedding module includes a first long short-term memory module, a second long short-term memory module and a linear projection module.

[0114] Specifically, the current station state is input into the input embedding module to obtain a station embedding containing global information of the current station and all subsequent stations and a node embedding containing information of all trains, including: inputting the station features into the first long short-term memory module to obtain a station embedding containing global information of the current station and all subsequent stations; inputting the train static features into the second long short-term memory module to obtain a static node embedding containing static information of all trains; inputting the train dynamic features into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding; and obtaining a node embedding containing information of all trains based on the static node embedding and the dynamic node embedding.

[0115] In the embodiment of the present application, considering that the length of each station state is different, a long short-term memory (LSTM) network (i.e., the first long short-term memory module) is selected. and the second long short-term memory module ) processes train features and station features. Specifically, the station embedding is obtained from the terminal station through the first long short-term memory module To the current station The global information of the station transmitted; for node embedding, it is necessary to consider the static characteristics of the train and the dynamic characteristics of the train. The second long short-term memory module is used to process the static characteristics of each train node, that is, the second long short-term memory module is used to obtain the static characteristics of the train from the terminal station. To the current station The global static information of each train is transmitted; at the same time, through the linear projection module The train dynamic features are linearly projected to the same embedding dimension as the static node embedding dimension.

[0116] Example, station embedded and node embedding It can be expressed as:

[0117]

[0118] Among them, LP represents the learnable parameters and Linear projection layer (Linear projection module), , represents the static characteristics of train i from the current station to the terminal station, represents the dynamic characteristics of train i at station k, Indicates the station characteristics from the current station to the terminal station.

[0119] Node embeddings obtained by inputting the embedding module Will be further processed at the encoder, station embedded Will be used directly as input to the decoder.

[0120] Step C2: The encoder extracts dependencies between trains from the node embeddings according to the attention mechanism to obtain an encoded train embedding containing global information of all trains.

[0121] In the embodiment of the present application, the node embedding is used to extract the dependency between trains according to the attention mechanism. Considering that the deep reinforcement learning method for existing high-speed rail scheduling problems usually ignores the relationship between different trains, the action decision lacks foresight and can only obtain local optimal strategies. Therefore, the embodiment of the present application refers to the Transformer architecture and uses the self-attention mechanism to complete the information exchange between train nodes. By aggregating information from all nodes, each train node obtains the overall perspective of the scheduling problem, rather than the local information of the train.

[0122] Specifically, the encoder includes multiple attention encoding layers, each of which includes a multi-head attention layer and a fully connected multi-layer perceptron feed-forward layer; in addition, residual connections and batch normalization (Batch Normalization, BN) layers are added to each multi-head attention (Multi-Head Attention, MHA) layer and fully connected multi-layer perceptron (Multi-Layer Perceptron, MLP) feed-forward layer.

[0123] Among them, the encoder extracts the dependency relationship between trains from the node embedding according to the attention mechanism to obtain the encoded train embedding containing the global information of all trains, including: inputting the node embedding into the multiple attention encoding layers for multi-layer attention encoding to obtain the encoded train embedding containing the global information of all trains.

[0124] For example, the multi-head attention layer and the fully connected multi-layer perceptron feed-forward layer can be expressed as:

[0125]

[0126] Among them, the attention encoding layer has L layers. represents the batch normalization layer, Indicates multi-head attention calculation for the node embedding of train i Represents a multi-head attention layer The train is at the station Node embedding, Represents a multi-head attention layer Output train At the station Node embedding, Represents a multi-layer perceptron feed-forward layer Represented by the attention encoding layer The given train At the station Node embedding of the final train node embedding go through The layer attention is obtained after the encoding layer and will be used as input in the decoder.

[0127] Step C3: Decoding is performed by the decoder according to the coded train embedding and the station embedding to obtain the departure order of all trains at the current station.

[0128] In the embodiment of the present application, the decoding of the decoder according to the encoded train embedding and station embedding is a step-by-step decoding process. The decoder selects the next departing train according to the train embedding and station embedding output by the first two modules and the partial departure sequence that has been determined currently, thereby gradually obtaining the departure order of all trains.

[0129] Specifically, the decoder decodes according to the coded train embedding and the station embedding to obtain the departure order of all trains at the current station, including: Step C3-1: At each decision time step, a context embedding is determined based on the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, wherein the sequence embedding represents a partially determined departure sequence, and the aggregate embedding of the coded train embedding represents the mean of the coded train embedding.

[0130] Step C3-2: Based on the context embedding and the encoded train embedding, determine the next departing train from the optional trains that meet the constraints according to the strategic probability distribution; the strategic probability distribution represents the probability distribution of each train being selected as the next departing train; the optional trains that meet the constraints include: unselected trains and N trains with the top arrival time at the current station.

[0131] Step C3-3: Repeat the above steps for multiple decision time steps until all trains leave the current station, and obtain the departure order of all trains at the current station.

[0132] In the embodiment of the present application, at each decision time step, the encoded train embedding and the context embedding are used as input, and then according to the partially determined departure order (included in the context embedding), the next departing train is sampled from the optional trains that meet the constraints according to the strategic probability distribution, until all trains leave the current station to obtain the complete departure order.

[0133] In order to improve decoding efficiency, a context embedding is used for decoding. It can be expressed as:

[0134] in, represents the aggregate embedding of the encoded train embeddings, i.e., the mean of the encoded train embeddings; Indicates station embedding; represents sequence embedding, i.e. the departure sequence that has been partially determined at the current decision time step t .

[0135] Sequence Embedding It can be based on long short-term memory network (i.e. the third long short-term memory module) to obtain the sequence embedding It can be expressed as:

[0136] in, represents the sequence embedding at decision time step t-1, Represents the coded train embedding corresponding to the departure train determined in decision time step t-1.

[0137] After the context embedding is obtained, step C3-2 is executed to determine the next departing train from the optional trains that meet the constraint conditions according to the strategic probability distribution based on the context embedding and the encoded train embedding.

[0138] In the embodiment of the present application, a train masking rule is designed for the driving safety constraints of high-speed rail scheduling to ensure the feasibility of the departure sequence, that is, according to the train masking rule, only the optional trains that meet the constraints can be selected as the next departure train. Specifically, the train masking rule includes: 1) the trains that have been dispatched are masked; 2) only the trains before the arrival time are allowed to be The next train to depart is selected from the trains (except the trains that have left the station), and other trains are masked; 3) For the train schedule rescheduling problem, the trains run according to the original scheduled schedule before the interference occurs, the departure order is unique, and other trains are masked. Based on the above train masking rules, the optional trains that meet the constraints include: the trains that are not selected, and the N trains with the first arrival time at the current station.

[0139] Specifically, based on the context embedding and the encoded train embedding, the next departing train is determined from the optional trains that meet the constraints according to the strategy probability distribution, including: inputting the context embedding and the encoded train embedding into the first decoding layer for attention interaction to obtain a new context embedding; wherein the first decoding layer determines the key vector and value vector of the attention interaction according to the encoded train embedding, and determines the query vector of the attention interaction according to the context embedding; inputting the new context embedding and the encoded train embedding into the second decoding layer to obtain attention weights, and the attention weights represent the weights of each train being selected as the next departing train; according to the attention weights, a strategy probability distribution is obtained, and the next departing train is determined according to the strategy probability distribution.

[0140] In the embodiment of the present application, the first decoding layer is a one-to-many multi-head attention layer, and the second decoding layer is a one-to-many single-head attention layer. For example, the new context embedding It can be expressed as:

[0141] in, Represents the first decoding layer, which is a one-to-many multi-head attention layer.

[0142] Attention Weight :

[0143] in, Indicates that the context node is at the station The query vector is Represents a train node At the station The key vector, constant C and function tanh clip the result to within the range.

[0144] According to the attention weight, the strategy probability distribution is obtained by treating the attention weight as the log-likelihood and using the SoftMax function to calculate the strategy probability distribution. It can be expressed as:

[0145] In this way, the agent distributes the probability of The next train leaving the station is selected randomly, and the above steps C3-1 and C3-2 are repeated until all trains have been dispatched, thereby obtaining the departure order of all trains at the current station.

[0146] For example, the framework of the policy network is as follows Figure 3 As shown, the policy network includes an input embedding module, an encoder and a decoder. The input embedding module includes a first long short-term memory module ( )、The second long short-term memory module ( ) and the linear projection module ( ), given the given station features and train features, the input embedding module can generate station embeddings containing the global information of the current station and all subsequent stations and node embeddings containing all train information. The encoder includes multiple attention encoding layers ( ), the node embedding is input into the multiple attention encoding layers for multi-layer attention encoding, and the encoded train embedding containing the global information of all trains is obtained. The decoder includes a first decoding layer (One to many ), the second decoding layer (One to many ) and SoftMax, at each decision time step, the decoder determines the context embedding according to the aggregate embedding of the encoded train embedding, the station embedding, and the sequence embedding, and then inputs the context embedding and the encoded train embedding into the first decoding layer for attention interaction to obtain a new context embedding; the new context embedding and the encoded train embedding are input into the second decoding layer to obtain the attention weight; the attention weight is then regarded as the log-likelihood, and the SoftMax function is used to calculate the strategy probability distribution, and the next departing train is determined according to the strategy probability distribution. By repeating multiple decision time steps, the departure order of all trains at the current station is obtained.

[0147] 1.4 Determine the train schedule based on the train state transition model: In combination with the above embodiments, in one implementation, the embodiments of the present application also provide a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning. In this method, the train state transition model can use a heuristic algorithm to simulate the train operation environment, give the accurate arrival and departure time of each train at the current station, and naturally determine the track occupancy. Specifically, in the above step S220, "based on the train state transition model, simulate the train operation environment according to the departure order of all trains at the current station, and determine the arrival time and departure time of all trains at the current station that meet the driving safety constraints" specifically includes steps B1 to B5: Step B1: Determine the initial arrival time of all trains at the current station according to the driving safety constraint and the departure time of all trains at the previous station.

[0148] Among them, the initial arrival time of all trains at the current station can be iteratively calculated based on the departure time of all trains at the previous station. For the current station k, the initial arrival time of all trains at the current station can be expressed as:

[0149] in, Indicates train At the station and The shortest running time between Indicates train Leaving the station The time of the train departure time at the previous station), Indicates train At the station The previous departing train Arrival at the station time.

[0150] Step B2: For each target train, determine the track occupancy of the current station at the target time, the target time being the initial arrival time of the target train minus the given shortest departure-arrival interval time, and the target train is any train selected in sequence according to the departure order of all trains.

[0151] The target time can be expressed as , For trains Arrival Station The initial time, The shortest departure-arrival interval between two adjacent trains occupying the same track. Determine the track occupancy of the current station at the target time, that is, calculate the current station at the target time. Number of occupied tracks .

[0152] Step B3: When the track occupancy condition is that there is no idle track, the arrival time of the target train at the current station is updated according to the track occupancy condition, and the arrival time of all subsequent trains at the previous station that depart later than the target train at the current station is updated.

[0153] Specifically, there is no free track indicating that the current station at the target time Number of occupied tracks Greater than or equal to station Total number of tracks At this time, the arrival time of the target train at the current station needs to be updated according to the track occupancy situation. The arrival time of the target train at the current station can be updated according to Formula 2:

[0154] Among them, train For the Listed at target time From the station The ID of the train that is departing, and therefore at the station Assign to train Tracks and trains same.

[0155] Furthermore, in order to avoid new conflicts, the arrival times of subsequent trains need to be modified one by one, and the arrival times of subsequent trains need to be updated according to Formula 3:

[0156] Among them, train Indicates at the station All trains later than The train is departing, Indicates train At the station Departure order The position in.

[0157] Step B4: When the track occupancy condition is that there is an idle track, a free track is randomly allocated to the target train, and the arrival time of the target train at the current station is the initial arrival time.

[0158] Step B5: Determine the ready time of the target train according to the driving safety constraint and the arrival time at the current station, and determine the departure time of the target train at the current station according to the ready time, wherein the ready time represents the time when the train has completed the stop operation and is ready to leave the station.

[0159] After the target train arrives at the current station, it waits for departure at the current station. Using the above formula 1, if the driving safety constraints are met, the target train will depart immediately, thus obtaining the actual departure time:

[0160] Among them, train It's a train At the station The previous train that left, for the rescheduling problem, if ,but .

[0161] At this time, the initial arrival time of all trains at the next station (i.e., the earliest arrival time of the train at the next station) can be determined according to the departure time of the target train at the current station. The initial arrival time of all trains at the next station can be expressed as:

[0162] For the rescheduling problem, if ,but .

[0163] In this way, based on the departure order of all trains at the current station and based on the train state transition model, a heuristic algorithm can be used to simulate the train operation environment, that is, according to the above steps B1 to B5, the exact arrival time and departure time of each train can be determined in turn, and the track occupancy can be determined naturally.

[0164] For example, the specific implementation algorithm of the above steps B1 to B5 can be implemented by the train state transition heuristic algorithm in Table 2.

[0165] Table 2 Train state transfer heuristic algorithm:

[0166] For example, Figure 4 As shown, Figure 4 This is a schematic diagram of the overall framework of a high-speed rail general scheduling model provided by an embodiment of the present application. Specifically, the high-speed rail train schedule optimization and rescheduling problem is modeled as a two-stage real-time decision model. The first stage is a Markov decision process model for high-speed rail scheduling, and the second stage is a train state transition model. In the first stage, which is the Markov decision process model for high-speed rail scheduling, a policy network is used to learn the action decision strategy of the Markov decision process model for high-speed rail scheduling. The policy network can generate the departure sequence of all trains based on the current state. The train operating environment is used to execute actions. Afterwards get reward , and transfer to the next decision station . In the second-stage train state transition model, the train operation environment can be simulated according to the departure order of all trains at the current station, and the arrival and departure times of all trains at the current station that meet the driving safety constraints can be determined. In this way, the process is iterated until a complete train timetable is generated. This method can handle scheduling problems under different operation plans and delay scenarios in real time by breaking down each decision step of high-speed rail train timetable optimization and rescheduling into two stages, and calculate the train departure order and precise arrival and departure times at each station in turn. Therefore, in actual large-scale problem scenarios, this method can optimize the initial timetable and have the flexibility required to respond to real-time disturbances, thereby ensuring service reliability and passenger satisfaction.

[0167] Further, in order to better illustrate the performance of the high-speed train schedule optimization and rescheduling method based on deep reinforcement learning in the embodiment of the present application, the Beijing-Shanghai high-speed railway passenger dedicated line with 23 stations is taken as an example to verify the effectiveness of solving the initial optimization and rescheduling problems of the high-speed train schedule optimization and rescheduling method based on deep reinforcement learning (general scheduling model for high-speed railway). Among them, the line allows mixed running of trains of various speed levels, with maximum speeds of 350km / h, 300km / h and 250km / h respectively. The basic data of the line are shown in Table 3.

[0168] Table 3 Basic data of Beijing-Shanghai high-speed railway line:

[0169] For the initial optimization problem, the maximum train speed is randomly selected from three levels, which determines the shortest running time of the interval. Taking into account the length of time the train stops, each train randomly selects Whether to stop, if so, the time is randomly set For the rescheduling problem, the delay scenario mainly focuses on minor interference and delay time. Randomly select values ​​from 5 to 60 minutes and assume that the interference occurs at the time is the earliest arrival time of the delayed train. In addition, other parameters are set as follows according to the actual situation: the shortest arrival or departure interval is 4 minutes, the shortest tracking time interval is 4 minutes, the shortest departure-arrival time interval is 2 minutes, and other required hyper parameters are shown in Table 4.

[0170] Table 4 Hyperparameters required for the high-speed rail general scheduling model:

[0171] The high-speed rail universal scheduling model proposed in the embodiment of the present application is applicable to two types of problems: initial optimization and rescheduling. Therefore, the initial optimization scheduling model USM is first trained. O And evaluate its performance, and then use USM O The model outputs the planned schedule and uses it as the input of the rescheduling instance to further train the rescheduling model USM R .

[0172] (1) Results of the initial optimization experiment of the train timetable: In the experiment, the size of the problem instance is defined as ,in is the number of stations, is the number of trains. The training case requires dispatching 20 trains at 13 stations (i.e., Beijing South-Bengbu South dispatching section and Xuzhou East-Shanghai Hongqiao dispatching section) to complete the planned timetable, recorded as The comparison method is a manually designed First Come First Served (FCFS) strategy, which determines the departure order based on the arrival time. In addition, the commercial solver Gurobi provides a benchmark for the exact algorithm. Due to its expensive computational cost, the calculation is terminated when the optimal gap is less than 5% or the calculation time reaches 300 seconds.

[0173] In order to verify the generalization and scalability of the method, the scheduling model is first trained on a small-scale problem, and then the trained policy network is transferred to solve large-scale initial optimization problems, including 13 stations and 50 trains, 13 stations and 80 trains, 23 stations (i.e., the complete Beijing-Shanghai high-speed railway line) and 50 trains. Greedy decoding is used to evaluate the performance of the method, and the results are shown in Table 5, including the response time and scheduling solution quality on 100 test instances.

[0174] Table 5 Initial optimization results of train timetables on various scale cases:

[0175] For the target value, USM OThe model is always better than the FCFS scheduling rule at different problem scales. The Gurobi solver only obtains better results on small-scale problems, but as the scale increases, it is unable to generate feasible scheduling solutions for large-scale instances within the time limit. Regarding the running time, both FCFS and the method proposed in the embodiment of the present application can give a planned schedule for problem instances of all scales within 0.1 seconds. Although FCFS consumes the shortest time, the huge gap with the optimal value is unacceptable. However, due to its precise solving characteristics, Gurobi's solution time is exponentially related to the number of trains and stations, and it performs poorly on large-scale problems.

[0176] (2) Train schedule rescheduling experiment results With the help of trained USM O Model, a large number of planned timetables can be obtained as training data for train rescheduling problems, and then The USMR rescheduling model is trained on a case scale of 100. In addition to the FCFS and Gurobi solvers, the comparison method also introduces the First Schedule First Served (FSFS) strategy. This method is also a manually designed scheduling rule that gives the departure order according to the planned schedule. Table 6 shows the USMR trained on 100 instances. R The effect of the model.

[0177] Table 6 Train schedule rescheduling results on various scale cases:

[0178] The results show that the policy network trained on small-scale instances can be effectively generalized to large-scale practical problems. The method proposed in the embodiment of the present application is much better than the two scheduling rules in terms of optimality gap, especially for large-scale instances. R The scheduling solutions generated by the model are always within 8% of the target value of the Gurobi baseline. If the Gurobi solution time is limited to 20 seconds to meet the real-time response requirements, USM R The effectiveness of the model is more obvious. Compared with the dispatching rules, the strategies learned by the model can improve the performance indicators by 60% to 90%. In addition, similar to the results of the initial optimization experiment, the learning-based model also shows significant advantages in running time compared with the exact solution method, which can meet the real-time requirements of high-speed rail re-dispatching.

[0179] The present application also provides a high-speed rail train schedule optimization and rescheduling device based on deep reinforcement learning, referring to Figure 5 As shown, Figure 5: is a structural schematic diagram of a high-speed railway train schedule optimization and rescheduling device based on deep reinforcement learning provided in an embodiment of the present application, the device comprising: A construction module 510 is used to construct a policy network based on the Markov decision process model of high-speed rail scheduling, wherein the policy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling, and the policy network includes an input embedding module, an encoder, and a decoder. The action decision strategy represents the scheduling strategy of the high-speed rail train, and the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train; An input module 520, for inputting the current station state into the policy network to obtain an action decision as the departure sequence of all trains at the current station, wherein the current station state includes station characteristics representing station capacity and station location, and train characteristics representing train operation information; A determination module 530 is used to simulate the train operation environment according to the departure sequence of all trains at the current station based on the train state transition model, and determine the arrival time and departure time of all trains at the current station that meet the driving safety constraints; The iteration module 540 is used to iteratively calculate the arrival time and departure time of all trains at each station on the running route in accordance with the above steps to obtain the train timetable.

[0180] The present application also provides an electronic device, referring to Figure 6 , Figure 6 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 600 includes: a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus communication. A computer program is stored in the memory 610. The computer program can be run on the processor 620, thereby implementing the steps of the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning described in the embodiment of the present application.

[0181] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning described in the embodiment of the present application are implemented.

[0182] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of high-speed rail train schedule optimization and rescheduling based on deep reinforcement learning as described in the embodiment of the present application.

[0183] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0184] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0188] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0189] The above is a detailed introduction to a high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A high-speed rail train schedule optimization and rescheduling method based on deep reinforcement learning, characterized in that: include: Based on the Markov decision process model of high-speed rail scheduling, a strategy network is constructed, wherein the strategy network learns the action decision strategy of the Markov decision process model of high-speed rail scheduling, wherein the strategy network includes an input embedding module, an encoder and a decoder, wherein the action decision strategy represents the scheduling strategy of the high-speed rail train, and the Markov decision process model of high-speed rail scheduling is constructed with the goal of minimizing the total travel time of the train or minimizing the total delay time of the train; Inputting the current station state into the strategy network, obtaining an action decision as the departure sequence of all trains at the current station, wherein the current station state includes station characteristics representing station capacity and station location, and train characteristics representing train operation information; Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival time and departure time of all trains at the current station that meet the driving safety constraints are determined; According to the above steps, the arrival time and departure time of all trains at each station on the route are iteratively calculated in sequence to obtain the train timetable.

2. The method according to claim 1, characterized in that Based on the Markov decision process model of high-speed rail scheduling, a strategy network is constructed, including: According to the Markov decision process model of high-speed rail dispatching, an action decision strategy for strategy network learning is designed; With the goal of learning the action decision strategy, the strategy network is trained through a deep reinforcement learning algorithm, and after the training end condition is met, a trained strategy network is obtained.

3. The method according to claim 2, characterized in that With the goal of learning the action decision strategy, the strategy network is trained by a deep reinforcement learning algorithm, including: The policy network generates the departure sequence of all trains at the target station based on the current policy network parameters and the target station status; Calculate the reward value and the next target station state according to the departure order of all trains at the target station through the train state transition model; The state of the target station, the departure order of all trains at the target station and the reward value are taken as a scheduling trajectory, and the complete scheduling trajectory is obtained by repeating the above steps until the destination station; A gradient is calculated according to the complete scheduling trajectory, and a policy network parameter of the policy network is updated.

4. The method according to claim 1, characterized in that: The current station status is input into the policy network, and the action decision is the departure order of all trains at the current station, including: Input the current station state into the input embedding module to obtain a station embedding including global information of the current station and all subsequent stations and a node embedding including information of all trains; The encoder extracts dependencies between trains from the node embeddings according to the attention mechanism to obtain an encoded train embedding containing global information of all trains; The decoder decodes the coded train embedding and the station embedding to obtain the departure order of all trains at the current station.

5. The method according to claim 4, characterized in that The current station status includes station characteristics and train characteristics, and the train characteristics include train dynamic characteristics and train static characteristics; The input embedding module includes a first long short-term memory module, a second long short-term memory module and a linear projection module; The current station state is input into the input embedding module to obtain a station embedding including global information of the current station and all subsequent stations and a node embedding including information of all trains, including: Inputting the station features into the first long short-term memory module to obtain a station embedding including global information of the current station and all subsequent stations; Inputting the train static features into the second long short-term memory module to obtain a static node embedding containing all train static information; Inputting the train dynamic features into the linear projection module to obtain a dynamic node embedding with the same dimension as the static node embedding; According to the static node embedding and the dynamic node embedding, a node embedding containing all train information is obtained.

6. The method according to claim 4, characterized in that The encoder includes a plurality of attention encoding layers, each of which includes a multi-head attention layer and a fully connected multi-layer perception feedforward layer; The encoder extracts dependencies between trains from the node embedding according to the attention mechanism to obtain an encoded train embedding containing global information of all trains, including: The node embedding is input into the multiple attention encoding layers for multi-layer attention encoding to obtain an encoded train embedding containing global information of all trains.

7. The method according to claim 4, characterized in that Decoding the encoded train embedding and the station embedding by the decoder to obtain the departure order of all trains at the current station, including: At each decision time step, a context embedding is determined according to the aggregate embedding of the coded train embedding, the station embedding, and the sequence embedding, wherein the sequence embedding represents a partially determined departure sequence, and the aggregate embedding of the coded train embedding represents a mean value of the coded train embedding; Based on the context embedding and the encoded train embedding, the next departing train is determined from the optional trains that meet the constraint conditions according to the strategy probability distribution; the strategy probability distribution represents the probability distribution of each train being selected as the next departing train; the optional trains that meet the constraint conditions include: trains that are not selected and N trains with the arrival time at the current station ranked first; According to the above steps, multiple decision time steps are repeated until all trains leave the current station, and the departure order of all trains at the current station is obtained.

8. The method according to claim 7, characterized in that Based on the context embedding and the encoded train embedding, determining a next departing train from optional trains satisfying the constraint condition according to a strategic probability distribution, including: Input the context embedding and the encoding train embedding into the first decoding layer for attention interaction to obtain a new context embedding; wherein the first decoding layer determines a key vector and a value vector of the attention interaction according to the encoding train embedding, and determines a query vector of the attention interaction according to the context embedding; Input the new context embedding and the encoded train embedding into the second decoding layer to obtain attention weights, where the attention weights represent the weights of each train being selected as the next train to leave the station; According to the attention weight, a strategy probability distribution is obtained, and the next departing train is determined according to the strategy probability distribution.

9. The method according to claim 1, characterized in that: Based on the train state transition model, the train operation environment is simulated according to the departure sequence of all trains at the current station, and the arrival time and departure time of all trains at the current station that meet the driving safety constraints are determined, including: Determine the initial arrival time of all trains at the current station according to the driving safety constraints and the departure time of all trains at the previous station; For each target train, determine the track occupancy of the current station at the target time, the target time being the initial arrival time of the target train minus the given shortest departure-arrival interval time, and the target train is any train selected in sequence according to the departure order of all trains; When the track occupancy condition is that there is no idle track, the arrival time of the target train at the current station is updated according to the track occupancy condition, and the arrival time of all subsequent trains at the previous station that depart later than the target train at the current station is updated; When the track occupancy condition is that there is an idle track, a free track is randomly allocated to the target train, and the arrival time of the target train at the current station is the initial arrival time; The ready time of the target train is determined according to the driving safety constraint and the arrival time at the current station, and the departure time of the target train at the current station is determined according to the ready time, wherein the ready time represents the time when the train has completed the stop operation and is ready to leave the station.

10. The method according to claim 1, characterized in that The Markov decision process model for high-speed rail scheduling is constructed in the following way: The station state composed of train characteristics and station characteristics is used as the state of the Markov decision process model of the high-speed rail scheduling; The departure order of all trains at the current station is used as the action of the Markov decision process model of the high-speed rail scheduling; The process of switching from the current station state to the next station state based on the departure sequence of all trains at the station is used as the state transition of the Markov decision process model of the high-speed rail scheduling; In the case of the initial optimization of the train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train travel time; or, in the case of the re-scheduling of the train schedule, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station with the goal of minimizing the total train delay time.

11. The method according to claim 10, characterized in that In the case of the initial optimization of the train schedule, with the goal of minimizing the total train travel time, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station, including: Under the condition of satisfying driving safety constraints, with the goal of minimizing the total travel time of the train, a first objective function is constructed according to the travel time of the train between two stations, wherein the travel time of the train between the two stations includes: the minimum stop time at the station, the minimum running time in the section and the buffer time; The negative objective function corresponding to the first objective function is used as the reward value of the Markov decision process model of the high-speed rail scheduling.

12. The method according to claim 10, characterized in that In the case of rescheduling the train schedule, with the goal of minimizing the total train delay time, the reward value of the Markov decision process model of high-speed rail scheduling is calculated according to the departure order of all trains at the station, including: Under the condition of satisfying the driving safety constraint, with the goal of minimizing the total train delay time, a second objective function is constructed according to the planned time for the train to arrive at the station and the planned time for the train to leave the station, wherein the planned time for the train to arrive at the station and the planned time for the train to leave the station are the times obtained by the initial optimization of the train schedule; The negative objective function corresponding to the second objective function is used as the reward value of the Markov decision process model of the high-speed rail scheduling.

13. The method according to any one of claims 1 to 12, characterized in that: The driving safety constraints include: Running time constraints, which are used to constrain the train's stop time at each station and the running time of the section to not be less than a minimum time limit; Overtaking constraint, which is used to restrict the train from overtaking in the section between stations; A time interval constraint, wherein the time interval constraint is used to constrain the time interval between adjacent trains to be no less than a safety time interval, wherein the safety time interval includes a station arrival time interval, a station departure time interval, and a section tracking time interval; Station capacity constraints, where each track of the station can only be allocated to a single train in the same time period, and the time interval between the departure time of the previous train occupying the same track and the arrival time of the next train occupying the same track is not less than the minimum departure-arrival time interval that meets train safety; Arrival and departure time constraints, the arrival and departure times are used to address the problem of rescheduling train timetables, constraining that trains run according to the train timetable when no disturbance occurs; or to constrain that in the event of a disturbance, trains are only allowed to arrive early and are not allowed to depart earlier than the original departure time in the train timetable.

Images