High-speed railway train diagram rescheduling method based on multi-agent reinforcement learning

Abstract

This application relates to the field of intelligent scheduling technology and discloses a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning. The method includes: sequentially identifying all stations on the target passenger line from the station where the interference event occurs to the station preceding the terminal station as decision stations; constructing a local observation feature sequence for each train at each decision station; inputting the local observation feature sequence of each train at the current decision station into the trained agent network to obtain the expected dwell time of each train at the current decision station; integrating and coordinating the expected dwell times of all trains based on safety constraints to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and identifying the next station as the new decision station. This method enables real-time scheduling optimization of railway passenger lines experiencing sudden interference, improving the operational efficiency of the line.

Description

Technical Field

[0001] This application relates to the field of intelligent scheduling technology, specifically to a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning. Background Technology

[0002] Train Timetable Rescheduling (TTR) is a core method for rapidly restoring operational order and mitigating passenger impact after sudden disruptions to passenger lines. It aims to quickly restore the original train schedule by adjusting departure times, stop times, and departure sequences. In real-world scenarios, rescheduling decisions for high-speed trains must be made within a limited timeframe to prevent the spread of delays. Furthermore, the resulting scheduling plan should minimize disruption and improve operational efficiency and passenger travel experience.

[0003] Traditional railway train operation scheduling relies on manual methods, which suffers from low efficiency and poor quality of generated scheduling plans. Related solutions employ mathematical programming methods or heuristic algorithms for train scheduling; however, the computational complexity of mathematical programming increases dramatically with scale, making it difficult to meet real-time requirements, while heuristic solutions have poor generalization performance and struggle to adapt to different delay scenarios. Furthermore, current railway train operation scheduling is largely operation-oriented, aiming to reduce total train delay time without considering the impact of passenger delays, leading to poor operational efficiency and severely impacting passenger travel experience. Summary of the Invention

[0004] The purpose of this application is to provide a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning, so as to realize real-time, efficient and flexible scheduling of high-speed railway passenger trains, reduce the impact of passenger delays and improve operational efficiency.

[0005] To achieve the above objectives, the technical solution of this application is as follows: In a first aspect, embodiments of this application provide a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning, the method comprising: All stations on the target passenger route, from the station where the interference event occurred to the station preceding the terminal station, are sequentially identified as decision stations. For each decision station, a local observation feature sequence corresponding to each train is constructed; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time and number of disembarking passengers; The local observation feature sequences of each train at the current decision station are input into the trained agent network to obtain the expected stopping time of each train at the current decision station. The agent network and the hybrid network together constitute the policy network of the QMIX architecture. The policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning. The agent network is used to generate the expected stopping time based on the local observation feature sequences of each train. The hybrid network is used to collaboratively optimize the expected stopping time of each train based on the global state features. Based on the safety constraints of the target passenger route, the expected stop time of all trains at the current decision station is integrated and coordinated to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and the next station is determined as the new decision station.

[0006] Secondly, embodiments of this application provide a high-speed railway train timetable rescheduling device based on multi-agent reinforcement learning, used to implement the steps in the method provided in the first aspect of this application. The device includes: The information construction module is configured to sequentially identify all stations on the target passenger route, from the station where the interference event occurred to the station preceding the terminal station, as decision stations; for each decision station, a local observation feature sequence corresponding to each train is constructed; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time, and number of disembarking passengers; The decision-making module is configured to input the local observation feature sequences of each train at the current decision station into a trained agent network to obtain the expected stopping time of each train at the current decision station. The agent network and the hybrid network together constitute the policy network of the QMIX architecture. The policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning. The agent network is used to generate the expected stopping time based on the local observation feature sequences of each train. The hybrid network is used to collaboratively optimize the expected stopping time of each train based on global state features. The environmental scheduling module is configured to integrate and coordinate the expected stop time of all trains at the current decision station based on the safety constraints of the target passenger line, so as to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and to determine the next station as the new decision station.

[0007] Thirdly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps in the method provided in the first aspect of embodiments of this application.

[0008] This solution pre-trains a strategy network based on minimizing passenger weighted delay time as the optimization objective. This ensures that the decisions made by the trained agent network during the application phase are more biased towards minimizing passenger delays. Consequently, the final global coordination of actual scheduling decisions prioritizes the departure times of trains with a higher number of passengers affected by delays, thereby reducing overall passenger delays on passenger routes and improving the passenger travel experience.

[0009] During the offline training phase, this solution employs the CTDE framework to train a policy network based on the QMIX architecture. By leveraging global features through a hybrid network for collaborative optimization, the agent network learns to make locally optimal decisions to approximate the global optimum when faced with sudden disturbances. Furthermore, in the application phase, for diverse delay scenarios on passenger routes, the agent network can make rapid decisions based on local observations and achieve multi-agent joint decisions that approximate the global optimum. Compared to traditional scheduling schemes, this solution eliminates the need for complex global communication or online collaborative computation during online scheduling in the application phase. It also demonstrates fast response efficiency and high decision-making quality in rescheduling after disturbances, meeting the demands for minimizing passenger delays, high-quality decision-making, and real-time response. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning proposed in an embodiment of this application; Figure 2 This is a schematic diagram of constructing a global directed graph in one embodiment of this application; Figure 3 This is a schematic diagram of a strategy network trained using the CTDE framework in one embodiment of this application; Figure 4 This is a schematic diagram of a high-speed railway train timetable rescheduling device based on multi-agent reinforcement learning proposed in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0014] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects as detailed in this application.

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0017] Traditional train scheduling methods that rely on manually designed fixed rules often produce poor-quality scheduling schemes. Improvements often involve mathematical programming and heuristic algorithms. However, the computational complexity of mathematical programming increases dramatically with scale, making it difficult to meet real-time requirements. While heuristic algorithms (such as genetic algorithms and hybrid particle swarm optimization) offer high response speeds, they suffer from poor generalization across different delay scenarios. Reinforcement learning-based train-to-train (TTR) schemes typically employ a single-agent framework, limiting their ability to coordinate multiple trains and facing the curse of dimensionality in state representation, making them unsuitable for multi-train railway passenger transport scenarios. Related urban rail transit (such as subway systems) scheduling schemes are operation-oriented, lacking explicit consideration of passenger demand. Furthermore, the periodic service and unpredictable passenger flow of urban rail transit systems prevent these schemes from being applied to railway passenger transport scenarios where trains and passengers are strongly coupled.

[0018] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] Figure 1 This is a flowchart of a high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning proposed in an embodiment of this application. Figure 1 As shown, the method includes: S1: Select all stations on the target passenger route, from the station where the interference event occurred to the station preceding the terminal station, as decision stations in sequence. S2: For each decision station, construct a local observation feature sequence corresponding to each train; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time and number of disembarking passengers; S3: Input the local observation feature sequence of each train at the current decision station into the trained agent network to obtain the expected stopping time of each train at the current decision station; the agent network and the hybrid network together constitute the policy network of the QMIX architecture; the policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning; the agent network is used to generate the expected stopping time based on the local observation feature sequence of each train; the hybrid network is used to collaboratively optimize the expected stopping time of each train based on the global state features; S4: Based on the safety constraints of the target passenger line, integrate and coordinate the expected stopping time of all trains at the current decision station to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and determine the next station as the new decision station.

[0020] This application embodiment considers the rescheduling of train timetables in a single direction (upward or downward) of a two-way double-track high-speed railway passenger dedicated line. The target passenger line has a seat reservation mechanism, and the passenger's departure-arrival (OD) information and the train number are predetermined through the ticketing system.

[0021] In this embodiment, a station-by-station iterative decision-making method is adopted. If a delay is caused by an interference event at a station in the target passenger line, starting from that station and continuing to the station before the terminal station (M-1), each station is determined as the decision station k according to the direction of travel. At each time, a corresponding joint decision is made for the current decision station until all trains arrive at the terminal station.

[0022] Specifically, the process of making a joint decision each time is as follows: First, based on the node information of each train on the line at the current decision station and all subsequent stations (including the terminal station), local observation features of the train at the decision station and each downstream station are constructed and combined according to the arrival order to obtain the local observation feature sequence of each train. The node information includes: planned arrival time, arrival delay time, and number of disembarking passengers. In this embodiment, the number of disembarking passengers for a train on the target passenger line at station A refers to the number of passengers who boarded the train at a station before station A and disembarked at station A. In practical applications, the planned arrival time, the number of boarding passengers at each station, and the number of disembarking passengers are known information in advance. After an interference event occurs, the actual arrival of each train at the decision station is affected by the interference, resulting in a delay. Based on the actual arrival time of the train and the known planned arrival time, the actual arrival delay time of the train can be calculated.

[0023] Then, the local observation feature sequences of each train are input into the pre-trained agent network, which makes the local optimal decision for the corresponding train based on the local observation feature sequences of each train (i.e., outputs the expected stopping time of the train at the current decision station).

[0024] Finally, based on the safety constraints of the target passenger route (such as station track capacity, departure intervals, etc.), the expected stop time of each train at the decision station is integrated and coordinated to determine the actual departure time of each train at the decision station and the actual arrival time at the next station.

[0025] After making a joint decision for the current decision station k, the iterative process continues, determining the next station k+1 as the new decision station, and continuing to execute the joint decision operation station by station until all trains reach the terminal station.

[0026] Actual railway passenger lines are typically large in scale, making it impossible for a single agent to acquire global information. Furthermore, single-agent reinforcement learning has limitations in collaborative optimization when faced with strong coupling between multiple trains and stations. Therefore, this application employs a multi-agent reinforcement learning approach. A policy network based on the information of stations and trains along the target passenger line, using a QMIX architecture (consisting of an agent network making local decisions for trains and a hybrid network for joint collaborative optimization), is pre-constructed. This policy network is then trained using a Centralized Training with Decentralized Execution (CTDE) framework. During training, minimizing passenger-weighted delay time is explicitly introduced as the optimization objective, enabling the trained agent network to independently make locally optimal decisions for multiple trains to jointly approximate the global optimum.

[0027] The core of QMIX-based multi-agent reinforcement learning is to decompose the global state-action Q-value into a monotonic combination of the local Q-values ​​of each agent, making the greedy choice during decentralized execution equivalent to the globally optimal action. In this embodiment, each train is treated as an independent agent. The agent network generates locally optimal decisions based on the local observation feature sequences of each agent (i.e., the train). Furthermore, a hybrid network collaboratively optimizes the locally optimal decisions of each agent based on global state features to approximate the globally optimal solution. In the application phase, the trained agent network predicts the expected stopping time for each train. The joint decision of all trains automatically approximates the global optimum, thereby achieving online, real-time, and efficient train rescheduling operations, meeting the needs of minimizing passenger delays, high-quality decision-making, and real-time response.

[0028] As one implementation of this application, for each decision station, a local observation feature sequence corresponding to each train is constructed, including: Based on the train's planned arrival time, arrival delay time, and number of disembarking passengers at the decision station, local observation features of the train at the decision station are constructed. Based on the planned arrival time, arrival delay time, and number of disembarking passengers of the train at each station downstream of the decision station, local observation features of the train at each station downstream of the decision station are constructed; wherein, the arrival delay time of the train at all stations downstream of the decision station is an estimated value obtained by iterative calculation based on the arrival delay time of the train at the decision station. Based on the local observation features of the train at the decision station and all downstream stations, a local observation feature sequence corresponding to the train is constructed.

[0029] In this embodiment, during the training and application phases, the local observation feature sequences of each train at the current decision station are constructed as input data for the agent network. Each train is considered an independent agent, and the local observation of agent i is defined as the node feature sequence of that train from the current decision station k to the destination station M: .

[0030] Each node This corresponds to a [train-station] event; the local observation feature vector of train i at station k. It includes three parameters: planned arrival time. Arrival delay time (in, (This refers to the actual arrival time of train i at station k) and the number of passengers disembarking at station k. (in, To take the train from the station The number of passengers who disembarked at station k.

[0031] It is worth noting that the arrival delay of each train at the current decision station k is a precise value, while the delay at downstream stations after the decision station is a more precise value. Delay Iterative estimation using the arrival delay time from the previous station: ; ; in, This represents the index of all downstream stations of the current decision station k; The buffer time for absorbing delays is calculated as follows: ; in, For train i at the station The planned arrival time; For train i at the station The section between the previous station The shortest running time; For train i at the station The scheduled departure time.

[0032] This approach enables agents to predict the propagation trend of delays caused by decision-making stations in subsequent downstream stations.

[0033] As one embodiment of this application, before inputting the local observation feature sequence of each train at the current decision station into the trained agent network, the method further includes: Construct a target model based on minimizing the passenger-weighted delay time of all trains on the target passenger route as the optimization objective; An environment simulator is constructed based on the safety constraints of the target passenger route; the safety constraints include: running time constraints, stop duration constraints, time interval constraints, station capacity constraints, and disturbance constraints. The target model is transformed into a distributed partially observable Markov decision process, wherein each train is considered as an independent agent. A global directed graph is constructed based on the local observation feature sequence of each train at the decision station, and global state features are extracted from the directed graph through a graph isomorphic network. Based on the local observation feature sequences of each agent and the global state features, the policy network is trained using the CTDE framework. During the training process, the environment simulator generates the state of each agent at the next station of the current decision station based on the joint actions of each agent.

[0034] In this embodiment, firstly, based on the station and train information of the target passenger route, a target model is constructed with the goal of minimizing the passenger-weighted delay time of all trains on the target passenger route. Then, this optimization problem is transformed into a Decentralized Partially Observable Markov Decision Process (Dec-POMDP), where each train acts as an independent agent. Next, a directed graph is designed to represent the global state, and a Graph Isomorphism Network (GIN) is used to extract the coupling relationships and complex operational constraints between trains, thereby enhancing the global state representation capability. Finally, within the CTDE framework, the policy network built on the QMIX architecture is trained offline end-to-end to obtain an agent network that can be applied online.

[0035] Since trains in the up and down directions operate independently on a double-track railway passenger line, either the up or down line can be considered an independent unidirectional line. This embodiment uses a mixed-integer linear programming mathematical model with the objective of minimizing passenger-weighted delay for a single-directional passenger line. The target passenger line contains N trains and M stations. Ideally, each train in the train set T... According to the planned operation schedule, the train will pass through each station in station set S in sequence. When a station experiences a sudden disruption, the train schedule needs to be adjusted in real time to restore train operation.

[0036] Key pre-known parameters of the line include: the set of tracks at all stations that can accommodate train stops. The number of tracks at station k The shortest travel time of train i in section k between station k and the next station k+1. The shortest stopping time of train i at station k Maximum stop time The shortest interval between adjacent trains arriving at the same station. The shortest interval between adjacent trains departing from the same station And, the shortest departure-arrival time interval between adjacent trains on the same track. The planned arrival time of train i at station k Scheduled departure time and initial delay time Take train i from a station before station k. Number of passengers getting off at station K The time of the interference is D.

[0037] In addition, with Correspondingly, an initial delay indicator variable was also set. , is used to indicate whether train i at station k has an initial delay. If train i has an initial delay at station k, it is 1, and if it does not have an initial delay, it is 0.

[0038] The decision variables that need to be determined include: the actual arrival time of train i at station k. Actual departure time Departure sequence indicator variable If train i, which departs earlier in the original departure sequence, leaves station k before train j, the indicator variable is 1; otherwise, the indicator variable is 0. (Track occupancy indicator variable) This variable is used to indicate whether a track is occupied. If track p at station k is occupied by train i, the variable is 1; otherwise, the variable is 0.

[0039] In this embodiment, the optimization objective is to minimize the overall weighted delay time of all passengers on the target passenger route. The objective function (model) is constructed by summing the arrival delay of each train at each station multiplied by the number of passengers disembarking at that station: ; Where N is the number of trains on the target passenger route; M is the number of stations on the target passenger route; Let i be the actual arrival time of train i at station k; Let i be the planned arrival time of train i at station k; To take the train from the station The number of passengers who disembarked at station k.

[0040] This target model imposes a greater penalty on delays that affect more passengers, guiding scheduling strategies to prioritize on-time performance for trains and stations with high passenger volume.

[0041] To ensure that the scheduling decisions of each train comply with the safe operation requirements of the target passenger line, this embodiment constructs an environment simulator based on multiple safety constraints of the line. During the reinforcement learning training phase, the environment simulator generates the state of each agent at the next station after the current decision station, based on the joint actions of all trains. In the application phase after training, the environment simulator, under safety constraints, integrates and coordinates the expected stopping times of each train (i.e., the set of local decisions made by the agent network) to determine the new departure times of each train and the actual arrival times at the next station after an interference occurs.

[0042] As one embodiment of this application, an environment simulator is constructed based on the safety constraints of the target passenger route, including: Based on the shortest running time between adjacent stations, a running time constraint is constructed; Based on the upper limit and lower limit of the stop duration, a stop duration constraint is constructed; Based on the arrival time interval and departure time interval of adjacent trains at the same station, the shortest interval duration corresponding to the departure time interval, and the uniqueness of the departure order, time interval constraints are constructed. Based on the shortest time interval between the departure of the previous train and the arrival of the next train on the same track, the track capacity of the station, and the uniqueness of track occupancy, a track capacity constraint is constructed. Construct disturbance constraints to limit the departure time of each train to no earlier than the original time after the occurrence of a disturbance event; An environment simulator is constructed based on the aforementioned runtime constraints, stop duration constraints, time interval constraints, track capacity constraints, and disturbance constraints.

[0043] In one embodiment, an environment simulator is constructed based on the multi-dimensional safety constraints of the target passenger route, specifically considering the following types of constraints: (1) Running time constraint: The running time of a train between two adjacent stations shall not be less than the minimum running time of the section specified therein, that is: .in, To determine the actual arrival time of the next station k+1 after station k; This refers to the actual departure time of station k; (2) Stopping time constraints: The actual stopping time of a train at a station shall not be less than the lower limit of the stopping time, and at the same time, to prevent excessive occupation of the limited platform capacity, it shall not exceed the upper limit of the stopping time. Considering the direct impact of the initial delay on the train, therefore: ; Where U is a sufficiently large positive number; This is the lower limit of the dwell time of train i at station k; This refers to the actual departure time of station k; The actual arrival time of station k; is the initial delay indicator variable, which is 1 if train i has an initial delay at station k, and 0 otherwise; TD is the upper limit of the stop time. (3) The arrival and departure time intervals of adjacent trains at the same station should exceed the minimum interval length to ensure safety and avoid collisions. Therefore: Arrival time interval constraint: ; in, , These are the actual arrival times of trains i and j at station k, respectively; The shortest interval between adjacent trains arriving at the same station; U is a sufficiently large positive number. This is an indicator variable for the departure order of trains i and j in station k-1. If train i, which is earlier in the original departure order, leaves station k before train j, which is later in the original order, the value is 1; otherwise, the indicator variable is 0. Departure time interval constraints: ; in, , These are the actual departure times of trains i and j at station k, respectively; The shortest interval between adjacent trains departing from the same station; U is a sufficiently large positive number. This is an indicator variable for the departure order of trains i and j in station k. If train i, which is earlier in the original departure order, leaves station k before train j, which is later in the original order, the value is 1; otherwise, the indicator variable is 0. Furthermore, to ensure the uniqueness of the train sequence, the following expression must be satisfied: ; ; (4) Track capacity constraint: At each station, a train needs to be allocated a track to stop or pass through. The time interval between the departure of the previous train and the arrival of the next train on the same track must not be less than the shortest departure-arrival time interval. Therefore: ; Furthermore, each train can only occupy one track at each station, that is: ; (5) Disturbance constraints: Before a disturbance occurs, the train runs as planned. After a disturbance occurs, the train must not depart earlier than the original scheduled time to prevent passengers from missing their trains. Therefore, the following applies: ; Where D represents the time when the interference occurs.

[0044] As one embodiment of this application, based on the safety constraints of the target passenger line, the expected dwell time of all trains at the current decision station is integrated and coordinated to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, including: Input the expected dwell time of all trains at the current decision station into the environment simulator, and then perform the following steps through the environment simulator: Based on the expected departure time of each train and the track capacity of the current decision station, determine the departure order of each train at the current decision station. Based on the departure order of each train at the current decision station, and provided that the aforementioned safety constraints are met, the actual departure time of each train at the current decision station and the actual arrival time at the next station are determined.

[0045] In one embodiment, during the application phase, an environment simulator integrates and coordinates the locally optimal decisions (i.e., expected stopping times) of each train output by the agent network to determine the actual departure and arrival times of each train. Specifically, the expected stopping times of all trains output by the agent network are input into the environment simulator. Based on pre-set safety constraints, the environment simulator first determines the departure order of each train at the decision station according to the expected stopping times of each train and the track capacity of the decision station. Then, based on the departure order of each train and the pre-set safety constraints, the environment simulator determines the actual departure time of each train at the decision station and the actual arrival time of the train at the next station.

[0046] As one embodiment of this application, the target model is converted into a distributed partially observable Markov decision process, including: The arrival and departure times of all trains on the target passenger route at the same station are considered as a decision event, and each train is considered as an independent intelligent agent. Tuples are constructed based on distributed partially observable Markov decision processes, and the tuples include the following elements: The collection of intelligent agents consists of the intelligent agents corresponding to all trains; The local observation space is composed of the local observation feature sequences of each agent. The global state space and joint observation space are composed of the local observation spaces of all agents. The joint action space consists of the selectable actions of all agents; The state transition is achieved by the environment simulator based on the joint action selected by all agents, the global state of the current decision station, the track capacity, and the safety constraints, to obtain the global state of the next station of the current decision station. Instant rewards are built based on the weighted delay time of passengers at the next station after the current decision station. A discount factor is used to control the importance of future rewards relative to the immediate rewards.

[0047] In this embodiment, starting from the affected station, each train needs to adjust its stopping time and arrival time at the next station station station by station. Since each train can only observe its own information, and all trains are coupled together through safety constraints, this scheme remodels the target model as a distributed partially observable Markov decision process (Dec-POMDP). Specifically, the arrival and departure times of all trains at each station are considered as decision events, and each train is considered as an independent intelligent agent. The target model is formalized as tuples. The details are as follows: (1) Set of finite intelligent agents It consists of intelligent agents corresponding to all N trains; (2) The local observation space consists of the local observation feature sequences of each agent, including the train's planned arrival time at the station, arrival delay time, and number of disembarking passengers. The local observation space of agent i is defined as follows: ,use This represents the local observation characteristics of train i at station k; (3) Global state space It consists of the local observation space of all agents, and the global state of decision station k. ; (4) The joint observation space is composed of the local observation spaces of all agents; (5) For joint action space, This represents the set of possible actions for agent i. The actions of agent i at decision station k. Indicates relative to its actual arrival time The expected stop time can be used to calculate the expected departure time. Each action originates from a discrete set. ( The maximum stop time is selected. This embodiment uses discrete stop time as the optional action, which effectively controls the scale of the action space while conforming to actual operating habits, saving computing resources and improving training efficiency.

[0048] (6) Let be the state transition probability. State transitions describe the state transition based on joint actions. From the current global state Transition to the next state The process is generated through an environment simulator. The environment simulator iteratively executes a two-stage process to determine the actual departure time of the current station and the actual arrival time of the next station, until all trains at the current decision station have departed.

[0049] Specifically, the environment simulator determines the departure order in the first stage: at the current decision station k, the expected departure time for each train is first calculated. Due to limited station track capacity, the train with the earliest actual arrival time is selected from all trains that have arrived but not yet departed. Trains form a candidate set. Then, the train i with the earliest expected departure time in the candidate set is selected as the next departing train.

[0050] In the second phase, the environment simulator calculates a feasible timetable. Based on the joint actions (stop time) of all agents at the current decision station, the global state of the decision station, track capacity, and safety constraints, it calculates the global state of the next station (i.e., the planned arrival time, arrival delay time, and number of disembarking passengers for each train). According to the departure sequence of each train, it sequentially checks whether there is an available track for each train to stop or pass through at station k. If no track is available, its arrival time must be postponed to a later train. Departure from the station must also satisfy the shortest departure-arrival time interval between adjacent trains on the same track. Then calculate the actual departure time of train i at the current station. : ; in This is the previous train to depart. After departure, train i will arrive at the next station. The actual arrival time is: .

[0051] This two-stage mechanism decouples the agent's action decisions from the satisfaction of safety constraints. The agent only needs to select discrete stop times, and the environment automatically ensures the feasibility of the scheduling scheme. This embodiment integrates station delay features and passenger delay information during state transitions, enabling the trained agent network to adapt to different interference scenarios and passenger needs.

[0052] Furthermore, when constructing the environment simulator in the above embodiments, it is also necessary to define decision variable constraints based on the inherent attributes of the agent's action space to limit the value range of continuous and integer decision variables. Specifically, the actual arrival time and actual departure time are continuous decision variables, while the departure sequence indication and track occupancy indication are represented using integer decision variables 0 / 1, which must satisfy the following constraints: Continuous decision variables: ; Integer decision variables: .

[0053] (7) The global reward function is shared by all agents. To guide the multi-agent system towards minimizing passenger-weighted delay, the global reward is defined as the negative value of the objective function: ; (8) This is a discount factor used to control the importance of future rewards relative to the immediate rewards during reinforcement training.

[0054] At the current decision-making station k, all trains make decisions based on their respective local observations. Make action decisions independently Subsequently, the environment simulator determines the current global state. Unified processing of all train actions at the current station, providing global rewards. And transfer to the next decision station. The iterative process continues until the complete train run is finished.

[0055] As one implementation of this application, a global directed graph is constructed based on the local observation feature sequences of each train at the decision station, including: Let a combination of a train and a station correspond to a node. Based on all stations from the decision station to the terminal station and the combinations of each train, generate nodes corresponding to each combination. The feature of each node corresponds to the local observation features of the train at the corresponding station in the combination of nodes. To generate the global directed graph of the decision station, all nodes are connected by directed edges. Specifically, this includes: using a first type of directed edge to connect nodes that arrive at adjacent stations in sequence with the same train, wherein the direction of the first type of directed edge is from the downstream station to the upstream station; and using a second type of directed edge to connect nodes that arrive at the same station in sequence with different trains, wherein the direction of the second type of directed edge is from the train that arrives at the station first to the train that arrives at the station later.

[0056] Because numerous factors influence train scheduling results in this scheme, such as planned timetables, delay scenarios, and passenger data, and because complex coupling relationships exist between trains, this complex and abundant information is difficult to fully integrate. In this embodiment, to enhance the ability to represent the global state of the target passenger line, a directed graph is used to structurally model the global state information, and a graph neural network is used for automatic feature extraction, thereby effectively capturing the coupling constraints between trains.

[0057] Specifically, each train and each station is paired up as an event node, and the global state of the current decision station k in the target passenger line is considered. Construct a global directed graph for decision station k. Node set It includes all [train-station] event nodes from the current station k to the destination station M. The characteristics of each node are defined in accordance with the local observation characteristics of the train at the corresponding station, i.e., node... The characteristics are: To encode the key dependencies between nodes, in this embodiment, the set of directed edges in the global directed graph... There are two categories. The first category is running constraint edges, which connect nodes of the same train at adjacent stations. The information transmission direction is from the downstream station to the upstream station, so that the information of subsequent nodes can assist the current decision. The second category is arrival order edges, which reflect the actual arrival order of trains at the current decision station. They are from the first arriving train to the last arriving train, capturing the arrival-departure competition relationship between trains.

[0058] It is worth noting that in this embodiment, when constructing the global directed graph, only the nodes of the current decision station and all downstream stations are considered, without considering the upstream nodes before the decision station. This is because the global state influence of the upstream stations is already reflected in the arrival delay of the current decision station. Furthermore, this approach effectively reduces the state space and further decreases the computational cost of training.

[0059] Figure 2 This is a schematic diagram illustrating the construction of a global directed graph in one embodiment of this application. For example... Figure 2 As shown, the current decision station is station 2. At this time, a global directed graph is constructed based on the nodes of station 2 and all downstream stations (station 3, station 4) and each train in pairs. Among them, the first type of directed edge (solid arrow) represents the information transmission order of the same train in adjacent stations (from downstream station to upstream station); and the second type of directed edge (dashed arrow) represents the actual arrival order of different trains to the same station (from the first arriving train to the last arriving train).

[0060] After constructing a global directed graph of decision station k to represent the global state of the current target passenger route, a graph isomorphic network (GIN) is used to extract features from this global directed graph to obtain a global state feature representation. Specifically, based on the global directed graph... An L-layer graph isomorphic network is used to iteratively update node embeddings, where the first... The layer update rules are as follows: ; in, It is a node Abbreviation; Represents a node In the Embedding of layered GIN networks; This is the initial embedding generated by linearly projecting the original node features; For nodes The set of neighbors; For node v's neighbor node u in the th... Embedding of layered GIN networks; The parameter is Multilayer perceptron; It is a learnable scalar; Indicates batch normalization; This is the activation function.

[0061] After L layers of iteration in the GIN network, the global graph embedding is obtained by averaging the embeddings of all nodes. (i.e., global state characteristics): ; This graph is embedded during training as input to the policy network, providing global state information to the hybrid network.

[0062] As one embodiment of this application, the policy network is trained using the CTDE framework based on the local observation feature sequences of each agent and the global state features, including: The inverse gated recurrent unit of the agent network processes the local observation feature sequence of each agent and outputs the Q value of the corresponding action; where different actions correspond to different expected stop times. use Greedy strategy, controlling each agent with probability Randomly select actions, and with probability Choose the action with the highest current Q value; where the probability... The probability value decreases linearly from the initial probability value to the first probability value as the number of training rounds increases; Based on the actions chosen by all agents, the next state is generated through the environment simulator. Calculate the immediate reward based on the next state, and determine the local observation feature sequence corresponding to each agent in the next state; Training samples are constructed based on the current state, action, reward, and next state of each agent. Based on multiple batches of training samples, the hybrid network integrates the Q values ​​of each agent based on the global state features corresponding to the training samples, and calculates the joint state action value corresponding to each empirical tuple; the hybrid network includes a feedforward neural network, and the weights and biases of each feedforward layer of the feedforward neural network are dynamically generated by the supernetwork according to the global state features, and the weights are non-negative values. The mean squared error loss is calculated based on the joint state action value, and the policy network is updated with the goal of minimizing the mean squared error loss.

[0063] In this embodiment, a policy network based on the QMIX architecture is constructed during the training phase, consisting of an agent network and a hybrid network. The agent network utilizes an inverse GRU to process local observations of each train; the hybrid network dynamically generates weights through a supernetwork to achieve monotonic decomposition of the joint state-action value function Q-value.

[0064] Figure 3 This is a schematic diagram of a strategy network trained using the CTDE framework in one embodiment of this application. Figure 3 As shown, at the current decision station k, the agent network determines the outcome based on the local observation feature sequence of each agent (train). The system outputs the Q-values ​​corresponding to each action. The agent network uses a back-gated recurrent unit (GRU) to transmit features back from the destination station M to the current station k station by station, thereby integrating all information from the decision station to the destination station and improving the foresight of local decisions. For train i, the Q-values ​​corresponding to all currently selectable actions are calculated as follows: ; MLP stands for Multilayer Perceptron; GRU stands for Reverse Gated Recurrent Unit. The local observation feature sequence of train i at decision station k.

[0065] In this embodiment, the hybrid network includes a feedforward neural network and a supernetwork, wherein the feedforward neural network stores the independent Q-values ​​of N train agents. The functions are integrated into a global joint state-action value function using a linear weighting method. To ensure monotonicity constraints The weights W and biases b of each feedforward layer are determined by the supernetwork (parameterized as follows). Dynamically generated. The hypernetwork is embedded with a global graph extracted from the global directed graph. The generated weights W are ensured to be non-negative through an absolute value activation function. By jointly training the agent network and the hybrid network, during the distributed execution phase in the application stage, each agent can approach the global optimum by making greedy choices independently through the agent network.

[0066] Policy network parameters are determined by The system is composed of multiple trains deployed in a distributed, centralized training and distributed execution (CTDE) framework, ensuring decision quality while meeting the practical needs of distributed multi-train deployments in real-world applications. During the training phase, global state features are used to optimize the scheduling strategy of the agent network; during the execution phase, each train agent independently determines its desired stopping time based solely on its local observations. During training, each agent is controlled using... Greedy strategy for action selection: The agent selects actions based on probability each time. Randomly select actions, and with probability Choose the action with the highest Q value.

[0067] In this embodiment, an initial probability value and a minimum probability threshold (i.e., a first probability value) are pre-specified before reinforcement learning training, and the probability during the training phase is... As the number of training rounds increases, from the initial probability value Linear decay to the first probability value In this way, the agent network fully explores the state space in the early stages of training, and gradually converges to a greedy strategy in the later stages. This approach can further improve the robustness of the agent network and adapt to different interference delay scenarios and unexpected situations.

[0068] After each action is performed, the experience tuples generated through interaction with the environment simulator (including the current state, action, global reward, and next state) are stored as training samples in the replay buffer. When updating the network, multiple batches of training samples are sampled from the replay buffer, and parameters are updated by minimizing the mean squared error loss. Mean squared error loss. The calculation method is as follows: ; in, Batch size; This represents the current global state corresponding to the b-th training sample. Let be the local observation feature sequence of agent i in the current state in the b-th training sample; For the b-th training sample, select the action to be performed by agent i in the current state; For the b-th training sample, the online policy network (with parameters) Based on the current global state Local observations of all intelligent agents and joint actions The calculated joint state-action Q-value; The time-series difference objective is calculated as follows: ; in, This indicates that in the b-th training sample, the target network (parameters) Based on the next global state Local observations of all agents in the next state and joint actions The calculated joint state-action Q-value. The target network and the online policy network have the same structure and parameters. Every F training epochs, the parameters of the online policy network are... Copy and update once (i.e.) F is usually set to 100) to alleviate the overestimation of Q value caused by temporal difference bootstrapping and to enhance the stability of training convergence.

[0069] In one embodiment, the algorithm flow for updating the policy network is as follows: Input: Policy network parameters Target network parameters Training epochs E, target network update frequency F, batch size B, exploration parameters and Learning rate ; Output: Learned policy network parameters ; (1) Initialize policy network parameters Target network parameters Playback buffer D; (2) In each round of training, repeat the following process until all trains reach the final station: Each agent constructs its own local observation feature sequence based on the current environmental state. The agent network outputs the Q-values ​​of all possible actions (expected stop duration) based on each agent's local observation feature sequence. Each agent selects actions using a greedy strategy. The joint actions of all agents are input into the environment simulator, which generates a new state and constructs the local observation feature sequences for each train corresponding to the next state. The global reward is calculated based on the new state. The current state of each agent, the local observation feature sequences of all agents, the selected actions, the obtained global reward, and the new state and local observation feature sequences are stored as a complete experience tuple in the replay buffer.

[0070] (3) After collecting a certain number of experience tuples, a batch of training samples (experience tuples) is randomly selected from the replay buffer. For each training sample in this batch of data, the agent network first calculates the Q value of each agent based on the observations in the tuple, and then integrates the Q values ​​of each agent into a global joint action value by using the global state features in the tuple through the hybrid network.

[0071] Then, the temporal difference objective is calculated, and the mean squared error loss between the current joint action value prediction and the temporal difference objective is calculated based on the temporal difference objective. Using the calculated mean squared error loss, all parameters of the policy network (including the agent network and the hybrid network) are updated using gradient descent. Multiple rounds of updates are performed iteratively using multiple batches of empirical tuples, with a fixed number of training rounds between each update to synchronously update the parameters of the policy network to the target network, thus stabilizing the training process.

[0072] (4) After training, in the application phase, each agent can meet the high-quality, real-time response requirements of online rescheduling by relying only on the Q value directly output by the agent network and executing a greedy strategy to select the corresponding expected downtime.

[0073] To verify the effectiveness of this solution, this embodiment uses a real-world passenger transport route as an example for testing. This route has 11 stations and a total length of 688 kilometers. Based on the actual planned operating schedule for August 2025, 21 trains departing from Beijing South Station between 8:30 and 11:30 were selected for study. The basic parameters of this route include the capacity of each station and the shortest travel time between adjacent stations, as shown in Table 1 below.

[0074] Table 1

[0075] This experiment uses a reasonable passenger flow assumption: if the train is scheduled to stop at both stations, then passengers taking train i from station i will be more likely to travel to both stations. The number of passengers getting off at station k is set as follows Other operating parameters are set as follows: the shortest interval between adjacent trains arriving at the same station and departing from the same station. / The shortest departure-arrival time interval between adjacent trains on the same track is 4 minutes. The minimum stop time for a train at a station is 2 minutes. The maximum stop time is 2 minutes. It lasts for 15 minutes.

[0076] To verify the robustness of this scheme, four representative delay scenarios were set up as test datasets in the experiment, covering initial delays of varying severity and impact. These delays were caused by station equipment failures or section congestion, resulting in delays in the departure times of multiple trains, which are typical disturbances in the daily operation of high-speed rail. The specific settings are shown in Table 2 below. In this scheme, the hyperparameter settings for training the policy network are shown in Table 3 below.

[0077] Table 2

[0078] Table 3

[0079] Two training paradigms were employed during training: ST training and GT training. ST training was used to train the policy network from scratch for fixed delay scenarios, while GT training was used to train the policy network offline for diverse random delay scenarios.

[0080] This experiment used two commonly used manual scheduling rules as comparison schemes: FCFS (Following the actual arrival order) and FSFS (Prioritizing trains with earlier scheduled departure times). The experimental comparison results of the solution quality, computation time, and relative differences of the three schemes under four delay scenarios are shown in Table 4 below. The relative difference is calculated based on the optimal scheme for each scenario (GMARL+ST: the scheme using a specific training paradigm). The target value (unit: min·person) is the objective function. The value of represents the cumulative weighted delay time caused by all trains to all passengers at all stations under this scheduling scheme. The smaller the target value, the smaller the impact of the scheduling scheme on the overall passenger delay, i.e., the higher the solution quality. GAP (Gap) represents the relative optimization gap between the scheme and the optimal solution (GMARL+ST) used for reference. The smaller the GAP, the closer the scheme is to the optimal solution of GMARL+ST. The GAP calculation method for each scheme in this experiment is as follows: .

[0081] Table 4

[0082] As shown in Table 4, the solution quality (i.e., target value) of manually calculated scheduling rules (FCFS, FSFS) is not ideal because these rules are based on fixed priority logic, which neither considers passenger demand nor captures the global coupling relationship between trains, leading to severe delay propagation and a large number of passenger delays. In contrast, the proposed solution (GMARL+GT, GMARL+ST in Table 4) successfully balances computational efficiency and solution quality, generating a high-quality rescheduling operation graph within a limited time. It significantly outperforms the manual rules in all four test scenarios, reducing the target value by more than 10%.

[0083] The training time and inference time for the four delay scenarios are shown in Table 5 below, under two training paradigms: specific training and generalized training.

[0084] Table 5

[0085] As shown in Table 5, comparing the two training paradigms, this scheme can generate the optimal rescheduling scheme under the specific training paradigm, while the policy network obtained by the generalized training strategy performs slightly weaker. The performance difference stems from the fact that generalized training optimizes the policy for diverse delay scenarios, resulting in a more robust and adaptable scheduling strategy, but with slightly weaker specificity for particular instances. However, the scheme using generalized training has strong practical application potential. After completing one-time training in the offline stage, it can achieve zero-shot online inference when facing sudden interference, without needing to retrain for each new scenario, significantly improving computational efficiency and meeting strict real-time requirements. The slight loss in solution quality is acceptable in practical applications.

[0086] In summary, experimental results demonstrate that this solution can achieve high-quality online real-time scheduling under various delay scenarios in railway passenger lines. The scheduling strategy obtained under the generalized training paradigm can achieve zero-sample online inference without fine-tuning when facing sudden interference, demonstrating strong application potential.

[0087] Based on the same inventive concept, one embodiment of this application provides a high-speed railway train timetable rescheduling device. Figure 4 This is a schematic diagram of a high-speed railway train timetable rescheduling device 100 according to an embodiment of this application. Figure 4 As shown, the device includes: The information construction module 101 is configured to sequentially identify all stations on the target passenger route, from the station where the interference event occurs to the station preceding the terminal station, as decision stations; for each decision station, a local observation feature sequence corresponding to each train is constructed; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time, and number of disembarking passengers; The decision module 102 is configured to input the local observation feature sequence of each train at the current decision station into the trained agent network to obtain the expected stopping time of each train at the current decision station. The agent network and the hybrid network together constitute the policy network of the QMIX architecture. The policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning. The agent network is used to generate the expected stopping time based on the local observation feature sequence of each train. The hybrid network is used to collaboratively optimize the expected stopping time of each train based on the global state features. The environmental scheduling module 103 is configured to integrate and coordinate the expected stop time of all trains at the current decision station based on the safety constraints of the target passenger line, so as to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and to determine the next station as the new decision station.

[0088] As one embodiment of this application, the information construction module 101 is specifically configured to perform the following steps: Based on the train's planned arrival time, arrival delay time, and number of disembarking passengers at the decision station, local observation features of the train at the decision station are constructed. Based on the planned arrival time, arrival delay time, and number of disembarking passengers of the train at each station downstream of the decision station, local observation features of the train at each station downstream of the decision station are constructed; wherein, the arrival delay time of the train at all stations downstream of the decision station is an estimated value obtained by iterative calculation based on the arrival delay time of the train at the decision station. Based on the local observation features of the train at the decision station and all downstream stations, a local observation feature sequence corresponding to the train is constructed.

[0089] As one embodiment of this application, a training module is also included, configured to perform the following steps: Construct a target model based on minimizing the passenger-weighted delay time of all trains on the target passenger route as the optimization objective; An environment simulator is constructed based on the safety constraints of the target passenger route; the safety constraints include: running time constraints, stop duration constraints, time interval constraints, station capacity constraints, and disturbance constraints. The target model is transformed into a distributed partially observable Markov decision process, wherein each train is considered as an independent agent. A global directed graph is constructed based on the local observation feature sequence of each train at the decision station, and global state features are extracted from the directed graph through a graph isomorphic network. Based on the local observation feature sequences of each agent and the global state features, the policy network is trained using the CTDE framework. During the training process, the environment simulator generates the state of each agent at the next station of the current decision station based on the joint actions of each agent.

[0090] In one embodiment of this application, the training module is configured to construct an environment simulator based on the safety constraints of the target passenger route, including: Based on the shortest running time between adjacent stations, a running time constraint is constructed; Based on the upper limit and lower limit of the stop duration, a stop duration constraint is constructed; Based on the arrival time interval and departure time interval of adjacent trains at the same station, the shortest interval duration corresponding to the departure time interval, and the uniqueness of the departure order, time interval constraints are constructed. Based on the shortest time interval between the departure of the previous train and the arrival of the next train on the same track, the track capacity of the station, and the uniqueness of track occupancy, a track capacity constraint is constructed. Construct disturbance constraints to limit the departure time of each train to no earlier than the original time after the occurrence of a disturbance event; An environment simulator is constructed based on the aforementioned runtime constraints, stop duration constraints, time interval constraints, track capacity constraints, and disturbance constraints.

[0091] In one embodiment of this application, the environment scheduling module 103 is specifically configured to perform the following steps: Input the expected dwell time of all trains at the current decision station into the environment simulator, and then perform the following steps through the environment simulator: Based on the expected departure time of each train and the track capacity of the current decision station, determine the departure order of each train at the current decision station. Based on the departure order of each train at the current decision station, and provided that the aforementioned safety constraints are met, the actual departure time of each train at the current decision station and the actual arrival time at the next station are determined.

[0092] In one embodiment of this application, the training module is configured to convert the target model into a distributed partially observable Markov decision process, including: The arrival and departure times of all trains on the target passenger route at the same station are considered as a decision event, and each train is considered as an independent intelligent agent. Tuples are constructed based on distributed partially observable Markov decision processes, and the tuples include the following elements: The collection of intelligent agents consists of the intelligent agents corresponding to all trains; The local observation space is composed of the local observation feature sequences of each agent. The global state space and joint observation space are composed of the local observation spaces of all agents. The joint action space consists of the selectable actions of all agents; The state transition is achieved by the environment simulator based on the joint action selected by all agents, the global state of the current decision station, the track capacity, and the safety constraints, to obtain the global state of the next station of the current decision station. Instant rewards are built based on the weighted delay time of passengers at the next station after the current decision station. A discount factor is used to control the importance of future rewards relative to the immediate rewards.

[0093] In one embodiment of this application, the training module is configured to construct a global directed graph based on the local observation feature sequences of each train at the decision station, including: Let a combination of a train and a station correspond to a node. Based on all stations from the decision station to the terminal station and the combinations of each train, generate nodes corresponding to each combination. The feature of each node corresponds to the local observation features of the train at the corresponding station in the combination of nodes. To generate the global directed graph of the decision station, all nodes are connected by directed edges. Specifically, this includes: using a first type of directed edge to connect nodes that arrive at adjacent stations in sequence with the same train, wherein the direction of the first type of directed edge is from the downstream station to the upstream station; and using a second type of directed edge to connect nodes that arrive at the same station in sequence with different trains, wherein the direction of the second type of directed edge is from the train that arrives at the station first to the train that arrives at the station later.

[0094] In one embodiment of this application, the training module is configured to train the policy network using the CTDE framework based on the local observation feature sequences of each agent and the global state features, including: The inverse gated recurrent unit of the agent network processes the local observation feature sequence of each agent and outputs the Q value of the corresponding action; where different actions correspond to different expected stop times. use Greedy strategy, controlling each agent with probability Randomly select actions, and with probability Choose the action with the highest current Q value; where the probability... The probability value decreases linearly from the initial probability value to the first probability value as the number of training rounds increases; Based on the actions chosen by all agents, the next state is generated through the environment simulator. Calculate the immediate reward based on the next state, and determine the local observation feature sequence corresponding to each agent in the next state; Training samples are constructed based on the current state, action, reward, and next state of each agent. Based on multiple batches of training samples, the hybrid network integrates the Q values ​​of each agent based on the global state features corresponding to the training samples, and calculates the joint state action value corresponding to each empirical tuple; the hybrid network includes a feedforward neural network, and the weights and biases of each feedforward layer of the feedforward neural network are dynamically generated by the supernetwork according to the global state features, and the weights are non-negative values. The mean squared error loss is calculated based on the joint state action value, and the policy network is updated with the goal of minimizing the mean squared error loss.

[0095] Based on the same inventive concept, one embodiment of this application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in any of the above embodiments of this application.

[0096] Regarding the apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here. The apparatus in the embodiments of this application can be a device, or a component, integrated circuit, or chip in a terminal. The apparatus can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc., and the embodiments of this application do not specifically limit the scope.

[0097] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0098] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and components involved are not necessarily essential to this application.

[0099] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0100] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0101] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0102] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] Although preferred embodiments of the embodiments of this application have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, this application is to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0104] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0105] The above provides a detailed description of the high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning, characterized in that, Applied to passenger transport lines with seat reservation mechanisms, including: All stations on the target passenger route, from the station where the interference event occurred to the station preceding the terminal station, are sequentially identified as decision stations. For each decision station, a local observation feature sequence corresponding to each train is constructed; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time and number of disembarking passengers; The local observation feature sequences of each train at the current decision station are input into the trained agent network to obtain the expected stopping time of each train at the current decision station. The agent network and the hybrid network together constitute the policy network of the QMIX architecture. The policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning. The agent network is used to generate the expected stopping time based on the local observation feature sequences of each train. The hybrid network is used to collaboratively optimize the expected stopping time of each train based on the global state features. Based on the safety constraints of the target passenger route, the expected stop time of all trains at the current decision station is integrated and coordinated to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and the next station is determined as the new decision station.

2. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, For each decision station, a local observation feature sequence corresponding to each train is constructed, including: Based on the train's planned arrival time, arrival delay time, and number of disembarking passengers at the decision station, local observation features of the train at the decision station are constructed. Based on the planned arrival time, arrival delay time, and number of disembarking passengers of the train at each station downstream of the decision station, local observation features of the train at each station downstream of the decision station are constructed; wherein, the arrival delay time of the train at all stations downstream of the decision station is an estimated value obtained by iterative calculation based on the arrival delay time of the train at the decision station. Based on the local observation features of the train at the decision station and all downstream stations, a local observation feature sequence corresponding to the train is constructed.

3. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that, Before inputting the local observation feature sequences of each train at the current decision station into the trained agent network, the following steps are also included: Construct a target model based on minimizing the passenger-weighted delay time of all trains on the target passenger route as the optimization objective; An environment simulator is constructed based on the safety constraints of the target passenger route; the safety constraints include: running time constraints, stop duration constraints, time interval constraints, station capacity constraints, and disturbance constraints. The target model is transformed into a distributed partially observable Markov decision process, wherein each train is considered as an independent agent. A global directed graph is constructed based on the local observation feature sequence of each train at the decision station, and global state features are extracted from the directed graph through a graph isomorphic network. Based on the local observation feature sequences of each agent and the global state features, the policy network is trained using the CTDE framework. During the training process, the environment simulator generates the state of each agent at the next station of the current decision station based on the joint actions of each agent.

4. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 3, characterized in that, Based on the safety constraints of the target passenger route, an environment simulator is constructed, including: Based on the shortest running time between adjacent stations, a running time constraint is constructed; Based on the upper limit and lower limit of the stop duration, a stop duration constraint is constructed; Based on the arrival time interval and departure time interval of adjacent trains at the same station, the shortest interval duration corresponding to the departure time interval, and the uniqueness of the departure order, time interval constraints are constructed. Based on the shortest time interval between the departure of the previous train and the arrival of the next train on the same track, the track capacity of the station, and the uniqueness of track occupancy, a track capacity constraint is constructed. Construct disturbance constraints to limit the departure time of each train to no earlier than the original time after the occurrence of a disturbance event; An environment simulator is constructed based on the aforementioned runtime constraints, stop duration constraints, time interval constraints, track capacity constraints, and disturbance constraints.

5. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 4, characterized in that, Based on the safety constraints of the target passenger route, the expected dwell time of all trains at the current decision station is integrated and coordinated to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, including: Input the expected dwell time of all trains at the current decision station into the environment simulator, and then perform the following steps through the environment simulator: Based on the expected departure time of each train and the track capacity of the current decision station, determine the departure order of each train at the current decision station. Based on the departure order of each train at the current decision station, and provided that the aforementioned safety constraints are met, the actual departure time of each train at the current decision station and the actual arrival time at the next station are determined.

6. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 3, characterized in that, The target model is transformed into a distributed partially observable Markov decision process, including: The arrival and departure times of all trains on the target passenger route at the same station are considered as a decision event, and each train is considered as an independent intelligent agent. Tuples are constructed based on distributed partially observable Markov decision processes, and the tuples include the following elements: The collection of intelligent agents consists of the intelligent agents corresponding to all trains; The local observation space is composed of the local observation feature sequences of each agent. The global state space and joint observation space are composed of the local observation spaces of all agents. The joint action space consists of the selectable actions of all agents; The state transition is achieved by the environment simulator based on the joint action selected by all agents, the global state of the current decision station, the track capacity, and the safety constraints, to obtain the global state of the next station of the current decision station. Instant rewards are built based on the weighted delay time of passengers at the next station after the current decision station. A discount factor is used to control the importance of future rewards relative to the immediate rewards.

7. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 3, characterized in that, A global directed graph is constructed based on the local observation feature sequences of each train at the decision station, including: Let a combination of a train and a station correspond to a node. Based on all stations from the decision station to the terminal station and the combinations of each train, generate nodes corresponding to each combination. The feature of each node corresponds to the local observation features of the train at the corresponding station in the combination of nodes. To generate the global directed graph of the decision station, all nodes are connected by directed edges. Specifically, this includes: using a first type of directed edge to connect nodes that arrive at adjacent stations in sequence with the same train, wherein the direction of the first type of directed edge is from the downstream station to the upstream station; and using a second type of directed edge to connect nodes that arrive at the same station in sequence with different trains, wherein the direction of the second type of directed edge is from the train that arrives at the station first to the train that arrives at the station later.

8. The high-speed railway train timetable rescheduling method based on multi-agent reinforcement learning according to claim 7, characterized in that, Based on the local observation feature sequences of each agent and the global state features, the policy network is trained using the CTDE framework, including: The inverse gated recurrent unit of the agent network processes the local observation feature sequence of each agent and outputs the Q value of the corresponding action; where different actions correspond to different expected stop times. use Greedy strategy, controlling each agent with probability Randomly select actions, and with probability Choose the action with the highest current Q value; where the probability... The probability value decreases linearly from the initial probability value to the first probability value as the number of training rounds increases; Based on the actions chosen by all agents, the next state is generated through the environment simulator. Calculate the immediate reward based on the next state, and determine the local observation feature sequence corresponding to each agent in the next state; Training samples are constructed based on the current state, action, reward, and next state of each agent. Based on multiple batches of training samples, the hybrid network integrates the Q values ​​of each agent based on the global state features corresponding to the training samples, and calculates the joint state action value corresponding to each empirical tuple; the hybrid network includes a feedforward neural network, and the weights and biases of each feedforward layer of the feedforward neural network are dynamically generated by the supernetwork according to the global state features, and the weights are non-negative values. The mean squared error loss is calculated based on the joint state action value, and the policy network is updated with the goal of minimizing the mean squared error loss.

9. A high-speed railway train timetable rescheduling device based on multi-agent reinforcement learning, characterized in that, For implementing the method as described in any one of claims 1-8, comprising: The information construction module is configured to sequentially identify all stations on the target passenger route, from the station where the interference event occurred to the station preceding the terminal station, as decision stations; for each decision station, a local observation feature sequence corresponding to each train is constructed; wherein, the local observation feature sequence corresponding to each train is constructed based on the node information of the train at the current decision station and all stations after the decision station; the node information includes: planned arrival time, arrival delay time, and number of disembarking passengers; The decision-making module is configured to input the local observation feature sequences of each train at the current decision station into a trained agent network to obtain the expected stopping time of each train at the current decision station. The agent network and the hybrid network together constitute the policy network of the QMIX architecture. The policy network is pre-trained under a centralized training and distributed execution framework, with the optimization objective of minimizing the passenger-weighted delay time of all trains on the target passenger line, and is trained through multi-agent reinforcement learning. The agent network is used to generate the expected stopping time based on the local observation feature sequences of each train. The hybrid network is used to collaboratively optimize the expected stopping time of each train based on global state features. The environmental scheduling module is configured to integrate and coordinate the expected stop time of all trains at the current decision station based on the safety constraints of the target passenger line, so as to determine the actual departure time of each train at the decision station and the actual arrival time at the next station, and to determine the next station as the new decision station.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-8.

Images