A method and system for dynamic scheduling of multiple trains based on the MAPPO algorithm

Abstract

This invention provides a method and system for dynamic scheduling of multiple trains based on the MAPPO algorithm, relating to the field of dynamic scheduling technology for multiple trains. The method includes acquiring the operating status information of multiple trains; identifying the status information based on a pre-trained neural network strategy model to generate discrete action decisions; generating adjustment instructions based on the discrete action decisions; calculating new operating status information and the composite reward feedback of the current decision step based on the adjustment instructions, and updating the train environment status information to proceed to the next decision step until the end of the current round; determining whether the scheduling strategy corresponding to the current round conforms to a preset scheduling strategy based on all rewards of the current round; if not, returning to the step of acquiring the operating status information of multiple trains, until the preset scheduling strategy is met, thereby dynamically and collaboratively optimizing the scheduling of multiple trains. This application solves the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms, lacking collaborative global scheduling capabilities.

Description

Technical Field

[0001] This invention relates to the field of dynamic scheduling technology for multiple trains, and in particular to a dynamic scheduling method and system for multiple trains based on the MAPPO algorithm. Background Technology

[0002] As the road network continues to expand and traffic density increases, the propagation effect of train delays caused by sudden disruptions such as equipment failures, natural disasters, and transportation safety issues is becoming increasingly prominent.

[0003] like Figure 1 As shown, under normal circumstances, high-speed trains operate efficiently and orderly on sections 1 to 4 between stations A and E according to a predetermined timetable. However, in reality, the operation of the railway system inevitably encounters various unexpected disruptions. For example... Figure 1 As shown, a sudden interference event occurred in section 3 near station C, where H start H indicates the start time of the sudden interference event. end This indicates the end time of the sudden disruption event. Due to the close spatiotemporal correlation of train operations within the railway system, this localized failure quickly triggered a chain reaction. Specifically, when train K2 was at station C, its actual departure time was forced to be adjusted, resulting in a planned departure interval with trains before and after it being less than the required safe interval. This directly prevented the train from departing as scheduled, inevitably causing a departure delay. This initial delay event did not exist in isolation but rather spread like dominoes to the subsequent operational chain. Under the impact of the sudden disruption event, a significant and continuously widening deviation appeared between the actual and planned trajectories of subsequent trains. The original timetable was disrupted, the arrival and departure times of subsequent trains were forced to be postponed across the board, the operational order was severely disrupted, and the overall transportation efficiency was significantly reduced.

[0004] To address the aforementioned issues, existing technologies primarily revolve around two categories of approaches: One category employs optimization models utilizing mathematical programming and intelligent algorithms. This involves constructing multi-objective mathematical models (such as mixed-integer programming, multi-objective particle swarm optimization, and the fireworks algorithm) to minimize delay time, resource adjustment frequency, energy consumption, and cost, and developing two- or multi-stage optimization strategies for specific scenarios (such as snowstorms and single-line faults in intervals). While these methods have achieved success in multi-objective collaboration and scenario adaptability, they generally rely on predefined rules, making it difficult to meet the real-time response requirements of highly dynamic environments. The other category utilizes reinforcement learning techniques, leveraging its interactive learning characteristics with the environment (such as Q-learning and its improvements, proximal policy optimization, and deep deterministic policy gradient algorithms). Through innovative methods such as designing reward mechanisms, lightweight state representations, model-free methods, and multi-agent competitive learning, these approaches demonstrate significant potential in improving learning efficiency, effectively suppressing delay propagation, and collaboratively optimizing objectives (such as delay and energy consumption). However, current research on reinforcement learning methods mostly focuses on single-train decision-making or simple competition and cooperation paradigms within a local scope. There is still no effective solution for how to achieve coordinated scheduling of global resources of the railway network (such as station arrival and departure tracks and section capacity), which has become a key bottleneck that urgently needs to be overcome. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a multi-train dynamic scheduling method and system based on the MAPPO algorithm, which solves the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms and lack the ability to coordinate global train scheduling under sudden interference events.

[0006] This invention provides a method for dynamic scheduling of multiple trains based on the MAPPO algorithm, comprising:

[0007] Acquire the operating status information of multiple trains, and identify the operating status information according to a pre-trained neural network strategy model to generate discrete action decisions;

[0008] Adjustment instructions are generated based on discrete action decisions. These instructions include speed adjustment and start / stop. New operating status information and composite reward feedback for the current decision step are calculated based on these instructions, and the train environment status information is updated to proceed to the next decision step until the end of the current round. Each round includes multiple decision steps, each decision step generates an action decision, and each round generates a scheduling strategy, with each scheduling strategy including multiple action decisions.

[0009] Determine whether the scheduling strategy for the current round conforms to the preset scheduling strategy based on all the rewards of the current round.

[0010] If not, return to the step of obtaining the operating status information of multiple trains until the scheduling strategy corresponding to the current round meets the preset scheduling strategy for dynamic collaborative optimization of multiple train scheduling.

[0011] The aforementioned multi-train dynamic scheduling method based on the MAPPO algorithm overcomes the scheduling conflict bottleneck caused by the lack of cooperative communication mechanisms in traditional single-agent systems through a dynamic scheduling framework based on Multi-Agent Proximal Policy Optimization (MAPPO). Specifically, with the goal of minimizing network-wide train delays and critical train delays, a dynamic avoidance strategy for high-priority trains is implemented through a distributed decision-making framework. Furthermore, relying on the distributed decision-making architecture, train autonomous collaborative scheduling is enabled by dynamically constructing each train as an agent with local perception and real-time decision-making capabilities. Specifically, autonomous decision-making under local observation is achieved based on the 11-dimensional state features covered by the operational status information. It innovatively integrates a multi-objective composite reward mechanism that includes safety operation rewards and penalties, speeding penalties, delay improvement rewards, on-time arrival rewards, and destination arrival rewards. It also establishes a centralized training and distributed execution collaborative paradigm to optimize group behavior coordination, significantly improving the efficiency of network resource utilization and operational safety. This solves the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms and lack collaborative global scheduling capabilities.

[0012] In addition, the multi-train dynamic scheduling method based on the MAPPO algorithm described above according to the present invention may also have the following additional technical features:

[0013] Furthermore, in the step of calculating the new operating status information and the composite reward feedback of the current decision step according to the adjustment instruction and updating the train environment status information, the calculation formula for the total reward integration mechanism of the composite reward feedback is:

[0014] ;

[0015] In the formula: k indicates whether the train is at the station, where 1 indicates that the train is at the station, and 0 indicates that the train is not at the station or the train has just arrived at the station; r in This represents the total reward returned by the environment when the train is inside the station, where... r in = r dwell , r dwell This indicates a reward for efficient station stops; r out This represents the total reward returned by the environment when the train is outside the station or has just arrived at the station, where...

[0016] ;

[0017] In the formula,r safe Indicates rewards and penalties for safe operation; r speed This indicates the penalties for speeding. r delay Indicates a reward for improving performance after delays; r punct This indicates a reward for arriving on time. r term ∩ represents the reward for reaching the finish line; ∩ represents the intersection.

[0018] Furthermore, the formula for calculating the delay improvement bonus is as follows:

[0019] ;

[0020] In the formula, r delay Indicates a reward for improving performance after delays; d t Indicates the current delay; d t-1 d0 represents the delay at the previous moment; d0 represents the difference between the current delay and the previous moment's delay; ∩ represents the intersection. l delay This indicates the incentive coefficient for improving delays.

[0021] Furthermore, the formula for calculating the on-time arrival bonus is as follows:

[0022] ;

[0023] In the formula, r punct This indicates a reward for arriving on time. This represents the calculation of the on-time linear decay function, where, This indicates the actual time the train arrives at station S. Indicates the train's scheduled arrival time at station S; l punct 's' represents the on-time arrival bonus coefficient; 's' represents the station number.

[0024] Furthermore, the formula for calculating the finish line completion bonus is as follows:

[0025] ;

[0026] In the formula, r term Indicates a reward for reaching the finish line; l term Indicates the reward coefficient for reaching the finish line; f This represents the gain coefficient for trains stopping at major stations; d t 60 indicates the current delay; 60 indicates 60 minutes, which is the maximum acceptable delay limit for the terminal station.

[0027] Furthermore, the formula for calculating rewards and penalties for safe operations is as follows:

[0028] ;

[0029] In the formula, r safe Indicates rewards and penalties for safe operation; d front Indicates the distance between the current train and the train in front; l safe This represents the safety operation reward coefficient.

[0030] Furthermore, the formula for calculating the stop efficiency bonus is as follows:

[0031] ;

[0032] In the formula, r dwell This indicates a reward for efficient station stops; l dwell This represents the stop efficiency bonus coefficient; t stop Indicates the actual stop time; t min Indicates the minimum stopping time; t target Indicates the target stopping time.

[0033] Furthermore, the formula for calculating speeding penalties is as follows:

[0034] ;

[0035] In the formula, r speed This indicates the penalties for speeding. l speed This indicates the reward coefficient for speeding penalties; v Indicates the current speed; v lim This indicates the maximum permissible speed.

[0036] Furthermore, the step of identifying the operational state information based on the pre-trained neural network policy model to generate discrete action decisions includes:

[0037] Obtain the model initialization instruction, and initialize the pre-trained neural network policy model according to the model initialization instruction;

[0038] The running state information is identified based on the initialized pre-trained neural network policy model to generate discrete action decisions.

[0039] This invention also provides a multi-train dynamic scheduling system based on the MAPPO algorithm, the system comprising:

[0040] The acquisition module is used to acquire the operating status information of multiple trains and identify the operating status information according to a pre-trained neural network strategy model to generate discrete action decisions.

[0041] The training module is used to generate adjustment instructions based on discrete action decisions. The adjustment instructions include speed adjustment and start / stop. Based on the adjustment instructions, the module calculates new operating status information and composite reward feedback of the current decision step and updates the train environment status information to enter the next decision step until the end of the current round. Each round includes multiple decision steps, each decision step generates an action decision, and each round generates a scheduling strategy, each scheduling strategy including multiple action decisions.

[0042] The judgment module is used to determine whether the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy based on all the rewards of the current round.

[0043] The first execution module is used to return to the step of obtaining the running status information of multiple trains if the scheduling strategy corresponding to the current round does not meet the preset scheduling strategy, until the scheduling strategy corresponding to the current round meets the preset scheduling strategy in order to dynamically and collaboratively optimize the scheduling of multiple trains. Attached Figure Description

[0044] Figure 1 Diagram showing train adjustments for unexpected interference events;

[0045] Figure 2 This is a flowchart of the decision-making process of the intelligent agent for adjusting high-speed railway train operation in an embodiment of the present invention;

[0046] Figure 3 This is a flowchart of a multi-train dynamic scheduling method based on the MAPPO algorithm in an embodiment of the present invention;

[0047] Figure 4 This is an optimized timetable for dynamic train scheduling under sudden interference events in this embodiment of the invention.

[0048] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0049] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0051] To address the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms under sudden interference events, lacking the ability for coordinated global train scheduling, this application provides a multi-train dynamic scheduling method and system based on the MAPPO algorithm. Addressing the limitations of existing technologies in collaborative and real-time decision-making, the architectural characteristics of Multi-Agent Proximal Policy Optimization (MAPPO) are deeply coupled with the needs of high-speed rail scheduling. High-speed rail dynamic scheduling requires train groups to minimize network delays through collaborative cooperation, which is naturally supported by MAPPO's Centralized Training with Decentralized Execution (CTDE) mechanism. The high efficiency of neural network forward computation directly supports millisecond-level response times for large-scale networks with more than 200 trains. Together, these two aspects constitute a systematic solution that breaks through the bottlenecks of existing technologies.

[0052] Based on this, this application adopts the MAPPO method. This method constructs a distributed decision-making architecture, enabling each train to become an independent intelligent agent and achieve autonomous decision-making based on local observations. It innovatively designs a composite reward mechanism integrating punctuality rate, safe distance, speeding rules, and minimum safe stopping time rules, establishing a collaborative paradigm of Centralized Training with Decentralized Execution (CTDE) to optimize group behavior coordination. An empirical study on the Shanghai-Kunming High-Speed ​​Railway in China demonstrates that this method exhibits significant advantages under typical interference scenarios such as equipment failure speed limits and severe weather-related line closures. It can also be scaled up to large-scale railway networks with more than 200 trains to enhance real-time response capabilities, providing a new theoretical paradigm and engineering practice basis for intelligent scheduling of complex transportation systems.

[0053] Specifically, the MAPPO dynamic scheduling framework, such as Figure 2As shown, its core is a closed-loop adaptive real-time interactive system composed of train intelligent agents. The system's operation mechanism is as follows: the train intelligent agent continuously collects 11-dimensional state feature vectors from the high-speed rail operating environment, covering interference event information, train status information, line resource information, and scheduling target information, achieving a holographic perception of the operating situation. Interference event information includes the location of the event, the duration of the event, and the estimated recovery time; train status information includes the train's section location, train speed, stopping status, and stopping time; line resource information includes the number of arrival and departure tracks at stations and the section's capacity; and scheduling target information includes key train identifiers and timetable offsets. Based on current local observations, the intelligent agent calls a neural network strategy model that has been frozen after online training using the MAPPO algorithm, generating five types of discrete action decisions in a distributed execution mode, covering acceleration, deceleration, and speed maintenance for actions outside the station; and continued stopping and departure for actions within the station. During interval operation, the system performs precise speed adjustments of ±10km / h (constrained within a safety range of 200km / h-352km / h) or maintains the speed. When stopping at stations, it chooses to force departure or extend the stop, and rigidly integrates safety constraints through a state-action association mechanism. After the generated speed adjustment or start / stop commands are applied to the high-speed rail simulation environment, the environment updates its operating status information based on the train dynamics model, tracking interval rules, and resource occupancy logic, and synchronously calculates composite reward feedback. The agent then observes the updated environmental status information, including delay time and safety distance, forming a millisecond-level closed-loop control flow of "perception → decision → execution → evaluation," where the maximum number of decision steps is greater than 200, until all trains arrive at the terminal station.

[0054] To facilitate understanding of the present invention, several embodiments are given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0055] Example 1

[0056] Please see Figure 3 The figure shows a multi-train dynamic scheduling method based on the MAPPO algorithm in the first embodiment of the present invention, the method including steps S101 to S104:

[0057] S101. Obtain the operating status information of multiple trains, and identify the operating status information based on the pre-trained neural network strategy model to generate discrete action decisions.

[0058] In this embodiment, the train acts as an intelligent agent, and its action decisions include actions outside the station and actions within the station. Actions outside the station include accelerating, decelerating, and maintaining speed; actions within the station include continuing to stop and departing. Figure 2As shown, the system acquires multi-train operation status information under sudden interference events. This operation status information includes four main types: interference event information, train status information, track resource information, and scheduling target information. Furthermore, each of these four status types comprises an 11-dimensional state feature vector. Specifically, interference event information includes the location of the event, the duration of the event, and the estimated recovery time; train status information includes the train's section location, train speed, stopping status, and stop time; track resource information includes the number of arrival and departure tracks at stations and the section's capacity; and scheduling target information includes key train identification and timetable offset. The 11-dimensional feature vector is used to accurately encode the train operation status, with core elements including train identification features, spatiotemporal status features, operation status features, station relationship features, safety status features, and departure control features.

[0059] In this embodiment, a reinforcement learning simulation environment is first constructed and a neural network strategy model is trained until convergence using train history or simulation data. Then, the scheduling execution stage is entered, where real-time operational status information, including interference events, train status, line resources, and scheduling objectives, is acquired and input into the trained strategy model to generate discrete action decisions for speed adjustment or start / stop.

[0060] To ensure strict compatibility between the pre-trained neural network policy model and the pre-trained architecture, the pre-trained neural network policy model must first be initialized according to the model initialization instructions. Then, the running state information is identified based on the initialized pre-trained neural network policy model to generate discrete action decisions. Specifically, the input layer of the pre-trained architecture matches an 11-dimensional state feature vector, and the output layer corresponds to the joint action space.

[0061] In this embodiment, the pre-trained neural network strategy model after initialization has the ability to output action decisions based on real-time collected multi-train operation status information. Its output is designed as follows: the first three neurons map actions outside the station, namely acceleration, deceleration, and maintaining speed; the last two neurons map actions within the station, namely continuing to stop and departing. After model initialization, it can be directly used for real-time inference, generating action probability distributions by processing state vectors containing core elements such as train identification, spatiotemporal state, and safety features.

[0062] S102. Generate adjustment instructions based on discrete action decisions, calculate new operating status information and composite reward feedback for the current decision step based on the adjustment instructions, and update train environment status information to enter the next decision step until the end of the current round.

[0063] In this embodiment, the neural network strategy model acquires real-time operational status information, including interference events, train status, line resources, and scheduling objectives, to generate discrete action decisions for speed adjustment or start / stop. Adjustment commands include speed adjustment (i.e., adjusting the operating speed within a section) and start / stop (i.e., whether to continue stopping or depart within a station). After the generated adjustment commands are applied to the high-speed rail simulation environment, the environment updates the train environment status information based on the train dynamics model, tracking interval rules, and resource occupancy logic, and simultaneously calculates composite reward feedback. The agent then observes the updated train environment status information, forming a millisecond-level closed-loop control flow of "perception → decision → execution → evaluation," where the maximum number of decision steps is greater than 200, until all trains arrive at the terminal station. The environmental status information includes stopping time, delay time, and safety distance.

[0064] In this embodiment, each round includes multiple decision steps, each decision step corresponds to generating an action decision, and each round generates a scheduling strategy, each scheduling strategy including multiple action decisions. Further, as a specific example, the reward function is mainly designed around safe operation and delay control. To ensure trains arrive on time, the reward function should penalize delays; the longer the delay, the greater the penalty, especially at key hub stations, to reduce the impact on overall scheduling. Simultaneously, arriving too early may lead to operational chaos, so appropriate penalties should also be applied to ensure trains run as planned. Safe operation is a fundamental prerequisite for scheduling optimization; train spacing must be maintained above a safe threshold. If the spacing is too small, a larger penalty is applied to avoid the risk of rear-end collisions. Furthermore, rapid changes in train speed may affect operational stability; therefore, penalties need to be applied to drastic acceleration and deceleration to maintain operational safety and stability. In summary, the design of the reward function should reduce train delays while ensuring safe distances and operational stability, enabling the agent to learn and optimize scheduling strategies to achieve reliable high-speed railway operation. Specifically, the calculation formula for the total reward integration mechanism (i.e., the reward function) of the composite reward feedback is:

[0065] ;

[0066] In the formula: k indicates whether the train is at the station, where 1 indicates the train is at the station, and 0 indicates the train is not at the station or the train has just arrived at the station; r in This represents the total reward returned by the environment when the train is inside the station, where... r in = r dwell , r dwell This indicates a reward for efficient station stops; r out This represents the total reward returned by the environment when the train is outside the station or has just arrived at the station, where...

[0067] ;

[0068] In the formula, r safe Indicates rewards and penalties for safe operation; r speed This indicates the penalties for speeding. r delay Indicates a reward for improving performance after delays; r punct This indicates a reward for arriving on time. r term ∩ represents the reward for reaching the finish line; ∩ represents the intersection.

[0069] Furthermore, the formula for calculating the delay improvement bonus is as follows:

[0070] ;

[0071] In the formula, r delay Indicates a reward for improving performance after delays; d t Indicates the current delay; d t-1 d0 represents the delay at the previous moment; d0 represents the difference between the current delay and the previous moment's delay, i.e. d t-1 - d t The difference terms; ∩ represents the intersection; l delay This indicates the incentive coefficient for improving delays. l delay =300; 1 / 15 represents the attenuation coefficient.

[0072] A decay mechanism for delay improvement rewards and an upper limit constraint on overspeed penalty items jointly suppress policy oscillations. To address the insufficient penalty strength of traditional methods in scenarios with continuous delays, such as cascading delay propagation, a lag-difference mechanism is designed. Furthermore, the difference term... d t-1 - d t Dynamically perceive delay trends and mitigate strategy oscillations caused by short-term deterioration with a decay coefficient of 1 / 15.

[0073] The internal logic of the delay improvement reward is as follows: when the delay is improved (i.e., d) t ≤d t-1 When using full-weighted incentives (λ), delay =300), strengthen positive feedback. When the delay worsens (i.e.: d t >d t-1 And it did not reach the serious delay threshold (i.e., d) tWhen the time limit is less than 15 minutes, the penalty intensity is reduced to prevent the agent from abandoning long-term optimization due to temporary deterioration. The threshold d is... t The time limit of 15 minutes is set according to the high-speed rail scheduling specifications to prevent small fluctuations from interfering with decision-making.

[0074] Furthermore, to drive station timing optimization, this embodiment provides an on-time arrival reward. Specifically, the calculation formula for the on-time arrival reward is as follows:

[0075] ;

[0076] In the formula, r punct This indicates a reward for arriving on time. This represents the calculation of the on-time linear decay function, where, This indicates the actual time the train arrives at station S. Indicates the train's scheduled arrival time at station S; l punct This represents the on-time arrival reward coefficient, balancing the weights of local and global objectives. ,l punct =50; s represents the station number.

[0077] Traditional binary punctuality judgment (on time / delayed) cannot guide micro-scheduling strategies. In this embodiment, the punctual arrival reward innovatively introduces a flexible time window (±8 minutes) to achieve fine-grained timing control. Here, "±8 minutes" is the passenger tolerance threshold during train operation, that is, the maximum acceptable deviation for high-speed rail passengers at intermediate stations.

[0078] The internal logic of the formula for calculating the on-time arrival bonus is as follows: a linear decay function maps the time deviation to a continuous bonus value (where the bonus is maximized when the deviation = 0).

[0079] Furthermore, the formula for calculating the finish line completion bonus is as follows:

[0080] ;

[0081] In the formula, r term Indicates a reward for reaching the finish line; l term This indicates the reward coefficient for reaching the finish line. l term =1500; f This represents the gain coefficient for trains stopping at major stations. f= 1.8; d t 60 indicates the current delay; 60 indicates 60 minutes, which is the maximum acceptable delay limit for the terminal station.

[0082] To resolve the conflict between prioritizing critical train services and ensuring efficiency for regular train services, this application innovatively uses a gain coefficient... f Quantitative injection scheduling priority breaks through the static limitations of traditional rule-based methods.

[0083] Specifically, the internal logic of the formula for calculating the reward upon reaching the destination is as follows: max(0.2,•) ensures that the lower limit of the base reward is 0.2, preventing strategy collapse (such as the reward becoming zero in the event of extreme delays). end / 60 compresses the endpoint delay into the [0,1] interval to achieve proportional penalty. f =1.8 Dynamically amplify rewards for key train services to drive resource allocation; where, when the train is a regular train, f =1.

[0084] To curb speeding and prevent cascading delays, this application provides a safety operation reward and penalty system. Furthermore, the calculation formula for the safety operation reward and penalty system is as follows:

[0085] ;

[0086] In the formula, r safe Indicates rewards and penalties for safe operation; d front Indicates the distance between the current train and the train in front; l safe This represents the safety operation reward coefficient. l safe =200.

[0087] In this embodiment, the physical safety rule (i.e., tracking interval ≥ 10 km) is transformed into a differentiable optimization objective, avoiding the infeasibility of decisions caused by traditional hard constraints. The safety operation reward and punishment calculation formula transforms the discrete safety rule (i.e., minimum interval 5 km) into a continuously differentiable function, supporting gradient optimization.

[0088] The internal logic of the safety operation reward and punishment calculation formula is: danger zone (i.e., d) front <8km): Penalty value - (8-d) front ) / 8•λ safe The demand for emergency braking increases exponentially as the distance decreases.

[0089] Buffer (i.e., 8≤d) front <16km): Positive reward min(1-d) front / 16)•λ safe / 2 Encourages proactive risk avoidance. The boundary value (i.e., 8 km) is aligned with the minimum safe distance D. min =5km, with a safety redundancy reserved.

[0090] To reduce resource conflicts, improve resource utilization, and support the efficient execution of the basic target layer, this application provides a stop efficiency reward to balance passenger boarding and alighting demand (minimum stop time τ). min ) and scheduling efficiency (target dwell time τ) target This resolves resource allocation conflicts. Furthermore, the formula for calculating the stop efficiency bonus is:

[0091] ;

[0092] In the formula, r dwell This indicates a reward for efficient station stops; l dwell This represents the stop efficiency bonus coefficient. l dwell =30, dynamically adjusted based on station level; t stop Indicates the actual stop time; t min Indicates the minimum stopping time; t target Indicates the target stopping time.

[0093] Specifically, the internal logic of the stop efficiency bonus calculation formula is as follows: when the target is not met (i.e., when τ... stop <τ min (Time), compensation reward (τ) stop / τ min )•λ dwell • 0.5, weight 0.5 to prevent overcompensation; after reaching the target (i.e., when τ min ≤τ stop ≤τ target (Time), linear decay function of dwell time (1-(τ)) stop -τ min ) / (τ target -τ min ))•λ dwell Approaching the efficiency limit.

[0094] To address the dynamic risks (e.g., sudden acceleration changes) caused by aggressive speed regulation, a progressive penalty mechanism is designed to overcome the limitations of traditional binary constraints. Specifically, the calculation formula for the overspeed penalty item is as follows:

[0095] ;

[0096] In the formula, r speed This indicates the penalties for speeding. l speed This indicates the reward coefficient for speeding penalties. l speed =-20;v Indicates the current speed; v lim This indicates the maximum permissible speed.

[0097] Specifically, the internal logic of the formula for calculating speeding penalties is: proportional term ((vv) lim ) / v lim )•λ speed For speeds exceeding the limit (i.e.) v >1.08 v lim Penalty based on the degree of deviation (i.e., λ) speed =-20). max(-100,•) limits the upper limit of the single-step penalty to avoid policy oscillation (e.g., drastic fluctuations in reward when there is a slight overspeed). v lim Dynamically bind speed limits to different sections to adapt to changing environments.

[0098] S103. Determine whether the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy based on all the rewards of the current round.

[0099] If the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy, then proceed to step S104; if the scheduling strategy corresponding to the current round does not conform to the preset scheduling strategy, then return to step S101 until the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy to dynamically and collaboratively optimize multi-train scheduling.

[0100] S104. Dynamically and collaboratively optimize multi-train scheduling based on the scheduling strategy corresponding to the current round.

[0101] like Figure 4 As shown in Table 1, the timetable for dynamic scheduling optimization of trains under sudden interference events is presented. Based on Table 1, the MAPPO algorithm demonstrates significant comprehensive advantages over the traditional PPO algorithm: In terms of punctuality, the multi-agent collaborative decision-making mechanism in this application improves the average on-time rate of the destination by 20% (77% vs 97%) and the average on-time rate of key trains by 15% (85% vs 100%). This is due to the effective implementation of intelligent adjustment strategies such as early identification of delay propagation paths and overtaking / path replanning by the centralized Critic network, significantly suppressing the spread of cascading delays. In terms of dynamic adaptation, the algorithm shortens the convergence rounds by 33%, verifying the rapid policy optimization characteristics of the centralized training and distributed execution architecture in non-steady-state environments. The PPO algorithm in Table 1 is the traditional Proximal Policy Optimization (PPO) algorithm.

[0102] Table 1:

[0103]

[0104] In summary, the multi-train dynamic scheduling method based on the MAPPO algorithm in the above embodiments of the present invention overcomes the scheduling conflict bottleneck caused by the lack of a cooperative communication mechanism in traditional single-agent systems by using a dynamic scheduling framework based on Multi-Agent Proximity Policy Optimization (MAPPO). Specifically, with the goal of minimizing network-wide train delays and critical train delays, a dynamic avoidance strategy for high-priority trains is implemented through a distributed decision-making framework. Furthermore, relying on the distributed decision-making architecture, train autonomous collaborative scheduling is enabled by dynamically constructing each train as an agent with local perception and real-time decision-making capabilities. Specifically, autonomous decision-making under local observation is achieved based on the 11-dimensional state features covered by the operating status information. It innovatively integrates a multi-objective composite reward mechanism that includes safety operation rewards and penalties, speeding penalties, delay improvement rewards, on-time arrival rewards, and destination arrival rewards. It also establishes a centralized training and distributed execution collaborative paradigm to optimize group behavior coordination, significantly improving the efficiency of network resource utilization and operational safety. It solves the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms and lack collaborative global scheduling capabilities.

[0105] Example 2

[0106] The second embodiment of the present invention provides a multi-train dynamic scheduling system based on the MAPPO algorithm, comprising:

[0107] The acquisition module is used to acquire the operating status information of multiple trains and identify the operating status information according to a pre-trained neural network strategy model to generate discrete action decisions.

[0108] The training module is used to generate adjustment instructions based on discrete action decisions. The adjustment instructions include speed adjustment and start / stop. Based on the adjustment instructions, the module calculates new operating status information and composite reward feedback of the current decision step and updates the train environment status information to enter the next decision step until the end of the current round. Each round includes multiple decision steps, each decision step generates an action decision, and each round generates a scheduling strategy, each scheduling strategy including multiple action decisions.

[0109] The judgment module is used to determine whether the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy based on all the rewards of the current round.

[0110] The first execution module is used to return to the step of obtaining the running status information of multiple trains if the scheduling strategy corresponding to the current round does not meet the preset scheduling strategy, until the scheduling strategy corresponding to the current round meets the preset scheduling strategy in order to dynamically and collaboratively optimize the scheduling of multiple trains.

[0111] In summary, the multi-train dynamic scheduling system based on the MAPPO algorithm in the above embodiments of the present invention overcomes the scheduling conflict bottleneck caused by the lack of a cooperative communication mechanism in traditional single-agent systems through a dynamic scheduling framework based on Multi-Agent Proximal Policy Optimization (MAPPO). Specifically, with the goal of minimizing network-wide train delays and critical train delays, a dynamic avoidance strategy for high-priority trains is implemented through a distributed decision-making framework. Furthermore, relying on the distributed decision-making architecture, train autonomous collaborative scheduling is enabled by dynamically constructing each train as an agent with local perception and real-time decision-making capabilities. Specifically, autonomous decision-making under local observation is achieved based on the 11-dimensional state features covered by the operating status information. It innovatively integrates a multi-objective composite reward mechanism that includes safety operation rewards and penalties, speeding penalties, delay improvement rewards, on-time arrival rewards, and destination arrival rewards. It also establishes a centralized training and distributed execution collaborative paradigm to optimize group behavior coordination, significantly improving the efficiency of network resource utilization and operational safety. It solves the technical problem that existing train scheduling methods are still limited to single-train decision-making or local competition paradigms and lack collaborative global scheduling capabilities.

[0112] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described above.

[0113] Furthermore, embodiments of the present invention also propose a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described above.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0115] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0116] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0117] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0118] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-train dynamic scheduling method based on the MAPPO algorithm, characterized in that, include: Acquire the operating status information of multiple trains, and identify the operating status information according to a pre-trained neural network strategy model to generate discrete action decisions; Adjustment instructions are generated based on discrete action decisions. These instructions include speed adjustment and start / stop. New operating status information and composite reward feedback for the current decision step are calculated based on these instructions, and the train environment status information is updated to proceed to the next decision step until the end of the current round. Each round includes multiple decision steps, each decision step generates an action decision, and each round generates a scheduling strategy, with each scheduling strategy including multiple action decisions. Determine whether the scheduling strategy for the current round conforms to the preset scheduling strategy based on all the rewards of the current round. If not, return to the step of obtaining the running status information of multiple trains until the scheduling strategy corresponding to the current round meets the preset scheduling strategy to dynamically and collaboratively optimize the scheduling of multiple trains. In the step of calculating the new operating status information and the composite reward feedback of the current decision step according to the adjustment instruction and updating the train environment status information, the calculation formula for the total reward integration mechanism of the composite reward feedback is as follows: ； In the formula: k indicates whether the train is at the station, where 1 indicates that the train is at the station, and 0 indicates that the train is not at the station or the train has just arrived at the station; r in This represents the total reward returned by the environment when the train is inside the station, where... r in = r dwell , r dwell This indicates a reward for efficient station stops; r out This represents the total reward returned by the environment when the train is outside the station or has just arrived at the station, where... ； In the formula, r safe Indicates rewards and penalties for safe operation; r speed This indicates the penalties for speeding. r delay Indicates a reward for improving performance after delays; r punct This indicates a reward for arriving on time. r term Indicates the reward for reaching the finish line; ∩ represents the intersection. The formula for calculating the delay improvement bonus is as follows: ； In the formula, r delay Indicates a reward for improving performance after delays; d t Indicates the current delay; d t-1 d0 represents the delay at the previous moment; d0 represents the difference between the current delay and the previous moment's delay; ∩ represents the intersection. λ delay This indicates the incentive coefficient for improving delays.

2. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The formula for calculating the on-time arrival bonus is as follows: ； In the formula, r punct This indicates a reward for arriving on time. This represents the calculation of the on-time linear decay function, where, This indicates the actual time the train arrives at station S. Indicates the train's scheduled arrival time at station S; λ punct 's' represents the on-time arrival bonus coefficient; 's' represents the station number.

3. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The formula for calculating the finish line completion bonus is: ； In the formula, r term Indicates a reward for reaching the finish line; λ term Indicates the reward coefficient for reaching the finish line; f This represents the gain coefficient for trains stopping at major stations; d t 60 indicates the current delay; 60 indicates 60 minutes, which is the maximum acceptable delay limit for the terminal station.

4. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The formula for calculating safety operation rewards and penalties is as follows: ； In the formula, r safe Indicates rewards and penalties for safe operation; d front Indicates the distance between the current train and the train in front; λ safe This represents the safety operation reward coefficient.

5. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The formula for calculating the stop efficiency bonus is as follows: ； In the formula, r dwell This indicates a reward for efficient station stops; λ dwell This represents the stop efficiency bonus coefficient; τ stop Indicates the actual stop time; τ min Indicates the minimum stopping time; τ target Indicates the target stopping time.

6. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The formula for calculating speeding penalties is as follows: ； In the formula, r speed This indicates the penalties for speeding. λ speed This indicates the reward coefficient for speeding penalties; v Indicates the current speed; v lim This indicates the maximum permissible speed.

7. The multi-train dynamic scheduling method based on the MAPPO algorithm according to claim 1, characterized in that, The steps for identifying the operational state information based on a pre-trained neural network policy model to generate discrete action decisions include: Obtain the model initialization instruction, and initialize the pre-trained neural network policy model according to the model initialization instruction; The running state information is identified based on the initialized pre-trained neural network policy model to generate discrete action decisions.

8. A multi-train dynamic scheduling system based on the MAPPO algorithm, characterized in that, The system includes: The acquisition module is used to acquire the operating status information of multiple trains and identify the operating status information according to a pre-trained neural network strategy model to generate discrete action decisions. The training module is used to generate adjustment instructions based on discrete action decisions. The adjustment instructions include speed adjustment and start / stop. Based on the adjustment instructions, the module calculates new operating status information and composite reward feedback of the current decision step and updates the train environment status information to enter the next decision step until the end of the current round. Each round includes multiple decision steps, each decision step generates an action decision, and each round generates a scheduling strategy, each scheduling strategy including multiple action decisions. The judgment module is used to determine whether the scheduling strategy corresponding to the current round conforms to the preset scheduling strategy based on all the rewards of the current round. The first execution module is used to return to the step of obtaining the running status information of multiple trains if the scheduling strategy corresponding to the current round does not meet the preset scheduling strategy, until the scheduling strategy corresponding to the current round meets the preset scheduling strategy in order to dynamically and collaboratively optimize the scheduling of multiple trains. In the step of calculating the new operating status information and the composite reward feedback of the current decision step according to the adjustment instruction and updating the train environment status information, the calculation formula for the total reward integration mechanism of the composite reward feedback is as follows: ； In the formula: k indicates whether the train is at the station, where 1 indicates that the train is at the station, and 0 indicates that the train is not at the station or the train has just arrived at the station; r in This represents the total reward returned by the environment when the train is inside the station, where... r in = r dwell , r dwell This indicates a reward for efficient station stops; r out This represents the total reward returned by the environment when the train is outside the station or has just arrived at the station, where... ； In the formula, r safe Indicates rewards and penalties for safe operation; r speed This indicates the penalties for speeding. r delay Indicates a reward for improving performance after delays; r punct This indicates a reward for arriving on time. r term Indicates the reward for reaching the finish line; ∩ represents the intersection. The formula for calculating the delay improvement bonus is as follows: ； In the formula, r delay Indicates a reward for improving performance after delays; d t Indicates the current delay; d t-1 d0 represents the delay at the previous moment; d0 represents the difference between the current delay and the previous moment's delay; ∩ represents the intersection. λ delay This indicates the incentive coefficient for improving delays.

Images