A bus station service interruption prevention method based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's bus station service interruption prevention method, comprising the following steps: S1, the bus operation simulation model of the combination of traffic and passenger flow is established, simulation model is as the environment of reinforcement learning, and state space and action space are designed;S2, the current operating state of bus system is collected, and the invalid action under the reinforcement learning algorithm is shielded according to operating state;S3, the environment state is predicted using long short-term memory model, so that the agent continuously interacts with the environment to learn the optimal strategy, and the algorithm parameters are updated until the set training round number is completed;S4, the trained reinforcement learning algorithm is used to simulate the calculation of the bus system, and the agent controls the vehicle to stay in the station, delays the departure time, and achieves the prevention and treatment of bus station service interruption. Compared with the traditional technology, the method makes real-time perception of passenger flow and vehicle position information in the bus system, and makes dynamic decision on bus station departure.

Description

Technical Field

[0001] This invention relates to the field of intelligent bus dispatching technology, and more specifically, to a method for preventing service interruptions at bus stations based on reinforcement learning. Background Technology

[0002] In a public transportation system, a service interruption occurs when the departure interval between two adjacent buses exceeds a given time value. Ideally, all buses should adhere to their pre-planned schedules. However, in practice, buses are severely affected by factors such as traffic flow, traffic lights, uneven passenger flow, and weather, causing deviations in the return time after completing previous trips from the pre-planned schedule. This results in buses being unable to fulfill their scheduled departures. Delayed buses typically need to serve more passengers accumulating at stops, leading to longer dwell times and further exacerbating delays. As the headway between buses gradually widens, it can lead to service interruptions, even disrupting the entire timetable and severely deteriorating the overall level of public transportation service.

[0003] Public transport service disruptions are a shared concern for both bus companies and passengers. When disruptions occur, uneven passenger loads occur on buses; leading buses experience overcrowding, causing passenger discomfort, while lagging buses waste capacity due to empty carriages. Furthermore, bus delays extend waiting and travel times for passengers. Existing technologies include strategies such as bus speed regulation, bus substitution, bus injection, station control, skip-stop control, short-stop services, and bus signal priority, or combinations of these strategies. Station control, which extends vehicle dwell time at stops, is the most widely studied method; however, its implementation increases passenger dwell time, potentially causing dissatisfaction, and its practicality in congested areas is limited. Speed ​​regulation places high demands on drivers, while skip-stop and short-stop services restrict passenger boarding, and other methods require additional resources or infrastructure. Moreover, traditional strategies, due to short-sighted decision-making and a lack of in-depth exploration of the problem structure, are not ideal for solving complex stochastic problems.

[0004] To improve the reliability of public transport services, researchers have invested significant effort in developing flexible control strategies, such as bus speed regulation, bus substitution, bus injection, station control, skip-stop control, short-distance buses, and bus signal priority, or combinations of these strategies. However, these strategies often negatively impact passenger travel experience or require additional resources or infrastructure, making them difficult to implement in many cities. How to better improve the reliability of public transport services, thereby enhancing passenger travel experience and increasing the attractiveness of public transport services, remains a crucial issue requiring further research.

[0005] Existing technology discloses a method for optimizing bus route congestion based on the combined scheduling of full-route buses and short-route buses. This method determines the starting station and departure number of short-route buses using historical passenger flow data and GPS data. Then, it establishes a mathematical model with the objective of minimizing departure time deviation, transforms the model into a mixed-integer linear programming problem, and solves it using the branch and bound method. Without changing the original departure numbers, it improves the stability of headway stability and alleviates bus congestion by combining full-route buses and short-route buses. The drawbacks of this method are that it relies on operations research modeling, requiring numerous ideal conditions that do not fit real-world operational scenarios. Furthermore, it consumes a significant amount of computation time, making it difficult to achieve real-time optimization and scale to large-scale real-world scenarios.

[0006] Therefore, in light of the above requirements and the shortcomings of existing technologies, this application proposes a method for preventing service interruptions at bus stations based on reinforcement learning. Summary of the Invention

[0007] This invention provides a method for preventing service interruptions at bus stations based on reinforcement learning. It can make dynamic decisions on bus departures from bus stations by sensing passenger flow and vehicle location information in the bus system in real time, and control the headway of buses within a reasonable range to solve the problem of service interruptions at stations. It has high operability and decision-making efficiency and is more suitable for real-world operation scenarios.

[0008] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows:

[0009] The first aspect of this invention provides a method for preventing service disruptions at bus stations based on reinforcement learning. This method includes the following steps:

[0010] S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space.

[0011] S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status;

[0012] S3. Use the long short-term memory model to predict the state of the environment, so that the agent can continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed.

[0013] S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

[0014] In this context, the intelligent agent is equivalent to a bus dispatcher who executes the departure plan, while the operating environment of the reinforcement learning algorithm is constructed using a bus operation simulation model.

[0015] Furthermore, the reinforcement learning algorithm needs to design a state space and an action space; the state space uses a long short-term memory model to predict bus travel time, enabling the agent to simultaneously perceive the headway of the vehicle to be decided and several vehicles in front and behind it; the action space of each bus departure station is an independent discrete action space, which includes two actions: departure and not departure.

[0016] In designing the state space, to integrate global information, passengers on board and waiting passengers are used as features input into a Long Short-Term Memory (LSTM) model to predict bus travel time.

[0017] Furthermore, the state space at time step t can be represented as:

[0018] s t ={τ t ,μ t ,f t ,b t} (1)

[0019] Where, τ t μ represents the current time at time step t. t ={0,1} indicates whether there are trains available to depart from the station, f t ={f t 1 ,f t 2 ,…,f t u} represents the departure time of the most recently dispatched vehicle u compared to τ. t Time difference, This represents the travel time required for vehicle v to reach the departure station, as predicted using a long short-term memory model.

[0020] Furthermore, the action space of each station is an independent discrete action space, consisting of two actions: "departure" and "no departure"; the action space of station k at time step t is represented as:

[0021]

[0022] Specifically, the algorithm parameters for setting the reinforcement learning algorithm are: setting the maximum number of training rounds e, the number of time steps T per round, the batch size d, initializing the replay buffer D, and the parameters of the Actor network and the Critic network.

[0023] Furthermore, step S2 specifically involves: when there are parked vehicles in a certain state of the station, both departure and non-departure are valid actions; when there are no parked vehicles in the station, departure is an invalid action; step S2 collects the current operating status of the public transportation system and uses invalid action masking technology to avoid invalid actions.

[0024] Sampling invalid actions can affect the decision-making of bus dispatchers, i.e., intelligent agents.

[0025] Furthermore, the invalid action masking technique introduces a state-dependent differentiable function into the policy gradient algorithm. Specifically, in the policy gradient algorithm, a neural network typically outputs non-standardized scores (logits), which are then converted into action probability distributions using a softmax operation or an equivalent operation. Let l(s) represent the non-standardized scores (logits) output by the policy network in state s. The invalid action masking process utilizes a state-dependent differentiable function inv s Invalid action a i The corresponding logits are represented by a very large negative number ξ (e.g., ξ = -1 × 10). 8 The action probability distribution π′ is replaced by another action, while the logits of other actions remain unchanged; at this time, the action probability distribution π′ is... θ (·|s t This can be represented as:

[0026] π′ θ (·|s t = softmax(inv) s (l(s))) (3)

[0027]

[0028] When ξ is a sufficiently large negative number, the above operation will make the probability of the policy network selecting an invalid action almost zero. When the policy network needs to be updated, the gradient corresponding to the logits of the invalid action is zero.

[0029] Furthermore, the reinforcement learning algorithm enables the agent to continuously interact with the environment to learn the optimal policy, and its training process is as follows:

[0030] S31, based on state s t Sampling action a t And execute a in the bus operation simulation environment t This enables intelligent agents to interact with their environment.

[0031] S32. Based on the state space designed in step S1, use the Long Short-Term Memory model to predict the bus travel time, and obtain the execution a. t Subsequent environmental state s t+1 .

[0032] S33. Calculate and execute a based on the reward function. t The reward r obtained later t and tuple (s t ,a t ,r t ,s t+1 Store in the experience replay library D.

[0033] S34. If the number of tuples in the experience replay library D reaches the batch size d, then calculate the dominance function. Update the parameters of the Actor network and Critic network, and clear the experience replay library D; otherwise, proceed to step S2.

[0034] S35. If the current round number reaches the maximum training round number e, then the algorithm model training ends; otherwise, proceed to step S2.

[0035] Furthermore, the reward function includes the departure interval and the station dwell time; its mathematical expression is as follows:

[0036]

[0037]

[0038] Where w1 and w2 are weight hyperparameters; k = 1, 2, ..., K is the kth departure station; n = 1, 2, ..., N is the nth vehicle; To prevent service interruptions due to excessively long departure intervals; ΔT t k Let T be the departure interval between two adjacent trains at station k in time step t; α This is the threshold for the time interval between train departures at the station; a service interruption is considered complete when the departure interval exceeds this value. β The relaxation time is used to penalize the departure intervals that are close to service interruption conditions, so that the agent can learn better to avoid service interruptions. This is a penalty for vehicles being parked at stations. On the one hand, it can prevent intelligent agents from choosing to park vehicles at stations for a long time in order to avoid service interruption. On the other hand, it can improve vehicle utilization efficiency and reduce passenger waiting time.

[0039] Furthermore, step S4 specifically involves: using a simulation model to simulate the service of the actual public transportation system, setting a service interruption threshold, and using a trained reinforcement learning strategy to make dynamic decisions on bus station departure behavior, so that the departure interval is controlled within the service interruption threshold, thereby achieving the prevention and control of bus station service interruptions.

[0040] A second aspect of the present invention provides a bus station service interruption prevention system based on reinforcement learning, comprising a memory and a processor. The memory includes a bus station service interruption prevention program based on reinforcement learning. When the processor executes the bus station service interruption prevention program based on reinforcement learning, it performs the following steps:

[0041] S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space.

[0042] S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status;

[0043] S3. Use the long short-term memory model to predict the state of the environment, so that the agent can continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed.

[0044] S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

[0045] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0046] This invention provides a method for preventing service interruptions at bus stations based on reinforcement learning. By simulating and modeling the bus system, and using a trained reinforcement learning algorithm, the method can make dynamic decisions on bus departures from bus stations by sensing passenger flow and vehicle location information in real time. This controls the headway between buses within a reasonable range, thereby solving the problem of service interruptions at bus stations. The method is highly operable and efficient in decision-making, and is more suitable for real-world operational scenarios. Attached Figure Description

[0047] Figure 1 This is a flowchart of a bus station service interruption prevention method based on reinforcement learning according to the present invention.

[0048] Figure 2 This is a schematic diagram of bus station departure intervals according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the driving trajectory of a bus according to an embodiment of the present invention.

[0050] Figure 4 This is a schematic diagram of a bus station service interruption prevention system based on reinforcement learning according to the present invention.

[0051] Figure 5 This is a flowchart of one embodiment of the present invention. Detailed Implementation

[0052] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0054] Example 1

[0055] like Figure 1 As shown, this invention provides a method for preventing service disruptions at bus stations based on reinforcement learning. This method includes the following steps:

[0056] S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space.

[0057] S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status;

[0058] S3. Use the long short-term memory model to predict the state of the environment, so that the agent can continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed.

[0059] S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

[0060] In this context, the intelligent agent is equivalent to a bus dispatcher who executes the departure plan, while the operating environment of the reinforcement learning algorithm is constructed using a bus operation simulation model.

[0061] Furthermore, the reinforcement learning algorithm needs to design a state space and an action space; the state space uses a long short-term memory model to predict bus travel time, enabling the agent to simultaneously perceive the headway of the vehicle to be decided and several vehicles in front and behind it; the action space of each bus departure station is an independent discrete action space, which includes two actions: departure and not departure.

[0062] In designing the state space, to integrate global information, passengers on board and waiting passengers are used as features input into a Long Short-Term Memory (LSTM) model to predict bus travel time.

[0063] Furthermore, the state space at time step t can be represented as:

[0064] s t ={τ t ,μ t ,f t ,b t} (1)

[0065] Where, τ t μ represents the current time at time step t. t ={0,1} indicates whether there are trains available to depart from the station, f t ={f t 1 ,f t 2 ,…,f t u} represents the departure time of the most recently dispatched vehicle u compared to τ. t Time difference, This represents the travel time required for vehicle v to reach the departure station, as predicted using a long short-term memory model.

[0066] Furthermore, the action space of each station is an independent discrete action space, consisting of two actions: "departure" and "no departure"; the action space of station k at time step t is represented as:

[0067]

[0068] Specifically, the algorithm parameters for setting the reinforcement learning algorithm are: setting the maximum number of training rounds e, the number of time steps T per round, the batch size d, initializing the replay buffer D, and the parameters of the Actor network and the Critic network.

[0069] Furthermore, step S2 specifically involves: when there are parked vehicles in a certain state of the station, both departure and non-departure are valid actions; when there are no parked vehicles in the station, departure is an invalid action; step S2 collects the current operating status of the public transportation system and uses invalid action masking technology to avoid invalid actions.

[0070] Sampling invalid actions can affect the decision-making of bus dispatchers, i.e., intelligent agents.

[0071] Furthermore, the invalid action masking technique involves introducing a state-dependent differentiable function into the policy gradient algorithm. Specifically, in the policy gradient algorithm, a neural network typically outputs non-standardized scores (logits), which are then converted into action probability distributions using a softmax operation or an equivalent operation. Let l(s) represent the non-standardized scores (logits) output by the policy network in state s. The invalid action masking process utilizes a state-dependent differentiable function inv s Invalid action a i The corresponding logits are represented by a very large negative number ξ (e.g., ξ = -1 × 10). 8 The action probability distribution π′ is replaced by another action, while the logits of other actions remain unchanged; at this time, the action probability distribution π′ is... θ (·|s t This can be represented as:

[0072] π′ θ (·|s t = softmax(inv) s (l(s))) (3)

[0073]

[0074] When ξ is a sufficiently large negative number, the above operation will make the probability of the policy network selecting an invalid action almost zero. When the policy network needs to be updated, the gradient corresponding to the logits of the invalid action is zero.

[0075] Furthermore, the reinforcement learning algorithm enables the agent to continuously interact with the environment to learn the optimal policy. In one embodiment, such as... Figure 5 As shown, the specific training process is as follows:

[0076] S31, based on state s t Sampling action a t And execute a in the bus operation simulation environment t This enables intelligent agents to interact with their environment.

[0077] S32. Based on the state space designed in step S1, use the Long Short-Term Memory model to predict the bus travel time, and obtain the execution a. t The subsequent environmental state s t+1 .

[0078] S33. Calculate and execute a based on the reward function. t The reward r obtained later t and tuple (s t ,a t ,r t ,s t+1 Store in the experience replay library D.

[0079] S34. If the number of tuples in the experience replay library D reaches the batch size d, then calculate the dominance function. Update the parameters of the Actor network and Critic network, and clear the experience replay library D; otherwise, proceed to step S2.

[0080] S35. If the current round number reaches the maximum training round number e, then the algorithm model training ends; otherwise, proceed to step S2.

[0081] Furthermore, the reward function includes the departure interval and the station dwell time; its mathematical expression is as follows:

[0082]

[0083]

[0084] Where w1 and w2 are weight hyperparameters; k = 1, 2, ..., K is the kth departure station; n = 1, 2, ..., N is the nth vehicle; To prevent service interruptions due to excessively long departure intervals; ΔT t k Let T be the departure interval between two adjacent trains at station k in time step t; α This is the threshold for the time interval between train departures at the station; a service interruption is considered complete when the departure interval exceeds this value. β The relaxation time is used to penalize the departure intervals that are close to service interruption conditions, so that the agent can learn better to avoid service interruptions. This is a penalty for vehicles being parked at stations. On the one hand, it can prevent intelligent agents from choosing to park vehicles at stations for a long time in order to avoid service interruption. On the other hand, it can improve vehicle utilization efficiency and reduce passenger waiting time.

[0085] Furthermore, step S4 specifically involves: using a simulation model to simulate the service of the actual public transportation system, setting a service interruption threshold, and using a trained reinforcement learning strategy to make dynamic decisions on bus station departure behavior, so that the departure interval is controlled within the service interruption threshold, thereby achieving the prevention and control of bus station service interruptions.

[0086] This invention proposes a reinforcement learning-based method for preventing service interruptions at bus stations. It utilizes an LSTM model to predict bus travel times, enabling the agent to perceive the headway between station vehicles and operating buses, thus better assessing the long-term impact of decisions. For scenarios where no buses are available at the station, an invalid action masking technique is employed to prevent the agent from issuing invalid commands. A reward function is used to penalize large departure intervals, and the model is trained using the Proximal Policy Optimization (PPO) algorithm.

[0087] Example 2

[0088] Based on the above embodiment 1, combined with Figures 2-3 This embodiment, combined with simulation experiments, elaborates in detail the process by which the present invention avoids station service interruptions.

[0089] In a specific embodiment, we consider a two-way bus route with two departure stations, 24 stops in each direction, and a total of 12 buses. The simulation period is [6:00, 10:00] for a total of 4 hours. A service interruption occurs when the departure interval exceeds 20 minutes. The travel time between stations follows a Gaussian distribution, and the passenger arrival time at the station follows a Poisson distribution.

[0090] Considering the uncertainties in travel time and passenger flow, the reliability of model-controlled departure was evaluated through 100 simulation experiments. The departure intervals for both the uphill and downhill directions were statistically analyzed, and the results are as follows: Figure 2 As shown in the figure. The results show that the departure intervals for both the up and down lines are controlled within 18 minutes, proving that the method proposed in this invention can effectively avoid station service interruptions.

[0091] The bus trajectory diagram obtained from the model control station departure is as follows: Figure 3 As shown, during the morning rush hour, the travel time between stations is relatively long. The intelligent agent controls the vehicles to remain at the station, delaying departure times, thereby avoiding service interruptions.

[0092] Example 3

[0093] like Figure 3 As shown, the present invention also provides a bus station service interruption prevention system based on reinforcement learning. The memory includes a bus station service interruption prevention program based on reinforcement learning. When the processor executes the bus station service interruption prevention program based on reinforcement learning, it performs the following steps:

[0094] S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space.

[0095] S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status;

[0096] S3. Use the long short-term memory model to predict the state of the environment, so that the agent can continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed.

[0097] S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

[0098] Furthermore, the reinforcement learning algorithm needs to design a state space and an action space; the state space uses a long short-term memory model to predict bus travel time, enabling the agent to simultaneously perceive the headway of the vehicle to be decided and several vehicles in front and behind it; the action space of each bus departure station is an independent discrete action space, which includes two actions: departure and not departure.

[0099] In designing the state space, to integrate global information, passengers on board and waiting passengers are used as features input into a Long Short-Term Memory (LSTM) model to predict bus travel time.

[0100] Furthermore, the state space at time step t can be represented as:

[0101] s t ={τ t ,μ t ,f t ,b t} (1)

[0102] Where, τ t μ represents the current time at time step t. t ={0,1} indicates whether there are trains available to depart from the station, f t ={f t 1 ,f t 2 ,…,f t u} represents the departure time of the most recently dispatched vehicle u compared to τ. t Time difference, This represents the travel time required for vehicle v to reach the departure station, as predicted using a long short-term memory model.

[0103] Furthermore, the action space of each station is an independent discrete action space, consisting of two actions: "departure" and "no departure"; the action space of station k at time step t is represented as:

[0104]

[0105] Specifically, the algorithm parameters for setting the reinforcement learning algorithm are: setting the maximum number of training rounds e, the number of time steps T per round, the batch size d, initializing the replay buffer D, and the parameters of the Actor network and the Critic network.

[0106] Furthermore, step S2 specifically involves: when there are parked vehicles in a certain state of the station, both departure and non-departure are valid actions; when there are no parked vehicles in the station, departure is an invalid action; step S2 collects the current operating status of the public transportation system and uses invalid action masking technology to avoid invalid actions.

[0107] Sampling invalid actions can affect the decision-making of bus dispatchers, i.e., intelligent agents.

[0108] Furthermore, the invalid action masking technique involves introducing a state-dependent differentiable function into the policy gradient algorithm. Specifically, in the policy gradient algorithm, a neural network typically outputs non-standardized scores (logits), which are then converted into action probability distributions using a softmax operation or an equivalent operation. Let l(s) represent the non-standardized scores (logits) output by the policy network in state s. The invalid action masking process utilizes a state-dependent differentiable function inv s Invalid action a i The corresponding logits are represented by a very large negative number ξ (e.g., ξ = -1 × 10). 8 The action probability distribution π′ is replaced by another action, while the logits of other actions remain unchanged; at this time, the action probability distribution π′ is... θ (·|s t This can be represented as:

[0109] π′ θ (·|s t = softmax(inv) s (l(s))) (3)

[0110]

[0111] When ξ is a sufficiently large negative number, the above operation will make the probability of the policy network selecting an invalid action almost zero. When the policy network needs to be updated, the gradient corresponding to the logits of the invalid action is zero.

[0112] Furthermore, the reinforcement learning algorithm enables the agent to continuously interact with the environment to learn the optimal policy, and its training process is as follows:

[0113] S31, based on state s t Sampling action a t And execute a in the bus operation simulation environment t This enables intelligent agents to interact with their environment.

[0114] S32. Based on the state space designed in step S1, use the Long Short-Term Memory model to predict the bus travel time, and obtain the execution a. tThe subsequent environmental state s t+1 .

[0115] S33. Calculate and execute a based on the reward function. t The reward r obtained later t and tuple (s t ,a t ,r t ,s t+1 Store in the experience replay library D.

[0116] S34. If the number of tuples in the experience replay library D reaches the batch size d, then calculate the dominance function. Update the parameters of the Actor network and Critic network, and clear the experience replay library D; otherwise, proceed to step S2.

[0117] S35. If the current round number reaches the maximum training round number e, then the algorithm model training ends; otherwise, proceed to step S2.

[0118] Furthermore, the reward function includes the departure interval and the station dwell time; its mathematical expression is as follows:

[0119]

[0120]

[0121] Where w1 and w2 are weight hyperparameters; k = 1, 2, ..., K is the kth departure station; n = 1, 2, ..., N is the nth vehicle; Penalties to prevent service interruptions due to excessively long departure intervals;

[0122] Let T be the departure interval between two adjacent trains at station k in time step t; α This is the threshold for the time interval between train departures at the station; a service interruption is considered complete when the departure interval exceeds this value. β The relaxation time is used to penalize the departure intervals that are close to service interruption conditions, so that the agent can learn better to avoid service interruptions. This is a penalty for vehicles being parked at stations. On the one hand, it can prevent intelligent agents from choosing to park vehicles at stations for a long time in order to avoid service interruption. On the other hand, it can improve vehicle utilization efficiency and reduce passenger waiting time.

[0123] Furthermore, step S4 specifically involves: using a simulation model to simulate the service of the actual public transportation system, setting a service interruption threshold, and using a trained reinforcement learning strategy to make dynamic decisions on bus station departure behavior, so that the departure interval is controlled within the service interruption threshold, thereby achieving the prevention and control of bus station service interruptions.

[0124] The icons depicting the structural positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0125] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for preventing service disruptions at bus stations based on reinforcement learning, characterized in that, Includes the following steps: S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space. S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status; S3. Utilize a Long Short-Term Memory (LSTM) model to predict the environmental state, enabling the agent to continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed; the specific algorithm parameters for the reinforcement learning algorithm are: setting the maximum number of training rounds. Time steps per round Batch size Initialize the experience replay library The parameters of the Actor network and Critic network; reinforcement learning algorithms enable agents to continuously interact with the environment to learn the optimal policy, and the training process is as follows: S31, According to the state Sampling action And execute in the bus operation simulation environment This enables intelligent agents to interact with their environment; S32. Based on the state space designed in step S1, use the Long Short-Term Memory model to predict the bus travel time and obtain the execution... Subsequent environmental conditions ; S33, Calculate and execute based on the reward function. The reward received later and tuple Store to experience replay library The reward function includes departure interval and station dwell time; its mathematical expression is as follows: in, and These are weight hyperparameters; For the first One departure station; For the first Vehicle; Penalties to prevent service interruptions due to excessively long departure intervals; For the station At time step The departure interval between two adjacent trains; This is the threshold for the time interval between service interruptions at the station. When the departure interval exceeds this value, it is considered a service interruption. The relaxation time is used to penalize the departure intervals that are close to service interruption conditions, so that the agent can learn better to avoid service interruptions. The penalty for vehicles parking at stations can prevent agents from choosing to park vehicles at stations for extended periods to avoid service interruptions, and can also improve vehicle utilization efficiency and reduce passenger waiting time. S34. If the experience replay library The number of tuples in the batch reaches the batch size. Then calculate the dominance function. Update the parameters of the Actor and Critic networks, and clear the experience replay library. Otherwise, proceed to step S2; S35. If the current round number reaches the maximum training round number. If the algorithm model training ends, then proceed to step S2; otherwise, go to step S2. S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

2. The method for preventing service disruptions at bus stations based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm requires the design of a state space and an action space. The state space uses a long short-term memory model to predict bus travel time, enabling the agent to simultaneously perceive the headway between the vehicle to be decided and several vehicles in front and behind it. The action space of each bus departure station is an independent discrete action space, which includes two actions: departure and non-departure.

3. The method for preventing service disruptions at bus stations based on reinforcement learning according to claim 2, characterized in that, At time step The state space at time can be represented as: in, Indicates at time step At the current moment, Indicates whether there are trains available at the station. Indicates the most recently issued The departure time of the vehicle and Time difference, This indicates predictions made using a long short-term memory model. The travel time required for the vehicle to reach the departure station.

4. The method for preventing service disruptions at bus stations based on reinforcement learning according to claim 3, characterized in that, Each station's operational space is an independent, discrete operational space, consisting of two actions: departure and non-departure; station At time step The action space is represented as: 。 5. A method for preventing service disruptions at bus stations based on reinforcement learning according to claim 4, characterized in that, Specifically, step S2 is as follows: when there are parked vehicles in a certain state of the station, both departure and non-departure are valid actions; when there are no parked vehicles in the station, departure is an invalid action; step S2 collects the current operating status of the bus system and uses invalid action masking technology to avoid invalid actions.

6. A method for preventing service disruptions at bus stations based on reinforcement learning according to claim 5, characterized in that, The invalid action masking technique involves introducing a state-dependent differentiable function into the policy gradient algorithm. Specifically, the policy gradient algorithm typically uses a neural network to output non-standardized scores, which are then converted into an action probability distribution using a softmax operation or an equivalent operation. Indicates the state The non-standardized score output by the policy network and the process of masking invalid actions utilize a state-dependent differentiable function. Invalid actions The corresponding unstandardized score is represented by a negative number. The non-standardized scores of other actions are replaced, while the non-standardized scores of other actions remain unchanged; at this time, the action probability distribution... It can be represented as: when If the value is a sufficiently large negative number, the above operation will make the probability of the policy network selecting an invalid action almost zero; when the policy network needs to be updated, the gradient corresponding to the logits of the invalid action will be zero.

7. A method for preventing service disruptions at bus stations based on reinforcement learning according to claim 6, characterized in that, Step S4 specifically involves: using a simulation model to simulate the service of the actual public transportation system, setting a service interruption threshold, and using a trained reinforcement learning strategy to make dynamic decisions on bus station departure behavior, so that the departure interval is controlled within the service interruption threshold, thereby achieving the prevention and control of bus station service interruptions.

8. A bus station service interruption prevention system based on reinforcement learning, comprising a memory and a processor, wherein the memory includes a bus station service interruption prevention program based on reinforcement learning, and the bus station service interruption prevention program based on reinforcement learning, when executed by the processor, performs the following steps: S1. Establish a public transport operation simulation model that combines vehicle flow and passenger flow, use the simulation model as a reinforcement learning environment, and design the state space and action space. S2. Collect the current operating status of the public transportation system and mask invalid actions under the reinforcement learning algorithm based on the operating status; S3. Utilize a Long Short-Term Memory (LSTM) model to predict the environmental state, enabling the agent to continuously interact with the environment to learn the optimal strategy and update the algorithm parameters until the set number of training rounds is completed; the specific algorithm parameters for the reinforcement learning algorithm are: setting the maximum number of training rounds. Time steps per round Batch size Initialize the experience replay library The parameters of the Actor network and Critic network; reinforcement learning algorithms enable agents to continuously interact with the environment to learn the optimal policy, and the training process is as follows: S31, According to the state Sampling action And execute in the bus operation simulation environment This enables intelligent agents to interact with their environment; S32. Based on the state space designed in step S1, use the Long Short-Term Memory model to predict the bus travel time and obtain the execution... Subsequent environmental conditions ; S33, Calculate and execute based on the reward function. The reward received later and tuple Store to experience replay library The reward function includes departure interval and station dwell time; its mathematical expression is as follows: in, and These are weight hyperparameters; For the first One departure station; For the first Vehicle; Penalties to prevent service interruptions due to excessively long departure intervals; For the station At time step The departure interval between two adjacent trains; This is the threshold for the time interval between service interruptions at the station. When the departure interval exceeds this value, it is considered a service interruption. The relaxation time is used to penalize the departure intervals that are close to service interruption conditions, so that the agent can learn better to avoid service interruptions. The penalty for vehicles parking at stations can prevent agents from choosing to park vehicles at stations for extended periods to avoid service interruptions, and can also improve vehicle utilization efficiency and reduce passenger waiting time. S34. If the experience replay library The number of tuples in the batch reaches the batch size. Then calculate the dominance function. Update the parameters of the Actor and Critic networks, and clear the experience replay library. ; Otherwise, proceed to step S2; S35. If the current round number reaches the maximum training round number. If the algorithm model training ends, then proceed to step S2; otherwise, go to step S2. S4. The trained reinforcement learning algorithm is used to simulate the bus system. The agent controls the vehicles to stay at the station and delays the departure time to prevent the interruption of bus station services.

Images