A bus stop dynamic equilibrium timetable generation method based on DQN

Abstract

The application discloses a kind of based on DQN realizes bus up-down action dynamic equilibrium timetable generation method, including initializing bus route information, obtaining the historical traffic conditions and passenger flow information of up-down line, initializing reinforcement learning DQN model parameters, construct the DQN model of bus up-down action dynamic equilibrium timetable that can be generated, including design the state matrix of DQN model of bus up-down action dynamic equilibrium timetable that can be generated, action state, reward function, the inference of the DQN model of the present application is output bus up-down action dynamic equilibrium timetable result, using direct and simulation environment interaction to train, decision-making is carried out in real-time simulation environment, realize real-time mode generation bus up-down action dynamic equilibrium timetable, it can be based on the passenger flow and station information after change inference generates new departure timetable, solve the problem that existing departure timetable cannot dynamically adapt to passenger flow and traffic condition change, lead to scheduling not to meet actual requirement and cannot be applied.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of bus operation system, in particular to a method for generating dynamic balanced timetable of bus boarding and alighting based on deep reinforcement learning (DQN). BACKGROUND

[0002] Optimizing bus departure timetable can effectively improve the quality of bus service and operation efficiency. In practical applications, the bus departure time point of the bus timetable is optimized according to passenger flow.

[0003] At present, the methods for optimizing bus timetable include genetic algorithm, graph heuristic algorithm and exhaustive method. These methods can generate bus timetable in offline mode according to historical passenger flow. In the actual operation process of the bus system, various unexpected situations will be encountered, including sudden increase of passenger flow caused by unexpected events, bus travel time caused by road congestion, and station change caused by road closure construction and other situations. The timetable generated based on historical data cannot meet the travel needs of passengers. In addition, the actual departure of the bus system requires two-way departure of the line and the vehicle cannot be empty. The existing reinforcement learning based method optimizes the departure timetable of a one-way line. The generated departure timetable cannot dynamically adapt to changes in passenger flow and traffic conditions, and cannot schedule both directions of a bus line at the same time, resulting in a scheduling that does not meet the actual requirements and cannot be applied. SUMMARY

[0004] The present application aims to provide a method for generating dynamic balanced timetable of bus boarding and alighting based on DQN. Based on the traffic conditions and bus passenger flow information of the bus operation, the method is trained by directly interacting with the simulation environment, makes decisions in the real-time simulation environment, and generates a dynamic balanced timetable of bus boarding and alighting. When the passenger flow changes due to unexpected events, a new departure timetable can be generated based on the changed passenger flow. When the bus travel time suddenly changes, a new departure timetable can be generated based on the changed travel time. When the bus station is temporarily adjusted or changed, a new departure timetable can be generated based on the changed station information, so as to solve the problems raised in the background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0006] A method for generating dynamic balanced timetable of bus boarding and alighting based on DQN, comprising the following steps:

[0007] S1: initialize bus line information, including setting line number ID, vehicle shortest departure interval t min , vehicle longest departure interval t max , vehicle carrying capacity C max , first departure time Start time, End time ;

[0008] S2: Obtain historical traffic conditions and passenger flow information on the line;

[0009] S3: Initialize the DQN model parameters of reinforcement learning, input dimension m input , output dimension m output , number of hidden layers m hidden , number of hidden layer neurons n hidden , activation function ReLU, learning rate lr, batch size batch_size, experience replay capacity C ER ;

[0010] S4: Construct a DQN model that can generate a dynamic equilibrium timetable for bus up and down;

[0011] S5: The DQN model that can generate a dynamic equilibrium timetable for bus up and down is inferred. Based on the trained model, the traffic conditions and passenger flow data of the current line are used for inference to obtain a dynamic equilibrium timetable for bus up and down. When the passenger flow changes due to a sudden event, a new departure timetable can be generated based on the changed passenger flow. When the bus travel time suddenly changes, a new departure timetable can be generated based on the changed travel time. When the bus station is temporarily adjusted and changed, a new departure timetable can be generated based on the changed station information.

[0012] S6: Output the results of the dynamic equilibrium timetable for bus up and down. Output the planned departure time point of each line in the up and down directions, and include the interval between the current departure time and the last departure time, the passenger capacity provided by each specific interval of the departure plan, the total number of departures, the number of passengers who cannot board due to full vehicles, and the average waiting time of passengers.

[0013] Further, the specific method in S2 is:

[0014] S201: Calculate the travel time between two adjacent stations for each bus based on the data of the bus entering and leaving the station;

[0015] S202: Calculate the travel time between two adjacent stations every 15 minutes in a day based on the travel time between two adjacent stations for each bus, as historical traffic condition data;

[0016] S203: Derive the passenger alighting station data based on the bus card data, and combine the bus entering and leaving station data to calculate the passenger boarding and alighting station information and time information as passenger flow information.

[0017] Further, the specific method in S4 is:

[0018] S401: Establish the state variable of the DQN model that can generate the dynamic balance timetable of the bus up and down;

[0019] S402: Establish the action state of the DQN model that can generate the dynamic balance timetable of the bus up and down;

[0020] S403: Establish the reward function of the DQN model that can generate the dynamic balance timetable of the bus up and down;

[0021] S404: Train the DQN model that can generate the dynamic balance timetable of the bus up and down, and in the training process, for each day, first update the environment state after forcibly sending a bus at the first bus time, and then for each minute, the DQN agent makes a decision to get an action, the action has a probability of p m The result generated by the model, and a probability of 1-p m The result of random value; if the current time is less than the minimum departure interval t min from the last departure time, forcibly do not depart; if the current time is greater than the maximum departure interval t max from the last departure time, forcibly depart;

[0022] S405: Simulate the DQN model that can generate the dynamic balance timetable of the bus up and down, apply the action to the simulation environment after the training in S404, get the next state vector and the reward value of the current action, and store the current state, current reward, current action and next state in the experience replay; when the experience replay is full, the DQN model will train once every 5 simulations, and update the target network in the DQN algorithm once every 100 training, and save the obtained model.

[0023] Further, the state variable in S401 will consider the time state, up and down vehicle passenger state, up and down passenger flow state, and up and down departure state. Define wherein m is different time in a day, X mu and X md represent the bus up state and down state respectively, define

[0024] X mu = [x u_h , x u_min , x u_pn , x u_wmin , x u_tpn , x u_vn ],

[0025] X md = [x d_h , x d_min , x d_pn, x d_wmin , x d_tpn , x d_vn ], x u_h and x d_h represent the hour of the uplink and downlink at the mth moment, x u_min and x d_min represent the minute of the uplink and downlink at the mth moment, x u_pn and x d_pn represent the number of passengers on the vehicle at the mth moment, x u_wmin and x d_wmin represent the waiting time of the passengers at the mth moment, x u_tpn and x d_tpn represent the passenger flow on the vehicle at the mth moment, x u_vn and x d_vn represent the number of departures of the line at the mth moment.

[0026] Further, the action vector of the action state in S402 defines four action elements: no departure of the uplink and downlink, departure of the uplink only, departure of the downlink only, and departure of the uplink and downlink, defines A = [0, 1, 2, 3], if the agent selects action 0, it represents that no departure of the uplink and downlink at the current moment; if the agent selects action 1, it represents that departure of the uplink only at the current moment; if the agent selects action 2, it represents that departure of the downlink only at the current moment; if the agent selects action 3, it represents that departure of the uplink and downlink at the current moment.

[0027] Further, the reward function in S403 contains two parts, reward = reward up + reward down -reward departure_std ; for the uplink, if no departure at the current moment reward up = 1-(o m / e m )-(ω×W m )-(β×ds m )+γ(c1-c2), wherein o m is the number of passengers getting on the vehicle if a vehicle is sent out at the mth minute, e m is the number of passengers on the vehicle, ω can take several values, if the value is larger, the model is biased to more departure, if the value is smaller, the model is biased to less departure, the value can be 1 / 1000; β is a constant, taking 0.2, ds m represents how many passengers will not get on the vehicle if the vehicle is full at the mth minute; γ is a constant, taking γ = 1e-3×200, when the simulation time is less than two hours from the last bus time, taking γ = 5e-3×200, if the current moment is departure reward up =(om / e m )-(β×ds m )-γ(c1-c2), for downlink if current time does not perform departure reward down =1-(o m / e m )-(ω×W m )-(β×ds m )-γ(c1-c2), for downlink if current time performs departure reward down =(o m / e m )-(β×ds m )+γ(c1-c2), for reward departure_std is the standard deviation of the departure interval.

[0028] Compared with the prior art, the bus dynamic equilibrium timetable generation method based on DQN has the following beneficial effects:

[0029] The bus dynamic equilibrium timetable generation method based on DQN of the present application is based on the traffic conditions and the bus passenger flow information of the bus operation, and a bus dynamic equilibrium timetable generation model based on DQN is designed, including the design of the state variables, action states and reward functions of the DQN model capable of generating the bus dynamic equilibrium timetable. The model is trained by directly interacting with the simulation environment, makes decisions in the real-time simulation environment, and generates the bus dynamic equilibrium timetable in real time. When the passenger flow changes due to a sudden event, a new departure timetable can be generated based on the changed passenger flow. When the bus travel time changes suddenly, a new departure timetable can be generated based on the changed travel time. When the bus station is temporarily adjusted or changed, a new departure timetable can be generated based on the changed station information. The generated departure timetable in the prior art cannot dynamically adapt to the changes of passenger flow and traffic conditions, and cannot simultaneously schedule two directions of a bus line, resulting in a scheduling that does not meet the actual requirements and cannot be applied. BRIEF DESCRIPTION OF DRAWINGS

[0030] Fig. 1 is the overall framework diagram of the present application;

[0031] Fig. 2 is the method flowchart of the second step of the present application;

[0032] Fig. 3 is the method flowchart of the fourth step of the present application. DETAILED DESCRIPTION

[0033] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0034] With reference to Figs. 1-3 , the embodiment of the present application provides a method for generating a dynamic balance timetable for bus boarding and alighting based on DQN, comprising the following steps:

[0035] Step 1: initialize bus line information, including setting line number ID, shortest departure interval t min , of the vehicle, longest departure interval t max , of the vehicle, vehicle carrying capacity C max , first departure time Start time , and last departure time End time . Specifically, set the line number ID as 211, the shortest departure interval t min of the vehicle as 10 minutes, the longest departure interval t max of the vehicle as 30 minutes, the vehicle carrying capacity C max as 84 people, the first departure time Start time as 6:30, and the last departure time End time as 22:30.

[0036] Step 2: obtain historical traffic conditions and passenger flow information on the line; the specific method is as follows:

[0037] Step 2-1: calculate the travel time between every two adjacent stations of each bus based on the data of the bus entering and leaving the station;

[0038] Step 2-2: calculate the travel time between every two adjacent stations within 15 minutes in a day based on the travel time between every two adjacent stations of each bus, as historical traffic condition data;

[0039] Step 2-3: derive the data of the passenger alighting station based on the bus card data, and combine the data of the bus entering and leaving the station to calculate the passenger boarding and alighting station information and time information, as passenger flow information;

[0040] Step 3: initialize the parameters of the reinforcement learning DQN model, including input dimension m input , output dimension m output , number of hidden layers m hidden , number of hidden layer neurons n hidden , activation function ReLU, learning rate lr, batch size batch_size, and experience replay capacity C ERSpecifically, set the input dimension m input is 12, the output dimension m output is 4, the number of hidden layers m hidden is 12, the number of hidden layer neurons n hidden is 500, the activation function is ReLU, the learning rate lr is 0.001, the batch size batch_size is 64, and the experience replay capacity C ER is 3000;

[0041] Step 4: Construct a DQN model that can generate a dynamic equilibrium timetable for bus up and down; the specific method is as follows:

[0042] Step 4-1: Design the state variables of the DQN model that can generate the dynamic equilibrium timetable for bus up and down, which will consider the time state, up and down vehicle passenger state, up and down passenger flow state, and up and down departure state. Define where m is different time of a day, X mu and X md represent the up and down state of the bus, respectively, and define

[0043] X mu = [x u_h , x u_min , x u_pn , x u_wmin , x u_tpn , x u_vn ],

[0044] X md = [x d_h , x d_min , x d_pn , x d_wmin , x d_tpn , x d_vn ], x u_h and x d_h represent the hour of up and down at m time, x u_min and x d_min represent the minutes of up and down at m time, x u_pn and x d_pn represent the number of people on the vehicle of up and down at m time, x u_wmin and x d_wmin represent the waiting time of passengers of up and down at m time, x u_tpn and x d_tpn represent the passenger flow of up and down at m time, x u_vn and x d_vn represent the number of departures of the line at m time;

[0045] Step 4-2: design the action state of the DQN model that can generate the dynamic balance timetable of the bus uplink and downlink, the action vector of the action state defines four action elements: no bus departure, only uplink departure, only downlink departure, and uplink and downlink departure, define A = [0, 1, 2, 3], if the agent selects action 0, it represents that no bus departs at the current time; if the agent selects action 1, it represents that only uplink departs at the current time; if the agent selects action 2, it represents that only downlink departs at the current time; if the agent selects action 3, it represents that uplink and downlink departs at the current time;

[0046] Step 4-3: design the reward function of the DQN model that can generate the dynamic balance timetable of the bus uplink and downlink, the reward function contains three parts, reward = reward up + reward down - reward departure_std ; for uplink, if no departure at the current time, reward up = 1 - (o m / e m ) - (ω × W m ) - (β × ds m ) + γ (c1-c2), wherein o m is the number of passengers getting on if a bus departs at the mth minute, e m is the carrying capacity of the vehicle, ω can take several values, if the value is larger, the model tends to have more departure number, if the value is smaller, the model tends to have less departure number, the value can be taken as 1 / 1000; β is a constant, taken as 0.2, ds m represents how many passengers will not be able to get on the bus due to full load of the current arrival vehicle at the mth minute; γ is a constant, taken as γ = 1e-3 × 200, when the simulation time is less than two hours from the end of the bus, taken as γ = 5e-3 × 200, if the current time departs, reward up = (o m / e m ) - (β × ds m ) - γ (c1-c2), for downlink, if no departure at the current time, reward down = 1 - (o m / e m ) - (ω × W m ) - (β × ds m ) - γ (c1-c2); if the current time departs, reward down = (o m / e m ) - (β × ds m ) + γ (c1-c2); reward departure_std is the standard deviation of the departure interval;

[0047] Step 4-4: DQN model training for generating dynamic balance timetable for bus boarding and alighting, in the training process, for each day, complete simulation is performed, first, after a bus is forced to be dispatched at the first bus time, the environment state is updated, thereafter, for each minute, the DQN agent makes a decision to obtain an action, the action has a probability P m of being the result generated by the model, and a probability 1-P m of being a random value, P m = 0.9 is set. If the time from the last departure is less than the minimum departure interval t min , then forced no departure is performed; if the time from the last departure is greater than the maximum departure interval t max , then forced departure is performed.

[0048] Step 4-5: DQN model simulation for generating dynamic balance timetable for bus boarding and alighting, after the training, the action is applied to the simulation environment to obtain the next state vector and the reward value obtained by making the current action, and the current state, the current reward, the current action and the next state are stored in the experience replay. When the experience replay is stored full, the DQN model will perform training every 5 simulations, and the target network in the DQN algorithm will be updated every 100 training times, and the obtained model is saved.

[0049] Step 5: DQN model inference for generating dynamic balance timetable for bus boarding and alighting, based on the trained model, the current traffic conditions and passenger flow data of the line are used for inference to obtain the dynamic balance timetable for boarding and alighting, when the passenger flow changes due to a sudden event, a new departure timetable can be generated based on the changed passenger flow, when the bus travel time changes suddenly, a new departure timetable can be generated based on the changed travel time, when the bus station is temporarily adjusted and changed, a new departure timetable can be generated based on the changed station information.

[0050] Step 6: Dynamic balance timetable for bus boarding and alighting result output, the planned departure time point of each line in the uplink and downlink directions is output, and the interval between the current departure time and the last departure time, the passenger transport capacity provided by the departure plan every 30 minutes, the total number of departures, the number of passengers who cannot board due to full vehicle, and the average waiting time of passengers are included.

[0051] From the above, it can be seen that the embodiment of the application provides a bus up-and-down dynamic balance schedule generation method based on DQN, a two-way bus schedule simultaneous generation optimization model based on DQN is designed based on traffic conditions and bus passenger flow information of bus operation, including designing a state matrix, an action state and a reward function of a DQN model capable of generating a bus up-and-down dynamic balance schedule, training through direct interaction with a simulation environment, making decisions in a real-time simulation environment, and realizing real-time generation of a bus up-and-down dynamic balance schedule. When a passenger flow has a sudden event change, a new departure schedule can be generated based on the changed passenger flow. When a bus travel time has a sudden change, a new departure schedule can be generated based on the changed travel time. When a bus station is temporarily adjusted and changed, a new departure schedule can be generated based on the changed station information. The method solves the problem that a generated departure schedule in the prior art cannot dynamically adapt to passenger flow and traffic condition changes and cannot simultaneously schedule two directions of a bus line, leading to scheduling that does not meet actual requirements and cannot be applied.

[0052] The above is only a preferred embodiment of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can make equivalent replacements or changes to the technical solution and the inventive concept of the application within the technical range disclosed by the application, and all of the above should be covered within the protection scope of the application.

Claims

1. A method for generating a dynamic equilibrium timetable for bus stops based on DQN, characterized in that, The method comprises the following steps: S1: initialize bus line information, including setting line number ID, shortest departure interval t min , longest departure interval t max , vehicle carrying capacity C max , first bus time Start time , last bus time End time ; S2: Obtain historical traffic conditions and passenger flow information on the line; S3: initialize reinforcement learning DQN model parameters, input dimension m input , output dimension m output , number of hidden layers m hidden , number of hidden layer neurons n hidden , activation function is ReLU, learning rate lr, batch size batch_size, experience replay capacity C ER ; S4: Construct a DQN model that can generate a dynamic equilibrium timetable for bus boarding and alighting, the DQN model includes a reward function, the reward function includes three parts, reward = reward up +reward down -reward departure_std ; for the uplink, if the current time does not make a departure, reward up = 1 - (o m / e m ) - (ω × W m ) - (β × ds m ) + γ (c1-c2), wherein o m is the number of passengers boarding if a vehicle is dispatched at the mth minute, e m is the carrying capacity of the vehicle, ω can take several values, if the value is larger, the model tends to make more departures, if the value is smaller, the model tends to make fewer departures; β is a constant, take 0.2, ds m represents how many passengers will not be able to board if the current arriving vehicle is full at the mth minute; γ is a constant, take γ = 1e-3 × 200, when the simulation time is less than two hours from the last bus time, take γ = 5e-3 × 200, if the current time makes a departure reward up = (o m / e m ) - (β × ds m ) - γ (c1-c2), for the downlink if the current time does not make a departure reward down = 1 - (o m / e m ) - (ω × W m ) - (β × ds m ) - γ (c1-c2); if the current time makes a departure reward down = (o m / e m ) - (β × ds m ) + γ (c1-c2); reward departure_std is the standard deviation of the departure interval; S5: A DQN model for generating a dynamic balance timetable for bus uplink and downlink can be generated. Based on the trained model, the current traffic conditions and passenger flow data of the line are used for reasoning to obtain a dynamic balance timetable for uplink and downlink. When a sudden event occurs in passenger flow, a new departure timetable can be generated based on the changed passenger flow. When the bus travel time suddenly changes, a new departure timetable can be generated based on the changed travel time. When the bus station is temporarily adjusted or changed, a new departure timetable can be generated based on the changed station information. S6: The dynamic balance timetable for bus uplink and downlink is output, and each departure time point of the line uplink and downlink direction plan is output, including the interval between the current departure time and the last departure time, the passenger capacity provided by the current departure plan every specific interval time, the total number of departures, the number of passengers who cannot board due to full load of the vehicle, and the average waiting time of passengers. 2.The method of claim 1, wherein the method is based on a deep Q-network (DQN) to generate a dynamic schedule for a bus stop. The specific method in S2 is: S201: Calculate the travel time between every two adjacent stations of each bus based on the data of the bus entering and leaving the station; S202: Calculate the travel time between every two adjacent stations within 15 minutes in a day based on the travel time between every two adjacent stations of each bus, as historical traffic condition data; S203: Derive the data of the passenger's alighting station based on the bus card swiping data, and combine the data of the bus entering and leaving the station to calculate the passenger's boarding and alighting station information and time information as passenger flow information. 3.The method of claim 1, wherein the method is characterized in that: The specific method in S4 is: S401: Establish the state variables of the DQN model for generating a dynamic balance timetable for bus uplink and downlink; S402: Establish the action state of the DQN model for generating a dynamic balance timetable for bus uplink and downlink; S403: Establish the reward function of the DQN model for generating a dynamic balance timetable for bus uplink and downlink; S404: The DQN model capable of generating dynamic equilibrium timetables for both directions of public transport is trained. During training, a complete simulation is performed for each day. First, a bus is forcibly dispatched at the first bus time, and the environmental state is updated. Then, for each minute thereafter, the DQN agent makes a decision, resulting in an action with p... m The probability is the result generated by the model, 1-p m The probability is the result of random selection; if the time since the last departure is less than the shortest departure interval t. min Then, departure will be forcibly cancelled; if the time since the last departure is greater than the longest departure interval t. max Then departure will be forced; S405: Simulate the DQN model for generating a dynamic balance timetable for bus uplink and downlink. After training in S404, apply this action to the simulation environment to obtain the next state vector and the reward value obtained by making the current action, and store the current state, current reward, current action, and next state in the experience replay. When the experience replay is full, the DQN model will be trained once every 5 simulations, and the target network in the DQN algorithm will be updated once every 100 training times, and the obtained model will be saved. 4.The method of claim 3, wherein the method is characterized in that: The state variable in S401 will consider time state, uplink and downlink vehicle passenger state, uplink and downlink passenger flow state, uplink and downlink departure state; define Wherein, m is different time in a day, X mu and X md respectively represent the bus uplink state, downlink state, define X mu =[x u_h , x u_min , x u_pn , x u_wmin , x u_tpn , x u_vn ], X md =[x d_h , x d_min , x d_pn , x d_wmin , x d_tpn , x d_vn ], x u_h and x d_h respectively represent the uplink and downlink at m time of the number of hours, x u_min and x d_min respectively represent the uplink and downlink at m time of the number of minutes, x u_pn and x d_pn respectively represent the uplink and downlink at m time of the number of people in the vehicle, x u_wmin and x d_wmin respectively represent the uplink and downlink at m time of the waiting time of passengers, x u_tpn and x d_tpn respectively represent the uplink and downlink at m time of the number of passengers on the vehicle, x u_vn and x d_vn respectively represent the uplink and downlink at m time of the number of line departure.

5. The method for generating a dynamic equilibrium timetable for bus stops and stations based on DQN according to claim 4, characterized in that: The action vector of the action state in S402 defines four action elements: no departure for uplink and downlink, departure for uplink only, departure for downlink only, and departure for uplink and downlink, defined as A=[0,1,2,3]. If the agent selects action 0, it means that no bus departs at the current time for uplink and downlink. If action 1 is selected, it means that only uplink departs at the current time. If action 2 is selected, it means that only downlink departs at the current time. If action 3 is selected, it means that uplink and downlink both depart at the current time.

Images