A bus scheduling method based on reinforcement learning

Abstract

The present invention discloses a bus scheduling method based on reinforcement learning, which belongs to the field of bus scheduling, and specifically comprises the following steps: first, the vehicle scheduling problem of a certain bus line is modeled as a Markov decision process, which includes three elements: state, action, and reward. Each time point in the departure schedule is taken as a decision point, and a reinforcement learning agent is used to select a currently available vehicle to be dispatched from the time point, thereby covering the time point. A simulation environment for bus scheduling is constructed, and the optimal strategy of the agent is obtained by using a reinforcement learning algorithm through the interaction between the agent and the simulation environment. Then, the agent is used to make decisions point by point in time sequence for each time point in the departure schedule, thereby obtaining a bus scheduling plan. The present invention provides an online scheduling method for buses, which can ensure that the departure schedule is fully covered in the event of uncertain events such as traffic congestion and traffic accidents, and a small number of vehicles are used.

Description

Technical Field

[0001] The invention belongs to the field of bus scheduling, and in particular relates to a bus scheduling method based on reinforcement learning. Background Art

[0002] Bus scheduling is a key link in improving the quality of bus services.

[0003] Currently, most of the methods used to solve the bus scheduling problem are based on precise algorithms and heuristic algorithms. Although precise algorithms can find the optimal solution, they require high computing power and cannot solve the more complex bus scheduling problems well. Although heuristic algorithms can solve some complex problems and obtain some good schedules, they lack real-time decision-making, resulting in poor ability to deal with emergencies, and sometimes cannot guarantee the important requirement of full coverage of time points.

[0004] Since existing methods are offline scheduling methods that generate a complete scheduling plan each time, it is impossible to make real-time decisions based on real-time information at a certain point in time. However, in actual application scenarios, due to the complexity of urban traffic problems, traffic congestion, traffic accidents and other uncertain events are very common. When encountering such phenomena, the original vehicle scheduling methods often cannot cope well.

[0005] Therefore, there is an urgent need to propose a bus scheduling method that can make real-time decisions and has good emergency response capabilities. Summary of the invention

[0006] In view of the above problems, the present invention provides a bus scheduling method based on reinforcement learning to achieve real-time decision-making, while being able to cope with dynamic events and provide an excellent scheduling schedule that ensures full coverage of time points.

[0007] The public transport scheduling method based on reinforcement learning has the following specific steps:

[0008] Step 1: For a given bus route departure schedule, use reinforcement learning to schedule vehicles. The bus scheduling problem is considered as a sequential decision process, with each time point in the departure schedule as a decision point. For each time point, the agent selects a vehicle to be dispatched from the control point at that time point based on the information of the vehicle in the control point, thereby covering the time point. The agent is used to make point-by-point decisions on all time points in the departure schedule, so that a vehicle is dispatched at each time point in the schedule, and finally a vehicle scheduling plan is formed.

[0009] Step 2: Model the bus scheduling problem as a Markov decision process consisting of three elements: state, action, and reward.

[0010] 1) State: For each available vehicle in the current control point, a vehicle representation vector is constructed for it, which contains the driving information related to the vehicle's schedule. Available vehicles refer to the vehicles that are currently parked at the control point and can be dispatched, as well as the vehicles that have not yet been dispatched. The matrix composed of the representation vectors of all available vehicles is used as the state. In order to ensure the convergence speed of the reinforcement learning algorithm, the state matrix is ​​further reduced in terms of row space and column space.

[0011] The dimensionality reduction process for row space is as follows: First, determine whether each vehicle meets the conditional constraints one by one. The constraint condition refers to whether the vehicle meets the conditions for departure at the current departure time, that is, whether it meets the constraints of location, rest time and working time. Then, the vehicles that meet the conditional constraints are divided into vehicles that have been dispatched and vehicles that have not been dispatched, and the vehicles that do not meet the conditional constraints are classified separately; finally, the vehicles that meet the conditional constraints and have not been dispatched at a certain departure time can be reduced to one row, and the vehicles that do not meet the conditional constraints are used to fill the state matrix to ensure that the shape of the matrix remains unchanged, thus ultimately reducing the number of rows in the state matrix.

[0012] The column space dimension reduction is as follows: construct a more concise vehicle representation vector to reduce the number of columns in the state matrix. By analyzing the information most relevant to the scheduling in the vehicle driving information, five pieces of information are selected: remaining driving time, rest time, number of trips, remaining working time and vehicle type.

[0013] 2) Action: The vehicle selected by the agent.

[0014] For vehicles that meet the constraints and have been dispatched, the vehicles are dispatched directly.

[0015] For vehicles that meet the condition constraints and have not been dispatched, a new vehicle pile is maintained. When such a vehicle is selected, a vehicle is randomly selected from the new vehicle pile and dispatched.

[0016] 3) Reward: A strategy combining mainline rewards and branchline rewards is adopted, and the reward function is constructed using the objectives of the bus scheduling problem. The objectives include the number of vehicles used, the number of vehicles with odd trips, and the coverage of time points.

[0017] In the reward design, the present invention adopts a reward setting method that combines main line rewards and branch line rewards. Branch line rewards are given after each decision-making step, and main line rewards are given at the last step of the decision-making step.

[0018] The main line rewards include: 1. Rewards for the number of unused vehicles N u ; 2. Penalty σ for the variance of vehicle running time h ; 3. The number of vehicles with odd trips is penalized N o; 4. Penalty for the number of short buses N s ; 5. The number of vehicles that complete the full journey is rewarded N f ;

[0019] The mainline reward is the weighted sum of these five items:

[0020] r m =w1×N u -w2×σ h -w3×N o -w4×N s +w5×N f

[0021] Among them, w1, w2, w3, w4, and w5 are positive real weights.

[0022] Branch rewards include: 1. Whether the selected car is a new car n ; 2. The idle time of the selected car ranks among all the available vehicles k ; 3. If the number of trips included in the vehicle is an even number, the reward is r o ; 4. Is the selected bus a long-distance bus? l ; 5. If you choose to send a new car when there is an available car, you will be penalized r e ; 6. Waiting time after the vehicle completes the trip r w .

[0023] The branch reward is the weighted sum of these six items:

[0024] r s =-w'1×r n -w'2×r k +w'3×r o +w'4×r l -w'5×r e -w'6×r w

[0025] Among them, w'1, w'2, w'3, w'4, w'5, and w'6 are positive real weights.

[0026] Step 3: Build a bus scheduling simulation environment. The intelligent agent obtains the optimal strategy by interacting with the simulation environment and using the reinforcement learning algorithm.

[0027] The simulation environment is a simulation of the vehicle operation process. The simulation environment contains the vehicle driving information at any time during the operation time, including the vehicle location, driving direction, running time, working time, rest time, etc. This information is used as the input of the intelligent agent. The intelligent agent generates output actions (i.e., selects the vehicle to be sent) based on the input, and the action acts on the simulation environment to update the vehicle driving information. Through the interaction between the intelligent agent and the environment, the simulation of the vehicle operation process is realized.

[0028] The agent obtains training data by interacting with the simulation environment and continuously learns. The reinforcement learning algorithm includes three types: value-based reinforcement learning algorithm, policy-based reinforcement learning algorithm, and Actor-Critic-based reinforcement learning algorithm. Under the constraints of the problem, the reinforcement learning algorithm is used to obtain the optimal strategy of the agent. The constraints of the problem are implemented by using two methods: invalid action masking and invalid action penalty.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] 1) A bus scheduling method based on reinforcement learning. Compared with traditional methods, the present invention models the bus scheduling problem as a Markov decision process. Through an online method, it solves the problem that offline methods cannot cope with uncertain events such as traffic congestion and vehicle failures. It has the ability to make real-time decisions and respond to emergencies in real time, and can provide an excellent scheduling schedule that guarantees full coverage of time points.

[0031] 2) A bus scheduling method based on reinforcement learning is proposed. The entire state space is reduced by performing row compression based on vehicle classification dimensionality reduction and column compression based on vehicle representation vector simplification, thereby ensuring the convergence of the reinforcement learning algorithm.

[0032] 3) A bus scheduling method based on reinforcement learning adopts a reward setting method combining main line rewards and branch line rewards, so that multiple goals of bus scheduling can be taken into account and balanced.

[0033] 4) A bus scheduling method based on reinforcement learning is proposed, which uses three types of algorithms: value-based reinforcement learning algorithm, strategy-based reinforcement learning algorithm and actor-critic based reinforcement learning algorithm to solve the problem respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a bus scheduling method based on reinforcement learning of the present invention;

[0035] Figure 2 A schematic diagram of the structural framework of a bus scheduling method based on reinforcement learning according to the present invention;

[0036] Figure 3 A state matrix construction diagram of a bus scheduling method based on reinforcement learning according to the present invention;

[0037] Figure 4 This is a diagram of the main network architecture in the D3QN reinforcement learning algorithm described in the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0039] The present invention discloses a bus scheduling method based on reinforcement learning, which includes: constructing a simulation environment according to the operation logic of the bus scheduling problem; modeling the bus scheduling control problem as a Markov decision process model, and defining the state, action and reward function therein; solving the optimal strategy by combining the reinforcement learning algorithm model with the constraint mechanism, and using the optimal strategy to select vehicles in real time when scheduling buses. The present invention improves the problems of the original method that cannot make real-time decisions, cannot cope with special situations, and is difficult to achieve full coverage of time points, and has certain advantages in major indicators such as the number of vehicles in use and the number of vehicles on odd trips.

[0040] The bus scheduling method based on reinforcement learning is as follows: Figure 1 As shown, the specific steps are as follows:

[0041] Step 1: For a given bus route departure schedule, use reinforcement learning to schedule vehicles. Model the bus scheduling problem as a sequential decision process, with each time point in the departure schedule as a decision point. For each time point, the agent selects a vehicle to be dispatched from the control point at that time point based on the information of the vehicle in the control point, thereby covering that time point. Use the agent to make point-by-point decisions on all time points in the departure schedule, so that a vehicle is dispatched at each time point in the schedule, and finally form a vehicle scheduling plan.

[0042] Step 2: Model the bus scheduling problem as a Markov decision process consisting of three elements: state, action, and reward.

[0043] 1) The state is the modeling of the information required for scheduling at the current decision point, including:

[0044] For each available vehicle in the current control point, a vehicle representation vector is constructed for it, which contains the driving information related to the vehicle and the shift. Available vehicles refer to the vehicles that are currently parked in the control point and can be dispatched, as well as the vehicles that have not yet been dispatched. The matrix composed of the representation vectors of all available vehicles is used as the state. In order to ensure the convergence speed of the reinforcement learning algorithm, the state matrix is ​​further reduced in terms of row space and column space.

[0045] For row space dimensionality reduction, first, determine whether each vehicle meets the conditional constraints one by one. The constraint condition refers to whether the vehicle meets the departure conditions at the current departure time, that is, whether it meets the constraints of location, rest time and working time. Then, the vehicles that meet the conditional constraints are divided into vehicles that have been dispatched and vehicles that have not been dispatched, and the vehicles that do not meet the conditional constraints are classified separately; After the vehicle classification and dimensionality reduction, since the priority of new vehicles that meet the conditional constraints is the same, this type of vehicle can be reduced to one row; in addition, the number of old vehicles that meet the conditional constraints at a certain departure time is also much smaller than the total number of vehicles. Therefore, this dimensionality reduction method can reduce the number of rows in the state matrix;

[0046] Column space dimension reduction is to reduce the number of columns of the state matrix by constructing a more concise vehicle representation vector. By analyzing the information most relevant to the scheduling in the vehicle driving information, five pieces of information are selected: remaining driving time, rest time, number of trips, remaining working time, and vehicle type.

[0047] 2) The action is the vehicle selected by the agent.

[0048] For vehicles that meet the constraints and have been dispatched, the vehicles are dispatched directly.

[0049] For vehicles that meet the condition constraints and have not been dispatched, a new vehicle pile is maintained. When this type of vehicle is selected, a vehicle is randomly selected from the new vehicle pile and dispatched.

[0050] 3) Rewards are the driving force for reinforcement learning agents to learn;

[0051] The reward function is constructed using the objectives of the bus scheduling problem, which include the number of vehicles used, the number of vehicles with odd trips, and the coverage of time points.

[0052] In the reward design, the present invention adopts a reward setting method that combines main line rewards and branch line rewards. The main line reward is the evaluation reward for the entire decision sequence after the decision sequence is completed; the branch line reward is the evaluation reward for each step in the decision-making process, which is responsible for guiding the intelligent agent to reach the goal of the main line reward faster.

[0053] The main line reward takes into account evaluation indicators such as the total number of vehicles in the entire scheduling plan and the number of vehicles on odd and even trips, while the branch line reward takes into account the suitability of the vehicle selected at the current decision point.

[0054] The main line rewards include: 1. Rewards for the number of unused vehicles N u ; 2. Penalty σ for the variance of vehicle running time h ; 3. The number of vehicles with odd trips is penalized N o ; 4. Penalty for the number of short buses N s ; 5. The number of vehicles that complete the full journey is rewarded Nf ;

[0055] The mainline reward is the weighted sum of these five items:

[0056] r m =w1×N u -w2×σ h -w3×N o -w4×N s +w5×N f

[0057] Among them, w1, w2, w3, w4, and w5 are positive real weights.

[0058] Branch rewards include: 1. Whether the selected car is a new car n ; 2. The idle time of the selected car ranks among all the available vehicles k ; 3. If the number of trips included in the vehicle is an even number, the reward is r o ; 4. Is the selected bus a long-distance bus? l ; 5. If you choose to send a new car when there is an available car, you will be penalized r e ; 6. Waiting time after the vehicle completes the trip r w .

[0059] The branch reward is the weighted sum of these six items:

[0060] r s =-w'1×r n -w'2×r k +w'3×r o +w'4×r l -w'5×r e -w'6×r w

[0061] Among them, w'1, w'2, w'3, w'4, w'5, and w'6 are positive real weights.

[0062] Step 3: Build a bus scheduling simulation environment. The intelligent agent obtains the optimal strategy by interacting with the simulation environment and using the reinforcement learning algorithm.

[0063] The simulation environment is a simulation of the vehicle operation process. The simulation environment contains the vehicle driving information at any time during the operation time, including the vehicle location, driving direction, running time, working time, rest time, etc. This information is used as the input of the intelligent agent. The intelligent agent generates output actions (i.e., selects the vehicle to be sent) based on the input, and the action acts on the simulation environment to update the vehicle driving information. Through the interaction between the intelligent agent and the environment, the simulation of the vehicle operation process is realized.

[0064] Under the condition that the problem constraints are met, the reinforcement learning algorithm is used to solve the optimal strategy. The reinforcement learning algorithm used in the present invention includes three types: value-based reinforcement learning algorithm, strategy-based reinforcement learning algorithm and actor-critic-based reinforcement learning algorithm.

[0065] In order to ensure that the action selected by the intelligent agent meets the constraints, the present invention uses two methods, invalid action masking and invalid action penalty, to implement the constraints of the problem.

[0066] Example:

[0067] like Figure 2 As shown in the figure, it is a structural framework of a bus scheduling method based on reinforcement learning, including three parts: environment model S1, Markov decision process model S2 and reinforcement learning algorithm model S3; the environment model S1 returns the updated state state and the corresponding reward reward according to the action action executed by the reinforcement learning algorithm model S3; and the three elements of action, state and reward constitute the main part of the Markov decision process model S2. In the training and learning process, with Markov decision process S2 as the basic learning framework, the reinforcement learning algorithm model S3 continuously interacts with the environment model S1 to obtain training data, and uses these training data to train the network. After the training is completed, the scheduling results of the training scene can be evaluated.

[0068] Environmental model S1: Model the logic of bus scheduling method and construct the environmental model.

[0069] Markov decision process model S2: The bus scheduling problem is modeled as a Markov decision process, and the states, actions and rewards are defined.

[0070] Reinforcement learning algorithm model S3: Use the D3QN reinforcement learning method to solve the optimal strategy and use the optimal strategy to select vehicles.

[0071] In this embodiment, the above-mentioned environment model S1 includes:

[0072] The environment is used to implement the bus scheduling environment logic for interacting with the reinforcement learning agent. In this invention, the environment model architecture of the gym library is used to implement the following functions, including:

[0073] The storage of information related to bus scheduling includes the driving information of each vehicle (whether it has started running, whether it is running, running direction, running time of the current trip, total driving time, rest time, number of trips, departure time, current time, vehicle type), departure schedule, single trip running time at different hours of the day and current time point, etc.

[0074] According to the actions output by the agent, the information related to the bus scheduling is updated by simulating the vehicle operation process. For example, at a certain decision point, vehicle V1 is selected as the departure vehicle, and the current decision step needs to simulate the changes in the entire environment from the current decision point to the next decision point, including the update of each vehicle's operation information and the current time point, etc.

[0075] Based on the action output by the agent, feedback for the action is generated. The agent is rewarded or punished based on the effect of the action taken by the agent on the final optimization goal. For example, if the selected vehicle is an old car and the car has been resting for a long time, a positive feedback will be given to the agent.

[0076] In this embodiment, the above-mentioned Markov decision process S2 model includes:

[0077] The three elements in the Markov decision process model: state, action, and reward.

[0078] The state element S21 includes a state matrix, such as Figure 3 As shown, in the state design, due to the large number of vehicles, it will cause problems such as difficulty in reinforcement learning training. For example, if there are 50 vehicles in the initial stage, and the vehicle vector corresponding to each vehicle is a 10-dimensional vector, then the size of the entire state matrix is ​​500 dimensions. Therefore, it is necessary to reduce the dimension of the state space. The present invention adopts a method of reducing the dimension of the row space and the column space respectively.

[0079] For row space dimensionality reduction, a method for state dimensionality reduction based on vehicle type is proposed. The vehicles are divided into three categories: vehicles that have been running and meet the condition constraints, new vehicles that meet the condition constraints, and vehicles that do not meet the condition constraints. Among them, new vehicles that meet the condition constraints can be classified into one row, and there will not be too many vehicles that have been running and meet the condition constraints. Finally, the invention reduces the original 50 vehicles to 16 vehicles using this method;

[0080] For column space dimensionality reduction, by analyzing the most relevant information in the column space for decision-making, five items of information, namely remaining driving time, rest time, number of trips, remaining working time, and vehicle type, were selected and summarized from the original 10-dimensional information (whether to start running, whether to run, running direction, current trip running time, total driving time, rest time, number of trips, departure time, current time, and vehicle type), and the column space of the state matrix was reduced in dimensionality.

[0081] Action element S22 includes:

[0082] In the action design, the dispatched vehicles that meet the conditional constraints can be dispatched directly; for the undispatched vehicles that meet the conditional constraints, the present invention needs to maintain a new vehicle pile, and when such a vehicle is selected, a vehicle is randomly selected from the new vehicle pile.

[0083] Reward element S23 includes:

[0084] In the reward design, a reward setting method combining main line rewards and branch line rewards is adopted.

[0085] The main line rewards include: 1. Rewards for the number of unused vehicles N u ; 2. Penalty σ for the variance of vehicle running time h ; 3. The number of vehicles with odd trips is penalized N o ; 4. Penalty for the number of short buses N s ; 5. The number of vehicles that complete the full journey is rewarded N f ;

[0086] The mainline reward is the weighted sum of these five items:

[0087] r m =w1×N u -w2×σ h -w3×N o -w4×N s +w5×N f

[0088] Among them, w1, w2, w3, w4, and w5 are positive real weights.

[0089] Branch rewards include: 1. Whether the selected car is a new car n ; 2. The idle time of the selected car ranks among all the available vehicles k ; 3. If the number of trips included in the vehicle is an even number, the reward is r o ; 4. Is the selected bus a long-distance bus? l ; 5. If you choose to send a new car when there is an available car, you will be penalized r e ; 6. Waiting time after the vehicle completes the trip r w .

[0090] The branch reward is the weighted sum of these six items:

[0091] r s =-w'1×r n -w'2×r k +w'3×r o +w'4×r l -w'5×r e -w'6×r w

[0092] Among them, w'1, w'2, w'3, w'4, w'5, and w'6 are positive real weights.

[0093] In an embodiment, the above-mentioned reinforcement learning algorithm model S3 can be specifically divided into a value-based reinforcement learning method, a strategy-based reinforcement learning method and an Actor-Critic-based reinforcement learning method.

[0094] The reinforcement learning methods attempted in the present invention mainly include the DQN series methods (Double DQN, Dueling DQN, D3QN, DRQN) and the like in the value-based reinforcement learning methods, TRPO, PPO and the like in the strategy-based reinforcement learning methods, and DDPG, TD3, SAC and the like in the Actor-Critic reinforcement learning methods.

[0095] Take the D3QN reinforcement learning algorithm as an example to solve the optimal strategy, specifically:

[0096] Initialize the replay buffer with a capacity of N to store samples for Q network training.

[0097] Initialize the main network and the target network, and randomly initialize the parameters.

[0098] The state matrix s composed of the acquired information is input into the main network to obtain the corresponding Q value vector under this state, and the epsilon-greedy method is used to select action a and obtain reward r from the environment. Each such state transition is recorded as a time step t, and the data (s, a, s', r) obtained in each time step is put into the replay buffer.

[0099] Sample a batch of data from the replay buffer, use the target Q value calculated by the target network as the label, use the Q value calculated by the main network as the prediction value, calculate the loss through MSE, and update the parameters through the Adam method.

[0100] In an embodiment, the main network S31 in the D3QN reinforcement learning algorithm includes:

[0101] With the flattened state matrix as input and the Q value vector as output, a four-layer network structure is constructed. In addition, a constraint addition model needs to be added to the last layer of the network to restrict unreasonable actions. The specific network structure is as follows Figure 4 The Neural Network part in the middle is shown.

[0102] In an embodiment, the constraint adding model S311 includes:

[0103] The invalid action masking method is to add a mask to the output of the Q network so that the Q value corresponding to the unreasonable action is set to a very small negative value so that the action cannot be selected. Figure 4 In the Invalid Action Masking section on the right, the values ​​of the four actions A1, A2, A3, and A4 are 200, 500, 400, and 250 respectively. If there is no constraint, action A2 with the largest corresponding value should be selected. However, after adding the constraint part, since action A2 does not meet the constraint, it is set to a very small negative value. Therefore, action A3 is finally selected.

[0104] The invalid action penalty method processes the reward. When the reinforcement learning agent selects a vehicle that does not meet the constraints, a small negative reward will be fed back to the agent, thereby guiding the agent to avoid this situation in the future.

[0105] The above-mentioned bus scheduling method based on reinforcement learning provided by the present invention interacts with the bus scheduling environment model through the reinforcement learning algorithm agent, obtains various training data and stores them in the replay buffer, and the reinforcement learning agent learns through these data to make better decisions. Compared with the prior art, the reinforcement learning method of the present invention can make real-time decisions and can handle some special scenarios, and has a certain degree of adaptability. Moreover, this method can be used not only for bus scheduling, but also for a series of similar scheduling problems such as subway scheduling and special car scheduling.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bus scheduling method based on reinforcement learning, characterized by: (1) For a given bus route departure schedule, reinforcement learning is used to schedule vehicles. The bus scheduling problem is modeled as a Markov decision process, with each time point in the departure schedule as a decision point. For each time point, the agent selects a vehicle to be dispatched from the control point at that time point based on the information of the vehicle at the control point, thereby covering the time point. The agent makes point-by-point decisions on all time points in the departure schedule, so that a vehicle is dispatched at each time point in the schedule, and finally forms a vehicle scheduling plan. (2) The Markov decision process model of the vehicle scheduling problem includes states, actions, and rewards; The state is the input of the agent, including: the information of all available vehicles at the current time point; the action is the output of the agent, that is, a vehicle is selected from the control point and sent out at that time point; the reward includes: the number of vehicles and the waiting time of the vehicle; The above mentioned rewards are a combination of mainline rewards and branchline rewards. Branchline rewards are given after each decision, and mainline rewards are given at the last step of the decision. The main line rewards include:

1. Rewards for the number of unused vehicles N u ; 2. Penalty σ for the variance of vehicle running time h ; 3. The number of vehicles with odd trips is penalized N o ; 4. Penalty for the number of short buses N s ; 5. The number of vehicles that complete the full journey is rewarded N f ; The mainline reward is the weighted sum of these five items: r m =w1×N u -w2×σ h -w3×N o -w4×N s +w5×N f Where w1, w2, w3, w4, and w5 are positive real weights; Branch rewards include:

1. Whether the selected car is a new car n ; 2. The idle time of the selected car ranks among all the available vehicles k ; 3. If the number of trips included in the vehicle is an even number, the reward is r o ; 4. Is the selected bus a long-distance bus? l ; 5. If you choose to send a new car when there is an available car, you will be penalized r e ; 6. Waiting time after the vehicle completes the trip r w ; The branch reward is the weighted sum of these six items: r s =-w‘1×r n -w‘2×r k +w‘3×r o +w‘4×r l -w‘5×r e -w‘6×r w Among them, w'1, w'2, w'3, w'4, w'5, and w'6 are positive real weights; (3) Construct a bus scheduling simulation environment, and the intelligent agent obtains the optimal strategy by interacting with the simulation environment and using the reinforcement learning algorithm.

2. A bus scheduling method based on reinforcement learning according to claim 1, characterized in that: The scheduling simulation environment is a simulation of the vehicle operation process; The simulation environment contains vehicle driving information at any time during the operating hours, including vehicle location, driving direction, operating time, working time and rest time. This information serves as the input of the intelligent agent. The intelligent agent generates output actions based on the input, that is, selects the vehicle to be sent out. The action acts on the simulation environment to update the vehicle driving information; through the interaction between the intelligent agent and the environment, the simulation of the vehicle operation process is realized.

3. A bus scheduling method based on reinforcement learning according to claim 1, characterized in that: The Markov decision process is characterized by: 1) State: For each available vehicle in the current control point, a vehicle representation vector is constructed for it. The vector contains the vehicle's driving information related to the scheduling. Available vehicles refer to the vehicles that are currently parked in the control point and can be dispatched, as well as the vehicles that have not yet been dispatched. The matrix composed of the representation vectors of all available vehicles is used as the state; 2) Action: For each decision point, the agent's action is to select an available vehicle from the schedule; the action space includes all available vehicles, where the vehicles that have not yet been dispatched are represented as one vehicle; 3) Rewards: including main line rewards and branch line rewards; the reward function is constructed according to the optimization goal of the vehicle scheduling problem. The factors that need to be considered in the reward include: the number of vehicles, the number of vehicles executing odd trips, and the coverage of time points.

4. A bus scheduling method based on reinforcement learning according to claim 1, characterized in that: The reinforcement learning algorithm includes three categories: value-based reinforcement learning algorithm, strategy-based reinforcement learning algorithm and actor-critic-based reinforcement learning algorithm; Under the constraints of the problem, the reinforcement learning algorithm is used to obtain the optimal strategy of the intelligent agent, and the constraints of the problem are implemented by using unreasonable action masks and unreasonable action penalties.

Images