Bus real-time scheduling method and system based on MAPPO reinforcement learning, and storage medium

Abstract

The invention relates to the technical field of reinforcement learning application, and discloses a bus real-time scheduling method and system based on MAPPO reinforcement learning and a storage medium, and the method comprises the following steps: constructing a multi-agent reinforcement learning model according to a bus road network, and enabling each route to correspond to one agent; obtaining a multi-agent state, generating candidate actions through an MAPPO action selection component, introducing the candidate actions into a configurable scene constraint pipeline to constrain the candidate actions, and generating a final execution action; according to the invention, decoupling of the business scene constraint and the algorithm model is realized, the multiplexing capability of the application is improved, and the model landing difficulty is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning application, and more specifically to a MAPPO reinforcement learning bus real-time scheduling method, system and storage medium. Background Art

[0002] The bus dispatching problem refers to how to reasonably arrange the number and departure time of buses on the specified traffic routes to meet the travel needs of passengers, while maximizing the conservation of public resources and improving the efficiency of bus operations. The planning and implementation of urban public transportation is a complex project that requires consideration of many factors, one of which is the bus dispatching problem. The dispatching and management of buses are characterized by regionality, complexity, and nonlinearity. In existing research and applications, there are the following two problems:

[0003] First, in industrial applications, real-time bus dispatching is mostly constructed based on complex and cumbersome rules. Second, in academic research, most heuristic algorithms such as genetic algorithms are proposed, and a small number of reinforcement learning algorithms are proposed for real-time bus dispatching scenarios, but most of them are based on deep reinforcement learning (DQN), SARSA, etc., and there is a lack of research on the application of multi-agent reinforcement learning to real-time bus dispatching scenarios. In the study of multi-agent algorithms, such as the multi-agent value decomposition network algorithm (VDN), the monotone value function decomposition algorithm (QMIX), and the multi-agent deep deterministic policy gradient algorithm (MADDPG), corresponding solutions have also been proposed. However, the effect is not good in the complex actual scenarios of buses. At the same time, due to the imperfect reward mechanism, it is impossible to accurately reflect the actual situation and optimization goals in bus dispatching, thus affecting the effect of the bus dispatching reinforcement learning model. In addition, there are problems such as low reusability and difficulty in model implementation.

[0004] Therefore, how to improve the effect of bus scheduling and reduce the difficulty of model implementation is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0005] In view of this, the present invention provides a MAPPO reinforcement learning real-time bus scheduling method, system and storage medium, which can achieve the decoupling of business scenario constraints and algorithm models, improve the reuse capability of applications, and reduce the difficulty of model implementation.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A bus real-time scheduling method based on MAPPO reinforcement learning comprises the following steps:

[0008] Build a multi-agent reinforcement learning model based on the bus network, so that each route corresponds to an agent;

[0009] Obtain multi-agent states and generate candidate actions through the MAPPO action selection component;

[0010] A configurable scene constraint pipeline is introduced, and scene constraints under the current configuration are obtained through the configurable scene constraint pipeline;

[0011] The dispatchable state of the corresponding public transport vehicle is confirmed according to the scenario constraints, the candidate actions are constrained, and a final execution action is generated.

[0012] Preferably, the obtaining of multi-agent states includes: predicting the next states corresponding to different actions under the observed states by the agents through a trained prediction model.

[0013] Preferably, the step further includes: using a preset reward function to calculate a corresponding action reward according to the final execution action.

[0014] Preferably, the training steps of the multi-agent reinforcement learning model include: calculating the value loss according to the reward function and optimizing the agent parameters.

[0015] Preferably, the reward function is:

[0016]

[0017] in, is the reward function value at time m, is the state environment at time m, To perform an action; It represents the actual number of passengers on the line at the mth moment. Indicates the number of people that the line can carry at the mth moment; Indicates the waiting time of all passengers on the line at time m; and are the carrying capacity utilization rate and the passenger waiting time cost penalty weight coefficients respectively.

[0018] Preferably, the prediction model includes a first prediction model and a second prediction model;

[0019] The first prediction model is used to predict the bus arrival time; the second prediction model is used to predict the passenger flow.

[0020] A MAPPO reinforcement learning bus real-time dispatching system, which adopts the above dispatching method, includes:

[0021] The bus network information monitoring module is used to obtain the bus traffic environment in real time;

[0022] MAPPO action selection component, used to obtain the public transportation environment to generate a preliminary scheduling plan;

[0023] The scenario configuration module is used to configure scenario constraints according to actual conditions;

[0024] The scheduling plan generation module is used to generate a final scheduling plan that meets the constraints based on the scenario constraints and the preliminary scheduling plan.

[0025] A computer-readable storage medium includes a computer program, and the computer program implements the above-mentioned scheduling method when executed.

[0026] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a MAPPO reinforcement learning bus real-time scheduling method, system and storage medium, which realizes the decoupling of business scenario constraints and algorithm models, improves the reuse capability of applications, and reduces the difficulty of model implementation. The ability to autonomously adjust scenario constraints according to actual conditions improves interactivity, practicality and accuracy, and has higher practical value. Reinforcement learning is carried out through a reward mechanism with adaptive penalty coefficient weights, which improves the model effect and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0028] Figure 1 A schematic diagram of a real-time bus scheduling method based on MAPPO reinforcement learning is provided for the present invention.

[0029] Figure 2 A schematic diagram of a configurable scene constraint pipeline in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example 1

[0032] like Figure 1 and Figure 2 The embodiment of the present invention discloses a real-time bus scheduling method based on MAPPO reinforcement learning, comprising the following steps:

[0033] A multi-agent reinforcement learning model is constructed according to the bus network, so that each route corresponds to an agent; the multi-agent state is obtained, and candidate actions are generated through the MAPPO action selection component; a configurable scenario constraint pipeline is introduced, and the scenario constraints under the current configuration are obtained through the configurable scenario constraint pipeline; the dispatchable state of the corresponding bus is confirmed according to the scenario constraints, the candidate actions are constrained, and the final execution action is generated.

[0034] In this embodiment, the algorithm model implements candidate actions, and the scenario constraint pipeline implements independent scenario constraints. Based on the candidate actions, the candidate actions are screened again to confirm the dispatch plan that meets the actual situation. The action selection needs to meet the actual business scenario constraints of bus dispatch, such as the maximum dispatch interval, the minimum dispatch interval, the driver's meal and rest (driver constraints in the autonomous driving scenario can be removed), vehicle charging (using the range prediction model), maintenance and other constraints, so that the model meets the actual application landing scenario. In the business constraint selection box, green means selecting the configuration, and gray means not selecting the configuration. Complex scenarios in bus dispatching are customized by a separate scenario constraint pipeline, which realizes the decoupling between complex scenarios and algorithm models, greatly reducing the learning difficulty while improving the model effect and reducing the landing difficulty.

[0035] The steps of generating candidate actions through the MAPPO action component specifically include:

[0036] Confirm the action space. The action of each agent is represented by 1 or 0, which means starting or not starting the car respectively. Therefore, in the original action space, there are all scheduling methods consisting of multiple agents executing actions.

[0037] The intelligent agent has an actor network for generating actions and a critic network for evaluating action rewards; the model parameters of the actor network and the critic network are transferred from a deep neural network, which is pre-trained to learn the action reward model under multi-agent information sharing.

[0038] According to the current state of the agent, the Actor network generates corresponding actions, namely candidate actions, and evaluates the action rewards through Critic.

[0039] In order to further implement the above technical solution, a plurality of different constraint modules are configured in the configurable scenario constraint pipeline, and each constraint module is connected to the pipeline through a standardized interface. The constraint modules may include driver meals, vehicle charging, vehicle maintenance, maximum departure interval, minimum departure interval, etc. The constraint module can obtain vehicle data in each route, such as basic vehicle status, route operation status, departure records, etc.

[0040] For the departure interval constraint module, the current time and the historical departure records of the corresponding line are obtained, and the verification is completed through the preset verification logic. For example, the most recent departure time of the line is read, the difference between the candidate departure time and the most recent departure time is calculated, and the difference is compared with the pre-set constraint parameters (such as minimum interval, maximum interval), and the departure status of the corresponding bus is confirmed based on the comparison result.

[0041] For the driver meal constraint module, the driver's on-site status or control signal can be obtained to determine whether the driver has a meal and form a constraint. Alternatively, the preset meal time can be combined with the current time to determine whether it is meal time and form a constraint.

[0042] The driver rest constraint module can obtain the driver's schedule and real-time working hours, and determine whether the driver is in a rest period, whether the driver has exceeded the continuous working hours, and the driver's rest time based on the current time. The driver's rest status is confirmed to form a constraint based on the above multiple factors.

[0043] For vehicle charging constraints, the vehicle's current SOC can be obtained to determine whether it can be fully scheduled, and charging constraints can be imposed when the battery is insufficient.

[0044] For vehicle maintenance constraints, computational maintenance parameters can be set to count the vehicle's mileage. When the preset mileage threshold is met, the vehicle can be maintained. The maintenance status can also be actively set according to the real-time situation to deal with sudden maintenance, or the bus operation status can be obtained in real time, and the vehicle fault code can be received to deal with sudden maintenance and intercept scheduling.

[0045] In this embodiment, an independent configuration center module is provided in the configurable scenario action constraint pipeline. The threshold parameters involved in all constraint modules are stored in the independent configuration center. Specific constraint modules are enabled or disabled through configuration files. For example, the driver rest constraint can be turned off in the autonomous driving scenario.

[0046] In order to further implement the above technical solution, when obtaining the current multi-agent state, in addition to the observed state obtained by direct monitoring of the bus traffic environment, the state trend at the next moment is also included. For the state trend at the next moment, a dual prediction model is used to implement it, with the first prediction model predicting the bus arrival time and the second prediction model predicting the passenger flow.

[0047] In the first prediction model, the input layer takes the vehicle's current GPS coordinates, real-time speed, road congestion coefficient (obtained through traffic big data API, such as the traffic congestion index provided by Amap or Baidu Map), and historical average travel time (statisticed by different time periods, weekdays / weekends, etc.) as input. The GPS coordinates are uniquely encoded so that they can be effectively processed by the model; the speed and congestion coefficient are normalized and mapped to the [0,1] interval. Then, features are extracted through a three-layer LSTM (Long Short-Team Meomory) layer to capture long-term dependencies in the time series. Finally, the predicted travel time for the vehicle to reach each station is output through the fully connected layer.

[0048] During the training process of the first prediction model, the vehicle driving data, including vehicle ID, timestamp, location, speed, actual arrival time, etc., is collected through the GPS device and on-board sensors on the bus. At the same time, the road congestion information of the corresponding period is obtained from the traffic data platform as training data. The mean square error can be used as the sample loss. The loss value is calculated and the gradient is calculated by the back propagation method to update the model parameters.

[0049] In the second prediction model, the input data includes the population density of the area around the station (obtained through census data and geographic information system), station type (such as commercial area, residential area, school, etc., one-hot encoding), historical passenger flow data (number of boardings counted by different time periods, weekdays / weekends), weather conditions (sunny, rainy, snowy, etc., one-hot encoding), date type (weekdays, weekends, holidays, etc., one-hot encoding). The population density and historical passenger flow data are normalized. After obtaining the input data, the spatial features are extracted through two convolutional layers and pooling layers, and then spliced ​​with time series features, such as historical passenger flow data, and input into the two-layer GRU to control the input degree of new information. Finally, the probability distribution of boarding demand for different lines at each station is output through the fully connected layer. After obtaining the boarding demand for different lines at each station, combined with the rated passenger capacity of the vehicle, the carrying capacity utilization rate is obtained.

[0050] During the training of the second model, the passenger flow data of different routes at each station can be used as training data, such as the number of people getting on and off, timestamp, station ID, etc. At the same time, relevant data such as population density data around the station, station type information, weather data, date information, etc. are collected. Similarly, after selecting a loss function, such as the cross entropy loss function; calculate the loss of the model prediction value, and update the model parameters through back propagation.

[0051] As a feasible implementation, the present invention provides a reward mechanism with an adaptive penalty coefficient weight, which can calculate the reward of the action after the agent takes action and provide feedback, which can intuitively reflect the execution quality of bus scheduling. In the process of reinforcement learning, the network parameters are updated based on the reward feedback. Specifically, the reward function can be expressed as:

[0052]

[0053] in, is the reward function value at time m, is the state environment at time m, To perform an action; It represents the actual number of passengers on the line at the mth moment. Indicates the number of people that the line can carry at the mth moment; Indicates the waiting time of all passengers on the line at time m; and are the carrying capacity utilization rate and the passenger waiting time cost penalty weight coefficients respectively.

[0054] In this embodiment, the reward function and As the training process progresses, the weights are adjusted adaptively, making the reward mechanism gradually perfect. The specific training process is as follows:

[0055] First, initialize the model parameters, the policy network, that is, the actor network V , and the comment network V, and set the learning rate .

[0056] Then, the trajectory data formed by multiple interactions between multiple agents and the bus network environment are collected. The process of generating trajectory data includes: initializing the hidden states of the actor network and the critic network, and then simulating the interaction of multiple agents in multiple time steps. In each interaction process, the agent calculates the action probability distribution to generate actions and updates the hidden state based on the current policy network π, its own observations and the current hidden state; then, sampling is performed in the action probability distribution to confirm the execution of the action; the evaluation network V calculates the state value and updates the hidden state. Finally, according to the current state, combined with the above-generated actions, the action is executed and the reward is obtained, and the state is transferred, that is, the next state and observation. Therefore, the trajectory point formed by one time step is recorded as the action executed in the current time step, the hidden state and reward of the network after the action, and the environmental state and observation state before and after the action is executed.

[0057] Finally, the model parameters are updated according to the trajectory. The advantage evaluation value and reward of the trajectory are calculated, and the hidden states of the policy network and the evaluation network are updated according to the first hidden state in the data block, and the model parameters are updated.

[0058] In order to further implement the above technical solution, the training data used includes the demand for vehicles in each period and each route during the scheduling period, as well as historical entry and exit data, route data, and bus card transaction data. Through these, the real-time bus network environment can be mastered, such as the number of people waiting at each station and the number of people on the bus. When the intelligent agent interacts with the external environment, it will obtain the current bus network environment, predict the arrival time and station passenger flow through the dual prediction model, and calculate the waiting time of passengers at the mth moment and the actual number of vehicles on the line, generate the current state, and use the current state as the input of the MAPPO action selection component. The Actor generates the action probability distribution and selects the execution action. Then, the immediate reward and the flag information of whether the trajectory ends after the action is executed are calculated.

[0059] Example 2

[0060] Based on the same inventive concept, an embodiment of the present invention discloses a real-time bus dispatching system based on MAPPO reinforcement learning. The system adopts the method in embodiment 1, including:

[0061] The bus network information monitoring module is used to obtain the bus traffic environment in real time;

[0062] MAPPO action selection component, used to obtain the public transportation environment to generate a preliminary scheduling plan;

[0063] The scenario configuration module is used to configure scenario constraints according to actual conditions;

[0064] The scheduling plan generation module is used to generate a final scheduling plan that meets the constraints based on the scenario constraints and the preliminary scheduling plan.

[0065] In order to further implement the above technical solution, it also includes an information perception module, which is used to predict the next state based on the observed state of the current public transportation, and send it to the MAPPO action selection component as a component of the public transportation environment, so that the MAPPO action selection component can fully grasp the direction of the public transportation environment when generating actions to form a scheduling plan.

[0066] Example 3

[0067] Based on the same inventive concept, an embodiment of the present invention discloses a computer-readable storage medium, including a computer program, and when the computer program is executed, the scheduling method in embodiment 1 is implemented.

[0068] In this embodiment, the storage medium is a solid-state hard disk, a distributed cloud storage node or a vehicle-mounted embedded memory, which has a high-speed access interface for real-time reading and writing of bus scheduling data; a dynamic constraint knowledge base is constructed when the program is executed to store constraint rule sets such as maximum vehicle capacity, minimum departure interval, and line operation time threshold, and supports dynamic updating of constraint conditions through XML configuration files.

[0069] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0070] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time bus dispatching method based on MAPPO reinforcement learning, characterized in that: The following steps are involved: Build a multi-agent reinforcement learning model based on the bus network, so that each route corresponds to an agent; Obtain multi-agent states and generate candidate actions through the MAPPO action selection component; A configurable scene constraint pipeline is introduced, and scene constraints under the current configuration are obtained through the configurable scene constraint pipeline; The dispatchable state of the corresponding public transport vehicle is confirmed according to the scenario constraints, the candidate actions are constrained, and a final execution action is generated.

2. A MAPPO reinforcement learning method for real-time bus scheduling according to claim 1, characterized in that: The obtaining of multi-agent states includes: predicting the next states corresponding to different actions of the agents under the observed state through a trained prediction model.

3. A MAPPO reinforcement learning method for real-time bus scheduling according to claim 2, characterized in that: The step also includes: using a preset reward function to calculate a corresponding action reward according to the final execution action.

4. A MAPPO reinforcement learning method for real-time bus scheduling according to claim 3, characterized in that: The training steps of the multi-agent reinforcement learning model include: calculating the value loss according to the reward function and optimizing the agent parameters.

5. A MAPPO reinforcement learning method for real-time bus scheduling according to claim 3 or 4, characterized in that: The reward function is: ； in, is the reward function value at time m, is the state environment at time m, To perform an action; It represents the actual number of passengers on the line at the mth moment. Indicates the number of people that the line can carry at the mth moment; Indicates the waiting time of all passengers on the line at time m; and are the carrying capacity utilization rate and the passenger waiting time cost penalty weight coefficients respectively.

6. A MAPPO reinforcement learning method for real-time bus scheduling according to claim 2, characterized in that: The prediction model includes a first prediction model and a second prediction model; The first prediction model is used to predict the bus arrival time; the second prediction model is used to predict the passenger flow.

7. A real-time bus dispatching system based on MAPPO reinforcement learning, characterized in that: The scheduling method described in any one of claims 1 to 6 is adopted.

8. A computer-readable storage medium, characterized in that: It comprises a computer program, which implements the scheduling method described in any one of claims 1 to 6 when being executed.

Images